JP2017091561A - Display control device, display control method, display control program, information provision system, information setting device, information setting method, and information setting program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an AR content individually depending on a business process to be performed.SOLUTION: According to one embodiment of the invention, a display control device includes: an acquisition unit for acquiring, from a display data group correlated to a plurality of work contents that include one or a plurality of display data correlated to a first work content and one or a plurality of display data correlated to a second work content, one or a plurality of display data correlated to the first work content out of the plurality of work contents; and a control unit for exercising control so that the acquired one or plurality of display data are displayed in a display mode based on a reference object recognized by image recognition. Any of the one or plurality of display data correlated at least to the first work content and any of the one or plurality of display data correlated at least to the second work content are associated with a common reference object.SELECTED DRAWING: Figure 9

Description

  The present invention relates to a technique for providing information by display.

  The model data of the three-dimensional object arranged in the three-dimensional virtual space corresponding to the real space is displayed so as to be superimposed on the captured image. This technology is called Augmented Reality (AR) because it extends information collected by human perception (such as vision). Model data of a three-dimensional object used for AR is called AR content.

  In the AR content, position coordinates in a virtual space with reference to a reference object (for example, an AR marker) existing in the real space is set. In addition, the positional relationship between the imaging device and the reference object is determined based on the image of the reference object reflected in the captured image. The position of the AR content with reference to the imaging device is determined according to the position coordinates set with reference to the reference object of the AR content and the positional relationship between the imaging device and the reference object. A projected image of the AR content is generated based on the position of the AR content with respect to the imaging device, and the generated projected image is displayed so as to be superimposed on the captured image. By matching the projected image with the real space image reflected in the captured image, the three-dimensional object shown in the AR content is displayed so as to exist in the real space.

  Also, according to a certain technology, conditions such as a provisionable period and a provisionable area are set for each AR content associated with one reference object, which is satisfied by the time and position when the reference object is recognized. AR content for which a condition is set is displayed (see, for example, Patent Document 1).

JP 2012-215989 A

  The technology for displaying AR content is used for supporting the execution of business processes. A business process is a series of operations performed in an organization such as a company. By displaying the AR content indicating the work guide and the items to be sent included in the business process, the work assistance is provided to the user, and the execution of the business process by the user is supported.

  In an organization's activities, multiple business processes are performed. For example, in an organization, a core business process (for example, a manufacturing process in a manufacturing company) that is the main business of the organization is performed, and a support business process (for example, a manufacturing device) that supports the core business process. An inspection process etc.) is also performed. In an IT system that supports execution of business processes, even the same business process is defined as a different business process depending on the user executing the business process. This is because, for example, the content of performance support differs between an inspection process performed by a skilled worker and an inspection process performed by a work beginner.

  When the execution support of a plurality of business processes is performed by displaying the AR content, the AR content relating to the whole of the plurality of business processes is provided, and even the assistance regarding the work of the business processes other than the business process being executed by the user is displayed. . Moreover, even if the AR content to be displayed on the condition of the position and time is determined by the above-described technology, the same AR content is provided to a plurality of users who execute different business processes in parallel. AR content corresponding to the process is not provided.

  An object of one aspect of the present invention is to provide AR contents individually according to a business process to be performed.

  According to one aspect, the display control device applies a plurality of work contents including one or more display data associated with the first work contents and one or more display data associated with the second work contents. An acquisition unit that acquires one or more display data associated with the first work content of the plurality of work contents from the associated display data group, and the acquired one or more display data, A control unit that performs display control in a display mode based on a reference object recognized by image recognition, and at least one of display data of one or a plurality of display data associated with the first work content And one of the display data associated with the second work content is associated with a common reference object.

  According to one aspect, the information providing system includes a first computer and a second computer, and the first computer includes one or more display data associated with the first work content and a second work. From a display data group associated with a plurality of work contents including one or a plurality of display data associated with contents, from a display data group associated with the plurality of work contents, among the plurality of work contents A transmission unit that transmits one or more display data associated with the first work content to the second computer, wherein the second computer receives the transmitted one or more display data; And a control for displaying the received one or more display data in a display mode based on a reference object recognized by image recognition, at least One of the display data associated with the first work content and one of the display data associated with the second work content and one of the display data associated with the second work content are: Corresponding to a common reference object.

  According to one aspect, the information management device associates the display data with the selection unit that selects the identification information indicating the work content, and the reference object recognized by the image recognition on the captured image obtained by the imaging unit. A setting unit that sets position information indicating a position relative to the reference object, and a management unit that associates the display data in which the position information is set in association with the reference object with the selected identification information. ,including.

  According to one aspect, AR content can be provided individually according to a business process to be performed.

FIG. 1 shows the relationship between the camera coordinate system and the marker coordinate system. FIG. 2 shows an example of AR content in the camera coordinate system and the marker coordinate system. FIG. 3 shows a transformation matrix E from the marker coordinate system to the camera coordinate system and a rotation matrix R in the transformation matrix E. FIG. 4 shows the rotation matrices R1, R2 and R3. FIG. 5 shows a display example of AR content. FIG. 6 shows a sequence example of AR content provision. FIG. 7 shows a sequence example of AR content provision. FIG. 8 shows a display example of AR content. FIG. 9 shows a group of AR content. FIG. 10 shows an example data structure of AR definition data. FIG. 11 shows a display example of AR content. FIG. 12 shows a group of AR content. FIG. 13 shows an example data structure of AR definition data. FIG. 14 shows a display example of AR content. FIG. 15 shows a sequence example of AR content provision. FIG. 16 shows a functional block configuration example of the computer 1. FIG. 17 shows a processing procedure example of the AR function. FIG. 18 shows a processing procedure example of the AR function. FIG. 19 shows an example of a mode selection screen and a scenario selection screen. FIG. 20 shows an example of the data structure of the route definition. FIG. 21 shows a processing procedure example of the download function. FIG. 22 shows an example of the data structure of the scenario definition. FIG. 23 shows an example of the data structure of the step definition. FIG. 24 shows an example of the data structure of the marker definition. FIG. 25 shows an example data structure of content definition. FIG. 26 shows an example of the reception state table. FIG. 27 shows an example of the data structure of the AR template. FIG. 28 shows an example of recognition marker information and marker movement information. FIG. 29 shows an example of scenario state information. FIG. 30 shows an example of an event detection processing procedure. FIG. 31 shows examples of markers displayed on the screen. FIG. 32 shows an example of the processing procedure of the projection image generation process. FIG. 33 shows examples of markers displayed on the screen. FIG. 34 shows an example of the data structure of the step definition. FIG. 35 shows an example of an event detection processing procedure. FIG. 36 shows an example of a marker displayed on the screen. FIG. 37 shows an example of an event detection processing procedure. FIG. 38 shows an example of the data structure of the scenario definition. FIG. 39 shows the processing procedure in the edit mode. FIG. 40 shows the processing procedure in the edit mode. FIG. 41 shows an example of a display screen in the edit mode. FIG. 42 shows an example of an editing palette. FIG. 43 shows an example of step selection. FIG. 44 shows an example of step selection. FIG. 45 shows an example of step selection. FIG. 46 shows an example of step selection. FIG. 47 shows an example of AR definition data. FIG. 48 shows a hardware configuration example of the computer 1. FIG. 49 shows a configuration example of a program operating on the computer 1. FIG. 50 shows an example of a system including the computer 1. FIG. 51 shows a functional configuration example of the computer 2. FIG. 52 shows an example of a processing procedure for providing AR content. FIG. 53 shows a hardware configuration example of the computer 2.

[Display AR content]
First, AR technology for displaying AR content, which is model data of a three-dimensional object, overlaid on a captured image captured by a camera which is an example of an imaging apparatus will be described.

  The AR content is model data configured using a plurality of point coordinates, for example. A pattern (texture) or image is set for each of a plurality of surfaces obtained by interpolating a plurality of points with straight lines or curves, and a plurality of surfaces are combined to form a three-dimensional model. The arrangement of the AR content is determined based on a reference object existing in the real space with the coordinates of each point constituting the AR content.

  On the other hand, the positional relationship between the camera and the reference object in the real space is obtained based on the appearance (image) of the reference object reflected in the captured image captured by the camera. The reference object is an object whose shape or pattern is known. For example, the reference object is located with respect to the camera by comparing the image of the reference object in the captured image with the known shape or pattern of the reference object. Is determined.

  The positional relationship between the camera and the coordinates of each point of the AR content is obtained based on the positional relationship between the coordinates relative to the reference object and the camera and the reference object. (Although the positional relationship between the camera and the AR content is shown, the AR content is virtually arranged, and the object of the AR content does not exist in the real space. Based on these positional relationships, an image (projected image) of the AR content obtained when the camera temporarily captures the AR content existing in the virtual space is generated. The calculation for generating the projected image of the AR content will be further described based on FIGS.

  FIG. 1 shows the relationship between the camera coordinate system and the marker coordinate system. The AR marker M shown in FIG. 1 is an example of a reference object used for AR content display. The AR marker M illustrated in FIG. 1 has a square shape and has a predetermined size (for example, the length of one side is 10 cm). Although the AR marker M shown in FIG. 1 has a square shape, another reference object having a shape in which the relative position and orientation from the camera can be determined based on an image obtained by imaging from any viewpoint is used. May be. As another example, feature points generated from captured images are used as reference objects.

  The camera coordinate system is configured in three dimensions (Xc, Yc, Zc). For example, the focal point of the camera is the origin (origin Oc). For example, the Xc-Yc plane of the camera coordinate system is a plane parallel to the image sensor surface of the camera, and the Zc axis is an axis perpendicular to the image sensor surface.

  The marker coordinate system is configured in three dimensions (Xm, Ym, Zm). For example, the center of the AR marker M is the origin (origin Om). For example, the Xm-Ym plane of the marker coordinate system is a plane parallel to the printed surface of the AR marker M pattern, and the Zm axis is perpendicular to the printed surface of the AR marker M. The origin Om is indicated by position coordinates V1c (X1c, Y1c, Z1c) in the camera coordinate system. Further, the rotation angle of the marker coordinate system with respect to the camera coordinate system is indicated by a rotation coordinate G1c (P1c, Q1c, R1c). P1c is a rotation angle around the Xc axis, Q1c is a rotation angle around the Yc axis, and R1c is a rotation angle around the Zc axis. Since the AR marker M illustrated in FIG. 1 rotates only around the Yc axis, P1c and R1c are zero.

  FIG. 2 shows an example of the AR content C in the camera coordinate system and the marker coordinate system. The AR content C illustrated in FIG. 2 is balloon-shaped model data, and includes text data “There is a crack!” In the balloon. The black circle at the tip of the balloon for AR content C is used as the reference point for AR content C. The coordinates of the reference point of the AR content C in the marker coordinate system are V2m (X2m, Y2m, Z2m). Furthermore, the direction of the AR content C is determined by the rotation coordinates G2m (P2m, Q2m, R2m), and the size of the AR content C is determined by the magnification D (Jx, Jy, Jz).

  The coordinates of each point constituting the AR content C are adjusted based on the coordinates V2m of the reference point, the rotation coordinates G2m, and the magnification D of the coordinates of each point defined in the AR content C template (AR template). Coordinates. The coordinates of the reference point defined in the AR template are (0, 0, 0), for example. Each coordinate included in the AR template is rotated based on the set rotation coordinate G2m, scaled by the magnification D, and further translated and adjusted based on the reference point coordinate V2m. The AR content C in FIG. 2 is configured based on points defined in the AR template that are adjusted in the marker coordinate system of the AR marker M based on the reference point coordinates V2m, the rotation coordinates G2m, and the magnification D. It is the state that was done.

  The coordinates in the marker coordinate system of each point of the AR content C are converted into the camera coordinate system, and further converted into positions on the display screen (coordinates in the screen coordinate system) based on the coordinates in the camera coordinate system. Based on the converted coordinates, a projected image of the AR content C is generated. The coordinates in the camera coordinate system are calculated by performing coordinate conversion (model-view conversion) based on the coordinates V1c and the rotation coordinates G1c with respect to the coordinates in the marker coordinate system. For example, when model-view conversion is performed on the coordinate V2m, the coordinate V2c (X2c, Y2c, Z2c) in the camera coordinate system of the reference point is obtained.

  FIG. 3 shows a transformation matrix E from the marker coordinate system to the camera coordinate system and a rotation matrix R in the transformation matrix E. The transformation matrix E is a 4 × 4 matrix. The column vector (Xc, Yc, Zc, 1) is obtained by the product of the transformation matrix E and the column vector (Xm, Ym, Zm, 1). The transformation matrix E is calculated based on the AR marker position coordinates V1c (X1c, Y1c, Z1c) and the rotational coordinates G1c (P1c, Q1c, R1c) in the camera coordinate system.

  A column vector including the point coordinates of the camera coordinate system by substituting the point coordinates of the marker coordinate system subject to coordinate transformation (model-view transformation) into the column vector (Xm, Ym, Zm, 1) and performing matrix calculation. (Xc, Yc, Zc, 1) is obtained. The sub-matrix (rotation matrix R) of the first to third rows and the first to third columns of the transformation matrix E acts on the coordinates of the marker coordinate system, thereby aligning the orientation of the marker coordinate system and the camera coordinate system. A rotation operation is performed. When the partial matrix in the first to third rows and the fourth column of the transformation matrix E acts, a translation operation for adjusting the orientation of the marker coordinate system and the position of the camera coordinate system is performed.

  FIG. 4 shows the rotation matrices R1, R2 and R3. The rotation matrix R shown in FIG. 3 is calculated by the product of the rotation matrices R1, R2, and R3 (R1, R2, R3).

  The coordinates (Xm, Ym, Zm) of the marker coordinate system of each point constituting the AR content C are converted into the coordinates (Xc, Yc, Zc) of the camera coordinate system by model-view conversion based on the conversion matrix E. . The position coordinate V2m illustrated in FIG. 2 is converted into the position coordinate V2c by model-view conversion.

  The screen coordinate system is configured in two dimensions (Xs, Ys). For example, the center of the captured image obtained by the imaging process of the camera is set as the origin (origin Os). The coordinates in the camera coordinate system of each point of the AR content C are converted into a screen coordinate system by perspective transformation. A projection image of the AR content C is generated based on the coordinates in the screen coordinate system of each point obtained by the perspective transformation.

  Coordinate transformation (perspective transformation) from the camera coordinate system to the screen coordinate system is performed based on the focal length f of the camera, for example. The Xs coordinate of the coordinate in the screen coordinate system corresponding to the coordinate (Xc, Yc, Zc) in the camera coordinate system is obtained by Xs = f · Xc / Zc. Further, the Ys coordinate of the coordinate in the screen coordinate system corresponding to the coordinate (Xc, Yc, Zc) in the camera coordinate system is obtained by Ys = f · Yc / Zc.

  Based on the position coordinates (screen coordinate system) obtained by perspective-transforming the position coordinates (camera coordinate system) of each point constituting the AR content C, a projection image of the AR content C is generated. The AR template that is the source of the AR content C defines which points are interpolated to form a surface and which texture (or image) is mapped to which surface. The projected image of the AR content C is generated by mapping a texture or image on a surface obtained by interpolating each position coordinate (screen coordinate system) according to the definition of the AR template.

  By the above-described model-view conversion and perspective conversion, coordinates on the captured image corresponding to the coordinates of the marker coordinate system are calculated, and a projection image of the AR content C corresponding to the viewpoint of the camera is generated by using the coordinates. Is done. When the projected image of the AR content C is combined with the captured image, it is displayed on the screen as if a three-dimensional object exists, so that the visual information provided to the user is expanded.

  As another example of AR content display, a projected image of AR content C may be displayed on a transmissive display. Also in this aspect, since the image of the real space obtained by the user passing through the display matches the projected image of the AR content, the visual information provided to the user is expanded. The process of combining and displaying the projected image and the captured image of the AR content in the present embodiment may be a process of displaying the AR content when a transmissive display is used.

  FIG. 5 shows a display example of AR content. The display screen S (1) illustrated in FIG. 5 displays an image of an AR marker M, an image of an object H (a pipe installed in a factory is illustrated), and a projected image of the AR content C. The AR content C is defined in a marker coordinate system with the AR marker M as a reference, and a projected image of the AR content C is generated by performing model-view conversion and perspective conversion on each of the defined coordinates. The AR marker M and the object H exist in the real space. When the coordinates of the AR content C are set according to a specific position in the real space, the AR content C provides information regarding the specific position. On the display screen S (1), the AR content C balloon is set so that the tip of the balloon of the AR content C points to the crack position of the object H. By providing the AR content C, the user can easily locate the crack position of the object H. Can be specified.

[Download AR content]
In a system that provides a plurality of types of AR content, if the client device (computer 1) that displays the AR content holds all types of AR content, the burden on the storage capacity of the computer 1 increases. For example, when the computer 1 is a mobile terminal, the computer 1 may not have a sufficient storage capacity for holding AR content due to the physical size limitation of the computer 1.

  Therefore, a management server (computer 2) that aggregates and stores provided AR content is used, and a system configuration in which the computer 1 downloads AR content from the computer 2 is used. In this case, for example, the computer 1 downloads AR content to be displayed among all types of AR content. When there are a plurality of client devices, the AR contents are collected and stored in the computer 2, so that the storage capacity for constantly holding the AR contents as a whole system is suppressed.

  Further, by imaging the reference object, not only the position and orientation of the reference object but also identification information of the reference object can be obtained. For example, in the AR marker, identification information (marker ID) of the AR marker is determined based on the pattern printed on the AR marker. In the method using the feature point of the image as the reference object, the identification information is determined based on a database in which the coordinates of the feature point are associated with the identification information.

  The AR content to be displayed may be individually set for each of the plurality of AR markers. Therefore, the AR content to be displayed is switched according to the recognized AR marker. In this case, which AR content is to be displayed among all types of AR content is determined after recognition of the AR marker. For example, in a system configuration in which AR content to be displayed is downloaded from the computer 2, the computer 1 downloads the AR content to be displayed after recognition of the AR marker.

  FIG. 6 shows a sequence example of AR content provision. In the sequence example shown in FIG. 6, the processing order of the computer 1 and the computer 2 is shown. After the AR content display program is started on the computer 1, the AR marker M1 is recognized, and then an AR content acquisition request corresponding to the AR marker M1 is made. In this acquisition request, for example, the marker ID of the AR marker M1 is notified. Upon receiving the acquisition request, the computer 2 searches the AR content corresponding to the AR marker M1 and transmits the AR content corresponding to the marker M1 to the computer 1. After receiving the AR content from the computer 2, the computer 1 displays the AR content corresponding to the AR marker M1.

  According to the sequence shown in FIG. 6, the AR content corresponding to the AR marker is requested and downloaded after the AR marker is recognized and before the AR content display process is started. For this reason, a time lag is greatly generated from when the AR marker is recognized until the AR content is displayed.

  If a time lag occurs until the AR content is displayed after the AR marker appears in the captured image, the perception of the user who views the AR marker and the perception that is expanded by viewing the AR content. Deviation increases. For this reason, the AR characteristic of extending the user's perception is impaired.

  FIG. 7 shows a sequence example of AR content provision. The sequence example shown in FIG. 7 shows the processing order of the computer 1 and the computer 2 as in FIG. When the AR content display program is started, the computer 1 first makes an AR content acquisition request prior to recognition of the AR marker M1. In the sequence example of FIG. 7, the computer 1 receives the AR content from the computer 2 and then displays the AR content according to the AR marker recognition.

  In the sequence of FIG. 7, since the AR content is not requested or downloaded between the recognition of the AR marker and the display of the AR content, the time lag from the recognition of the AR marker to the display of the AR content is reduced. Therefore, the difference between the perception of the user viewing the AR marker and the perception of the user viewing the AR content is reduced.

  On the other hand, in the sequence example of FIG. 7, regardless of which AR marker is recognized, all types of AR content are downloaded to the computer 1 in order to display AR content corresponding to the AR marker. As a result, since all types of AR content are held in the computer 1, the load on the storage capacity of the computer 1 increases.

  Even when the AR content is downloaded from the computer 2 to the computer 1 in the sequence shown in FIGS. 6 and 7, a situation such as an increase in time lag and an increase in load on the storage capacity occurs.

[Selective provision according to the scenario]
The technology for displaying AR content is used for supporting the execution of business processes. As described above, a plurality of business processes are performed in an organization such as a company. Each business process is often a combination of multiple operations. In this embodiment, this work combination is regarded as one unit, which is called a scenario. In the case where the business process is configured by one work, the work is similarly referred to as a scenario. Further, even in the same business process, different scenarios may be set for different users. For example, different scenarios are provided depending on whether a user who performs a business process is a skilled worker or a beginner. The content of the AR content corresponding to the business process differs depending on the work content included in the scenario. That is, when an AR content group is set for a plurality of business processes with different scenarios, the AR content group includes individual AR contents that are applied only to a single scenario.

  FIG. 8 shows a display example of AR content. The display example shown in FIG. 8 shows a screen S (2) on which AR content associated with the AR marker M1 is displayed for a plurality of business processes. The screen S (2) includes an image of the AR marker M1, and further shows projected images of the AR contents CA11 to CA14 and the AR contents CB11 to CB13, C11, and C12 generated based on the image of the marker M1. .

  The AR contents CA11 to CA14 are AR contents unique to the inspection work scenario. The AR contents CA11, CA12, and CA13 indicate the work contents of the inspection work. The AR content CA11 indicates a work content instruction “turn the valve to the left”. The AR content CA12 indicates a work content instruction “flow meter check”. The AR content CA13 indicates a work content instruction “turn the valve to the right and confirm that the oil amount is in the range of XX to YY”. The AR content CA14 illustrates the work content of the work content shown in the AR content CA13.

  On the other hand, the AR contents CB11 to CB13 are a group of AR contents unique to the scenario of the manufacturing work. The AR contents CB11, CB12, and CB13 indicate the work contents of the manufacturing work. The AR content CB11 indicates a work content instruction “set board”. The AR content CB12 indicates a work content instruction “confirm remaining solvent”. The AR content CA13 indicates a work content instruction “pressing the“ execute ”button”.

  The AR content C11 and the AR content C12 are not information related to a specific scenario but information directed to a user who works near the AR marker M1. The AR content C11 is a warning message “Warning about water leakage” and further indicates the position of water leakage. Further, the AR content C12 indicates a message of “confirmation of water leak from pipe A on AA month BB”.

  As a result of the control to display the AR content corresponding to the recognized AR marker, on the screen S (2), both the original AR content in the inspection work process and the original AR content in the manufacturing work scenario are displayed. ing. Even when the user performs the business process of the inspection work, the unique AR content is displayed to the user in the manufacturing work scenario. Conversely, when the user performs the manufacturing work process, the AR content unique to the inspection work scenario is displayed to the user. That is, only by displaying AR content according to the recognized AR marker, AR content not related to the business process performed by the user is also displayed.

  On the screen S (2), there are many AR contents to be displayed. For example, the AR content CA11, the AR content CB13, and the AR content CA13 are displayed in an overlapping manner. In a general Web page or the like, even if the amount of content is large, if the area of the Web page is enlarged, each content can be browsed by a scroll operation or the like. However, the AR content display area is limited within the captured image. For this reason, if there are many AR contents, they are likely to be displayed overlapping each other.

  Further, when the user performs the inspection work process, the AR content unique to the manufacturing process is downloaded from the computer 2 to the computer 1. That is, even AR content that does not assist the user's work is downloaded to the computer 1. By downloading AR content that does not assist the user's work, the time lag from AR marker recognition to AR content display is increased, and the load on the storage capacity of the computer 1 is increased.

  FIG. 9 shows a group of AR content. A set of AR content to be displayed when the AR marker M1 is recognized is an AR content group G1. The AR content group G1 includes AR contents CA11 to CA13 for procedure display, AR content CA14 for operation assistance, AR contents C11 and C12 indicating a message, and AR contents CB11 to CB13 for procedure display. A set of AR contents to be displayed when the AR marker M2 is recognized is an AR content group G2. The AR content group G2 includes procedure-displayed AR contents CA21 to CA23 and procedure-displayed AR content CB21.

  Simply by controlling to display AR content according to the recognized AR marker, the AR content group G1 is displayed when the AR marker M1 is recognized, and the AR content group G2 is displayed when the AR marker M2 is recognized. Is done. In this control, as shown in FIG. 8, the entire AR content group G1 related to a plurality of business processes is displayed according to the recognition of the AR marker M1, and even AR content that does not support the business processes performed by the user is displayed. Will be. In the sequence shown in FIG. 6, it is determined by recognition of the AR marker which of the AR content group G1 and the AR content group G2 should be downloaded. Therefore, the AR content is downloaded from the recognition of the AR marker to the display of the AR content, and as a result, a time tag is generated from the recognition of the AR marker to the display of the AR content. Furthermore, when the AR content is downloaded in the sequence shown in FIG. 7, the entire AR content shown in FIG. 9 is downloaded to the computer 1, so that the load on the storage capacity of the computer 1 is large.

  Thus, for example, AR content is managed and provided in units corresponding to scenarios. That is, a group (AR content group G3) is formed by the AR contents CA11 to CA14 and AR contents CA21 to CA23 unique to the inspection work scenario and the AR contents C11 and C12 common to the scenarios. Further, a group (AR content group G4) is formed by the AR contents CB11 to CB13 and AR content CB21 unique to the scenario of the manufacturing work and the AR contents C11 and C12 common to the scenarios. By providing the AR content in units corresponding to the scenario, it is possible to suppress the display of the unique AR content in a scenario not related to the business process performed by the user. For example, the AR content group G3 is selectively provided to the user who performs the business process of the inspection work, while the AR content CB11 to CB13 is not provided to the user.

  By managing the AR content in groups of the AR content group G3 and the AR content group G4, the AR content is individually provided to the user according to the scenario. The AR content group G3 and the AR content having the same AR marker as a display reference are not limited so that provision of some AR content itself is not performed depending on conditions such as position and time. The group G4 can be provided in parallel to different users who perform corresponding business processes. That is, it is possible to provide the AR content group G4 to other users while providing the AR content group G3 to a certain user.

  The user decides which scenario to perform before starting a series of work (business process). That is, the scenario is selected before the AR marker is recognized. Further, the AR content group (G3 and G4) provided in units according to the scenario is a part of all types of AR content. Compared with the sequence example shown in FIG. 7, since the AR content group corresponding to the scenario is selectively downloaded, the data amount of the AR content held in the computer 1 is suppressed. Therefore, if the computer 1 holds the AR content group corresponding to the scenario before the recognition of the AR marker, the computer 1 displays the AR content corresponding to the recognition of the AR marker on the AR held in the computer 1. This can be done using content. Since the downloading of the AR content from the recognition of the AR marker to the display of the AR content is avoided, the occurrence of a time lag from the recognition of the AR marker to the display of the AR content is suppressed.

  FIG. 10 shows an example data structure of AR definition data. With the data structure shown in FIG. 10, AR content is managed and provided in units corresponding to scenarios. The data structure example shown in FIG. 10 has a tree structure.

  The root definition N of the tree structure indicates a plurality of scenarios which are AR content provision targets. In the example of FIG. 10, the identification information (scenario ID) of the inspection work scenario is 001, and the scenario ID of the manufacturing work scenario is 002. The route definition N includes information in which the scenario ID: 001 is associated with a pointer indicating a storage destination of information related to the scenario with the scenario ID: 001. Further, the route definition N includes information in which a scenario ID: 002 is associated with a pointer indicating a storage destination of information (scenario definition) regarding the scenario with the scenario ID: 002. Thus, when a scenario ID is selected, information related to the scenario can be referred to according to the selected scenario ID.

  The scenario definition N11 is information related to the scenario with scenario ID 001. The scenario definition N11 indicates a plurality of AR markers used for displaying AR content in the scenario with the scenario ID 001. For example, the identification information (marker ID) of the AR marker M1 is 001, and the marker ID of the AR marker M2 is 002. The scenario definition N11 includes information in which the marker ID of each marker used in the scenario with the scenario ID: 001 is associated with a pointer indicating the storage destination of information related to each marker (marker definition). Thereby, when a marker ID is selected, information related to the AR marker can be referred to according to the selected marker ID. Similarly to the scenario definition N11, the scenario definition N12 also makes it possible to refer to the marker definition of the AR marker used for the scenario ID: 002 according to the selection of the marker ID.

  The marker definition N301 is information regarding the AR marker M1 with the marker ID: 001. The marker definition N301 indicates AR content to be displayed in response to recognition of the AR marker M1. For example, the identification information (content ID) of the AR contents CA11 to CA14 is 001 to 004, and the content IDs of the AR contents C11 and C12 are 005 and 006. Further, for example, the content IDs of the AR contents CB11 to CB13 are 007 to 009, respectively. The marker definition N301 includes the content ID of each AR content displayed when the AR marker with the marker ID: 001 is recognized in the scenario with the scenario ID: 001, and the storage location of information (content definition) regarding each AR content. The information which linked | related with the pointer to show is contained. Thereby, when a content ID is selected, information related to the AR content can be referred to according to the selected content ID. Similarly to the marker definition N301, the marker definition N302 allows the AR content definition to be displayed when the AR marker with the marker ID: 002 is recognized in the scenario with the scenario ID: 001 to be referred to according to the selection of the content ID. . Similarly to the marker definitions N301 and N302, the marker definitions N303 and N304 also make it possible to refer to the AR content definition displayed when each marker is recognized in the scenario with the scenario ID: 002 according to the selection of the content ID.

  The content definition N401 is information related to the AR content with the content ID: 001. The content definition N401 includes definitions of the position, orientation, and size of the AR content with the content ID: 001 based on the marker ID: 001. The content definition N401 further includes identification information (template ID) of an AR template in which the shape and pattern of the AR content with the content ID: 001 is defined. Similarly to the content definition N401, each of the content definitions N402 to N406 includes the definition of the position, orientation, and size of the AR content with the content ID: 002 to 006 based on the marker ID: 001, and the template ID of the AR template. It is. Each of the content definitions N407 to N409 includes a definition of the position, orientation, and size of the AR content of the content ID: 007 to 009 with reference to the marker ID: 002, and a template ID of the AR template. Similarly to the content definition N401, each of the content definitions N410 to N412 includes the definition of the position, orientation, and size of the AR content of the content ID: 010 to 012 with reference to the marker ID: 001, and the template ID of the AR template. It is. Similarly to the content definition N401, each of the content definitions N413 and N414 includes the definition of the position, orientation, and size of the AR content with the content IDs: 005 and 006 based on the marker ID: 001 and the template ID of the AR template. It is. The content definition N415 also includes the definition of the position, orientation, and size of the AR content with the content ID: 013 based on the marker ID: 002, and the template ID of the AR template.

  The content definition N405 and the content definition N413 are content definitions related to the same AR content. The content definition N405 and the content definition N413 may be separate data or single data. When the overlapping content definition is a single piece of data, the same value is stored as a pointer that references the content definition. For example, the pointer that refers to the content definition N405 included in the marker definition N301 and the pointer that refers to the content definition N413 included in the marker definition N303 have the same value.

  In order to selectively provide the AR content group related to a specific scenario, a specific scenario may be selected in the route definition N. The AR content group related to a specific scenario exists under the scenario definition of the selected scenario ID. With the data structure shown in FIG. 10, AR content is managed and provided in units corresponding to scenarios.

  FIG. 11 shows a display example of AR content. The display example shown in FIG. 11 shows a screen S (3) on which the AR content associated with the recognized AR marker M1 is displayed for the inspection work scenario (scenario ID: 001 scenario). The screen S (3) includes an image of the AR marker M1, and the projected images of the AR contents CA11 to CA14, C11, and C12 generated based on the image of the marker M1 are displayed. Since the AR contents CA11 to CA14, C11, and C12 are defined under the scenario definition of the scenario ID: 001, they are displayed. The AR contents CB11 to CB13 are displayed on the screen S (2) shown in FIG. 8, whereas the AR contents CB11 to CB13 are not displayed on the screen S (3). This is because the AR contents CB11 to 13 are defined under the scenario definition of scenario ID: 002. By defining the AR content under the scenario as shown in FIG. 10, the AR content corresponding to the scenario is individually provided for each scenario as shown in the screen S (3).

[Selective provision according to step]
As described above, a scenario is configured by a plurality of operations. However, a plurality of operations may be configured in sequence as a series of steps (steps). If multiple steps are taken in order in a scenario, providing the user with AR content for a step that is not being performed by the user does not help the user to work on the step being performed. For example, in the screen S (3) of FIG. 11, if the user is a step of confirming the oil amount, the AR content CA13 indicating the procedure may be displayed, and the AR content CA11 or CA12 indicating the procedure is displayed. Even if it is done, it does not assist the work. Furthermore, the AR content (CA11 or CA12) targeted for a step different from the current step may be displayed overlapping the AR content to be displayed at the current step. For example, in the screen S (3), the operation instruction by the AR content CA13 to turn the valve for checking the oil amount to the right is not easily seen by the AR content CA11. Further, the display of the AR contents CA11 and CA12 blocks the field of view secured when the user views the captured image.

  Thus, for example, AR content is managed in units corresponding to steps. Within the group of AR content related to a specific scenario, a group for each step included in the scenario is further formed.

  FIG. 12 shows a group of AR content. AR content groups G31 to G35 shown in FIG. 12 are examples of AR content groups corresponding to the respective steps included in the scenario with the scenario ID: 001.

  In the example shown in FIG. 12, the scenario with scenario ID 001 is composed of five steps 1-5. In the operation of Step 1, the user performs the operation based on the AR content CA11 displayed as a procedure. In the operation of step 2, the user performs the operation based on the AR content CA12 displayed as a procedure. The AR contents CA11 and CA12 are displayed with the AR marker M1 as a reference. In the operation of Step 3, the user performs the operation based on the AR content CA21 displayed as a procedure. The AR content CA21 is displayed with the AR marker M2 as a reference. In the operation of Step 4, the user performs the operation based on the AR content CA13 of procedure display and the AR content CA14 of operation assistance. The AR contents CA13 and CA14 are displayed with the AR marker M1 as a reference. In the operation of Step 5, the user performs the operation based on the AR content CA22 of procedure display and the AR content CA23 of operation assistance. The AR contents CA22 and CA23 are displayed with the AR marker M2 as a reference.

  In the display control in the group for each step shown in FIG. 12, the status of the current step is managed, and for example, control for displaying the AR content of the current step is performed. At this time, the AR content related to other steps is not displayed.

  FIG. 13 shows an example data structure of AR definition data. With the data structure shown in FIG. 13, AR content is managed and provided in units corresponding to steps. The data structure shown in FIG. 13 has a tree structure similar to the data structure example shown in FIG. Since a step is a component included in a scenario, it is defined by data elements that are lower than the scenario and higher than the AR content in the tree structure.

  The route definition N of the tree structure in FIG. 13 is the same as the route definition N shown in FIG. 10, and by referring to the route definition N, information related to the scenario can be referred to according to the selected scenario ID.

  Each scenario definition illustrated in FIG. 13 includes a reference destination of information (step definition) regarding each step included in each scenario. The scenario definition N13 is information related to the scenario with the scenario ID: 001 and includes a pointer to the step definition for each of steps 1 to 5 included in the scenario. Referring to the scenario definition N13, the corresponding step definition can be referred to according to the step number.

  The step definition shown in FIG. 13 includes the reference destination of the marker definition of the AR marker that is a reference for AR content display in the step. For example, the step definition N21 includes a pointer to the marker definition N311 of the AR marker with the marker ID: 001. When the step definition N311 is referred to, the corresponding marker definition can be referred to according to the marker ID.

  The marker definition shown in FIG. 13 includes a pointer to the content definition of the AR content displayed in response to the recognition of the AR marker in the step defined in the step definition above the marker definition. For example, the marker definition N311 includes a pointer to the content definition of the AR content to be displayed when the AR marker M1 with the marker ID: 001 is recognized in step 1.

  In order to display AR contents in units of steps with the data structure shown in FIG. 13, the steps may be sequentially selected in the scenario definition N13. The AR content group to be displayed in each step is defined under the step definition that is sequentially selected.

  FIG. 14 shows a display example of AR content. Screen S (4) shown in FIG. 14 is a screen example when the AR content group under the selected step definition is displayed in the data structure shown in FIG. On the screen S (4), the AR contents defined in the content definitions N403 and N404 under the step definition N24 (corresponding to step 4 in FIG. 12) are displayed. As with the AR contents CA13 and CA14 (corresponding to the content definitions N403 and N404), the AR contents CA11 and CA12 (corresponding to the content definitions N401 and N402) using the AR marker M1 as a display reference are not displayed. In FIG. 14, the AR contents CA11 and CA12 are not displayed by the dotted line display of the AR contents. This is because the content definitions N401 and N402 do not exist under the step definition N24. That is, by using the data structure managed by the AR content in units corresponding to the steps, display of the AR content relating to a process different from the current process is suppressed.

  As another display control example, AR content related to a process different from the current process may not be completely hidden, for example, and only a wire frame may be displayed without mapping a texture. For example, in addition to displaying the AR content group under the step definition N24 of the current step, only the wire frame is displayed without texture mapping for the AR content groups under the step definitions N21 to N23 and N25 of the other steps. Also good. Thereby, it is suppressed that it becomes difficult to see the AR content of the current process due to the display of the AR content regarding the process different from the current process. Further, control may be performed in which the AR content of only the wire frame relating to a process different from the current process is displayed on the back surface, and the AR content of the current process subjected to texture mapping is displayed on the front surface.

  FIG. 15 shows a sequence example of AR content provision. By managing the AR content in units corresponding to the steps, the AR content can be downloaded in stages. When the scenario is selected by the computer 1 after the program is started (t1), the selected scenario definition and AR definition data (scenario definition, step definition, marker definition and content definition) under the scenario definition are downloaded. Since the AR content is displayed using the AR template data indicated in the content definition, the AR template is sequentially downloaded after the AR definition data download is completed (t2). At this time, by preferentially downloading the AR template used in the first step in the selected scenario, the start timing of the AR content display can be made earlier than the time point when the download of all AR content included in the scenario is completed. . At least when the AR template related to step 1 is downloaded (t3), the AR content display related to step 1 can be started.

[Functional configuration of this embodiment]
FIG. 16 shows a functional block configuration example of the computer 1. The computer 1 includes a control unit 10, an acquisition unit 11, a recognition unit 12, a calculation unit 13, a generation unit 14, a storage unit 15, an imaging unit 16, a display unit 17, a communication unit 18, and an input unit 19. The control unit 10 includes a selection control unit 101, an event detection unit 102, a display control unit 103, and a setting management unit 104.

  The control unit 10 controls provision of AR content and editing of AR content by controlling other functional blocks. The acquisition unit 11 downloads AR content from the computer 2. The recognition unit 12 recognizes an AR marker based on the captured image captured by the imaging unit 16. The recognition unit 12 reads the marker ID for identifying the AR marker and calculates the coordinates of the AR marker in the camera coordinate system in the recognition of the AR marker. The calculation unit 13 converts the AR content position information (marker coordinate system) into position information (screen coordinate system). In addition, the calculation unit 13 calculates the position of the marker coordinate system corresponding to the position on the screen based on the coordinates of the AR marker calculated by the recognition unit 12. The generation unit 14 generates a projected image of the AR content based on the position information calculated by the calculation unit 13.

  The storage unit 15 stores information used in other functional blocks. For example, the AR content downloaded by the acquisition unit 11 and management information used for displaying / editing the AR content are stored. The imaging unit 16 performs an imaging process. A captured image obtained by the imaging process of the imaging unit 16 is also stored in the storage unit 15. The display unit 17 displays an image. The communication unit 18 performs data communication with other devices (for example, the computer 2) via the network 3. The input unit 19 detects the operation of the operator of the computer 1 and transmits a signal corresponding to the detection to the control unit 10.

  The selection control unit 101, the event detection unit 102, the display control unit 103, and the setting management unit 104 are functional blocks that execute a part of the processing procedure of the AR content display / editing process executed by the control unit 10. The selection control unit 101 generates a scenario list display event, and generates an AR content acquisition event according to the selection in the scenario list. The scenario list will be described later. The event detection unit 102 generates a switching event based on the recognition result of the recognition unit 12. The display control unit 103 changes the screen displayed on the display unit 17 in accordance with the state transition definition. The screen transition by the display control unit 103 is performed according to the generation of the display event. When the generation unit 13 generates a projection image, the control unit 10 generates a projection image display event. When the display event of the projection image is generated, the display control unit 103 performs control to cause the display unit 17 to display the projection image so as to overlap the captured image captured by the imaging unit 16. The setting management unit 104 manages the setting of calculation information generated by the calculation unit 13.

[AR content display processing procedure]
17 and 18 show the processing procedure of the AR function. The AR function whose processing procedure is shown in FIGS. 17 and 18 provides two modes: an AR display mode and an AR editing mode. The AR function is called, for example, when a program start instruction is given by user input. When the AR function is called, the control unit 10 generates a display event of the mode selection screen S (5), and the display unit 17 displays the mode selection screen S (5) according to the generation of the display event (S101).

  FIG. 19 shows an example of a mode selection screen and a scenario selection screen. Mode selection screen S (5) includes button B1 and button B2. An area in the screen is assigned to each of the buttons B1 and B2. Display indicating the AR display mode is performed in the area assigned to the button B1, and the AR display mode is selected by input to the area assigned to the button B1. Similarly to the button B1, the button B2 displays the AR editing mode in the area assigned to the button B2, and the AR editing mode is selected by input to the area assigned to the button B2.

  The scenario selection screen S (6) is a screen including the button B3, the button B4, and the button group B5, and is displayed when the AR display mode is selected on the mode selection screen S (5). An area in the screen is allocated to the button B3, and a display for returning to the mode selection screen S (5) is performed in the allocated area. Further, the mode selection screen S (5) is displayed by input into the area assigned to the button B3. An area in the screen is assigned to the button B4, and a display for ending the AR function is displayed in the assigned area. Further, the AR function end processing is executed by input into the area assigned to the button B4. An area in the screen is allocated to each button group B5, and a scenario name labeled with the scenario is displayed in each allocated area. A scenario is selected according to the input in the area where the scenario name is displayed.

  The scenario selection screen S (7) also includes a button B3 and a button B4, and further includes a button group B6. The button group B6 includes buttons similar to the button group B5 and buttons corresponding to the new scenario. When the new scenario button is selected, a new scenario is created.

  When the input unit 19 detects an input operation in the state where the mode selection screen is displayed in S101 of FIG. 17, the input unit 19 transmits the user input on the mode selection screen to the control unit 10, and the control unit 10 receives the user input. To determine whether or not the AR display mode is selected (S102). When the AR display mode is selected on the mode selection screen, the AR content display process corresponding to the scenario is executed. On the other hand, when the AR edit mode is selected, the AR content edit process is executed.

  When the AR display mode is selected on the mode selection screen, the selection control unit 101 generates a scenario list acquisition event, and the acquisition unit 11 generates a scenario list from the computer 2 via the communication unit 18 in response to generation of the acquisition event. get. The scenario list is generated based on the route definition N included in the AR definition data shown in FIGS.

  Further, a plurality of types of scenario lists may be prepared, and the scenario list provided may be switched according to information (for example, an account) related to the user who is the subject of the work. For example, before S101, a login operation based on input of user account information, a password, and the like is performed, and a scenario list corresponding to the user account information input at the time of login is provided. Thereby, it is possible to provide a scenario according to the authority of the user assigned in an organization that handles a plurality of business processes.

  FIG. 20 shows an example of the data structure of the route definition. Each data included in the AR definition data includes, for example, a list (information unit 1 and information unit group 2) of information (information unit) having a predetermined data length (for example, 4 bytes) as a unit. In the first information unit (information unit 1) of the route definition N, the number Ns of scenarios existing under the route is indicated. The route definition N includes Ns information units subsequent to the information unit indicating the number of scenarios Ns. The second and subsequent information units (information unit group 2) include a scenario ID for identifying a scenario under the route and a pointer indicating a storage location of the scenario definition.

  The scenario list acquired by the acquisition unit 11 may be the route definition itself, or may be obtained by removing the pointer to the scenario definition under the route. Further, information in which a corresponding scenario name is associated with each scenario ID under the route may be used as a scenario list. Further, for example, in a configuration in which a plurality of types of scenario lists are switched according to a user account, a plurality of AR definition data are prepared in a plurality of ways.

  When the scenario list is acquired in S103 of FIG. 17, the selection control unit 101 generates a display event of the scenario selection screen S (6), and the display control unit 103 selects a scenario on the display unit 17 in accordance with the generation of the display event. Screen S (6) is displayed (S104). When the input unit 19 detects an input operation in a state where the scenario selection screen is displayed, the input unit 19 transmits the user input on the scenario selection screen to the control unit 10, and the control unit 10 determines the selection content by the user input. (S105).

  In S105, when the button B3 for displaying the return to the mode selection screen S (5) ("Return") is selected (S105: "Return"), the process of S101 is performed again. Further, when the button B4 for displaying that the AR function is to be ended (“END”) is selected in S105 (S105: “END”), the AR function is ended. In S105, when any scenario shown in the button group B5 is selected (S105: “scenario”), the control unit 10 provides the acquisition unit 11 with the AR content group provided in the scenario selected in S105. Is started (S106).

  FIG. 21 shows a processing procedure example of the download function. The acquisition unit 11 transmits the scenario ID of the selected scenario to the computer 2 via the communication unit 18 (S201). The computer 2 transmits AR definition data corresponding to the scenario ID transmitted by the acquisition unit 11 to the computer 1. The processing procedure of the computer 2 is illustrated in FIG. The acquisition unit 11 receives the AR definition data transmitted from the computer 2 via the communication unit 18 (S202).

  An example of the data structure of the AR definition data is shown in FIG. 10 or FIG. A route definition N which is a part of the AR definition data is shown in FIG. For each of the scenario definition, step definition, marker definition, and content definition in the data structure example shown in FIG. 13, examples of the data structure are shown in FIGS.

  FIG. 22 shows an example of the data structure of the scenario definition. The scenario definition N13 shown in FIG. 22 has a storage position indicated by a pointer included in the route definition N. The scenario definition is composed of information units (information unit 1, information unit group 2, information unit 3, information unit group 4) having a predetermined data length. The first information unit (information unit 1) of the scenario definition N13 includes the number of step definitions under the scenario definition N13 (5 in the scenario definition N13) and a pointer indicating the storage location of the route definition N. The number of step definition information units (information unit group 2) indicated in the first information unit is included in the scenario definition N13 following the first information unit. These information units include the step number of the step definition under the scenario definition N13 and a pointer indicating the storage location of the step definition. Further, the number of marker definitions under the scenario definition N13 (1 in the scenario definition N13) is shown in the information unit (information unit 3) subsequent to the information related to the step definition under the scenario definition N13. Furthermore, the number of marker definitions under the scenario definition N13 (information unit group 4) is subsequently included in the scenario definition N13. These information units include a marker ID of the marker definition under the scenario definition N13 and a pointer indicating the storage location of the marker definition.

  FIG. 23 shows an example of the data structure of the step definition. In the step definition N21 shown in FIG. 23, the storage position is indicated by a pointer included in the scenario definition N13. The step definition is composed of information units (information unit 1 and information unit group 2) having a predetermined data length. The first information unit (information unit 1) of the step definition N21 includes the number of marker definitions under the step definition N21 (1 in the step definition N21) and a pointer indicating the storage position of the scenario definition N13. The number of marker definition information units (information unit group 2) shown in the information unit 1 is included in the step definition N21 following the first information unit. These information units include a marker ID of the marker definition under the step definition N21 and a pointer indicating the storage location of the marker definition.

  FIG. 24 shows an example of the data structure of the marker definition. The marker definition N311 shown in FIG. 24 has a storage position indicated by a pointer included in the step definition N21. The marker definition is composed of information units (information unit 1 and information unit group 2) having a predetermined data length. The first information unit (information unit 1) of the marker definition N311 includes the number of content definitions under the marker definition N311 (1 in the marker definition N311) and a pointer indicating the storage location of the step definition N21. The number of content definition information units (information unit group 2) shown in the first information unit is included in the marker definition N311 following the first information unit. These information units include the content ID of the content definition under the marker definition N311 and a pointer indicating the storage location of the content definition.

  FIG. 25 shows an example data structure of content definition. In the content definition N401 shown in FIG. 25, the storage position is indicated by a pointer included in the marker definition N311. The content definition is composed of information units (information units 1 to 5) having a predetermined data length. The first information unit (information unit 1) of the content definition N401 includes information indicating content definition and a pointer to the marker definition N311. The second information unit (information unit 2) of the content definition N401 indicates the content ID of the content definition N401 and the template ID of the AR template used for the AR content defined by the content definition N401. In the third information unit (information unit 3) of the content definition N401, position coordinates (Xm, Ym, Zm) in the marker coordinate system are indicated. The reference point of the AR content is set at this position coordinate. In the fourth information unit (information unit 4) of the content definition N401, rotational coordinates (Pm, Qm, Rm) in the marker coordinate system are indicated. Furthermore, the fifth information unit (information unit 5) of the content definition N401 indicates the magnification (Jx, Jy, Jz) of the AR content.

  When the AR definition data under the selected scenario is acquired, the acquisition unit 11 generates the reception state table T1 in the work area of the storage unit 15 (S203).

  FIG. 26 shows an example of the reception state table. The reception state table T1 is a table that associates, for each AR content included in the selected scenario, the step number of the step in which the AR content is displayed, the content ID, and information indicating whether or not the reception state is received. In the reception state, the value “0” indicates that the content data has not been received, and the value “1” indicates that the content data has been received. The example of the reception state table shown in FIG. 26 is an example when scenario ID: 001 is selected. When reception of content data (AR template and image file) of AR content is completed, the reception state is changed from “0” to “1”.

  When the reception state table T1 is generated, the acquisition unit 11 refers to the reception state table T1 and determines whether there is unreceived content data (S204). If there is unreceived content data (S204: YES), the acquisition unit 11 receives the content data (S205). As will be described later with reference to FIG. 52, the computer 2 sequentially transmits content data of AR content included in the selected scenario for each step.

  Content data is information indicating the shape and pattern of AR content whose position, orientation, and size are defined by content definition. The content data includes an AR template and an image file shown in the AR template.

  FIG. 27 shows an example of the data structure of the AR template. The AR template T2 includes the template ID of the AR template, coordinate data T21 of each vertex constituting the AR template, and surface configuration data T22 of each surface constituting the AR template. The information of each surface included in the surface configuration data includes a vertex order that is the order of the vertices constituting the surface and designation of the texture ID. The texture ID indicates the identification information (image file identification information) of the texture mapped to the surface. The reference point of the AR template is, for example, the 0th vertex.

  When a predetermined time elapses after S205 is executed, the acquisition unit 11 updates the reception state table T1 (S206). The acquisition unit 11 changes the reception state of the reception state table T1 from “0” to “1” for the AR content whose content data has been received. When the AR template T2 corresponding to the AR content and the image file specified by the AR template T2 are all downloaded, it is determined that the reception of the AR content has been completed. When the process of S206 is performed, the acquisition unit 11 performs the determination of S204 again.

  If it is determined by reference to the reception state table T1 that there is no unreceived content data (S204: NO), the acquisition unit 11 ends the download function.

  When the acquisition unit 11 starts the download process, the control unit 10 causes the recognition unit 12 to determine whether an AR marker is detected (S107). The recognition unit 12 determines that the AR marker has been detected when the image of the AR marker is included in the captured image captured by the imaging unit 16. Whether or not an AR marker image is included is determined, for example, based on whether or not a parallelogram-shaped image is included in the captured image. When the recognition unit 12 receives a determination result indicating that the AR marker is not detected (S107: NO), the process of S107 is executed again after a predetermined time has elapsed. The imaging unit 16 periodically performs an imaging process, and when the recognition unit 12 performs the process of S107 again, a determination is made based on another captured image.

  When the determination result that the AR marker is detected is received from the recognition unit 12 (S107: YES), the control unit 10 causes the recognition unit 12 to execute a marker recognition process (S108). In the marker recognition process, for the AR marker detected in S107, calculation of the AR marker coordinates (position coordinates and rotation coordinates) in the camera coordinate system and reading of the marker ID of the AR marker are performed. Based on the calculated AR marker coordinates and the read marker ID, the recognition marker information D1 and the marker movement information D2 are updated. The recognition marker information D1 and the marker movement information D2 are updated in the event detection process of S109.

  FIG. 28 shows an example of recognition marker information and marker movement information. The recognition marker information D1 shown in FIG. 28 indicates the number of AR markers recognized by the marker recognition processing in S108, the marker ID for each recognized AR marker, and a pointer to movement information. The marker movement information D2 indicated by the pointer included in the recognition marker information D1 indicates the position coordinates (Xs, Ys) and the rotation coordinates (R1c) of the AR marker image calculated by the most recent predetermined number of marker recognitions. The marker movement information D2 indicates, for example, the position coordinates (Xs, Ys) and the rotation coordinates (R1c) of the image of the AR marker up to the previous marker recognition process. The unit of position coordinates is, for example, a pixel, and the unit of rotational coordinates is, for example, degrees. The recognition marker information D1 and the marker movement information D2 are stored in, for example, a register.

  When the marker recognition process is performed, the control unit 10 executes an event detection process (S109). Depending on the detection of an event by the event detection process, the step in the selected scenario proceeds or retreats.

  FIG. 29 shows an example of scenario state information. The state information D0 shown in FIG. 29 indicates the progress of the scenario. The state information D0 is stored in, for example, a register. The state information D0 indicates the scenario ID of the scenario selected in S105, the number of steps included in the selected scenario, and the current step. When a step progresses or reverses due to the detection of an event, information on the current step in the state information D0 is updated.

  FIG. 30 shows an example of an event detection processing procedure. In the event detection process shown in FIG. 30, the step number is incremented or decremented when the AR marker image moves laterally and out of the frame. When the frame is out in the positive direction in the screen coordinate system, the step number is incremented, and when the frame is out in the negative direction, the step number is decremented.

  When the event detection process is started, the event detection unit 102 refers to the recognition marker information D1 and determines whether or not there is an AR marker that has been recognized up to the previous time and that has not been recognized in the current marker recognition process. Is determined (S301). The recognition marker information D1 referred to in S301 does not reflect the result of the marker recognition process (current marker recognition process) performed immediately before the event detection process of FIG. In other words, the recognition marker information D1 referred to in S301 includes the results of the marker recognition process up to the immediately preceding marker recognition process (current marker recognition process). When there is an unrecognized AR marker in S301 (S301: YES), the event detection unit 102 acquires marker movement information of the unrecognized AR marker detected in S301 based on the recognized marker information (S302). ).

  The event detection unit 102 further determines the absolute maximum value of the coordinates in the Xs direction and the amount of change in the Ys direction (the maximum value of Ys and the minimum value of Ys) based on the position coordinates included in the marker movement information acquired in S302. Is calculated (S303). Subsequently, the event detection unit 102 determines whether or not the amount of change in the Ys direction calculated in S303 is equal to or less than a predetermined value (S304). If the amount of change in Ys calculated in S303 is less than or equal to a predetermined value, it is determined that the camera of the computer 1 has not moved significantly in the vertical direction. If the amount of change in Ys calculated in S303 is less than or equal to a predetermined value (S303: YES), the event detection unit 102 determines whether or not the maximum absolute value in the Xs direction calculated in S303 is greater than or equal to a predetermined value. Is determined (S305). If the maximum value of the absolute value in the Xs direction calculated in S303 is equal to or greater than a predetermined value, it is determined that the AR marker image is present in the right end region or the left end region of the screen. The screen coordinate system is configured in two dimensions (Xs, Ys) as described above, and for example, the center of the captured image obtained by the imaging process of the camera is set as the origin (origin Os). As the predetermined value, a value smaller than the maximum value of Xs (corresponding to the right end of the screen) and larger than 0 is used, for example, a value such as 3/4 of the maximum value of Xs is used. If the maximum value of the absolute value in the Xs direction calculated in S303 is greater than or equal to a predetermined value (S305: YES), the event detection unit 102 determines whether Xs that maximizes the absolute value in the Xs direction is a positive value. Is determined (S306).

  When Xs that maximizes the absolute value in the Xs direction is a positive value (S306: YES), the event detection unit 102 increments the current step number of the state information D0 (S307). On the other hand, when Xs that maximizes the absolute value in the Xs direction is a negative value (S306: NO), the event detection unit 102 decrements the current step number of the state information D0 (S308).

  When there is no unrecognized marker among the AR markers recognized in the previous marker recognition process (S301: NO), and when the change amount of Ys is larger than a predetermined value (S304: NO), the Xs direction calculated in S303 When the maximum absolute value is smaller than the predetermined value (S305: NO), when the process of S307 or S308 is performed, the event detection unit 102 performs the current marker recognition process S108. The recognition result is reflected in the recognition marker information D1 and the marker movement information D2 (S309). When the process of S309 is performed, the event detection process ends.

  FIG. 31 shows examples of markers displayed on the screen. Screen S (8) shown in FIG. 31 is an example of the display screen in step 1 of scenario ID: 001. Since it is the state of step 1, AR content CA11 is displayed, and the display of AR content CA12 and CA13 is suppressed (or only a part is displayed). When the image of the AR marker M1 displayed on the screen S (8) moves in the direction of the arrow and goes out of the frame, the step number is incremented because the maximum value of Xs is a positive value.

  A screen S (9) shown in FIG. 31 is an example of a screen when the AR marker M1 enters the frame again after being out of the frame as shown in the screen S (8). When the AR marker is framed in, the marker is recognized by the marker recognition process, and AR content corresponding to the recognized marker is displayed. In that case, since one step has progressed from the screen S (8), the AR content CA12 corresponding to step 2 of the scenario ID: 001 is displayed. Further, the display of the AR content CA11 displayed on the screen (8) is suppressed (or only a part is displayed).

  According to the switching of the steps according to the event detection process shown in FIGS. 30 and 31, the user simply changes the direction of the camera while holding the computer 1 (with the posture of browsing the display of the AR content). Steps can be switched. In addition, according to the procedure of FIG. 30, the frame-out direction is determined only at the timing when the AR marker is no longer recognized.

  For example, the step switching may be performed according to a transition action such as a button tap or a display tap / flick. However, when the computer 1 is a tablet terminal or the like, the user holds the tablet terminal with one hand in order to perform operations such as tapping and flicking. As the size of the terminal increases, it becomes difficult to hold it stably with one hand.

  Further, for example, switching of steps may be performed according to a gesture of the main body of the computer 1. The determination by the gesture is performed according to the measurement value by the acceleration sensor, the analysis of the captured image, or the like. However, in order to determine the timing when the gesture is started, numerical calculation processing is always performed.

  In the above description using FIG. 31, the step number is incremented when the image of the AR marker M1 is out of the frame on the right side of the screen, but it may be decremented. In this case, for example, when the image of the AR marker M1 is out of the frame on the left side of the screen, the step number is incremented.

  Moreover, the trigger of determination may be frame-in instead of frame-out. For example, when an AR marker that has not been recognized up to the previous time is recognized in S301 of FIG. 30, a flag is set, and the step number is set according to the moving direction of the image of the AR marker in the state where the flag is set. Incremented or decremented.

  When the event detection process (S109) is performed by the event detection unit 102, the control unit 10 controls the calculation unit 13 and the generation unit 14 to execute a projection image generation process (S110).

  FIG. 32 shows an example of the processing procedure of the projection image generation process. The control unit 10 determines whether or not the recognition target AR marker defined in the step definition of the current step number indicated in the scenario state information exists in the recognition marker information (S401). That the AR marker to be recognized in the current step is not indicated in the recognition marker information means that there is no AR content to be displayed in the current step. Therefore, when the recognition target AR marker is not indicated in the recognition marker information (S401: NO), the marker detection procedure of S107 is performed. At this time, the display control unit 103 may cause the display unit 17 to display guide information indicating that the AR marker to be recognized is different.

  When there is an AR marker to be recognized at the current step (S401: YES), the control unit 10 stores information on the AR marker to be recognized at the current step (marker definition, content definition under the marker definition). Read (S402). The control unit 10 determines whether or not there is an AR content that has not been received from the content data among the AR content corresponding to the content definition read in S402 (S403). The control unit 10 confirms the content reception status for the current step number indicated in the scenario status information by referring to the reception status table T1, and performs the determination in S403. When the AR content is not received (S403: NO), the control unit 10 waits until the AR content is received and performs the determination of S403 again. Or the display control part 103 may display on the display part 17 that it is waiting for reception, and the procedure of the marker detection of S107 may be performed after that.

  When the AR content to be displayed is received (S403: YES), the control unit 10 performs the processing of S405 to S407 for each AR content to be displayed (repeated by S404 and S408). The AR content to be displayed is AR content defined in the content definition under the step definition of the current step.

  When the AR content is selected in S404, the control unit 10 causes the calculation unit 13 to perform coordinate conversion on each vertex coordinate defined in the AR template indicated in the content definition of the selected AR content based on the content definition ( S405). The calculation unit 13 rotates each vertex coordinate based on the content-defined rotation coordinates (Pm, Qm, Rm), expands / contracts the rotated vertex coordinates according to the magnification (Jx, Jy, Jz), and further the position coordinates. Translation is performed according to (Xm, Ym, Zm). The control unit 10 further causes the calculation unit 13 to execute model-view conversion and perspective conversion on each vertex coordinate whose coordinates are converted in S405 (S406). Through S405 and S406, each vertex coordinate of the AR content selected in S404 is expressed in the screen coordinate system.

  The control unit 10 causes the generation unit 14 to perform texture mapping for each surface definition defined in the AR template based on the vertex coordinates that have undergone coordinate conversion in S405 and S406 (S407). The generation unit 14 maps the image file (image specified by the texture ID) defined in the surface definition to an area obtained by interpolating the coordinates in the screen coordinate system of each vertex defined in the surface definition with a line.

  When the generation unit 14 generates a plane image in which the shape and position for each plane definition are specified, the display control unit 103 arranges each plane image on the transparent image layer (S408). The transparent image layer is an image having a display screen size and is transparent by default. For this reason, the image of the layer existing behind the transparent image layer is displayed as it is. The surface image arranged in S408 is displayed in front of the image of the back layer.

  When the processing of S405 to S408 is performed on each AR content to be displayed, the control unit 10 ends the projection image generation processing, and the display control unit 103 updates the projection image (S111). The display control unit 103 controls the display unit 17 to display a transparent image layer in which each surface image of each AR content is arranged in front of the captured image in the process of S110.

  Thereafter, the control unit 10 determines whether or not to end the scenario (S112). The control unit 10 refers to the scenario state information, and determines that the scenario is to end when the current step number is larger than the number of steps. If the current step number is equal to or smaller than the number of steps, the control unit 10 does not end the scenario. When the scenario is not terminated in S112 (S112: NO), the control unit 10 performs the marker detection process in S107.

  When ending the scenario in S112, the display control unit 103 displays a selection screen on the display unit 17 (S113). On the selection screen displayed in S113, options of “end”, “other scenario”, and “other mode” are shown. The control unit 10 determines whether the selection input received by the input unit 19 in a state where the selection screen is displayed in S113 is “End”, “Other scenario”, or “Other mode” (S114). ).

  When the selection input is “end” in S114 (S114: “end”), the control unit 10 ends the AR function. When the selection input is “other scenario” in S114 (S114: “other scenario”), the display control unit 103 causes the display unit 17 to display the scenario selection screen in S104. When the selection input is “other mode” in S114 (S114: “other mode”), the display control unit 103 causes the display unit 17 to display the mode selection screen in S101.

[Other methods of switching event detection]
In the event detection procedure of FIG. 30, a step switching event is detected depending on the direction of the marker frame out. Another procedure for step switching determination according to the recognized marker mode will be described.

  For example, a step switching indicator (index image) is displayed on the screen, and the step is switched in accordance with the fact that the marker image in the captured image is aligned with the indicator. The indicator is, for example, a square frame shape.

  FIG. 33 shows examples of markers displayed on the screen. A screen S (10) shown in FIG. 33 is an example of the screen in step 1 of a scenario. An indicator I1 is displayed on the screen S (10). The indicator I1 is a square frame-shaped figure and indicates the number “2”. Further, the image of the AR marker M1 is reflected on the screen S (10). The displayed captured image is sequentially updated, and marker recognition processing (S108) is performed on the updated captured image.

  A screen S (11) shows a screen when the position of the AR marker M1 obtained by the marker recognition process overlaps the display position of the indicator I1. When the state of the screen S (11) is entered, switching from step 1 to step 2 is performed. When switched to step 2, display corresponding to step 2 is performed according to recognition of the AR marker M1.

  Screen S (12) is an example of a screen on which display corresponding to step 2 is performed. On the screen S (12), an image of the AR marker M1, the AR contents C31 and C32 to be displayed in step 2, and indicators I2 and I3 are displayed. Indicators I2 and I3 are also rectangular frame shapes like indicator I1, and indicate numbers “3” and “1”, respectively. When the image of the AR marker M1 is recognized at the position within the screen of the indicator I2, the step proceeds (switching from step 2 to step 3), and the image of the AR marker is recognized at the position within the screen of the indicator I3. Then, step backwards (switching from step 2 to step 1) is performed.

  When the AR marker image is recognized at the position in the screen of the indicator I2, the screen S (13) is displayed. Screen S (13) is an example of a screen on which display corresponding to step 3 is performed. The AR contents C33 and C34 to be displayed in step 3 and indicators I4 and I1 are displayed. The indicator I4 is also a square frame shape like the indicator I1, and indicates the number “4”. When the image of the AR marker M1 is recognized at the position within the screen of the indicator I4, the step proceeds (switching from step 3 to step 4), and the image of the AR marker is recognized at the position within the screen of the indicator I1. Then, step backwards (switching from step 3 to step 2) are performed.

  FIG. 34 shows an example of the data structure of the step definition. In the step switching shown in FIG. 33, the display position of the indicator is designated at each step. The display position of the indicator is defined in each step definition. The step definition N22a corresponding to step 2 shown in FIG. 33 is composed of a series of information (information units 1 to 4) with a predetermined data length (for example, 4 bytes) as a unit, like the step definition N21.

  In the information unit 1, the number of AR markers used as a reference for AR content display in step 2 and a pointer to the scenario definition are shown. In the information unit 2, the marker ID of the AR marker and the pointer to the marker definition used as the reference for displaying the AR content in step 2 are shown. In the information unit 3, the display position (Xs, Ys) of the indicator used for the determination to reverse the step from step 2 is shown. In addition, the information unit 4 indicates the display position (Xs, Ys) of the indicator used for the determination of proceeding from step 2 to step. When the step is switched, an indicator is displayed according to the information units 3 and 4.

  FIG. 35 shows an example of an event detection processing procedure. In the method of determining step switching according to the display position of the indicator and the AR marker, when the event detection process is started, the control unit 10 displays the position (Xs, AR) of the AR marker recognized by the marker recognition process. Ys) is acquired (S501).

  The control unit 10 refers to the step definition of the current step, and determines whether or not the position of the AR marker acquired in S501 is a position for step backward (S502). If it is determined in S502 that the position of the AR marker is the backward position (S502: YES), the control unit 10 decrements the current step number of the scenario state information D0 (S503).

  If it is determined in S502 that the position of the AR marker is not a backward position (S502: NO), the control unit 10 refers to the step definition of the current step, and the position of the AR marker acquired in S501 is the same. It is determined whether or not the position is a step forward position (S504). If it is determined in S504 that the position of the AR marker is a forward position (S504: YES), the control unit 10 increments the current step number of the scenario state information D0 (S505).

  When S503 or S505 is performed, the control unit 10 generates an indicator indicating a backward position and an indicator indicating a forward position based on the step definition of the current step number indicated in the state information D0. (S506). Furthermore, the display control unit 103 updates the indicator displayed on the display unit 17 to the indicator generated in S506 (S507). When S507 is performed or when it is determined that the position of the AR marker is not the forward position (S504: NO), the control unit 10 ends the event detection process.

  In another example of the event detection method, the step is switched according to the rotation angle of the AR marker image displayed on the screen.

  FIG. 36 shows an example of a marker displayed on the screen. When the AR marker M1 displayed at the square position indicated by M1 (1) is displayed at the square position indicated by M1 (2) at another timing, the AR marker M1 is recognized in a state rotated by an angle Rs. It will be done. When the rotation angle is larger than a predetermined value, it is possible to determine step switching according to the direction of rotation. Incidentally, the AR marker is also rotated by an operation in which the user rotates the camera with respect to the fixed AR marker.

  FIG. 37 shows an example of an event detection processing procedure. When the event detection process is started, the control unit 10 acquires rotational coordinate information from the marker movement information D1 of the recognized AR marker (S601). Next, the control unit 10 determines whether or not the recognized AR marker has been rotated (S602). The determination in S602 is made according to whether or not the rotation angle Rs is greater than or equal to a predetermined value. Since the rotation angle R1c is calculated by the marker recognition processing, for example, the difference between the maximum value and the minimum value of the rotation angle (R1c) included in the marker movement information may be used instead of the rotation angle Rs.

  When it is determined in S602 that rotation has occurred (S602: YES), the control unit 10 determines whether or not the direction of rotation is a positive direction (S603). The direction of rotation can be determined based on which of the maximum value and the minimum value of the rotation angle (R1c) included in the marker movement information is the result of the previous marker recognition process. If the minimum value is the processing result recognized earlier, it is determined that the rotation is in the positive direction because the rotation is in the direction of increasing the rotation angle.

  When it is determined in S603 that the rotation is in the positive direction (S603: YES), the control unit 10 increments the current step number of the state information D0 (S604). On the other hand, when it is determined in S603 that the rotation is not in the positive direction (S603: NO), the control unit 10 decrements the current step number of the state information D0 (S605).

  When the process of S604 or S605 is performed, or when it is determined in S602 that no rotation has occurred (S602: NO), the control unit 10 displays the marker movement information based on the result of the marker recognition process of S108. Update (S606). When S606 is performed, the event detection process ends.

  Although three types of methods have been described for detecting a step switching event, other methods may be used. In addition, any of the above three detection methods may be appropriately modified and implemented. For example, in the method of detecting a switching event by marker out, the event may be detected based on the marker frame-in. Further, in the above-described example, a step advance event is detected when marker out is performed at the right end of the screen, and a step backward event is detected when performed at the left end of the screen, but the direction may be reversed. Further, the determination may be made based on the frame out / in at the upper end of the screen, or may be made based on the frame out / in at the lower end of the screen.

  Further, in the example of event detection according to the rotation of the marker, the step advance event is detected by the forward rotation, but conversely, the step backward event may be detected by the forward rotation. In that case, for example, a step progress event is detected by rotation in the negative direction.

[Exception scenario occurred]
In the present embodiment, one scenario is selectively provided from a plurality of scenarios, but it may be configured to transition to another scenario while the scenario proceeds. In one example, the transition destination scenario is associated with the transition condition, and the transition destination scenario is started when the transition condition is satisfied while the scenario is in progress.

  FIG. 38 shows an example of the data structure of the scenario definition. In the scenario definition N13a shown in FIG. 38, a first information unit, a second information unit group, a third information unit, a fourth information unit (group), a fifth information unit, and a sixth information unit group are included. included.

  The first information unit indicates the number of steps included in the scenario (the number of second information units) and a pointer to the route definition. The second information unit group is data subsequent to the first information unit, and each of the second information units indicates a step number for each step and a pointer to the step definition. The third information unit is data subsequent to the second information unit group, and indicates the number of marker definitions (number of fourth information units) immediately below the scenario. The fourth information unit is data subsequent to the third information unit, and indicates a marker ID of the marker definition immediately below the scenario and a pointer to the marker definition. The fifth information unit is data subsequent to the fourth information unit, and indicates the number of scenario transition definitions (the number of sixth information units). The sixth information unit is data subsequent to the fifth information unit, and indicates the scenario ID of the transition destination scenario and the transition condition.

  Detection that the transition condition is satisfied is performed at any stage from S107 to S112 in FIG. For example, when determining whether or not the scenario is ended in S112, it may be determined based on whether a specific input satisfying the transition condition has been made by the user, or parameters (remaining battery power etc.) inside the computer 1 May be determined according to whether the transition condition is satisfied.

  When the transition condition is satisfied, the control unit 10 changes the scenario ID of the state information D0 to the scenario ID of the transition destination scenario, and executes the procedure of S106 in FIG. The AR content of the transition destination scenario is acquired by the process of S106.

[AR content editing procedure]
If the AR display mode is not selected in S102 of FIG. 17 (S102: NO), the AR edit mode is called. In the present embodiment, the AR content is edited for each scenario.

  39 and 40 show an example of the processing procedure in the edit mode. When the AR editing mode is called, the control unit 10 causes the acquisition unit 11 to acquire a scenario list (S701). The acquisition unit 11 acquires a scenario list generated according to the route definition N in the computer 2 via the communication unit 18. The display control unit 103 generates a scenario selection screen S (7) shown in FIG. 19 based on the acquired scenario list, and displays the generated scenario selection screen S (7) on the display unit 17 (S702).

  When the input unit 19 detects an input operation in a state where the scenario selection screen is displayed, the input unit 19 transmits the user input on the scenario selection screen to the control unit 10, and the control unit 10 determines the selection content by the user input. (S703).

  In S703, when the button B3 for displaying the return to the mode selection screen S (5) ("Return") is selected (S703: "Return"), the process of S101 in FIG. 17 is performed again. In S703, when the button B4 for displaying that the AR function is to be ended (“END”) is selected (S703: “END”), the AR function is ended. In S703, when any scenario shown in the button group B6 is selected (S703: “scenario”), the control unit 10 provides the acquisition unit 11 with the AR content group provided in the scenario selected in S703. Download processing is started (S704). In S704, the acquisition unit 11 performs the process of FIG. 21 and downloads the AR content group of the selected scenario. S703: When “Scenario” is selected to create a new scenario, a new scenario ID is downloaded.

  When the control unit 10 causes the acquisition unit 11 to start the download process in S704, the control unit 10 causes the recognition unit 12 to determine whether or not an AR marker has been detected (S705). When the AR marker is not detected by the recognition unit 12 (S705: NO), the process of S705 is executed again after a predetermined time has elapsed. When the AR marker is detected (S705: YES), the control unit 10 determines whether or not a user input for switching to the still image mode is transmitted from the input unit 19 (S706). When the switching input to the still image mode is not performed (S706: NO), the processing of S705 is executed again after a predetermined time has elapsed.

  When an input to the still image mode is detected (S706: YES), the display control unit 103 switches the screen display of the display unit 17 to the still image mode, and the control unit 10 switches the recognition unit 12 to the display unit 17. Marker recognition processing is executed for the displayed still image (S707). The marker recognition process of S707 is the same process as S108. When the marker recognition process is performed by the recognition unit 12, the control unit 10 edits the AR content based on the recognized AR marker (S708).

  The AR content is edited by selecting an AR template and arranging the AR template. The AR template is selected by selecting an AR template from the editing palette displayed on the display unit 17. The AR template is arranged by designating the display position / size / orientation of the selected AR template and inputting text. The selection of the AR template and the editing of the AR template are performed according to the input to the edit mode display screen.

  FIG. 41 shows an example of a display screen in the edit mode. On the display screen S (14), a captured image captured by the imaging unit 16 and menu groups B11 to B14 and B17 to B19 are displayed. The menu groups B11 to B14 include an edit end button B11, an edit palette button B12, a cancel button B13, and a re-execute button B14. The captured image of the screen S (14) includes an image of the marker M and the object H. Further, a projected image of the AR object C is further displayed on the screen S (14). The menu groups B17 to B19 include an enlargement / reduction movement mode button B17, a rotation mode button B18, and a numerical value designation mode button B19.

  The edit end button B11 is a display element indicating an input position of an operation instruction for confirming editing of the AR object C. The palette edit button B12 is a display element indicating an input position of an operation instruction for calling an edit palette from which an AR template can be selected. The cancel button B13 is a display element indicating the input position of the operation instruction for returning to the state before the previous operation. The re-execution button B14 is a display element that is displayed when an operation is performed on the cancel button B13 and that indicates an input position of an operation instruction for re-execution of the recovered operation.

  The enlargement / reduction movement mode button B17 is a display element indicating an input position of an operation instruction for designating an operation mode. When an input to the display position of the enlargement / reduction movement mode button B17 is performed, the operation mode is switched to the enlargement / reduction movement mode. The rotation mode button B18 is a display element that indicates an input position of an operation instruction that specifies an operation mode. When an input to the display position of the rotation mode button B18 is performed, the operation mode is switched to the rotation mode. The numerical designation mode button B19 is a display element indicating the input position of the operation instruction for designating the operation mode. When an input to the display position of the numerical designation mode button B19 is performed, the operation mode is switched to the numerical designation mode.

  In the enlargement / reduction movement mode, the position coordinates and the magnification of the AR content C are adjusted according to the input operation. For example, the enlargement / reduction movement mode is automatically set as the operation mode at the start of the arrangement of the AR content C. For example, the position coordinates are designated according to the position in the screen on which the tap or double tap operation has been performed, and the position of the AR content C is adjusted by the drag operation. Further, the magnification of the AR content C is adjusted according to a pinch-out or pinch-in operation. In the rotation mode, the rotation coordinates of the AR content C are adjusted according to the input operation. For example, the rotation coordinates are adjusted according to a flick operation or the like. In the numerical specification mode, a numerical input screen is displayed. For example, the position of the reference point of AR content C, the rotation coordinates, and the magnification are set with numerical values input on the numerical value input screen.

  For example, in the editing of AR content in S708, each time a position is designated, a projected image of the AR content placed at the designated position is displayed.

  When the position is designated by the user's input to the screen S (14), the above-described perspective transformation is inversely transformed with respect to the designated position coordinates in the screen. The position coordinate in the marker coordinate system is obtained by performing the inverse transformation of the model-view conversion based on the recognized AR marker with respect to the position coordinate in the camera coordinate system subjected to the inverse transformation of the perspective transformation. It is done.

  When the position coordinates in the marker coordinate system are obtained, the AR content arranged at the designated position is displayed by the processing of S110 and S111 in FIG. For example, if the position is designated before the input to the edit end button B11 is performed, the inverse transformation of the position coordinates and the generation and display of the projection image are performed again.

  FIG. 42 shows an example of an editing palette. In the selection of the AR template in S708 shown in FIG. 39, an editing palette is displayed. The display screen S (15) illustrated in FIG. 42 is a display screen in a state where the editing palette is called. In the editing palette exemplified in FIG. 42, a group of buttons (button 1 to button 9) capable of selecting an AR template type is displayed. In the explanation of the AR template menu, the buttons are identified and explained based on the numbers shown in the button area. For example, a square button with a number “1” is “button 1”. The edit palette includes, for example, selectable button groups other than the button group shown in FIG. 42, and includes a scroll button B21 for performing a scroll operation for displaying those button groups. The edit palette also includes a close button B22 that terminates selection of the AR template.

  Buttons 1 to 9 correspond to individual template IDs. When an input is made to any button, a template ID corresponding to that button is selected, and the AR template of the selected template ID is called.

  The button 1 is associated with a balloon type AR template. In the balloon type AR template, text information is added in a balloon-shaped figure. Button 2 is associated with a drawer box type AR template. The drawer box type AR template includes a leader line and a square figure joined to the tip of the leader line, and text information is added to the square figure. The button 3 is associated with a text box type AR template. In the text box type AR template, text information is added to a rectangular frame-shaped figure. The button 4 is associated with a photo type AR template. In the photo type AR template, image data is mapped in a rectangular frame-like figure. As this image data, an image file stored in the storage unit 15 is used. The button 5 is associated with an imaging type AR template. The imaging type AR template is also the same AR template as the photo type AR template, but the acquisition source of the image data is different. When the imaging type AR template is used, the imaging mode is called, and imaging processing by the imaging unit 16 is performed. When the imaging type AR template is used, the image data captured by the imaging process is mapped in a rectangular frame shape. The image data mapped by the photo type or the imaging type may be a still image or a moving image. The button 6 is a handwritten type AR template. The handwritten type AR template is a rectangular and transparent graphic object, and a graphic pattern is edited by a handwriting operation. The button 7 is a link type AR template, which is the same template as the text box type AR template. When the link type AR template is selected, a list of Web pages is displayed, and access information to the Web page selected from the list is added to the AR template. The list of Web pages is acquired from, for example, a bookmark or access history of a Web browser. The button 8 is associated with a graphic type AR template. In the graphic type AR template, a three-dimensional solid model graphic is defined. For example, a screen display for selecting the shape of the three-dimensional model figure in accordance with an input to the button 8 may be performed. The shape of the three-dimensional model figure is, for example, a cube, a rectangular parallelepiped, a cylinder, a sphere, a cone, a triangular prism, or the like. Also, CAD data may be called in response to an input to the button 8. As the CAD data, for example, a CAD data file stored in the storage unit 15 is selected. The button 9 is associated with a file type AR template. The file type AR template is a rectangular figure to which an icon image indicating a file is mapped. When the file type AR template is selected, a file in the storage unit 15 is selected, and a link to the selected file is added to the AR template.

  In step S708, an AR template is sequentially selected from the editing palette, and a position in the screen is sequentially specified, whereby a plurality of AR contents are edited. When the position in the screen is designated, the calculation unit 13 performs reverse transformation of perspective transformation and inverse transformation of model-view transformation with respect to the designated position (screen coordinate system) in the screen, and the position of the marker coordinate system Calculate the coordinates. The calculated position coordinates are set as the reference point of the AR content. The AR content is added by the AR content editing process in S708. When the input to the edit end button B11 is performed, the process of S708 ends.

  When S708 ends, the display control unit 103 cancels the still image mode, and the control unit 10 determines whether or not to end the arrangement of the AR content (S709). S709 is determined depending on, for example, whether or not the input to the edit end button B11 has been performed again. If the arrangement is not finished (S709: NO), the marker detection process of S705 is executed again. Thereby, AR content editing for a plurality of markers becomes possible.

  When the arrangement is finished in S709 (S709: YES), the control unit 10 performs a setting process of steps included in the scenario. For example, as shown in FIG. 12, information on each step included in the scenario is displayed, and insertion / addition / deletion of a step is performed according to a user input. In the case of creating a new scenario, since step information is not displayed at the beginning, step information is displayed in response to a user input for adding a step.

  When the step is set, the display control unit 103 displays a step selection screen on the display unit 17 (S710). In the step selection screen, a group of steps set in S709, a button indicating transition to the mode selection screen S (5), and a button indicating transition to the scenario selection screen S (7) are shown. In a state where the step selection screen is displayed, the control unit 10 determines that the user input transmitted from the input unit 19 is a button indicating the transition to the mode selection screen S (5), the scenario selection screen S ( It is determined which of the buttons indicates the transition to 7) (S711).

  When an input to a button indicating transition to the mode selection screen S (5) is performed (S711: “other mode”), the display control unit 103 performs the process of S101 for displaying the mode selection screen S (5). Do. When an input to a button indicating a transition to the scenario selection screen S (7) is performed (S711: “other mode”), the display control unit 103 performs the process of S702 for displaying the scenario selection screen S (7). Do.

  When a step is designated on the step selection screen (S711: “Step”), marker detection processing is performed (S712). The marker detection process of S712 is the same process as the process of S705. If the AR marker is not detected by the recognition unit 12 in S712 (S712: NO), the process of S712 is executed again after a predetermined time has elapsed. When the AR marker is detected by the recognition unit 12 in S712 (S712: YES), the control unit 10 causes the recognition unit 12 to perform marker recognition processing (S713). The marker recognition process in S713 is the same process as the process in S707.

  The control unit 10 executes a projection image generation process based on the result of the marker recognition process in S713 (S714). The projection image generation process of S714 is the same process as the process of S110. The display control unit 103 causes the display unit 17 to display the projection image generated in S714 (S715).

  Further, in a state where the projection image is displayed on the screen by the process of S714, the control unit 10 determines whether or not the projection image is selected by an input on the screen (S716). When the projection image is selected (S716: YES), the setting management unit 104 registers the AR content corresponding to the projection image selected in S716 in association with the step selected in S711 (S717).

  When S717 is executed or when a projection image is not selected in S716 (S716: NO), the process of S710 is performed again.

  Further, the step selection method in S711 and the projection image selection method in S715 will be described.

  FIG. 43 shows an example of step selection. For example, in S711, the display control unit 103 displays the screen S (16) and selects a step. On the screen S (16), information on each step is listed. When step 2 is selected, the display control unit 103 displays a screen S (17) in S715, for example. On the screen S (17), the recognized image of the AR marker M1, the projected images C41 and C42 of the AR content arranged in S708, and the step number I5 selected on the screen S (16) are displayed. The When the projected image in the screen S (17) is selected, the AR content corresponding to the selected projected image is associated with step number 2.

  FIG. 44 shows an example of step selection. For example, step numbers are automatically selected in S710 and S711. That is, the selected step number is sequentially incremented from 1 every time a step is selected. Screen S (18) in the selection of the projection image in S716 includes a display I7 of the automatically selected step number. In addition, the screen S (18) shows the recognized image of the AR marker M1 and the projected images C41 and C42 of the AR content arranged in S708. For example, when the projection image C42 is selected, the AR content corresponding to the selected projection image C42 is associated with step number 3.

  The screen S (19) in the selection of the projection image in S716 after the step number is automatically selected again includes the display I9 of the automatically selected step number. The step number shown on the display I9 is a number obtained by incrementing the step number shown on the display I7. In addition, the screen S (19) shows the recognized image of the AR marker M1 and the projected image C41 of the AR content arranged in S708. Since the projection image C42 of the AR content is previously associated with step number 3, it may not be displayed on the screen S (19).

  FIG. 45 shows an example of step selection. For example, a screen S (20) including both a projected image display and a timeline display may be used. On the screen S (20), the recognized image of the AR marker M1, the projected images C41, C42 and C43 of the AR content arranged in S708, and the time zones I10 to I13 corresponding to each step are displayed. The time zones I10 to I13 are a time zone I10 corresponding to step number 1, a time zone I11 corresponding to step number 2, a time zone I12 corresponding to step number 3, and a time zone I13 corresponding to step number 4.

  For example, when the projected image is dragged and dropped into the time zone, control such as associating the AR content corresponding to the projected image with the step number corresponding to the time zone is performed. In the example of the screen S (20), the projection image C41 is dragged and dropped into the time zone I11, the projection image C42 is dragged and dropped into the time zone I13, and the projection image C43 is dragged and dropped into the time zone I10. -Dropped.

  For example, when dragging and dropping, a projected image may be displayed in the time zone. The display mode of the projected image displayed in the time zone may be changed. For example, the projection image C41 is displayed in the time zone I11 with the display mode changed from the image C41a. The projected image C42 is displayed in the time zone I13 with the display mode changed from that of the image C42a. The projected image C43 is displayed in the time zone I10 with the display mode being changed from that of the image C43a.

  FIG. 46 shows an example of step selection. For example, a screen S (21) including both a projected image display and a timeline display may be used. On the screen S (21), the recognized image of the AR marker M1, the projected images C41, C42 and C43 of the AR content arranged in S708, and the time zones I10 to I13 corresponding to each step are displayed. The time zones I10 to I13 are a time zone I10 corresponding to step number 1, a time zone I11 corresponding to step number 2, a time zone I12 corresponding to step number 3, and a time zone I13 corresponding to step number 4.

  Further, the projected image of the AR content is displayed in the time zone corresponding to the order arranged in S708. The example of the screen S (21) illustrates a case where the projected images C43, C41, and C42 of the AR content are arranged in the order in S708. In this case, the projection image C43b corresponding to the projection image C43 arranged first in the arrangement order of S708 is displayed in the time zone I10. In addition, a projection image C41a corresponding to the projection image C41 arranged next is arranged in the time zone I11. Further, a projection image C42a corresponding to the projection image C42 arranged next is arranged in the time zone I12. The projection images C41a, 42a, and 43a initially arranged on the screen S (12) are moved to another time zone, for example, by a drag-and-drop operation by a user input. The example shown in the screen S (21) shows that the projection image C43b initially arranged in the time zone I10 has been moved to the time zone I13.

[AR definition data structure]
FIG. 10 and FIG. 13 illustrate a data structure in a tree format. As shown in FIG. 10 and FIG. 13, AR content related to a specific scenario can be extracted by referring to data under the scenario definition of the specific scenario. As shown in FIG. 13, AR content related to a specific step can be extracted by referring to data under the step definition of the specific step.

  Instead of the tree structure shown in FIGS. 10 and 13, other data structures that can extract AR content related to a specific scenario or a specific step may be used.

  FIG. 47 shows an example of AR definition data. In the data structure of the AR definition data N0 illustrated in FIG. 47, attribute information (scenario ID, step number, and marker ID) indicating the work status is associated with the content ID of each AR content and the content definition. ing. The content definition is a content definition of each AR content, and is data having a structure similar to the data structure shown in FIG. A plurality of pieces of attribute information are associated with the AR content with content IDs 004 and 005.

  Each AR content included in the AR definition data N0 illustrated in FIG. 47 is extracted and displayed when attribute information matching the user state information D0 and the recognition marker information D1 is given. The AR content associated with a plurality of attribute information is extracted when any one of the plurality of attribute information is satisfied. In addition, for AR contents in which the scenario ID and the marker ID match but the step numbers do not match, the display is performed such that the display mode is not displayed or the display mode is changed.

[Means for Realizing this Embodiment]
FIG. 48 shows a hardware configuration example of the computer 1. Each functional block shown in FIG. 16 is realized by the hardware configuration shown in FIG. 48, for example. The computer 1 includes, for example, a processor 301, a RAM (Random Access Memory) 302, a ROM (Read Only Memory) 303, a drive device 304, a storage medium 305, an input interface (input I / F) 306, an input device 307, an output interface ( Output I / F) 308, output device 309, communication interface (communication I / F) 310, camera module 311, acceleration sensor 312, angular velocity sensor 313, display interface 314, display device 315, bus 316, and the like. Each piece of hardware is connected via a bus 316.

  The communication interface 310 controls communication via the network 3. The communication controlled by the communication interface 310 may be a mode of accessing the network 3 via the radio base station 4 (see FIG. 50) using radio communication, or a mode of accessing the network 3 by wire. The input interface 306 is connected to the input device 307 and transmits an input signal received from the input device 307 to the processor 301. The output interface 308 is connected to the output device 309 and causes the output device 309 to execute output in accordance with an instruction from the processor 301. The input device 307 is a device that transmits an input signal according to an operation. The input signal is, for example, a key device such as a keyboard or a button attached to the main body of the computer 1, or a pointing device such as a mouse or a touch panel. The output device 309 is a device that outputs information according to the control of the processor 301. The output device 309 is, for example, an audio output device such as a speaker. The display interface 314 is connected to the display device 315. The display interface 314 causes the display device 315 to display the image information written by the processor 301 in the display buffer provided in the display interface 314. The display device 315 is a device that outputs information according to the control of the processor 301. As the display device 315, an image output device such as a display, a transmissive display, or the like is used. When a transmissive display is used, the projected image of the AR content may be controlled so as to be displayed at an appropriate position in the transmissive display, for example, instead of being combined with the captured image. As a result, the user can obtain a vision in which the real space and the AR content are aligned. For example, an input / output device such as a touch screen is used as the input device 307 and the display device 315. Further, instead of the input device 307 and the display device 315 being incorporated in the computer 1, for example, the input device 307 and the display device 315 may be connected to the computer 1 from the outside.

  The RAM 302 is a readable / writable memory device. For example, a semiconductor memory such as SRAM (Static Random Access Memory) or DRAM (Dynamic Random Access Memory), or a flash memory other than the RAM may be used. The ROM 303 includes a PROM (Programmable Read Only Memory) and the like. The drive device 304 is a device that performs at least one of reading and writing of information stored in the storage medium 305. The storage medium 305 stores information written by the drive device 304. The storage medium 305 is at least one of storage media such as a hard disk, an SSD (Solid State Drive), a CD (Compact Disc), a DVD (Digital Versatile Disc), and a Blu-ray disc. For example, the computer 1 includes a drive device 304 corresponding to the type of the storage medium 305 in the computer 1.

  The camera module 311 includes an image sensor (image sensor). For example, the camera module 311 reads a value measured by the image sensor and writes it in an image buffer for an input image included in the camera module 311. The acceleration sensor 312 measures acceleration acting on the acceleration sensor 312. The angular velocity sensor 313 measures the angular velocity of the operation by the angular velocity sensor 313.

  The processor 301 reads a program (eg, a program illustrated in FIG. 49) stored in the ROM 303 or the storage medium 305 to the RAM 302, and performs processing according to the read program procedure.

  For example, the function of the control unit 10 is that the processor 301 controls other hardware based on an AR control program (display control program and information generation program) which is a part of the program shown in FIG. Realized. The function of the acquisition unit 11 is realized by the processor 301 controlling the communication interface 310 to execute data communication and storing the received data in the storage medium 305. The function of the recognition unit 12 is realized by the processor 301 performing arithmetic processing based on the image data stored in the image buffer for the input image of the camera module 311. The function of the calculation unit 13 is realized by the processor 301 performing a calculation based on data such as coordinates stored in the RAM 302 and data such as a transformation matrix. The function of the generation unit 14 is realized by writing image data generated by the processor 301 using data (including image data) stored in the storage medium 305 into the display buffer.

  The functions of the storage unit 15 are realized by the ROM 303 and the storage medium 305 storing program files and data files, and the RAM 302 being used as a work area of the processor 301. For example, AR definition data and the like are stored in the RAM 302. The function of the imaging unit 16 is realized by the camera module 311 writing image data into the image buffer for input images and the processor 301 reading out the image data in the image buffer for input images. In the monitoring mode, for example, the image data is written in the image buffer for the input image and is written in parallel in the display buffer of the display device 315. The function of the display unit 17 is realized by writing the image data generated by the processor 301 into a display buffer provided in the display interface 314 and causing the display device 315 to display the image data in the display buffer. Is done. The function of the communication unit 18 is realized by the processor 301 controlling the communication interface 310. The function of the input unit 19 is realized by the input interface 306 transmitting an input signal from the input device 307 to the processor 301.

  The functions of the selection control unit 101, the event detection unit 102, the display control unit 103, and the setting management unit 104 are realized by the processor 301 controlling hardware such as the RAM 302 based on the AR control program. This function is realized by the processor 301 controlling hardware such as the RAM 302.

  FIG. 49 shows a configuration example of a program operating on the computer 1. In the computer 1, an OS (Operating System) 502 that controls a hardware group 501 (hardware shown in FIG. 48) operates. The processor 301 operates in accordance with the procedure in accordance with the OS 502, and the hardware 501 is controlled and managed, whereby processing by the application program 504 and the middleware 503 is executed on the hardware 501. In the computer 1, programs such as the OS 502, middleware 503, and application program 504 are read into the RAM 302 and executed by the processor 301, for example. The AR control program is a program called from the application program 504 as the middleware 503, for example. Alternatively, for example, the AR control program is a program that realizes an AR function as the application program 504. The AR control program is stored in the storage medium 305. The storage medium 305 can be distributed separately from the main body of the computer 1 while storing the AR control program.

  FIG. 50 shows an example of a system including the computer 1. The system shown in FIG. 50 includes a computer 1, a computer 2, a network 3, and a radio base station 4. The computer 1 and the computer 2 can communicate by at least one of wired or wireless communication methods.

  FIG. 51 shows a functional configuration example of the computer 2. The computer 2 includes a control unit 21, a storage unit 22, and a communication unit 23. The control unit 21 controls the storage unit 22 and the communication unit 23. The communication unit 23 communicates with the communication unit 18 of the computer 1. The storage unit 22 stores AR definition data and content data. The control unit 21 includes a provision control unit 211. When the scenario list is requested from the computer 1 via the communication unit 23, the provision control unit 211 generates a scenario list based on the AR definition data, and causes the communication unit 23 to return the scenario list. When receiving the scenario ID from the computer 1, the provision control unit 211 causes the communication unit 23 to transmit the AR definition data under the scenario definition of the received scenario ID and the content data used in the scenario to the computer 1.

  FIG. 52 shows an example of a processing procedure for providing AR content. When provision of AR content is started by the computer 2, the provision control unit 211 determines whether an acquisition request is received from the computer 1 (S801). If the acquisition request has not been received (S801: NO), the provision control unit 211 performs the process of S801 again after a predetermined time has elapsed. When an acquisition request is received from the computer 1 (S801: YES), the provision control unit 211 determines whether the acquisition request from the computer 1 is a scenario list request (S802). When the scenario list is requested from the computer 1 (S802: YES), the provision control unit 211 generates a scenario list based on the route definition N of the AR definition data stored in the storage unit 22, and the communication unit The scenario list is transmitted to 23 (S803). When the scenario list is transmitted, the provision control unit 211 performs the process of S801 again.

  If the request is not a scenario list request (S802: NO), the provision control unit 211 determines that the request is an AR content request for a specific scenario, and stores AR definition data according to the scenario ID specified in the acquisition request. The data is read from the unit 22 (S804). The AR definition data read in S804 is AR definition data (step definition, marker definition, content definition, etc.) under the scenario definition of the designated scenario ID. If a specific scenario is not selected in S804, the provision control unit 211 determines that a new scenario is generated, generates a scenario ID that does not conflict with any scenario ID shown in the scenario list, Send to. The provision control unit 211 transmits the AR definition data read in S804 to the request source of the acquisition request (S805).

  Next, the provision control unit 211 causes the communication unit 23 to transmit the AR content displayed in each step (S807). For example, the provision control unit 211 causes the communication unit 23 to transmit the content ID and information indicating that the transmission of the content data is completed, every time the content data is transmitted in S807. The provision control unit 211 repeats the process for each step included in the designated scenario (S806 and S808).

  When the content data in each step is transmitted, the control unit 21 determines whether or not an instruction to end AR content provision has been made (S809). If no termination instruction has been issued (S809: NO), the provision control unit 211 performs the process of S801 again. If an end instruction has been given (S809: YES), the control unit 21 ends the provision of the AR content.

  FIG. 53 shows a hardware configuration example of the computer 2. Each functional block shown in FIG. 29 is realized by the hardware configuration shown in FIG. 30, for example. The management server 2 includes, for example, a processor 401, a RAM 402, a ROM 403, a drive device 404, a storage medium 405, an input interface (input I / F) 406, an input device 407, an output interface (output I / F) 408, an output device 409, A communication interface (communication I / F) 410, a SAN (Storage Area Network) interface (SAN I / F), a bus 412, and the like are included. Each piece of hardware is connected via a bus 412.

  For example, the processor 401 is the same hardware as the processor 301. The RAM 402 is hardware similar to the RAM 302, for example. The ROM 403 is hardware similar to the ROM 303, for example. The drive device 404 is hardware similar to the drive device 304, for example. The storage medium 405 is hardware similar to the storage medium 305, for example. The input interface (input I / F) 406 is hardware similar to the input interface 306, for example. The input device 407 is hardware similar to the input device 307, for example. The output interface (output I / F) 408 is hardware similar to the output interface 308, for example. The output device 409 is hardware similar to the output device 309, for example. The communication interface (communication I / F) 410 is hardware similar to the communication interface 310, for example. A SAN (Storage Area Network) interface (SAN I / F) is an interface for connecting the management server 2 to the SAN, and includes an HBA (Host Bus Adapter).

  The processor 401 reads the management program stored in the ROM 403 or the storage medium 405 to the RAM 402 and performs the processing of the processing unit 21 according to the procedure of the read management program. At that time, the RAM 402 is used as a work area of the processor 401. The function of the storage unit 22 is realized by the ROM 403 and the storage medium 405 storing program files and data files (such as the management table T3), or by using the RAM 402 as a work area of the processor 401. Further, the function of the communication unit 23 is realized by the processor 401 controlling the communication interface 410 to perform communication processing.

[Use for entertainment content]
In the above embodiment, a business process is exemplified as a situation that provides a series of AR contents (scenarios). In other situations where a series of AR content is provided, the AR content may be provided according to the scenario described in the above embodiment.

  For example, game content in which a game event progresses by a user action also includes one type of scenario. For example, it is required to provide AR content included in game content individually to a plurality of users who advance different game content in parallel via a common AR marker. This mechanism is used for, for example, experience-type game content that a plurality of users play individually in this attraction facility. For that purpose, if the scenario ID is assigned to each game content and the mechanism of the above-described embodiment is used, the AR content is individually provided to each user.

The above-described embodiment is an aspect of the present invention, and design changes can be made as appropriate without departing from the spirit of the present invention. Regarding the above embodiment, the following additional notes are disclosed.
(Appendix 1)
An acquisition unit that acquires one or more display data associated with a first work content among the plurality of work contents, from a display data group associated with the plurality of work contents;
A control unit that performs control to display the acquired one or more display data in a display mode based on a reference object recognized by image recognition;
A display control apparatus comprising:
(Appendix 2)
Other display data different from any of the one or more display data is associated with the second work content of the plurality of work contents.
The display control apparatus according to Supplementary Note 1, wherein
(Appendix 3)
The first work content is selected from the plurality of work content according to an input,
The input is information specifying the first work content or information specifying an execution subject of the work indicated in the first work content.
The display control apparatus according to Supplementary Note 1 or Supplementary Note 2, wherein
(Appendix 4)
The control unit performs the control when the reference unit is recognized after the acquisition unit starts acquiring the one or more display data.
The display control apparatus according to any one of Supplementary notes 1 to 3, wherein:
(Appendix 5)
The one or more display data is display data associated with a combination of execution subjects of the work indicated by the first work content and the first work content in the display data group.
The display control apparatus according to any one of supplementary notes 1 to 4, characterized in that:
(Appendix 6)
The one or more display data includes first display data corresponding to a first reference object, and second display data corresponding to a second reference object,
The controller, when recognizing the first reference object, displays the first display data in a display mode based on the first reference object, and when recognizing the second reference object, Control to display the second display data in a display mode based on the second reference object,
The display control apparatus according to any one of Supplementary notes 1 to 5, wherein:
(Appendix 7)
The first work content includes a plurality of steps having order,
A first step of the plurality of steps is associated with the first display data;
The control unit displays the first display data in a display mode based on the first reference object when the process of recognizing the first reference object is the first process.
The display control apparatus according to appendix 6, wherein:
(Appendix 8)
The one or more display data further includes third display data corresponding to a first reference object,
A second step of the plurality of steps is associated with the third display data;
The control unit suppresses display of at least a part of the third display data when the step of recognizing the first reference object is the first step;
The display control apparatus according to appendix 7, characterized by:
(Appendix 9)
The control unit displays a display indicating that there is no display data to be displayed when the display data associated with the step of recognizing the first reference object is not included in the one or more display data. Do,
9. The display control device according to appendix 7 or appendix 8, wherein
(Appendix 10)
The control unit determines the progress or retreat of the process in the first work content based on the reference object recognized by the image recognition;
The display control apparatus according to any one of appendices 7 to 9, characterized in that:
(Appendix 11)
The control unit advances or retreats the process according to the moving direction of the reference object when the recognition or non-recognition state of the reference object is switched.
The display control apparatus according to appendix 10, wherein
(Appendix 12)
The recognition or non-recognition state is switched by the reference object being framed in or out of the image recognition target image.
The display control apparatus according to Supplementary Note 11, wherein:
(Appendix 13)
The control unit displays an index image,
When the reference object is displayed at the display position of the index image, the control unit advances or retracts the process.
The display control apparatus according to appendix 10, wherein
(Appendix 14)
When the control unit is recognized in a state where the reference object is rotated by a predetermined angle or more, the control unit proceeds or reverses the process.
The display control apparatus according to appendix 10, wherein
(Appendix 15)
On the computer,
Obtaining one or more display data associated with the first work content of the plurality of work contents;
Performing control to display the acquired one or more display data in a display mode based on a reference object recognized by image recognition;
A display control method characterized by causing
(Appendix 16)
On the computer,
Obtaining one or more display data associated with the first work content of the plurality of work contents;
Performing control to display the acquired one or more display data in a display mode based on a reference object recognized by image recognition;
A display control program for executing processing.
(Appendix 17)
An information providing system including a first computer and a second computer,
The first computer is
A transmission unit that transmits one or more display data associated with the first work content of the plurality of work contents to the second computer;
The second computer is
A receiving unit for receiving the one or more display data;
Performing control to display the received one or more display data in a display mode based on a reference object recognized by image recognition;
An information providing system characterized by that.
(Appendix 18)
The second computer is
A selection unit for selecting identification information indicating the first work content;
A setting unit that sets position information indicating a position of the first display data with reference to the reference object in association with the reference object recognized by the image recognition;
A transmitter that transmits the first display data in which the position information is set in association with the reference object in association with the first work content indicated in the selected identification information;
The one or more display data includes the first display data.
The information providing system according to claim 17.
(Appendix 19)
A selection unit for selecting identification information indicating the work content;
A setting unit that sets position information indicating a position with reference to the reference object of display data in association with the reference object recognized by image recognition;
A management unit that associates the display data in which the position information is set in association with the reference object, with the selected identification information;
An information setting device comprising:
(Appendix 20)
On the computer,
Select identification information indicating the work content,
In association with the reference object recognized by the image recognition, set position information indicating the position of the display data based on the reference object,
Associating the display data in which the position information is set in association with the reference object with the selected identification information;
An information setting method characterized by causing
(Appendix 21)
On the computer,
Select identification information indicating the work content,
In association with the reference object recognized by the image recognition, set position information indicating the position of the display data based on the reference object,
Associating the display data in which the position information is set in association with the reference object with the selected identification information;
An information setting program for executing a process.
(Appendix 22)
A display control device including a processor,
The processor is
Obtaining one or more display data associated with the first work content of the plurality of work contents;
Performing control to display the acquired one or more display data in a display mode based on a reference object recognized by image recognition;
A display control device characterized by that.

DESCRIPTION OF SYMBOLS 1 Computer 2 Computer 3 Network 4 Wireless base station 10 Control part 11 Acquisition part 12 Recognition part 13 Calculation part 14 Generation part 15 Storage part 16 Imaging part 17 Display part 18 Communication part 19 Input part 21 Control part 22 Storage part 23 Communication part

Claims (14)

From the display data group associated with a plurality of work contents including one or more display data associated with the first work contents and one or more display data associated with the second work contents, the plurality of display data An acquisition unit that acquires one or a plurality of display data associated with the first work content of the work content;
A control unit that performs control to display the acquired one or more display data in a display mode based on a reference object recognized by image recognition;
Including
Display data of at least one of display data associated with the first work content and display data of one or more display data associated with the second work content Are associated with a common reference,
A display control device characterized by that. The first work content is selected from the plurality of work content according to an input,
The input is information specifying the first work content or information specifying an execution subject of the work indicated in the first work content.
The display control apparatus according to claim 1. The control unit performs the control when the reference unit is recognized after the acquisition unit starts acquiring the one or more display data.
The display control device according to claim 1, wherein the display control device is a display control device. The acquired one or more display data is display data associated with a combination of execution subjects of the first work content and the work content indicated in the first work content in the display data group.
The display control apparatus according to any one of claims 1 to 3. The acquired one or more display data includes first display data corresponding to a first reference object, and second display data corresponding to a second reference object,
The controller, when recognizing the first reference object, displays the first display data in a display mode based on the first reference object, and when recognizing the second reference object, Control to display the second display data in a display mode based on the second reference object,
The display control apparatus according to claim 1, wherein the display control apparatus is a display control apparatus. The one or more display data associated with the first work content is associated with one or more reference objects,
The display control apparatus according to claim 1, wherein the display control apparatus is a display control apparatus. The acquisition unit transmits a request including information for identifying the first work content to a device storing the display data group, and the first work transmitted from the device based on the request. Retrieve one or more data associated with the content over the network;
The display control apparatus according to any one of claims 1 to 6. On the computer,
From the display data group associated with a plurality of work contents including one or more display data associated with the first work contents and one or more display data associated with the second work contents, the plurality of display data Obtaining one or more display data associated with the first work content of the work content;
Performing control to display the acquired one or more display data in a display mode based on a reference object recognized by image recognition;
To do that,
Display data of at least one of display data associated with the first work content and display data of one or more display data associated with the second work content Are associated with a common reference,
A display control method characterized by the above. On the computer,
From the display data group associated with a plurality of work contents including one or more display data associated with the first work contents and one or more display data associated with the second work contents, the plurality of display data Obtaining one or more display data associated with the first work content of the work content;
Performing control to display the acquired one or more display data in a display mode based on a reference object recognized by image recognition;
Let the process run,
Display data of at least one of display data associated with the first work content and display data of one or more display data associated with the second work content Are associated with a common reference,
A display control program characterized by that. An information providing system including a first computer and a second computer,
The first computer is
From the display data group associated with a plurality of work contents including one or more display data associated with the first work contents and one or more display data associated with the second work contents, the plurality of display data A transmission unit that transmits one or more display data associated with the first work content of the work content to the second computer;
The second computer is
A receiving unit for receiving the transmitted one or more display data;
Performing control to display the received one or more display data in a display mode based on a reference object recognized by image recognition;
Display data of at least one of display data associated with the first work content and display data of one or more display data associated with the second work content Are associated with a common reference,
An information providing system characterized by that. The second computer is
A selection unit for selecting identification information indicating the first work content;
A setting unit that sets position information indicating a position of the first display data with reference to the reference object in association with a reference object recognized by image recognition on a captured image obtained by imaging by the imaging unit;
A transmission unit that transmits the first display data in which the position information is set in association with the reference object in association with the first work content indicated in the selected identification information;
The information providing system according to claim 10. A selection unit for selecting identification information indicating the work content;
A setting unit that sets position information indicating a position of the display data with reference to the reference object in association with a reference object recognized by image recognition on a captured image obtained by imaging by the imaging unit;
A management unit that associates the display data in which the position information is set in association with the reference object, with the selected identification information;
An information setting device comprising: On the computer,
Select identification information indicating the work content,
In association with a reference object recognized by image recognition for a captured image obtained by imaging by the imaging unit, position information indicating a position based on the reference object of display data is set,
Associating the display data in which the position information is set in association with the reference object with the selected identification information;
An information setting method characterized by causing On the computer,
Select identification information indicating the work content,
In association with a reference object recognized by image recognition for a captured image obtained by imaging by the imaging unit, position information indicating a position based on the reference object of display data is set,
Associating the display data in which the position information is set in association with the reference object with the selected identification information;
An information setting program for executing a process.

Images