JP2020030704A - Guidance system and guidance control method - Google Patents

Abstract

To appropriately guide the operation of a user using AR for even use requiring high accuracy.SOLUTION: In a guidance system 1, when executing a scenario that includes position alignment to a prescribed object, a scenario control device (scenario control environment 20) sends information relating to a scenario together with auxiliary information relating to the position alignment to a smart device 10. The smart device 10 having received the information relating to the scenario measures the object of the scenario by a sensor 12. As display on a guidance screen by a display 14, information for guiding the object shown on the display 14 to an appropriate position for position alignment is displayed at least partly on the display 14 by superimposition on the object, on the basis of the auxiliary information and the result of object measurement by the sensor 12.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a guidance system and a guidance control method, and is suitably applied to a guidance system and a guidance control method for inducing a user's operation using AR (augmented reality).

  In recent years, services using AR (Augmented Reality) technology have attracted attention. The AR technology has an advantage that visual information can be effectively provided by superimposing and displaying images, characters, and the like on a real environment viewed through a camera or the like.

  As a specific example of use, in security work in which inspection targets and inspection procedures are defined, information such as what order should be inspected and what information should be inspected is superimposed on the captured image of the object, so that security work can be performed. The procedure can be easily taught. For example, Patent Literature 1 discloses a maintenance and inspection system that can improve work efficiency of on-site patrol and maintenance and inspection work. In the maintenance and inspection system disclosed in Patent Literature 1, inspection character information and inspection area information extracted from image data captured by a camera provided in a portable information terminal are superimposed on a device display image prepared in advance, and a projector is used. By projecting and displaying, clear and accurate image data is provided to the worker as equipment inspection history information, and work efficiency is to be improved.

JP 2010-003190 A

  However, as with the maintenance and inspection system disclosed in Patent Literature 1, the service using the conventional AR includes an image prepared in advance in the image of the real environment (the inspection area information in Patent Literature 1) (the device display image). Or superimposed character information is generally displayed. In other words, as one form of the user interface, existing visual information is simply added in one direction. Therefore, with the existing services in the related art, it is difficult to use the AR to provide appropriate user support for applications such as work requiring high-precision correspondence in a real-time changing real environment.

  Specifically, for example, when the security work includes a high-precision work in which the shooting angle and the shooting distance of the inspection target are strictly determined, it is preferable that the shooting position in the work can be accurately supported by reflecting the sensor data. However, a technology that effectively deploys AR for such a sensing application has not been devised yet. On the other hand, in recent years, the accuracy of sensors and the performance of CPUs have been increasing, and AR applications are expected to be expanded to applications that were difficult in the past, such as the sensing applications described above. I have.

  The present invention has been made in consideration of the above points, and proposes a guidance system and a guidance control method that can appropriately guide a user's operation even in a highly accurate application by using AR. Things.

  In order to solve this problem, in the present invention, a smart device having a sensor and a display, and a scenario in which a series of operations using the smart device are managed by a plurality of scenarios, and information on each of the scenarios is transmitted to the smart device The following guidance system including a control device is provided. In the guidance system, at the time of executing a scenario including alignment with respect to a predetermined target among the plurality of scenarios, the scenario control device transmits the information related to the scenario to the smart device including the accompanying information regarding the alignment. Sending, the smart device that receives the information about the scenario, measures the object by the sensor, as a guidance screen display on the display, based on the accompanying information and the measurement result of the object by the sensor, At least a portion of information for guiding the target object projected on the display to the proper position for the alignment is displayed on the display so as to be superimposed on the target object.

  Further, in order to solve such a problem, the present invention provides the following guidance control method by a guidance system for displaying a guidance screen on a smart device. The guidance system, the smart device having a sensor and a display, a scenario control device that manages a series of operations using the smart device by a plurality of scenarios, and transmits information on each of the scenarios to the smart device. Have. In the guidance control method, when executing a scenario including alignment with respect to a predetermined target among the plurality of scenarios, the scenario control device may include information related to the scenario, An information sending step to be sent to a smart device, and after the information sending step, the smart device receiving the information on the scenario measures the object by the sensor; and, after the measuring step, the smart device However, as the guidance screen display on the display, based on the accompanying information obtained in the information sending step and the measurement result of the object obtained in the measurement step, the object displayed on the display is moved to the position Information to guide you to the appropriate position for alignment, Without even characterized by and a guidance screen display step by superposing on the object displayed on the display part.

  ADVANTAGE OF THE INVENTION According to this invention, a user's operation | movement can be appropriately induced | guided | achieved also in a highly accurate use using AR.

It is a block diagram showing the example of composition of the guidance system concerning one embodiment of the present invention. FIG. 3 is a block diagram illustrating a functional configuration example of a smart device. FIG. 3 is a block diagram illustrating a functional configuration example of a scenario control environment. FIG. 3 is a diagram showing a specific example of table data stored in a database. FIG. 4 is a sequence diagram illustrating an overall outline of guidance processing up to object recognition. FIG. 9 is a sequence diagram illustrating an overall outline of guidance processing after object recognition. It is a flowchart which shows the example of a processing procedure of AR display preparation processing. It is a sequence diagram showing a detailed processing procedure in the AR display preparation processing. It is a flowchart which shows the example of a processing procedure of AR display processing. It is a sequence diagram showing a detailed processing procedure in the AR display processing. It is a flowchart which shows the example of a processing procedure of a photography process. FIG. 9 is a sequence diagram illustrating a detailed processing procedure in a photographing process. It is a flowchart which shows the example of a processing procedure of the next scenario discrimination processing. FIG. 14 is a sequence diagram illustrating a detailed processing procedure in a next scenario determination process. It is a figure for explaining an example of a guidance screen displayed in guidance processing until object recognition. FIG. 11 is a diagram (part 1) for explaining an example of a guidance screen displayed in the guidance process after object recognition. It is FIG. (2) for demonstrating an example of the guidance screen displayed in the guidance process after object recognition. It is FIG. (3) for demonstrating an example of the guidance screen displayed in the guidance process after object recognition.

  Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

(1) Configuration of Guidance System FIG. 1 is a block diagram illustrating a configuration example of a guidance system according to an embodiment of the present invention. The guidance system 1 shown in FIG. 1 records a real object (subject) with a smart device 10 in a series of operations such as security and trail management according to a scenario for managing each process of the operation with high accuracy. The operator of the smart device 10 uses an AR (Augmented Reality) display on the smart device 10 so that the smart device 10 has a predetermined positional relationship with the target (so that the target can be positioned at an appropriate position). A guidance system for guiding a user (hereinafter, a user) includes a smart device 10, a scenario control environment 20, a database 30, and a file server 40.

  The smart device 10 is a smart device that can be carried (worn) by a user, and specifically, for example, a smart glass or a smartphone can be used. The smart device 10 illustrated in FIG. 1 includes an AR display function unit 11 for controlling an AR display function, a sensor 12 for acquiring various sensor data, a camera 13 having a photographing function, a display 14 having a display function, and audio output. A speaker 15 having a function is provided. The smart device 10 is communicably connected to the scenario control environment 20 via a network 50 (mainly a wireless network).

  In this example, as an embodiment of the present invention, a case will be described in which the smart device 10 and the scenario control environment 20 are configured separately. However, the guidance system and the guidance control method of the present invention employ such a configuration. The configuration is not limited to the configuration. For example, the configuration may be integrated such that a configuration corresponding to the scenario control environment 20 is placed inside the smart device 10. Similarly, the database 30 and the file server 40 may have a configuration integrated with the smart device 10 and the scenario control environment 20. Further, communication via the network 50 is not required between the integrated configurations as described above.

  FIG. 2 is a block diagram illustrating a functional configuration example of the smart device. As shown in FIG. 2, as a functional configuration of the smart device 10, the AR display function unit 11 includes a communication control unit 111 that communicates with the outside (the scenario control environment 20) and a process control that controls a process related to the AR display function. And a local folder 113 that is a local storage unit of the AR display function unit 11.

  The communication control unit 111 is, for example, a communication board mounted on the smart device 10, and the processing control unit 112 is, for example, a control board mounted on the smart device 10. Each processing by the smart device 10 described later is mainly executed by the processing control unit 112 of the AR display function unit 11, and more specifically, a CPU (Central Processing Unit) mounted on a control board is stored in a memory. This is realized by executing a program.

  In FIG. 2, as a specific example of the sensor 12 of the smart device 10, a microphone 121 for collecting surrounding sound conditions and inputting a voice from a target, and a depth sensor 122 for measuring a distance (depth) to the target ( Specifically, for example, a LiDAR (Light Detection and Ranging) and the like, a gyro sensor 123 for measuring the posture, angular velocity, and the like of the target, and an optical sensor 124 for measuring the amount of ambient light are provided. Note that the sensor 12 of the smart device 10 does not necessarily need to include all the various sensors illustrated in FIG. 2, and may use another type of sensor (for example, an inertial measurement device that measures the movement of an object). You may have.

  FIG. 3 is a block diagram illustrating a functional configuration example of the scenario control environment. The scenario control environment 20 responds to the purpose of the guidance by the guidance system 1 (for example, acquiring image data of a predetermined object in security work, recording a trail of the predetermined object for trail management, and the like). Has a function of controlling the execution of a plurality of scenarios prepared in advance, and is realized by a computer such as a server or a PC.

  As shown in FIG. 3, the scenario control environment 20 includes, as functional components, a communication control unit 21 that communicates with the outside (the smart device 10) and a scenario that performs selection of a scenario, control of AR display based on the scenario, and the like. The control unit 22 includes a control processing unit 22 and a determination unit 23 that performs analysis processing and determination processing of an image file. Although not shown, the scenario control environment 20 has a storage unit (local folder) used for temporarily storing data when the process is executed by the scenario control processing unit 22 or the determination unit 23. May be.

  Although specific processing will be described later, at the time of execution of processing by the scenario control environment 20, the scenario control processing unit 22 or the determination unit 23 accesses the database 30 or the file server 40 as necessary, and stores desired data. It can be referenced / acquired and stored.

  The database 30 is an example of a storage unit, and in the present embodiment, mainly stores table data and the like that are often partially used in the processing by the scenario control environment 20. Specifically, for example, the AR display information table 310, the accompanying information table 320, and the scenario control information table 330 illustrated in FIG. Further, the file server 40 is an example of a relatively large-capacity storage unit. In the present embodiment, an object file (image file, image setting information, etc.) in which the entire scenario control environment 20 data is often used is used. ) Is stored at the center. Specifically, for example, image data and the like prepared in advance for causing the smart device 10 to perform AR display can be stored in the file server 40, and image data and the like captured by the smart device 10 are also stored in the file server 40. Can be stored.

  The database 30 and the file server 40 are an example of a storage unit in the guidance system 1 according to the present embodiment, and their division is not particularly limited in the present invention. That is, in the present invention, the number of storage means corresponding to the database 30 and the file server 40 may be one or three or more, and the method of realizing them is not limited to either physical storage or virtual storage.

  FIG. 4 is a diagram illustrating a specific example of table data stored in the database.

  The AR display information table 310 illustrated in FIG. 4A is a table of AR display information that is referred to and acquired when the scenario control environment 20 inquires AR display information, and includes a scenario ID 311 indicating a scenario identifier, and a Object file path 312 indicating the storage location (file path) of the corresponding object file, work instruction information 313 indicating the contents of the work instruction in the scenario, and accompanying information indicating the identifier (associated information ID) of the accompanying information in the scenario An ID list 314 is provided.

  The information described in the work instruction information 313 is displayed on the guidance screen displayed on the display 14 of the smart device 10 when the scenario is executed. A specific description of the guidance screen will be described later with reference to FIGS. The incidental information ID described in the incidental information ID list 314 is an identifier common to the incidental information ID described in the incidental information ID 322 of the incidental information table 320 illustrated in FIG.

  The incidental information table 320 illustrated in FIG. 4B is a table of the incidental information referred to and acquired at the time of querying the incidental information by the scenario control environment 20, and shows a scenario ID 321 and an identifier of the incidental information corresponding to the scenario. Accompanying information ID 322, a sensor ID 323 indicating an identifier of the sensor 12 used for obtaining the “current value” in the accompanying information, an accompanying information target value 324 indicating a “target value” in the accompanying information, and a display unit of the accompanying information Information unit 325 indicating the “error range” in the associated information, and display contents when the user is out of the associated information range 326 (excess / under) when displaying the associated information Are provided.

  The scenario control information table 330 illustrated in FIG. 4C is a table of the scenario control information which is referred to and obtained when the scenario control environment 20 inquires the scenario ID (or the next scenario ID). It is configured to include branch determination information 332 indicating information used for branch determination from one scenario to the next scenario, and a next scenario ID 333.

  In each of the above tables, an identifier (scenario ID) that can uniquely identify a scenario prepared in advance is commonly used as the scenario IDs 311, 321 and 331 and the next scenario ID 333. Further, an identifier (attached information ID) that can uniquely identify the attached information is commonly used for the attached information ID list 314 and the attached information ID 322. Actual usage examples of the AR display information table 310, the accompanying information table 320, and the scenario control information table 330 illustrated in FIG. 4 will be shown in a guidance process described later.

(2) Guidance Processing In the guidance system 1 according to the present embodiment, when carrying out work such as security or trail management, a user who carries (or wears) the smart device 10 carries a predetermined object (subject). The object is projected on the display 14 by being within the shooting range of the camera 13. Then, on the display screen of the display 14, a “guidance screen” using AR is displayed (including addition / replacement by voice notification from the speaker 15) for alignment of the object projected on the display 14. . By moving and operating the smart device 10 in accordance with the instructions on the guidance screen, an object (subject) can be recorded with high accuracy by the smart device 10 in accordance with a work scenario such as security or trail management. . The processing of the guidance system 1 in such a series of operations is referred to as guidance processing, and will be described in detail below.

  Note that, in the present embodiment, “the object is projected on the display 14” means that the entity of the object itself is visible through the display 14 and that the object is photographed. This means that the stereoscopic video is displayed on the display screen of the display 14 and made visible. More specifically, the former corresponds to, when the display 14 is transparent or translucent, displaying an entity of an object through the display 14 on a guidance screen described later. In the latter case, when the display 14 is non-transparent, it corresponds to displaying a substantial image of the object photographed by the camera 13 on the display screen of the display 14 on the guidance screen. In the section (3) that specifically details the guidance screen, the latter case will be described as an example.

(2-1) Overall Overview First, the overall outline of the guidance processing will be described separately until the target of the scenario is recognized (until object recognition) and after the scenario is recognized (after object recognition).

  FIG. 5 is a sequence diagram showing an overall outline of the guidance process up to object recognition. In FIG. 5, the inquiry process is indicated by a solid line, and the answer process is indicated by a broken line. This is the same in other sequence diagrams described later.

  According to FIG. 5, first, the smart device 10 prepares to start the object recognition (step S11). Next, the smart device 10 transmits the scenario ID for object recognition to the scenario control environment 20 via the network 50 as an AR display information inquiry (Step S12).

  The scenario control environment 20 that has received the AR display information inquiry from the smart device 10 refers to the AR display information table 310 of the database 30 based on the scenario ID for object recognition received in the inquiry (step S13), and Is obtained (step S14).

  When the accompanying information ID related to the AR display information acquired in step S14 is described (the accompanying information ID list 314), the scenario control environment 20 uses the accompanying information of the database 30 based on the accompanying information ID. With reference to the table 320 (step S15), the associated information corresponding to the associated information ID is acquired (step S16).

  Further, the scenario control environment 20 refers to the file server 40 based on the object file path 312 included in the AR display information acquired in step S14 (step S17), and acquires the object file stored in the file path. (Step S18).

  Thereafter, the scenario control environment 20 transmits the AR data (AR display information, accompanying information, object file) acquired in steps S14, S16, S18 to the smart device 10 via the network 50 (step S19).

  The smart device 10 that has received the AR data in step S19 performs the AR display (display of the guidance screen) on the display 14 using the received AR data and starts object recognition using the camera 13 and the sensor 12. . At this time, for example, the relative distance to the object recognized by the camera 13 is obtained using the sensor 12 (step S20).

  When the object recognition is completed, the smart device 10 inquires the scenario control environment 20 of the next scenario ID via the network 50 (Step S21).

  The scenario control environment 20 that has received the next scenario ID inquiry from the smart device 10 refers to the scenario control information table 330 of the database 30 (step S22) and acquires the scenario control information corresponding to the current scenario ID (step S23). ). Then, the scenario control environment 20 transmits the next scenario ID to the smart device 10 based on the scenario control information acquired in step S23 (step S24).

  When the processing in step S24 is completed, the guidance processing up to the object recognition is completed, and the guidance processing based on the scenario after the object recognition is started.

  FIG. 6 is a sequence diagram illustrating an overall outline of the guidance process after object recognition.

  According to FIG. 6, first, the smart device 10 transmits the scenario ID (for example, see step S24 in FIG. 5) previously transmitted from the scenario control environment 20 as the AR display information inquiry via the network 50 to the scenario control environment 20. (Step S31).

  The subsequent processing (steps S32 to S38 in FIG. 6) performed by the scenario control environment 20 that has received the AR display information inquiry from the smart device 10 in step S31 is the same as the processing in steps S13 to S19 described in FIG. Description is omitted. In step S38, the scenario control environment 20 transmits the AR data (AR display information, accompanying information, object file) acquired from the database 30 and the file server 40 to the smart device 10 via the network 50.

  The smart device 10 that has received the AR data in step S38 performs an AR display (display of a guidance screen) on the display 14 using the received AR data to establish an appropriate positional relationship with the target in the current scenario. Guide the user (smart device 10). Further, at this time, the smart device 10 acquires the sensor data value for the target object by the sensor 12 (Step S39). Thereafter, when the positional relationship with the target object becomes appropriate according to the guidance, the smart device 10 shoots the target object with the camera 13 (step S40), and transmits the shooting data (photos and moving images) to the scenario control environment 20. (Step S41).

  Note that, in the present embodiment, the transmission of the imaging data from the smart device 10 to the scenario control environment 20 may be the transmission of the imaging data directly via the network 50 or the indirect transmission of the imaging data. There may be. As an indirect transmission method, for example, the smart device 10 uploads photographing data to a predetermined storage destination (for example, a file server 40 or a WEB server) via the network 50, and a file path (photograph file path) of the upload destination Is transmitted to the scenario control environment 20. In this case, the scenario control environment 20 can acquire the photographing data by accessing the storage location based on the received photograph file path and downloading the photographing data. In addition, as described above, when the components of the guidance system 1 (the smart device 10, the scenario control environment 20, the file server 40, and the like) are integrated, communication via the network 50 is not required. It becomes possible to directly acquire, in the scenario control environment 20, photographing data photographed by the smart device 10.

  The scenario control environment 20, which has obtained the photographing data of the target object of the scenario in step S41, analyzes the image of the photographing data (step S42), and determines the next scenario using the result of the image analysis. Specifically, the scenario control environment 20 refers to the scenario control information table 330 of the database 30 (step S43), acquires scenario control information corresponding to the current scenario ID (step S44), and acquires the acquired scenario control information. The next scenario ID is determined by applying the result of the image analysis to the branch determination information 332 in. Then, the scenario control environment 20 transmits the determined next scenario ID to the smart device 10 via the network 50 (Step S45).

  Upon completion of the processes in steps S31 to S45, the guidance process for the scenario ID specified in step S31 ends, and thereafter, the guidance process for the next scenario ID starts.

  The overall outline of the guidance process according to the present embodiment has been described above with reference to FIGS. Of these guidance processes, the processes of steps S31 to S38 in FIG. 6 (the same applies to steps S13 to S19 in FIG. 5) are referred to as “AR display preparation process”, and the processes of steps S38 to S39 in FIG. The processing of steps S40 to S41 in FIG. 6 is referred to as “photographing processing”, and the processing of steps S41 to S45 in FIG. 6 is referred to as “next scenario determination processing”. This will be described with reference to FIG.

(2-2) AR Display Preparation Processing FIG. 7 is a flowchart illustrating an example of a processing procedure of the AR display preparation processing. FIG. 7 shows an example of a processing procedure by the scenario control environment 20 in the AR display preparation processing. It should be noted that a detailed processing procedure in each configuration of the guidance system 1 in the AR display preparation processing is shown in a sequence diagram of FIG. 8 described later.

  According to FIG. 7, first, the scenario control environment 20 acquires AR display information (step S101). This acquisition of the AR display information corresponds to the processing of steps S32 to S33 in FIG. 6 (or steps S13 to S14 in FIG. 5).

  Next, the scenario control environment 20 determines whether or not the AR display information acquired in step S101 has accompanying information (step S102). Specifically, when the associated information ID is described in the associated information ID list 314 of the AR display information table 310, it is determined that the AR display information has the associated information (YES in step S102), and step S103 is performed. Proceed to. In step S103, the associated information is acquired from the database 30 based on the associated information ID described in the AR display information. Thereafter, the process proceeds to step S104. On the other hand, if the incidental information ID is not described in the incidental information ID list 314 of the AR display information in step S102, it is determined that there is no incidental information (NO in step S102), and the process skips step S103 and proceeds to step S104. .

  In step S104, the scenario control environment 20 acquires an object file from the file server 40 based on the object file path 312 described in the AR display information acquired in step S101.

  Thereafter, the scenario control environment 20 transmits the set of data (AR display information, accompanying information, and object file) acquired in steps S101 to S104 to the smart device 10 as AR data (step S105), and thereafter, described later. AR display processing is performed.

  FIG. 8 is a sequence diagram showing a detailed processing procedure in the AR display preparation processing.

  According to FIG. 8, first, the scenario ID is transmitted from the smart device 10 to the communication control unit 21 of the scenario control environment 20 (step S201). Step S201 corresponds to transmitting the scenario ID in the AR display information inquiry shown in step S31 of FIG. 6 (or step S12 of FIG. 5).

  Next, the communication control unit 21 decodes the received scenario ID (Step S202) and sends it to the scenario control processing unit 22 (Step S203).

  Then, the scenario control processing unit 22 refers to the AR display information table 310 of the database 30 based on the scenario ID received in step S203 (step S204), and acquires AR display information corresponding to the scenario ID (step S205). ).

  Next, the scenario control processing unit 22 determines whether or not the AR display information acquired in step S205 has accompanying information (confirmation of the accompanying information ID list 314). The associated information table 320 of the database 30 is referred to based on the associated information ID (step S206), and the associated information corresponding to the associated information ID is obtained (step S207).

  Further, the scenario control processing unit 22 refers to the file server 40 based on the object file path 312 included in the AR display information acquired in step S205 (step S208), and acquires the object file stored in the file path. (Step S209).

  Thereafter, the scenario control processing unit 22 sends the set of information (AR data) acquired in steps S205, S207, and S209 to the communication control unit 21 (step S210), and the communication control unit 21 transmits the received AR data. Encode (step S211) and transmit to the smart device 10 (step S212).

  By performing the AR display preparation process as described above, the scenario control environment 20 collects information (AR data) necessary for performing the AR display according to the scenario in the smart device 10 and the smart device 10 Can be sent to

(2-3) AR Display Processing FIG. 9 is a flowchart illustrating an example of a processing procedure of the AR display processing. FIG. 9 illustrates a processing procedure example by the smart device 10 in the AR display processing. The detailed processing procedure of each component of the guidance system 1 in the AR display processing is shown in a sequence diagram of FIG. 10 described later.

  According to FIG. 9, first, by transmitting the AR data in step S105 in FIG. 7, the smart device 10 acquires the AR data from the scenario control environment 20 (step S111). This acquisition of the AR display information corresponds to the processing of step S38 in FIG. 6 (or step S19 in FIG. 5).

  Next, the smart device 10 AR-displays the object file included in the AR data received in step S111 on the display 14 (step S112).

  Then, the smart device 10 determines whether or not the AR data received in step S111 includes accompanying information (step S113). If it is determined that there is accompanying information (YES in step S113), step S114 is performed. To S117. On the other hand, when it is determined that the received AR data does not include the accompanying information (NO in step S113), the AR display process ends.

  In the sensor processing, first, the smart device 10 acquires a sensor value based on the accompanying information (Step S114). Specifically, sensing is performed on the target using the sensor 12 corresponding to the sensor ID described in the sensor ID 323 of the accompanying information table 320, and a sensor value is obtained.

  Thereafter, the smart device 10 performs a proper position determination to determine whether or not the positional relationship with the target is a proper position based on the sensor value acquired in Step S114 (Step S115). As the criterion for determining the proper position, the associated information target value 324, the associated information unit 325, and the associated information range 326 of the associated information table 320 are used. More specifically, in the proper position determination, when the acquired sensor value exists within the range of the accompanying information range 326 of the accompanying information table 320, it is determined that the position is the proper position. On the other hand, if the acquired sensor value is out of the range of the accompanying information range 326, it is determined that the position is not a proper position, and further, it is determined whether the position is over or under the accompanying information range 326.

  Next, the smart device 10 corrects the accompanying information for display based on the results of steps S114 and S115 (step S116). Here, the additional information for display is the additional information of the local data created based on the additional information included in the AR data, and is stored in, for example, the local folder 113. Then, in the next step S117, the smart device 10 causes the display 14 to AR-display the display-related information corrected in step S116 (display of the accompanying information). In the present embodiment, by using the accompanying information of the local data (the accompanying information for display) in this manner, even if a new sensor value is acquired in step S114, appropriate information can be obtained without communicating with the scenario control environment 20. Since the position determination is performed and the AR display of the accompanying information can be realized using the updated latest accompanying information for display, an effect of suppressing the occurrence of the processing delay can be expected.

  In the accompanying information display of step S117, not only the current sensor value is displayed, but also a text display based on the determination result (proper position, inappropriate position due to excess / under) based on the proper position determination is performed. For example, in the case of an inappropriate position due to excess, the excess display information 327 of the accompanying information table 320 is displayed, and in the case of an inappropriate position due to too small, the undersized display information 328 of the accompanying information table 320 is displayed.

  The sensor processing in steps S114 to S117 is repeatedly executed at a sensor value update interval (for example, a predetermined time interval) until the appropriate position is determined in the appropriate position determination in step S115 and the next step of photographing processing is started. You.

  FIG. 10 is a sequence diagram illustrating a detailed processing procedure in the AR display processing.

  According to FIG. 10, first, AR data (AR display information, accompanying information, object file) is transmitted from the scenario control environment 20 to the communication control unit 111 of the smart device 10 (step S221). Step S221 corresponds to the process shown in step S38 of FIG.

  Next, the communication control unit 111 decodes the received AR data (step S222) and sends the data to the processing control unit 112 (step S223).

  Then, the processing control unit 112 transmits the object file of the AR data received in step S223 to the display 14 (step S224), and causes the display 14 to display the AR (step S225).

  Next, the processing control unit 112 determines whether or not the AR data received in step S223 includes additional information (step S226). If there is additional information, as described with reference to FIG. The sensor process is executed.

  In the sensor processing, first, the processing control unit 112 inquires of the sensor 12 corresponding to the sensor ID described in the sensor ID 323 of the accompanying information table 320 about a sensor value (step S227). The sensor 12 that has received the inquiry performs sensing, and returns a detection result (sensor value) to the processing control unit 112 (step S228).

  Next, the processing control unit 112 performs an appropriate position determination based on the sensor value acquired in step S228 (step S229).

  Thereafter, the processing control unit 112 transmits the display-related information to the display 14 (step S230), and the display 14 updates the display content of the AR display in step S205 and displays the display-related information in an AR manner (step S231). ). As described above with reference to FIG. 9, in the AR display of the accompanying information for display, in addition to the display of the sensor value, a text display according to the determination result of the appropriate position determination can be performed.

  The processes in steps S227 to S231 are repeatedly executed and updated at sensor value update intervals (for example, predetermined time intervals) until the next stage of the photographing process is started. Note that the screen displayed in AR on the display 14 in the AR display processing is a guidance screen for guiding the user of the smart device 10 so as to have a predetermined positional relationship (appropriate position) with the target of the scenario. Specific examples thereof will be described later with reference to FIGS.

  By performing the AR display process as described above, the smart device 10 uses the guidance screen using the AR display so that the positional relationship between the user (the smart device 10) and the target of the scenario becomes an appropriate position. Can be guided.

(2-4) Photographing Process FIG. 11 is a flowchart illustrating an example of a processing procedure of the photographing process. FIG. 11 shows an example of a processing procedure by the smart device 10 in the photographing processing. A detailed processing procedure of each component of the guidance system 1 in the photographing process is shown in a sequence diagram of FIG. 12 described later.

  The photographing process is executed when the user (smart device 10) is at a proper position with respect to the target object by the proper position determination in step S115 in FIG. Further, in this example, the smart device 10 automatically executes the photographing process based on a voice instruction from the user, but the method of realizing the photographing process in the present embodiment is not limited to this. Instead, for example, the user himself may perform a shooting operation of the smart device 10. Further, a moving image may be taken in the photographing process.

  According to FIG. 11, first, the smart device 10 acquires and recognizes a voice instruction of a user who instructs shooting of an object (step S131). Next, the smart device 10 takes a photograph of the object (Step S132).

  Thereafter, the smart device 10 uploads the photographing data of the photograph taken in step S132 to a predetermined storage destination (for example, the file server 40) (step S133). Then, the smart device 10 transmits the file path (photograph file path) of the upload destination to the scenario control environment 20, and ends the photographing process. Thereafter, in the scenario control environment 20, a next scenario determination process described later is performed.

  Note that the shooting data uploaded in step S133 may include environment information (for example, a sensor value or the like) at the time of shooting. By adding such environment information, shooting can be performed in the next scenario determination process described later. The sensor value at the appropriate position at the time can also be transmitted to the scenario control environment 20 together.

  FIG. 12 is a sequence diagram illustrating a detailed processing procedure in the photographing process.

  According to FIG. 12, first, in the smart device 10, a user's voice instruction to instruct shooting of an object is input from the microphone 121 to the processing control unit 112 (step S <b> 241).

  In step S241, since the voice data of the voice instruction is input, the processing control unit 112 performs voice recognition processing on the voice data (step S242), and instructs the camera 13 to take a photograph based on the recognition content. (Step S243). Upon receiving the photographing instruction in step S243, the camera 13 photographs the target (step S244), and sends the photographed data to the processing control unit 112 (step S245). Note that the voice recognition processing in step S242 can use various known techniques.

  Next, the processing control unit 112 uploads the photographing data sent from the camera 13 to the file server 40 (step S246), and receives the upload destination file path (photograph file path) from the file server 40 (step S247). .

  Then, the processing control unit 112 sends the photo file path received in step S247 to the communication control unit 111 (step S248), and the communication control unit 111 encodes the photo file path (step S249), and transmits the photo file path via the network 50. To the scenario control environment 20 (step S250).

  By performing the photographing process as described above, the smart device 10 photographs the target at the appropriate position, and displays data representing the photographed data (in this example, the file path of the upload destination, but the photographed data is directly transferred). In the case of transmission, the image data itself) can be transmitted to the scenario control environment 20.

(2-5) Next Scenario Determination Processing FIG. 13 is a flowchart illustrating an example of a processing procedure of the next scenario determination processing. FIG. 13 shows an example of a processing procedure by the scenario control environment 20 in the next scenario determination processing. It should be noted that a detailed processing procedure by each component of the guidance system 1 in the next scenario determination processing is shown in a sequence diagram of FIG. 14 described later.

  According to FIG. 13, first, the scenario control environment 20 acquires a photo file path from the smart device 10 by transmitting the photo file path in step S134 of FIG. 11 (step S141). The acquisition of the photo file path corresponds to the processing in step S41 in FIG.

  Next, the scenario control environment 20 acquires photographing data (photograph file) from a predetermined storage location (in this example, the file server 40) based on the photo file path received in step S141 (step S142). In the processing of steps S141 to S142, in the AR display processing described in FIG. 11, the photographing data is uploaded to the file server 40, and the file path (photograph file path) of the upload destination is transmitted to the scenario control environment 20. This is to explain the processing in the case. Therefore, for example, in the case where the imaging data is directly transmitted to the scenario control environment 20 in the AR display processing, it is sufficient to simply acquire the imaging data instead of steps S141 to S142.

  Next, the scenario control environment 20 analyzes the image of the photo file, and determines the next scenario using the result of the image analysis (step S143). Various known techniques can be used for image analysis. The determination of the next scenario is a process for determining a scenario to be executed next, and information corresponding to the branch determination information 332 of the scenario control information table 330 is extracted from the result of the image analysis.

  Then, the scenario control environment 20 acquires the ID of the scenario to be executed next (the next scenario ID) by referring to the scenario control information table 330 of the database 30 based on the result of step S143 (step S144).

  Then, the scenario control environment 20 transmits the next scenario ID acquired in step S144 to the smart device 10 via the network 50 (step S145), and ends the next scenario determination processing. Thereafter, guidance processing based on the next scenario ID is started (for example, step S11 in FIG. 5).

  FIG. 14 is a sequence diagram illustrating a detailed processing procedure in the next scenario determination processing.

  According to FIG. 14, first, the photo file path is transmitted from the smart device 10 to the communication control unit 21 of the scenario control environment 20 (step S251). Step S251 corresponds to step S250 in FIG. 12 described above.

  The communication control unit 21 decodes the received photo file path (Step S252), and sends it to the processing control unit 112 (Step S253). Then, the processing control unit 112 transfers the photo file path to the determination unit 23 (Step S254).

  Next, the determination unit 23 inquires of the file server 40 about the photo file (image file) using the photo file path (step S255), and the image file stored in the photo file path, that is, FIG. The photographing data (photograph) stored in step S246 is obtained from the file server 40 (step S256).

  Next, the determination unit 23 performs image analysis on the photographed data acquired in step S256, and determines and extracts at least “predetermined information” necessary for branch determination of the next scenario (step S257). The determination unit 23 sends the result of the determination to the scenario control processing unit 22 (Step S258).

  Here, the “predetermined information” required for the branch determination of the next scenario can be specified from the branch determination information 332 of the scenario control information table 330. For example, in the scenario control information table 330 of FIG. 4, regarding the scenario ID “001”, “A, B” is described in the branch determination information 332, and if the branch determination information is “A”, the next scenario ID Is "002A", and when the branch determination information is "B", the next scenario ID is "002B". Therefore, in the next scenario determination processing when the scenario ID is “001”, information corresponding to “A” or “B” described in the branch determination information 332 is extracted as the “predetermined information”.

  In the present embodiment, various branches can be realized for the following scenarios depending on what information is described in the branch determination information 332. Regarding the color of a predetermined area (for example, a lamp) in the photographed object, “predetermined area is red” is information “A” of branch determination information 332, and “predetermined area is blue” is branch determination information. If the information 332 is “B”, the next scenario can be branched according to the color of the lamp. Further, for example, it is also possible to perform a predetermined diagnosis process and to perform a branch setting such as setting information “A” and “B” according to the diagnosis result (OK / NG). The number of branches is not limited to two, and a scenario can be set by arbitrarily setting the number of branches and branch conditions.

  Returning to the description of FIG. The scenario control processing unit 22 that has received the determination result in step S258 refers to the scenario control information table 330 of the database 30 based on the current scenario ID and the above determination result (step S259), and acquires the next scenario ID. (Step S260). To explain a specific example using the scenario control information table 330 of FIG. 4, if the current scenario ID is “001” and the determination result is “A”, “002A” is determined as the next scenario ID. Is done.

  Thereafter, the scenario control processing unit 22 sends the next scenario ID obtained in step S260 to the communication control unit 21 (step S261), and the communication control unit 21 encodes the received next scenario ID (step S262), The data is transmitted to the smart device 10 (step S263).

  By performing the next scenario determination process as described above, the scenario control environment 20 can acquire the image data of the target object of the current scenario imaged by the smart device 10 and obtain the next image data from the image analysis result of the image data. The scenario to be executed is determined and transmitted to the smart device 10.

(3) Guidance Screen As described above, in the guidance system 1 according to the present embodiment, the guidance screen is AR-displayed on the display 14 of the smart device 10 in order to instruct the user to perform a predetermined operation (operation). Hereinafter, the guidance screen will be described with reference to specific examples.

  First, the guidance screen displayed in the guidance process up to object recognition will be described. As described in step S20 in FIG. 5 and the like, in the guidance processing up to the object recognition, a guidance screen for recognizing the target object is displayed on the display 14 of the smart device 10 in an AR based on the scenario ID of the object recognition. Is done.

  FIG. 15 is a diagram for explaining an example of a guidance screen displayed in the guidance process up to object recognition. As shown in FIG. 15, the guidance screen 400 includes a first display screen 410 for superimposing (AR display) an appropriate position of an object in the current scenario on an object (subject) displayed in the screen, And a second display screen 420 for AR display of work instructions and accompanying information in the current scenario.

  The guidance screen 400 shown in FIG. 15 is confirmed more specifically. On the first display screen 410, the target object is indicated by a solid line. As described above, in the present embodiment, the “object displayed on the display 14 (the guidance screen 400, the first display screen 410)” means that the substance of the object is transmitted through the display 14. Both the case where the image is projected and the case where the real image of the object passing through the camera 13 (the real image captured by the camera 13) is displayed on the display screen of the display 14 (here, the first display screen 410). In the sense of. Although there is a difference between the two in terms of image processing and the like, in any case, when the user views the guidance screen 400 (first display screen 410), the target object is projected and superimposed on the target object. Is still displayed on the display. Therefore, hereinafter, the description will be continued by taking as an example a case where a substantial image of the target object is displayed through the camera 13.

  The specific description of the guidance screen 400 shown in FIG. 15 will be continued. On the first display screen 410, a substantial image of the object captured through the camera 13 is indicated by a solid line. Further, for the target in the current scenario (here, the object recognition scenario), an image (3D guide) of an appropriate position for alignment for performing object recognition is superimposed and displayed by a broken line. This 3D guide is AR-displayed by using an object file included in AR data transmitted from the scenario control environment 20 (for example, see step S19 in FIG. 5).

  The second display screen 420 of the guidance screen 400 includes a work instruction display area 421 in which the contents of the work instruction in the current scenario are displayed in text, and an objective position of the current user (smart device 10) and the object. And an associated information display area 422 in which an image of the relationship is shown.

  As the text displayed in the work instruction display area 421, information described in the work instruction information 313 (see FIG. 4A) of the AR display information included in the AR data transmitted from the scenario control environment 20 is displayed. You. Specifically, in the case of FIG. 15, the message “Please align the subject with the 3D guide and recognize the object” is displayed in the work instruction display area 421, so the user is prompted on the first display screen 410. The user can be informed that the position of himself / herself (smart device 10) needs to be changed so that the stereoscopic image of the target object matches the position of the 3D guide.

  The image of the objective positional relationship between the current user (smart device 10) and the object displayed in the accompanying information display area 422 is the object file and the accompanying information included in the AR data transmitted from the scenario control environment 20. AR display is performed based on the display information (for example, see step S116 in FIG. 9) updated with the acquired sensor value and stored in the local folder 113. Specifically, in the case of FIG. 15, in the accompanying information display area 422, as an appropriate positional relationship between the user and the object, a specific angle of “30 ± 5 °” and a distance of “25 ± 1 cm” are displayed. The values are shown, and image images of the user and the object are also displayed. The display of the accompanying information makes it easier for the user to visually recognize and judge the appropriate positional relationship with the object.

  Next, a guidance screen displayed in the guidance process after object recognition will be described. As described in step S39 in FIG. 6 and the like, in the guidance process after the object recognition, the guidance screen for positioning the target to the appropriate position for shooting is displayed on the display of the smart device 10 based on the current scenario ID. 14 is displayed as an AR.

  16 to 18 are diagrams (parts 1 to 3) for explaining an example of a guidance screen displayed in the guidance processing after object recognition. Each of the guidance screens 500, 600, and 700 shown in FIGS. 16 to 18 is a first display in which an appropriate position of an object in the current scenario is superimposed (AR display) on a real image captured through the camera 13. Screens 510, 610, and 710, and second display screens 520, 620, and 720 for AR display of work instructions and accompanying information in the current scenario. The display contents on the first display screens 510, 610, and 710 are common to the above-described first display screen 410 of the guidance screen 400, and thus description thereof is omitted. On the other hand, the second display screens 520, 620, and 720 show different variations of the display mode of the accompanying information relating to the positioning and the photographing with respect to the object. Hereinafter, the display modes on the respective second display screens will be described. Will be described.

  The display mode of the second display screen 520 in FIG. 16 will be described. The second display screen 520 shows a work instruction display area 521 in which the contents of the work instruction in the current scenario are text-displayed, and an image of an objective positional relationship between the current user (smart device 10) and the object. An accompanying information display area 522 is provided. Further, the accompanying information display area 522 includes an accompanying information text display area 523 in which an operation necessary for alignment is displayed in text.

  As the text displayed in the work instruction display area 521, information described in the work instruction information 313 (see FIG. 4A) of the AR display information included in the AR data transmitted from the scenario control environment 20 is displayed. You. Specifically, in the case of FIG. 16, the work instruction display area 521 displays "Please align the subject with the 3D guide, input [Photo] and voice, and obtain the current photo". From this display content, the user can recognize that it is necessary to position the stereoscopic image of the target object displayed on the first display screen 510 with the 3D guide, and then perform voice input of “photograph”.

  The image of the objective positional relationship between the current user (smart device 10) and the object displayed in the accompanying information display area 522 is the object file and the accompanying information included in the AR data transmitted from the scenario control environment 20. AR display is performed based on the display information (for example, see step S116 in FIG. 9) updated with the acquired sensor value and stored in the local folder 113.

  In the case of FIG. 16, the numerical display related to the positional relationship between the user and the object (subject) is displayed in the format of “[current value] / [target value] ± [error range]”. Specifically, “17.5 / 30 ± 5 °” displayed in the accompanying information display area 522 indicates that the viewing angle of the user with respect to the target at the appropriate position is “30 ± 5 ° (that is, 25 ° to 35 °). )), Whereas the current user's viewing angle is “17.5 °”, which means that the viewing angle is too small. Similarly, in “22.8 / 25 ± 1 cm” displayed in the accompanying information display area 522, the distance to an object at an appropriate position is determined to be “25 ± 1 cm (that is, 24 cm to 26 cm)”. On the other hand, the distance from the current user to the object is “22.8 cm”, which means that the distance is too close. By performing such an accompanying information display area 522, the current state of the positioning can be accurately transmitted to the user, and the user's operation required for the positioning can be assisted.

  Further, in the case of FIG. 16, in the accompanying information text display area 523, the text display of “··············································· / These text displays are information described in the excess display information 327 or the under display information 328 of the incidental information table 320, and when the current value acquired by the sensor 12 is out of the incidental information range (excess information). / Small), text display according to the situation is performed. By performing such an accompanying information text display area 523, the user can clearly teach an operation to be performed for positioning to an appropriate position, and the user's operation can be more appropriately guided. .

  Next, a display mode of the second display screen 620 in FIG. 17 will be described. The second display screen 620 shows a work instruction display area 621 in which the contents of the work instruction in the current scenario are displayed in text, and an image of an objective positional relationship between the current user (smart device 10) and the target. An accompanying information display area 622 is provided. The display contents of the work instruction display area 621 are the same as those of the work instruction display area 521 described with reference to FIG.

  In the accompanying information display area 622 of FIG. 17, a numerical display related to the positional relationship between the user and the object (subject) is performed, as in the accompanying information display area 522 of FIG. The specific numerical display is the same as that in FIG. 16 and the description is omitted. Further, in FIG. 17, as a characteristic display mode, in the accompanying information display area 622, an operation to be performed by the user (a destination of the user) for alignment to an appropriate position is indicated by a broken line of the user image. Taught by images.

  Specifically, in the case of FIG. 17, the image of the user indicated by the dashed line is displayed at the upper left of the image of the user indicated by the solid line. From such an image of the dashed line, the user: You will be guided to "go a little further away" and "look a little more from above." That is, this broken line image is an image of the content of the text display in the accompanying information text display area 523 in FIG. By displaying such an image, it is possible to visually and clearly teach an operation to be performed by the user for positioning to an appropriate position, and to guide the user's operation more appropriately. .

  Next, a display mode of the second display screen 720 in FIG. 18 will be described. The second display screen 720 shows a work instruction display area 721 in which the content of the work instruction in the current scenario is text-displayed, and an image of an objective positional relationship between the current user (smart device 10) and the object. An accompanying information display area 722 is provided. The display contents of the work instruction display area 721 are the same as those of the work instruction display area 521 described with reference to FIG.

  In the accompanying information display area 722 of FIG. 18, unlike the above-described accompanying information display area 622 of FIG. 16 and the accompanying information display area 722 of FIG. 17, as a characteristic display mode, numerical display is not performed. Instead, an operation to be performed by the user for the position adjustment to the appropriate position (a destination of the user) is displayed by an image with an arrow. Specifically, in the case of FIG. 18, a rightward arrow image is displayed from the user to guide the operation of approaching the target object, and an arrow image that curves from top to bottom in front of the user's face is displayed. By doing so, the operation of seeing the object from above is guided. By adopting the display mode as shown in FIG. 18, it is possible to simply teach the operation to be performed by the user, so that an effect of appropriately guiding the user's operation can be expected.

(4) Conclusion As described above, the guidance system 1 according to the present embodiment manages the smart device 10 having the sensor 12 and the display 14 and a series of operations using the smart device 10 by a plurality of scenarios. A scenario control environment 20 is provided. In the guidance system 1, when a scenario including positioning with respect to a predetermined target among a plurality of scenarios is executed, the scenario control environment 20 uses AR data (AR display information, accompanying information, Object file) to the smart device 10. Then, in the smart device 10 that has received the AR data, the target object of the scenario is measured by the sensor 12, the guidance information is displayed on the display 14 (for example, the guidance screen 500), and the accompanying information and the measurement of the target object by the sensor 12 are measured. Based on the result, the object to be projected on the screen (the substance of the object that is transmitted through or the substance image of the object that is photographed by the camera 13 and displayed on the screen) is properly positioned. (A 3D guide at an appropriate position, accompanying information, and work instructions) are displayed at least partially superimposed on the target object (first display screen 510).

  According to such a guidance system 1, it is possible to appropriately guide a user's operation by using AR for a series of tasks managed in a scenario. In particular, in a high-precision application in which alignment with the target is required, the display that displays the actual image of the target displays the associated information display based on the alignment information and the sensor measurement value in an AR display. Accordingly, it is possible to appropriately guide the user's operation so that the target object is positioned at an appropriate position.

  In the guidance system 1 according to the present embodiment, in order to appropriately guide the user's operation on the above-described guidance screen, the following various devices are devised.

  For example, as shown in the first display screen 410 of the guidance screen 400, etc., the entity image captured through the camera 13 (or the entity of the object shown through the display 14) includes the object. By superimposing and displaying an image of the proper position of the object (3D guide), it is easy for the user to imagine an operation necessary for positioning the object at the proper position.

  Also, for example, as shown in the second display screen 520 of the guidance screen 500, the work instruction and the accompanying information in the current scenario are displayed in an AR, so that what kind of work is required of the user. Alternatively, the difference between the current state of the object and the appropriate position can be visually recognized. In particular, on the second display screen 520 and the like, an image showing the positional relationship between the current user (smart device 10) and the target object from an objective viewpoint is displayed (for example, the accompanying information display area 522). By helping to objectively grasp the relative position of the user with respect to the target object and making it easy to understand the operation required for the alignment, an effect of appropriately guiding the user's operation can be expected.

  Further, as illustrated in FIGS. 16 to 18, the accompanying information display of the guidance screen can be realized in various display modes, and has a wide range of practicalities according to the purpose of the work, the request of the user, and the like. . For example, in the case of a work requiring more precise positioning, the work efficiency of the positioning can be improved by additionally writing a numerical display as in the accompanying information display area 522 and the like. Further, for example, when a special operation such as a visual angle adjustment is to be induced, the user can visually teach necessary operations as in the accompanying information text display area 523 and the accompanying information display area 722, thereby allowing the user to perform appropriate operations. Can be easily guided.

  Further, since at least a part of the accompanying information display displayed on the second display screen 520 or the like can be replaced with audio output from the speaker 15, the user's operation can be appropriately performed using not only the viewing angle information but also the auditory information. Can be expected to be effective.

  Note that, in the photographing process described with reference to FIGS. 11 and 12, the smart device is triggered by a voice instruction (for example, voice input of “photograph” into the microphone 121) of the user instructing photographing of the target object. Although the photographing of the target object can be executed by the automatic control of the ten processing control units 112, such a photographing process may be further automated in the present embodiment. Specifically, when it is determined that the target object is at the proper position (positioning is completed) in the proper position determination (for example, step S115 in FIG. 9) in the AR display processing, without waiting for the user's voice instruction. Alternatively, the processing control unit 112 of the smart device 10 may automatically execute photographing of an object. By doing so, the user can complete the work of the scenario only by performing positioning of himself (smart device 10) in accordance with the guidance on the guidance screen, so that a series of work can be performed in a simpler procedure. It is possible to proceed.

  Further, in the service using the conventional AR, the display content transmitted by one-way communication is generally displayed as an AR, whereas in the guidance system 1 according to the present embodiment, the smart device is used. The display content of the AR in 10 is not changed by using the AR data such as the object data sent from the scenario control environment 20 as it is, but is dynamically changed by reflecting the measurement result by the sensor 12, and photographed by the smart device 10. The scenario control environment 20 obtains the captured image data of the target object, and analyzes / determines the captured data to determine the next scenario, and transmits AR data relating to the determined next scenario. One of the features is that an AR display using bidirectional communication is realized. As a result, according to the guidance system 1 according to the present embodiment, it is possible to provide a highly accurate guidance screen, which is difficult to realize with the above-described conventional service, and to position the object at an appropriate position. The effect of appropriately guiding the user's operation can be enhanced.

  In addition, in the guidance system 1 according to the present embodiment, while using the above-described two-way communication and using the accompanying information for display stored in the local folder 113, during the AR display during the execution of the scenario, Without communication from the smart device 10 to the scenario control environment 20, it is possible to dynamically change the content of AR display on the guidance screen by reflecting the measurement result by the sensor 12, thereby reducing the processing load on the entire system. The effect is obtained.

  The guidance system 1 according to the present embodiment as described above can be applied to all techniques for guiding a user's operation from one scenario to the next scenario in a series of operations in which a plurality of processes are managed in a scenario. .

  Note that the present invention is not limited to the above-described embodiment, and includes various modifications. For example, the above-described embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described above. For example, for a part of the configuration of the embodiment, it is possible to add, delete, or replace another configuration.

  In addition, each of the above-described configurations, functions, processing units, processing means, and the like may be partially or entirely realized by hardware, for example, by designing an integrated circuit. In addition, the above-described configurations, functions, and the like may be realized by software by a processor interpreting and executing a program that realizes each function. Information such as a program, a table, and a file for realizing each function can be stored in a memory, a hard disk, a recording device such as an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.

  In the drawings, control lines and information lines are considered to be necessary for explanation, and not all control lines and information lines are necessarily shown on a product. In fact, it can be considered that almost all components are connected to each other.

1 Guidance System 10 Smart Device 11 AR Display Function Unit 12 Sensor 13 Camera 14 Display 15 Speaker 20 Scenario Control Environment 21 Communication Control Unit 22 Scenario Control Processing Unit 23 Judgment Unit 30 Database 40 File Server 111 Communication Control Unit 112 Processing Control Unit 113 Local Folder 121 Microphone 122 Depth sensor 123 Gyro sensor 124 Optical sensor 310 AR display information table 311 Scenario ID
312 Object file path 313 Work instruction information 314 Associated information ID list 320 Associated information table 321 Scenario ID
322 Associated information ID
323 Sensor ID
324 Additional information target value 325 Additional information unit 326 Additional information range 327 Excess display information 328 Under display information 330 Scenario control information table 331 Scenario ID
332 branch decision information 333 next scenario ID

Claims (12)

A smart device having a sensor and a display;
A scenario control device that manages a series of operations using the smart device by a plurality of scenarios, and transmits information on each of the scenarios to the smart device,
With
When executing a scenario including alignment with respect to a predetermined target among the plurality of scenarios,
The scenario control device sends the information related to the scenario to the smart device, including the accompanying information related to the alignment,
The smart device receiving the information on the scenario,
Measuring the object by the sensor,
As a guidance screen display on the display, based on the accompanying information and a measurement result of the object by the sensor, at least one information for guiding the object projected on the display to an appropriate position for the alignment is provided. A guidance system, wherein a part is superimposed on the object and displayed on the display. The guidance system according to claim 1, wherein the guidance screen display includes a display of an image showing a positional relationship between the object and the smart device from an objective viewpoint. The smart device further has a camera,
In the guidance screen display, the target object transmitted through the display, or, on the display of the real image of the target object photographed by the camera on the display, the appropriate position of the alignment, The guidance system according to claim 1, wherein the guide is superimposed. The smart device further has a speaker,
When the scenario includes the positioning of the object and the next operation after the positioning,
In the guidance screen display in the scenario, work instruction information for instructing the work contents of the positioning and the next operation is displayed on the display or output as voice from the speaker. Guidance system described. The smart device further has a camera,
After the object is guided to the proper position of the alignment by the guidance screen display, when the camera of the smart device has photographed the object by performing the next operation,
The said scenario control apparatus acquires the imaging | photography data of the said object image | photographed by the said smart device, and determines the said scenario which starts execution next based on the said imaging | photography data. Guidance system described. The smart device includes:
The display accompanying information including the accompanying information received from the scenario control device and the measurement result of the object by the sensor is stored in its own storage unit, and the display accompanying information is stored for each measurement by the sensor. 2. The guidance system according to claim 1, wherein the updated display accompanying information stored in the storage unit is updated and updated on the guidance screen display. 3. A guidance control method by a guidance system for displaying a guidance screen on a smart device,
The guidance system includes:
The smart device having a sensor and a display;
A series of operations using the smart device is managed by a plurality of scenarios, and a scenario control device that transmits information on each of the scenarios to the smart device,
When executing a scenario including alignment with respect to a predetermined target among the plurality of scenarios,
An information sending step in which the scenario control device sends information to the smart device to the information on the scenario, including accompanying information on the alignment,
After the information sending step, the smart device receiving the information on the scenario, a measuring step of measuring the object by the sensor,
After the measuring step, the smart device displays a guidance screen on the display, based on the accompanying information obtained in the information transmitting step and the measurement result of the object obtained in the measuring step, on the display. A guidance screen display step of displaying information for guiding the object to be projected to the proper position for the alignment, at least a part of the information being superimposed on the object and displayed on the display.
A guidance control method, comprising: The said guidance screen display displayed in the said guidance screen display step includes the display of the image image which shows the positional relationship between the said object and the said smart device from an objective viewpoint. The claim 7 characterized by the above-mentioned. Guidance control method. The smart device further has a camera,
In the guidance screen display step, an appropriate position of the alignment with respect to a display of the target image transmitted through the display or a real image of the target object captured by the camera on the display. The guidance control method according to claim 7, wherein a guide indicating is displayed in a superimposed manner. The smart device further has a speaker,
When the scenario includes the positioning of the object and the next operation after the positioning,
8. The guidance screen displaying step in the scenario, wherein work instruction information for instructing the contents of the work of the positioning and the next operation is displayed on the display or output as voice from the speaker. Guidance control method. The smart device further has a camera,
After the object is guided to the proper position of the alignment through the guidance screen display step, when the camera of the smart device has photographed the object by performing the next operation,
The scenario control device further includes a determination step of acquiring shooting data of the object shot by the smart device, and determining the scenario to start execution next based on the shooting data. The guidance control method according to claim 10, wherein: The smart device retains, in its own storage means, display-related information including the accompanying information received from the scenario control device in the information sending step and the measurement result of the object measured in the measuring step. And a display accompanying information update step of updating the display accompanying information for each measurement by the sensor,
The guidance control method according to claim 7, wherein, in the guidance screen displaying step, the latest display accompanying information held in the storage unit is displayed on the guidance screen.

Images