JP2018160036A - Control program, control method, and information processing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a control program capable of outputting object information dependent on work that is an object of implementation.SOLUTION: A control program allows a computer to execute processing of: referencing a first storage unit, which stores an implementation history of work, so as to identify work that is an object of implementation when acquiring a taken image containing any of plural reference images; referencing a second storage unit, which stores first object information in association with a combination of the reference image and work, so as to acquire the first object information associated with the combination of the reference image, which is contained in the acquired taken image, and the identified work; and outputting the acquired first object information.SELECTED DRAWING: Figure 8

Description

  This case relates to a control program, a control method, and an information processing apparatus.

  Mixed reality (MR: mixed reality) aimed at seamless fusion of the real world and the virtual world is known. Further, a head mounted display (HMD: head-mounted display device) is known as a device for providing MR. Furthermore, work support is known as an application field of MR. In particular, work support is known in which a product assembly procedure is displayed as a virtual space image superimposed on an actual product in a factory (see, for example, Patent Document 1).

  On the other hand, Augmented Reality (AR) technology is increasingly introduced for the purpose of improving the efficiency of various operations such as inspection and maintenance of equipment. The AR technology is a technology for virtually setting a three-dimensional virtual space corresponding to a real space and arranging and displaying object information (so-called AR object) that does not exist in the real space in the virtual space. It is known that the object information includes, for example, work content instruction information (see, for example, Patent Document 2).

JP-A-2005-260724 Japanese Patent Laying-Open No. 2015-082830

  By the way, in the work mentioned above, it may work by moving a part of equipment according to a series of work processes. Specifically, a panel provided in the equipment may be opened and closed, or a lever provided in the equipment may be turned. In such a case, if object information corresponding to the outer shape is arranged and displayed in the virtual space based on a change in the outer shape of the equipment, appropriate object information may not be displayed. For example, if the current outline of the equipment in the series of work steps matches the past outline of the equipment already finished, object information corresponding to the past outline can be displayed for the current outline There is sex. As a result, the person in charge of the work may perform an incorrect work.

  In view of this, an object of one aspect is to provide a control program, a control method, and an information processing apparatus that can output object information according to work to be performed.

  In one embodiment, the control program obtains a captured image including any one of a plurality of reference images, and refers to a first storage unit that stores a work execution history, and performs an execution target. The reference image included in the acquired captured image with reference to the second storage unit that stores the first object information in association with the combination of the reference image and the work The first object information associated with the combination with the work is acquired, and the computer executes the process of outputting the acquired first object information.

  Object information corresponding to the work to be performed can be output.

FIG. 1 is a diagram for explaining an example of an information processing system. FIG. 2 is an example of an HMD. FIG. 3 shows an example of the hardware configuration of the HMD. FIG. 4 is an example of a hardware configuration of the server apparatus. FIG. 5 is an example of a block diagram of the HMD and server apparatus according to the first embodiment. FIG. 6 is an example of the implementation history storage unit. FIG. 7 is an example of a work process storage unit. FIG. 8 is an example of a processing sequence of the information processing system according to the first embodiment. FIG. 9A is an example of a captured image. FIG. 9B is another example of the captured image. FIG. 10 is a diagram for explaining the processing until the work AR object to be transmitted is specified. FIGS. 11A and 11B are diagrams for explaining the update of history information. FIG. 12A is an example of a reference image when the equipment is looked down from above. FIG. 12B is an example of a reference image when the equipment is viewed from the front. FIG. 12C is an example of a reference image when the equipment is looked up from below. FIG. 13 is an example of a block diagram of the HMD and server apparatus according to the third embodiment. FIG. 14 shows an example of the recovery work storage unit. FIG. 15 is an example of a processing sequence of the information processing system according to the third embodiment.

  Hereinafter, an embodiment for carrying out this case will be described with reference to the drawings.

(First embodiment)
FIG. 1 is a diagram for explaining an example of the information processing system S. FIG. 2 is an example of the HMD 100. The information processing system S includes an HMD 100 and a server device 200 as an information processing device. A smart device such as a tablet terminal or a smartphone may be used instead of the HMD 100. However, by using the HMD 100, the person in charge of the work 10 installs the equipment 20 as a work execution target placed on the pedestal 30. Can work with both hands.

  The HMD 100 and the server device 200 are connected to each other. More specifically, the HMD 100 and the server apparatus 200 are connected via an access point AP and a communication network NW. An example of the communication network NW is a local area network (LAN). Therefore, the HMD 100 can be connected to the server apparatus 200 using wireless communication and wired communication. The server device 200 may be deployed in a data center on the cloud. In this case, the Internet is used as the communication network NW.

  As shown in FIG. 1, the HMD 100 is attached to the head of the worker 10 with a belt or the like. As shown in FIG. 2, the HMD 100 includes a camera 100G and a display 100I. The display 100I displays an image captured by the camera 100G. Further, when the display 100I receives the AR object transmitted from the server device 200, the display 100I displays the received AR object on the captured image.

  Although details will be described later, the HMD 100 includes a central processing unit (CPU), a radio frequency (RF) circuit, and the like. When the camera 100G captures an area within the imaging range, the CPU captures an image corresponding to the area. And the RF circuit transmits an image of the part of the equipment 20 included in the captured image to the server device 200. When the server device 200 receives the image, the server device 200 transmits to the HMD 200 an AR object corresponding to the outer shape of the equipment device 20 based on the received image. When the RF circuit receives the AR object, the CPU displays the AR object received by the RF circuit on the display 100I.

  Hereinafter, details of various configurations and operations of the HMD 100 and the server device 200 described above will be described with reference to the drawings.

  FIG. 3 shows an example of the hardware configuration of the HMD 100. As shown in FIG. 3, the HMD 100 includes a CPU 100A, a random access memory (RAM) 100B, a read only memory (ROM) 100C, an electrically erasable programmable read only memory (EEPROM) 100D, and an RF circuit 100E. An antenna 100E ′ is connected to the RF circuit 100E. A CPU that realizes a communication function may be used instead of the RF circuit 100E.

  The HMD 100 includes a sensor 100F, a camera 100G, a speaker 100H, a display 100I, and a microphone 100J. The CPU 100A to the microphone 100J are connected to each other by an internal bus 100K. At least the CPU 100A and the RAM 100B cooperate to realize a computer.

  In the RAM 100B described above, the program stored in the ROM 100C or the EEPROM 100D is stored by the CPU 100A. When the stored program is executed by the CPU 100A, various functions described later are realized, and various processes are executed. In addition, what is necessary is just to make a program according to the flowchart mentioned later.

  FIG. 4 is an example of a hardware configuration of the server device 200. As shown in FIG. 4, the server apparatus 200 includes at least a CPU 200A, a RAM 200B, a ROM 200C, and a network I / F (interface) 200D. The server device 200 may include at least one of a hard disk drive (HDD) 200E, an input I / F 200F, an output I / F 200G, an input / output I / F 200H, and a drive device 200I as necessary. The CPU 200A to the drive device 200I are connected to each other by an internal bus 200J. At least the CPU 200A and the RAM 200B cooperate to realize a computer.

An input device 710 is connected to the input I / F 200F. Examples of the input device 710 include a keyboard and a mouse.
A display device 720 is connected to the output I / F 200G. An example of the display device 720 is a liquid crystal display.
A semiconductor memory 730 is connected to the input / output I / F 200H. Examples of the semiconductor memory 730 include a universal serial bus (USB) memory and a flash memory. The input / output I / F 200 </ b> H reads programs and data stored in the semiconductor memory 730.
The input I / F 200F and the input / output I / F 200H include, for example, a USB port. The output I / F 200G includes a display port, for example.

A portable recording medium 740 is inserted into the drive device 200I. Examples of the portable recording medium 740 include a removable disk such as a Compact Disc (CD) -ROM and a Digital Versatile Disc (DVD). The drive device 200I reads a program and data recorded on the portable recording medium 740.
The network I / F 200D includes, for example, a LAN port. The network I / F 200D is connected to the communication network NW described above.

  In the above-described RAM 200B, a program stored in the ROM 200C or the HDD 200E is stored by the CPU 200A. In the RAM 200B, the program recorded on the portable recording medium 740 is stored by the CPU 200A. When the stored program is executed by the CPU 200A, various functions to be described later are realized, and various processes to be described later are executed. In addition, what is necessary is just to make a program according to the flowchart mentioned later.

  FIG. 5 is an example of a block diagram of the HMD 100 and the server device 200 according to the first embodiment. FIG. 5 illustrates functional configurations of the HMD 100 and the server device 200. FIG. 6 is an example of the implementation history storage unit 210. FIG. 7 is an example of the work process storage unit 220.

  First, the HMD 100 will be described. The HMD 100 includes an input unit 110, a display unit 120, a control unit 130, an imaging unit 140, and a communication unit 150. The input unit 110 is realized by a button switch (not shown). The display unit 120 is realized by the display 100I described above. The control unit 130 is realized by the CPU 100A described above. The imaging unit 140 is realized by the camera 100G described above. The communication unit 150 is realized by the RF circuit 100E and the antenna 100E ′ described above.

  When the input unit 110 receives an instruction from the worker 10, the input unit 110 outputs the received instruction to the control unit 130. For example, when the worker 10 performs an operation of pressing a button switch as the input unit 110, the button switch outputs an instruction to the control unit 130 that the work process is finished. Note that the microphone 100 </ b> J may be used as the input unit 110 and an instruction to the effect that the work process is completed may be output to the control unit 130 by the voice of the worker 10. The display unit 120 displays the captured image and the AR object.

  The control unit 130 controls the operation of the entire HMD 100. For example, the control unit 130 outputs the instruction output from the input unit 110 to the communication unit 150. The control unit 130 acquires the image output from the imaging unit 140 as a captured image and outputs the acquired image to the display unit 120. The control unit 130 recognizes the part of the equipment device 20 from the captured image using a known object recognition technology (or object detection technology), and uses the recognized part as a state image for specifying the state of the equipment device 20 to the communication unit 150. Output. The control unit 130 outputs the AR object output from the communication unit 150 to the display unit 120.

  The imaging unit 140 captures an area within the imaging range, generates an image corresponding to the area, and outputs the image to the control unit 130. Therefore, if the equipment device 20 is included in the region within the imaging range, the image corresponding to the region includes the image of the equipment device 20. When the communication unit 150 receives the instruction and the state image output from the control unit 130, the communication unit 150 transmits the received instruction and the state image to the server device 200. In addition, when the communication unit 150 receives the AR object transmitted from the server device 200, the communication unit 150 outputs the AR object to the control unit 130.

  Next, the server device 200 will be described. The server device 200 includes an execution history storage unit 210, a work process storage unit 220, an information processing unit 240, and a communication unit 250. The execution history storage unit 210 and the work process storage unit 220 are realized by the HDD 200E described above, for example. The information processing unit 240 is realized by the CPU 200A described above. The communication unit 250 is realized by the network I / F 200D described above.

  The implementation history storage unit 210 stores history information. The history information is information for managing an execution history of a series of operations for the equipment device 20. Based on the history information, it is possible to determine whether or not the work determined by each work process has been completed. Specifically, as shown in FIG. 6, the implementation history storage unit 210 manages history information using a history table T1. The history information includes an equipment device ID, a work process number, and a process completion flag as components. The equipment device ID is information for identifying the equipment device 20. The work process number is a number for identifying a series of work processes. The process completion flag is information for determining whether or not the work of the work process is completed. For example, when the work of the work process is completed, the information processing unit 240 updates the process completion flag “NULL” to “◯”.

  The work process storage unit 220 stores the work AR object in association with the combination of the reference image representing the outer shape of the equipment device 20 and the work process number. The work AR object includes information for supporting various kinds of work by the worker 10. For example, the work AR object includes information instructing the next work. When the work AR object appears on the captured image, the work person in charge 10 can smoothly work even if the work on the equipment 20 is unfamiliar.

  Here, as shown in FIG. 7, the work process storage unit 220 manages work AR objects using a process table T2. The reference image in the process table T2 is image data representing the state of the equipment 20 at the time of work specified by the work process number. In particular, the reference image in the first embodiment is image data when the equipment device 20 is viewed from the front. Although details will be described later, the reference image is a matching target of the state image. On the other hand, the work AR object in the process table T2 represents an AR object displayed at the time of work specified by the work process number. As described above, the work AR object is associated with the combination of the work process number and the reference image.

  Here, as shown in FIG. 7, even if the work process numbers are different, the same reference image may be associated. For example, the reference image with the work process number “1” and the reference image with the work process number “4” are the same. In such a case, if the work AR object is to be specified based only on the reference image, it cannot be determined which reference image the work AR object associated with is to be specified. However, in this embodiment, since the work AR object is associated with the combination of the reference image and the work process number, even if the work image number is different even if the reference image is the same, the work AR object is changed. It can be uniquely identified.

  The information processing unit 240 accesses the execution history storage unit 210 and the work process storage unit 220 to execute various information processing. For example, when receiving the state image transmitted from the HMD 100, the information processing unit 240 refers to the history information stored in the execution history storage unit 210 and identifies the current work process number based on the reception of the state image. When the information processing unit 240 specifies a work process, the information processing unit 240 acquires the work AR object associated with the specified work process and the received state image from the work process storage unit 220 and outputs the work AR object to the communication unit 250. Other information processing executed by the information processing unit 240 will be described in detail later.

  The communication unit 250 controls communication between the HMD 100 and the server device 200. For example, when the communication unit 250 receives a state image or instruction transmitted from the HMD 100, the communication unit 250 outputs the received state image or instruction to the information processing unit 240. For example, when the communication unit 250 receives the work AR object output from the information processing unit 240, the communication unit 250 transmits the work AR object to the HMD 100.

  Subsequently, the operation of the information processing system S will be described with reference to FIGS. 8 to 11.

  FIG. 8 is an example of a processing sequence of the information processing system S according to the first embodiment. FIG. 9A is an example of a captured image. FIG. 9B is another example of the captured image. FIG. 10 is a diagram for explaining the processing until the work AR object to be transmitted is specified. FIGS. 11A and 11B are diagrams for explaining the update of history information.

  First, as shown in FIG. 8, the control unit 130 of the HMD 100 acquires a captured image (step S101). For example, when the worker 10 presses a power button (not shown) of the HMD 100, the imaging unit 140 starts imaging, and the control unit 130 acquires an image captured by the imaging unit 140 as a captured image. Then, the control unit 130 outputs the acquired captured image to the display unit 120, and the display unit 120 displays the captured image.

  For this reason, for example, when the worker 10 tries to start the work of the work process number “3” and the facility device 20 is included in the imaging area of the imaging unit 140, as shown in FIG. Displays the captured image IM1 including the equipment 20 placed on the pedestal 30. The external shape of the equipment 20 when the work process number “3” is started is such that the left panel 21 is closed, the upper panel 22 is opened, and the lever 23 is connected to the bottom surface of the equipment 20. It is in a vertical state. The lever 23 can rotate around the rotation shaft 25 by turning the handle 24 clockwise or counterclockwise.

  When the process of step S101 is completed, the control unit 130 transmits a state image to the server device 200 via the communication unit 150 (step S102). More specifically, the control unit 130 recognizes the part of the equipment device 20 from the captured image based on the object recognition technology, and transmits the recognized part as a state image representing the state of the equipment device 20.

  When the information processing unit 240 of the server device 200 acquires the state image received by the communication unit 250, the information processing unit 240 identifies the work process number (step S201). More specifically, the information processing unit 240 accesses the execution history storage unit 210, and refers to the process completion flag in order from the beginning of the work process number in the history table T1 as shown in the upper part of FIG. The work process number when the flag “NULL” appears is specified and acquired. In the present embodiment, the information processing unit 240 specifies and acquires the work process number “3”.

  When the process of step S201 is completed, the information processing unit 240 then performs matching (step S202). More specifically, the information processing unit 240 accesses the work process storage unit 220 and, as shown in the lower part of FIG. 10, the work process number acquired in the process of step S201 from the work process numbers of the process table T2. Identify. When the information processing unit 240 identifies the work process number, the information processing unit 240 performs matching with the received state image using the reference image associated with the identified work process number as a matching target.

  If the reference image matches the state image as a result of the matching (step S203: YES), the information processing unit 240 transmits the work AR object to the HMD 100 (step S204). That is, in the present embodiment, as illustrated in the lower part of FIG. 10, the information processing unit 240 acquires the work AR object associated with the work process number “3” as a transmission target, and passes through the communication unit 250. To the HMD 100. Note that if the reference image data and the state image do not match as a result of the matching (step S203: NO), the information processing unit 240 stops the subsequent processing in the first embodiment.

  As shown in FIG. 8, the control unit 130 of the HMD 100 displays the work AR object via the communication unit 150 on the display unit 120 (step S103). Accordingly, as illustrated in FIG. 9B, the display unit 120 displays the work AR object OB in association with (for example, synthesis) the captured image IM2. The work person in charge 10 closes the upper panel 22 of the equipment 20 based on the contents designated by the work AR object OB, and turns the lever 23 90 degrees clockwise. As a result, the equipment 20 shifts to the same state as the reference image associated with the work process number “4” in the process table T2.

  As shown in FIG. 8, when the process of step S103 is completed, the control unit 130 then transmits a process end instruction (step S104). More specifically, when the worker 10 finishes the work process, the worker 10 instructs the input unit 110 to end the work process. When the input unit 110 receives the instruction, the control unit 130 transmits a process end instruction via the communication unit 150.

  When receiving the process end instruction, the information processing unit 240 updates the history information (step S205). More specifically, the information processing unit 240 accesses the execution history storage unit 210 and updates the process completion flag of the work process number acquired in the process of step S201. In the present embodiment, the information processing unit 240 obtains the work process number “3” in the process of step S201, and therefore, as shown in FIG. 11A, the process completion flag “NULL” of the work process number “3” is set. As shown in FIG. 11B, the process completion flag is updated to “◯”. Thereby, the server apparatus 200 manages the work completion of the work process number.

  As shown in FIG. 8, when the process of step S205 is completed, the information processing unit 240 then determines whether or not it is a final process (step S206). For example, when the information processing unit 240 accesses the execution history storage unit 210 and updates the process completion flag of the last work process number in a series of work processes, it determines that the process is the final process (step S206: YES). In this case, the information processing unit 240 initializes a process completion flag (step S207). When step S207 is completed, the information processing unit 240 may notify the HMD 100 of the completion of work defined by a series of work processes. When the completion of work is notified from the server device 200, the control unit 130 of the HMD 100 may output the fact by voice or image, for example. Thereby, the worker 10 recognizes the completion of the work.

  On the other hand, when the information processing unit 240 accesses the execution history storage unit 210 and updates the process completion flag of the work process number other than the last in the series of work processes, it determines that the process is not the final process (NO in step S206). In this case, the control unit 130 and the information processing unit 240 repeat the processes in steps S101 to S104 and the processes in steps S201 to S207, respectively. As a result, the control unit 130 displays the next work AR object on the display unit 120, and the worker 10 performs the work of the next work process.

  As described above, according to the first embodiment, the server device 200 includes the information processing unit 240. When the information processing unit 240 obtains a captured image including any one of the plurality of reference images, the information processing unit 240 refers to the execution history storage unit 210 and identifies the work process number to be executed. Then, the information processing unit 240 refers to the work process storage unit 220 to acquire and acquire the work AR object associated with the combination of the reference image included in the acquired captured image and the specified work process number. The completed work AR object is output. Thereby, the work AR object corresponding to the work to be performed can be output.

(Second Embodiment)
Next, a second embodiment of the present case will be described with reference to FIG.
FIG. 12A is an example of a reference image when the equipment device 20 is looked down from above. FIG. 12B is an example of a reference image when the equipment 20 is viewed from the front. FIG. 12C is an example of a reference image when the equipment device 20 is looked up from below.

  In the second embodiment, the process table T2 manages a plurality of types of reference images having different viewpoint positions. A plurality of types of reference images managed by the process table T2 are associated with one work process number. For example, the work process number “1” is associated with three types of reference images. Specifically, as shown in FIG. 12A, the work process number “1” is associated with a reference image when the equipment 20 is looked down from above. As shown in FIG. 12B, the work process number “1” is also associated with a reference image when the equipment 20 is viewed from the front. Further, the work process number “1” is also associated with a reference image when the equipment device 20 is looked up from below. The process table T2 may manage, for example, a reference image when the facility device 20 is viewed from the side or a reference image when viewed from an oblique direction.

  As described above, according to the second embodiment, the process table T2 manages a plurality of types of reference images having different viewpoint positions, so that an imaging field angle or a viewing angle with respect to the equipment device 20 according to the height of the worker 10 can be obtained. Even in the case of changing, the possibility that the state image and the reference image match is improved. In addition, even in an environment where the equipment device 20 cannot be viewed from the front, upper front, or lower front, the possibility that the state image matches the reference image is improved. Thereby, the work AR object is easily displayed.

(Third embodiment)
Subsequently, a third embodiment of the present case will be described with reference to FIGS.
FIG. 13 is an example of a block diagram of the HMD 100 and the server device 200 according to the third embodiment. FIG. 14 shows an example of the recovery work storage unit 230. In addition, the same code | symbol is attached | subjected to the structure similar to each part of HMD100 which concerns on 1st Embodiment shown in FIG. 4, and the server apparatus 200, and the description is abbreviate | omitted. As shown in FIG. 13, the server device 200 according to the third embodiment is different from the server device 200 according to the first embodiment in that a recovery work storage unit 230 is provided.

  The restoration work storage unit 230 stores the restoration AR object in association with the combination of the reference image representing the outer shape of the equipment device 20 and the work process number. The recovery AR object includes information for supporting the recovery work by the worker 10. When the recovery AR object appears on the captured image, the worker 10 can correct a work error for the equipment device 20.

  Here, the recovery work storage unit 230 manages the AR object for recovery using the recovery table T3 as shown in FIG. The reference image in the recovery table T3 is several types of image data representing the state of work error of the equipment 20 assumed in the work process number. On the other hand, the recovery AR object in the recovery table T3 represents an AR object displayed at the time of recovery specified by the work process number. As described above, the recovery AR object is associated with the combination of the work process number and the reference image.

  FIG. 15 is an example of a processing sequence of the information processing system S according to the third embodiment. In particular, FIG. 15 illustrates a part of the processing sequence of the information processing system S. In the first embodiment, if the reference image and the state image do not match in the processing of step S203, the subsequent processing is stopped. However, in the third embodiment, as shown in FIG. The information processing is executed.

  Specifically, when the reference image and the state image do not match in the process of step S203, the information processing unit 240 first transmits the mismatched portion to the HMD 100 through the communication unit 250 (step S211). For example, the information processing unit 240 identifies and transmits a mismatched portion based on the difference between the reference image and the state image. When the control unit 130 of the HMD 100 receives the mismatched portion transmitted from the server device 200 through the communication unit 150, the control unit 130 displays the received mismatched portion on the display unit 120 (step S111). Thereby, the person in charge of work 10 can recognize the work error and can recognize the wrong place.

  When the process of step S211 is completed, the information processing unit 240 then checks the recovery table T3 (step S212). More specifically, the information processing unit 240 accesses the recovery work storage unit 230, and based on the combination of the work process number immediately preceding the work process number acquired in the process of step S201 and the state image transmitted from the HMD 100, Check the recovery table T3. And the presence or absence of the combination of the work process number and reference | standard image corresponding to the combination is confirmed.

  When the combination of the work process number and the reference image corresponding to the combination of the previous work process number and the state image transmitted from the HMD 100 is in the recovery table T3, the information processing unit 240 combines the work process number and the reference image. The recovery AR object associated with is transmitted (step S213). The control unit 130 of the HMD 100 receives the recovery AR object through the communication unit 150, and displays the received recovery AR object on the display unit 120 (step S112). Thereby, the work person in charge 10 performs the restoration work for the equipment 20 based on the contents designated by the restoration AR object.

  When the process of step S112 is completed, the control unit 130 then transmits a recovery end instruction (step S113). More specifically, when the worker 10 finishes the restoration work, the worker 10 instructs the input unit 110 to end the restoration work. When the input unit 110 receives the instruction, the control unit 130 transmits a recovery end instruction via the communication unit 150. The information processing unit 240 ends the process when receiving the recovery work instruction. Therefore, when the worker 10 starts imaging the equipment device 20 with the HMD 100 again, the processing of the HMD 100 and the server device 200 described with reference to FIG. 8 is started. Thereby, a series of work steps proceeds again.

  As described above, according to the third embodiment, not only the work AR object corresponding to the work to be performed but also the recovery AR object corresponding to the recovery work can be output.

  The preferred embodiments of the present invention have been described in detail above, but the present invention is not limited to the specific embodiments according to the present invention, and various modifications are possible within the scope of the gist of the present invention described in the claims.・ Change is possible.

  For example, in the above-described embodiment, the server device 200 is used as an information processing device, but the HMD 100 may be used as an information processing device. That is, in the above-described embodiment, the information processing unit 240 executes various types of information processing, but the control unit 130 of the HMD 100 may execute some or all of various types of information processing executed by the information processing unit 240.

  In the above-described embodiment, the server device 200 includes the execution history storage unit 210, the work process storage unit 220, and the recovery work storage unit 230. However, the implementation history storage unit 210, the work process storage unit 220, and the recovery Another server device 200 different from the server device 200 may be provided with the work storage unit 230.

  Furthermore, in the above-described embodiment, the state image and the reference image are matched. However, the control unit 130 transmits the captured image to the server device 200, and the information processing unit 240 recognizes the captured image or the captured image from the captured image using object recognition technology. You may make it judge whether the image of the equipment 20 applicable to a reference image is included by matching an image and a reference image. Furthermore, the first to third embodiments may be appropriately combined. Furthermore, in the first to third embodiments, the matching between the state image and the reference image has been described. However, when the state image and the reference image are similar within an error range, the work AR object and the restoration AR object are added. You may make it display.

In addition, the following additional notes are disclosed regarding the above description.
(Supplementary Note 1) When a captured image including any one of the plurality of reference images is acquired, the work to be performed is specified with reference to the first storage unit that stores the work execution history. , Referring to the second storage unit that stores the first object information in association with the combination of the reference image and the work, and the combination of the reference image included in the acquired captured image and the identified work A control program for causing a computer to execute a process of acquiring the first object information associated with the first object information and outputting the acquired first object information.
(Supplementary note 2) The supplementary note 1 is characterized in that the second storage unit stores the first object information in association with a combination of a plurality of types of reference images with different viewpoint positions and work. Control program.
(Supplementary Note 3) When the combination of the reference image included in the acquired captured image and the identified work does not exist, the processing outputs second object information that is different in content from the first object information. The control program according to appendix 1 or 2, characterized by:
(Supplementary Note 4) When there is no combination of the reference image included in the acquired captured image and the identified work, the processing uses second object information different from the first object information as the reference image. The third storage unit that stores the reference image and work that are different from each other in association with each other is referred to, and the reference image included in the acquired captured image is associated with the identified combination of work. The control program according to any one of appendices 1 to 3, wherein second control information is acquired and the acquired second control information is output.
(Supplementary Note 5) When a captured image including any one of the plurality of reference images is acquired, the work to be performed is specified with reference to the first storage unit that stores the work execution history. , Referring to the second storage unit that stores the first object information in association with the combination of the reference image and the work, and the combination of the reference image included in the acquired captured image and the identified work A computer executes a process of acquiring first object information associated with the first object information and outputting the acquired first object information.
(Appendix 6) When a captured image including any one of the plurality of reference images is acquired, the work to be performed is specified with reference to the first storage unit that stores the work execution history. , Referring to the second storage unit that stores the first object information in association with the combination of the reference image and the work, and the combination of the reference image included in the acquired captured image and the identified work An information processing apparatus comprising: a processing unit that executes processing for acquiring first object information associated with the first object information and outputting the acquired first object information.
(Supplementary note 7) The supplementary note 6 is characterized in that the second storage unit stores the first object information in association with combinations of a plurality of types of reference images with different viewpoint positions and work. Information processing device.
(Supplementary Note 8) When there is no combination of the reference image included in the acquired captured image and the identified work, the processing unit obtains second object information different in content from the first object information. The information processing apparatus according to appendix 6 or 7, wherein the information processing apparatus outputs the information.
(Supplementary Note 9) When there is no combination of the reference image included in the acquired captured image and the specified work, the processing unit uses the second object information different from the first object information as the reference. With reference to the third storage unit that stores the reference image different from the image in association with the combination of the work, the reference image included in the acquired captured image is associated with the identified combination of the work. The information processing apparatus according to any one of appendices 6 to 8, wherein the second object information is acquired and the acquired second object information is output.

S Information processing system 100 HMD
DESCRIPTION OF SYMBOLS 110 Input part 120 Display part 130 Control part 140 Imaging part 150 Communication part 200 Server apparatus 210 Execution history memory | storage part 220 Work process memory | storage part 230 Restoration work memory | storage part 240 Information processing part 250 Communication part

Claims (6)

When a captured image including any one of the plurality of reference images is acquired, the work to be performed is identified with reference to the first storage unit that stores the work execution history, and the first With reference to the second storage unit that stores the object information in association with the combination of the reference image and the work, the object information is associated with the combination of the reference image included in the acquired captured image and the identified work. Obtain the first object information,
Outputting the acquired first object information;
A control program for causing a computer to execute processing. The second storage unit stores the first object information in association with a combination of a plurality of types of reference images with different viewpoint positions and work,
The control program according to claim 1, wherein: When the combination of the reference image included in the acquired captured image and the identified work does not exist, the process outputs second object information having a content different from the first object information.
The control program according to claim 1 or 2, characterized by the above. In the process, when there is no combination of the reference image included in the acquired captured image and the specified work, the second object information different from the first object information is different from the reference image. The second object associated with the combination of the reference image included in the acquired captured image and the identified operation with reference to the third storage unit stored in association with the combination of the work and the work Obtaining information and outputting the obtained second object information;
The control program according to any one of claims 1 to 3, wherein: When a captured image including any one of the plurality of reference images is acquired, the work to be performed is specified with reference to the first storage unit that stores the work execution history, and the object information is stored. Object information associated with the combination of the reference image included in the acquired captured image and the identified work with reference to the second storage unit stored in association with the combination of the reference image and the work Get
Outputting the acquired object information;
A control method characterized in that a computer executes a process. When a captured image including any one of the plurality of reference images is acquired, the work to be performed is specified with reference to the first storage unit that stores the work execution history, and the object information is stored. Object information associated with the combination of the reference image included in the acquired captured image and the identified work with reference to the second storage unit stored in association with the combination of the reference image and the work Get
Outputting the acquired object information;
An information processing apparatus comprising a processing unit that executes processing.

Images