JP2016062336A - Operation instruction system, operation instruction method, attached terminal, and operation instruction management server - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an operation instruction system and an operation instruction method, capable of instructing an object person to perform an operation, while suppressing the decrease of the efficiency of the operation by the object person and to provided an attached terminal and an operation instruction management server, enabling an instruction for the operation by the operation instruction system and the operation instruction method.SOLUTION: An attached terminal 10 includes an acquisition part for acquiring medium information including identification information from an identification mark 40 and a projection part for projecting an image. An operation instruction management server 20 includes a receiving part for receiving the medium information from the attached terminal 10, a storage part for storing association data where the identification information is associated with image information indicating the image including an operation instruction, an information specification part for specifying the image information associated with the identification information included in the medium information by using the medium information received by the receiving part and the association data, and a transmission part for instructing the attached terminal 10 to project the image indicated by the image information specified by the information specification part.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an operation instruction system, an operation instruction method, and an attachment terminal and an operation instruction management server used in the operation instruction system and the operation instruction method.

  2. Description of the Related Art Conventionally, a system used for instructing work contents and work procedures to a worker who performs work in a factory or the like has been developed. In such a system, for example, work information for instructing work contents and work procedures is displayed on a computer terminal assigned to each worker, and the worker performs work according to the work information (see, for example, Patent Document 1).

JP 2013-73511 A

  By the way, in the above system, when a computer terminal is installed and used, there are restrictions on where the terminal can be installed at the work site, so the terminal is installed near the place where the worker is working It is not always possible. Then, when the installation location of the terminal is not secured near the worker, every time the work information is confirmed, the worker must interrupt the work and move to the terminal.

  On the other hand, when the computer terminal is a portable terminal such as a tablet terminal, the worker does not need to move, but the worker takes out the terminal every time work information is confirmed, or , You have to work while holding the terminal.

  As a result, in any case, the act of confirming the work information causes a reduction in work efficiency. In addition, such a problem is not limited to a system for instructing work at a factory, but in a system for instructing an operation to a target person, such as a system for instructing work at a construction site or a system for instructing treatment at a medical site, This is a common issue.

  The present invention provides an operation instruction system, an operation instruction method, and an operation instruction by such an operation instruction system and an operation instruction method capable of instructing an operation to the object while suppressing a decrease in operation efficiency by the object person. It is an object of the present invention to provide a mounting terminal and an operation instruction management server that are made possible.

  An operation instruction system that solves the above-described problem includes a mounting terminal that is mounted on a subject who receives an operation instruction, and an operation instruction management server that is connected to the mounting terminal via a network. An acquisition unit that acquires medium information including the identification information from a medium that holds the identification information; and a projection unit that projects an image, and the operation instruction management server receives the medium information from the mounting terminal. Using a receiving unit, a storage unit that stores correspondence data in which the identification information and image information indicating an image including an operation instruction are associated, and the medium information and the correspondence data received by the receiving unit An information specifying unit that specifies the image information associated with the identification information included in the medium information, and a projection of the image indicated by the image information specified by the information specifying unit. And a transmission unit that instructs the.

  An operation instruction method that solves the above-described problem includes a mounting terminal that is provided with a projection unit that projects an image and is mounted on a subject who receives an operation instruction, and an operation instruction management server that is connected to the mounting terminal via a network. , Wherein the mounting terminal acquires medium information including the identification information from a medium holding the identification information; and the operation instruction management server is transmitted from the mounting terminal. The operation instruction management server receives the medium information, and the operation instruction management server uses correspondence data in which the identification information is associated with image information indicating an image including the operation instruction. A step of specifying the image information associated with the identification information included in the medium information; and the operation instruction management server indicates the specified image information. A step of instructing the projection of serial images to the mounting terminal, according to an instruction of the operation instruction management server, comprising the steps of: projecting the image showing the mounting terminals of the image information.

  According to the above configuration, since the image including the operation instruction is projected from the wearing terminal worn by the subject, the location of the terminal where the subject displays the instruction for confirming the operation instruction as in the past It is possible to prevent the terminal from being taken out every time the operation is confirmed, or to be operated while holding the terminal. Therefore, in a situation where the subject performs an operation while receiving an instruction, the subject can be instructed to perform an operation while suppressing a decrease in the efficiency of the operation by the subject.

  The operation instruction system is a management terminal connected to the operation instruction management server via a network, and further includes the management terminal having a display unit for displaying an image. Including a region for projecting the area around the mounting terminal, and in the operation instruction management server, the receiving unit captures the imaging data generated by the imaging of the mounting terminal. Received from the mounting terminal, the transmission unit transmits the imaging data received by the receiving unit to the management terminal, and the management terminal is an imaging based on the imaging data received from the operation instruction management server It is preferable to display an image on the display unit.

  According to the above configuration, shooting data generated by shooting by the mounting terminal is transmitted to the management terminal via the operation instruction management server, and a shot image is displayed on the management terminal. Therefore, even if the user of the management terminal is in a remote place, the user's behavior can be grasped through the captured image.

  In the operation instruction system, the management terminal receives an input of an auxiliary instruction that is an instruction for a situation indicated by the captured image, transmits auxiliary data indicating the input auxiliary instruction to the operation instruction management server, and In the operation instruction management server, it is preferable that the reception unit receives the auxiliary data from the management terminal, and the transmission unit instructs the mounting terminal to project an image including the auxiliary instruction indicated by the auxiliary data. .

  According to the above configuration, the image including the auxiliary instruction input to the management terminal is projected from the mounting terminal. Therefore, even if the user of the management terminal is in a remote place, the user can give an instruction according to the situation to the target person while viewing the captured image displayed on the management terminal.

  The operation instruction system includes a plurality of the management terminals. In the operation instruction management server, the reception unit receives the auxiliary data from each of the plurality of management terminals, and the transmission unit includes a plurality of the management terminals. It is preferable to instruct the wearing terminal to project an image including a plurality of the auxiliary instructions indicated by the plurality of auxiliary data received from each of the terminals.

  According to the above configuration, an auxiliary instruction is input to each of the plurality of management terminals, and an image including the plurality of auxiliary instructions is projected from the wearing terminal, so that a plurality of users using the plurality of management terminals are targeted. An instruction according to the situation can be given to the person. As a result, the opinions of a plurality of users are gathered, and it becomes possible to instruct the target person more accurately.

In the operation instruction system, it is preferable that the mounting terminal further includes a projection auxiliary unit that collects information used for correcting the projection state of the image indicated by the image information.
According to the above configuration, information for correcting the projection state of an image can be collected. Therefore, it is possible to correct the image projection state.

  In the operation instruction system, the projection assisting unit collects information for correcting the stereoscopic effect of the image according to the surface shape of the three-dimensional object arranged at the place where the image is projected, and It is preferable that at least one gyro sensor that collects information for correcting the projection position of the image is included.

  According to the above configuration, in the configuration in which the projection auxiliary unit includes the infrared camera, the projected image is obtained by correcting the stereoscopic effect of the image in accordance with the surface shape of the three-dimensional object arranged at the place where the image is projected. Can be made easier to see. Further, in the configuration in which the projection assisting unit includes the gyro sensor, it is possible to correct the projection position of the image according to the inclination or shaking of the wearing terminal, and to prevent the image from blurring due to the movement of the target person. it can.

In the operation instruction system, the wearing terminal is preferably worn on the head of the subject.
According to the above configuration, it is easy to project an image at a position that is easy for the subject to view. In addition, it becomes easier for the subject to perform the operation compared to the case where the wearing terminal is worn on the trunk or extremities.

  In the stem that indicates the movement instruction, it is preferable that the medium functions as an AR marker, and the projection position of the image indicated by the image information by the wearing terminal is defined with reference to the position of the medium.

  According to the above configuration, it is possible to easily display an operation instruction at a preset position by setting the position of the medium. Therefore, it is possible to instruct a more accurate operation.

  A mounting terminal that solves the above problem is a mounting terminal that is connected to an operation instruction management server that manages an operation instruction via a network and is attached to a target person who receives the operation instruction, and holds identification information From the operation instruction management server, an acquisition unit that acquires medium information including the identification information and the medium information acquired by the acquisition unit are transmitted to the operation instruction management server. A communication unit that receives an instruction for projecting the image including an operation instruction, and a projection unit that projects the image according to the instruction from the operation instruction management server. .

  By using the mounting terminal having the above configuration, an image including an operation instruction is projected from the mounting terminal worn by the subject. Therefore, in the operation instruction system or the operation instruction method using such a wearing terminal, the operation of the subject by the subject is performed. It is possible to instruct the subject to perform an operation while suppressing a decrease in efficiency.

  An operation instruction management server that solves the above problems includes a projection unit that projects an image, a mounting terminal that is worn by a subject who receives the operation instruction, and a management terminal and a network that include a display unit that displays an image. A storage unit that stores correspondence data in which the identification information of the medium that holds the identification information and the image information indicating the image including the operation instruction are associated with each other. A receiving unit that receives medium information including the identification information of the medium from the mounting terminal, and shooting data generated by shooting around the mounting terminal by the mounting terminal, and the medium information received by the receiving unit And the correspondence data, the information specifying unit for specifying the image information associated with the identification information included in the medium information, and the image specified by the information specifying unit A transmission unit that instructs the wearing terminal to project the image indicated by the report, and transmits the captured data received by the reception unit to the management terminal, and further, the reception unit from the management terminal, Auxiliary data indicating an auxiliary instruction, which is an instruction for a situation indicated by an image displayed on the management terminal, is received based on the photographing data, and the transmission unit projects an image including the auxiliary instruction indicated by the auxiliary data. Instruct the wearing terminal.

  By using the operation instruction management server configured as described above, an image including an operation instruction is projected from the wearing terminal worn by the subject. Therefore, in the operation instruction system or operation instruction method using such an operation instruction management server, It is possible to instruct the subject to perform the operation while suppressing a decrease in the operation efficiency by the user. In addition, since the image including the auxiliary instruction input to the management terminal is projected from the mounting terminal, the user of the management terminal can view the captured image displayed on the management terminal even when the user is in a remote place. The user can be instructed according to the situation.

  According to the present invention, an operation can be instructed to a subject while suppressing a decrease in operation efficiency by the subject.

1 is a diagram illustrating an overall configuration of an operation instruction system according to an embodiment. It is a figure which shows the structure of the mounting terminal of one Embodiment. It is a figure which shows an example of the mounting terminal of one Embodiment. It is a figure which shows the structure of the operation instruction management server of one Embodiment. It is a figure which shows an example of the data structure of the corresponding data in one Embodiment. It is a sequence diagram which shows the procedure of the operation | movement instruction | indication by the operation | movement instruction | indication system of one Embodiment, Comprising: It is a figure which shows the procedure until a basic image is projected. It is a sequence diagram which shows the procedure of the operation | movement instruction | indication by the operation | movement instruction | indication system of one Embodiment, Comprising: It is a figure which shows the procedure until an integrated image is projected. (A)-(e) is a figure which shows an example of a mode that a subject receives the instruction | indication of the operation | movement by the operation | movement instruction | indication system of one Embodiment.

  An embodiment of an operation instruction system, an operation instruction method, a mounting terminal, and an operation instruction management server will be described with reference to FIGS.

[Configuration of operation instruction system]
The overall configuration of the operation instruction system will be described with reference to FIG.
As shown in FIG. 1, the operation instruction system includes a mounting terminal 10, an operation instruction management server 20, and one or more management terminals 30. The mounting terminal 10, the operation instruction management server 20, and the one or more management terminals 30 are connected to a network NW such as the Internet. The mounting terminal 10 and the operation instruction management server 20 transmit and receive data to each other, and the management terminal 30 and the operation instruction management server 20 transmit and receive data to each other.

  The network to which the mounting terminal 10 and the operation instruction management server 20 are connected and the network to which the management terminal 30 and the operation instruction management server 20 are connected may be different networks.

  The wearing terminal 10 is a terminal having a recognition function of the identification mark 40, an image projection function, and an image photographing function as main functions, and is worn by a subject who receives an operation instruction. The mounting mode is not particularly limited, and the mounting terminal 10 may have a helmet-shaped mounting portion and may be mounted on the subject's head, or may have a ring-shaped mounting portion through which the neck passes. It may be hung on the person's neck, or may be worn on the face of the subject with a spectacle-shaped wearing part. In addition, it is preferable that the mounting terminal 10 is mounted on the head from the viewpoint of improving the ease of operation and the ease of viewing the projected image.

  The identification mark 40 has a pattern for each identification mark 40, thereby holding information unique to each identification mark 40. The identification mark 40 may be a barcode or a two-dimensional code, or the pattern of the identification mark 40 may constitute a picture, a pattern, or a character. Information unique to the identification mark 40 held by the identification mark 40 functions as identification information of the identification mark 40.

  The mounting terminal 10 transmits medium information that is identification information of the identification mark 40 acquired based on the recognition of the identification mark 40 to the operation instruction management server 20. Then, the mounting terminal 10 projects an image including an operation instruction based on the data received from the operation instruction management server 20. The identification mark 40 is an AR marker, and preferably functions as a mark for designating the projection position of the image by the mounting terminal 10.

  The projected image includes a basic image that is projected first after the identification mark 40 is recognized, and an integrated image that reflects an instruction from the management terminal 30. The basic image and the integrated image may be a still image or a moving image. Further, the basic image and the integrated image may be a planar image or a stereoscopic image. The operation instruction included in the basic image and the integrated image is an operation instruction for the target person and includes the content and procedure of the operation, and is embodied as an operation instruction by characters or an operation instruction by a figure such as an arrow. . The basic image and the integrated image to be projected may be content with audio.

  In addition, the mounting terminal 10 captures the surroundings of the mounting terminal 10 including the basic image and the integrated image projected by the mounting terminal 10, and transmits the captured data to the operation instruction management server 20. The shooting data is preferably moving image data.

  Based on the medium information received from the mounting terminal 10, the operation instruction management server 20 transmits basic image data associated with the medium information in the operation instruction management server 20 to the mounting terminal 10. In addition, the operation instruction management server 20 transmits the photographing data received from the mounting terminal 10 to the management terminal 30 and receives the auxiliary data received from the management terminal 30 and the basic image data transmitted to the mounting terminal 10. The integrated image data is integrated and transmitted to the mounting terminal 10.

  The management terminal 30 is a computer terminal used by an administrator who manages the progress of the operation of the target person. The management terminal 30 includes a display unit that displays an image and an operation unit that receives an operation of the administrator, and is embodied in, for example, a personal computer, a tablet terminal, or a smartphone. For example, the management terminal 30 is used by the administrator at a location different from the location where the operation by the subject is performed. When there are a plurality of management terminals 30, each of the plurality of managers may use the management terminal 30 at a different place.

When receiving the photographic data from the operation instruction management server 20, the management terminal 30 displays a photographic image that is an image based on the photographic data on the display unit, and accepts an input of an auxiliary instruction for the displayed photographic image. The auxiliary instruction is a more detailed instruction or an additional instruction with respect to the operation of the target person, and is embodied in, for example, characters or drawings, and is input to the management terminal 30 by the operation of the operation unit by the administrator. It is preferable that the auxiliary instruction is input using, for example, a mouse or a touch panel as an operation unit, and is written over the captured image displayed on the display unit. When the auxiliary instruction is input, the management terminal 30 generates auxiliary data indicating the content of the auxiliary instruction, the input position of the auxiliary instruction with respect to the captured image, and the like, and transmits the auxiliary data to the operation instruction management server 20.
[Configuration of wearing terminal]
With reference to FIG. 2 and FIG. 3, the structure of the mounting | wearing terminal 10 is demonstrated in detail.
As shown in FIG. 2, the mounting terminal 10 includes a communication unit 11, a photographing unit 12, a projection unit 13, a projection auxiliary unit 14, a control unit 15, and a storage unit 16.

  The communication unit 11 executes a connection process between the mounting terminal 10 and the operation instruction management server 20 through the network NW. In addition, the communication unit 11 transmits and receives data between the wearing terminal 10 and the operation instruction management server 20.

  The imaging unit 12 is embodied in, for example, a Web camera, and mainly shoots a part of the periphery of the mounting terminal 10 that enters the field of view of the subject wearing the mounting terminal 10. Shooting data generated by shooting by the shooting unit 12 is transmitted to the operation instruction management server 20 via the communication unit 11 with real time or a time difference close to real time.

  The imaging unit 12 also serves as an acquisition unit that acquires medium information from the identification mark 40, and images the identification mark 40 in the imaging range of the imaging unit 12. The image captured by the imaging unit 12 is analyzed by the control unit 15, and based on such analysis, the identification mark 40 is recognized and the medium information is extracted from the identification mark 40. The medium information acquired by the mounting terminal 10 in this way is transmitted to the operation instruction management server 20 via the communication unit 11.

The projection unit 13 is embodied by, for example, a projector, and displays an image by projecting it on a wall or the like. As described above, the image projected by the projection unit 13 includes a basic image and an integrated image.
The projection auxiliary unit 14 collects information for correcting the projection state of the image by the projection unit 13.

  For example, the projection assisting unit 14 is an infrared camera and collects information for correcting the stereoscopic effect of the image. Specifically, the projection auxiliary unit 14 performs three-dimensional measurement of a three-dimensional object existing in a range where the image is projected, and based on data obtained from the three-dimensional measurement, the three-dimensional effect of the projected image. Is corrected. That is, when the number of dimensions of the space on the surface on which the image is projected and the number of dimensions of the space required for displaying the image are different from each other, the image projected on the surface is as if the space required for displaying the image. The image data is corrected in the mounting terminal 10 so that the image is displayed as if it was projected onto the camera.

  For example, when the basic image is a plane image, the image displayed on the part including the three-dimensional object is corrected so that it appears as a plane. When the basic image is a three-dimensional image, the part including the three-dimensional object is corrected. The projected image is corrected so as to appear with a predetermined stereoscopic effect. In a work site where an operation is performed by a subject, it may be difficult to secure a plane as an image projection location. Even in such a case, according to the above configuration, the image is projected by the projection unit 13. This makes it easier to see the images. In addition, when the unevenness information of the image projection location can be acquired in advance, it is preferable that the data of the basic image is created so that the operation instruction is displayed in an easy-to-read manner in consideration of the unevenness.

For example, the projection auxiliary unit 14 is a gyro sensor and collects information for correcting the projection position of the image. Specifically, the projection assisting unit 14 detects the angle of inclination and the angular velocity of the mounting terminal 10 and corrects the position of the image such as the angle and height of the image based on the detection result. As a result, it is possible to prevent the image from being blurred due to the movement of the subject.
Note that the mounting terminal 10 may include a plurality of projection assisting units 14 having different functions.

  The control unit 15 includes a CPU, and includes control of communication in the communication unit 11, reading and writing of information in the storage unit 16, various arithmetic processes, and the like based on programs and data stored in the storage unit 16. Control of each part of the terminal 10 is performed.

  For example, the control unit 15 analyzes an image captured by the imaging unit 12, recognizes the identification mark 40, and acquires medium information. Further, the control unit 15 receives an instruction from the operation instruction management server 20 and causes the projection unit 13 to project a basic image or an integrated image. Further, the control unit 15 corrects the basic image and the integrated image to be projected by the projection unit 13 based on the information collected by the projection assist unit 14.

  The storage unit 16 temporarily or permanently stores programs and data necessary for the processing of the control unit 15. For example, the storage unit 16 stores programs and data necessary for recognizing the identification mark 40 and programs and data necessary for correcting the basic image and the integrated image.

  The configuration for realizing the function of the control unit 15 may be embodied by various hardware such as a plurality of CPUs and memories, or by software that provides a plurality of functions to one common hardware. Also good.

An example of the mounting terminal 10 will be described with reference to FIG. In FIG. 3, a terminal that is mounted on the subject's head is illustrated as the mounting terminal 10.
The mounting terminal 10 includes a helmet-shaped mounting unit 18, and a Web camera as the photographing unit 12, a projector as the projection unit 13, and an infrared camera as the projection auxiliary unit 14 are attached to the mounting unit 18. . The projection unit 13 is also assembled with a gyro sensor as the projection auxiliary unit 14.

  The projection unit 13 is disposed at a position where an image can be projected in front of the mounting unit 18, that is, toward a portion that enters the field of view of the subject. The imaging unit 12 and the infrared camera as the projection assisting unit 14 are arranged at a position where a part that enters the field of view of the subject including an area where the image is projected by the projection unit 13 can be captured.

The imaging unit 12, the projection unit 13, and the projection auxiliary unit 14 are connected to the control device 19. The control device 19 functions as the communication unit 11, the control unit 15, and the storage unit 16, and the control device 19 may be a tablet terminal, for example. The imaging unit 12, the projection unit 13, and the projection assisting unit 14 may be connected to the control device 19 by wire or wirelessly. This is preferable because the degree of freedom in size increases.
The subject wears the wearing terminal 10 by putting the wearing portion 18 on his / her head and placing the control device 19 in the pocket of the subject's clothes or in the vicinity of the subject.

The arrangement of each part in the mounting terminal 10 is not limited to the above example. For example, when the control device 19 is small, the control device 19 may also be attached to the mounting unit 18. Further, the shape of the mounting portion 18 when the mounting terminal 10 is mounted on the subject's head is not limited to the helmet shape as described above, and the mounting portion 18 has a headband shape and has the head of the subject. It may be wound around the part.
[Configuration of operation instruction management server]
The configuration of the operation instruction management server 20 will be described in detail with reference to FIGS. 4 and 5.
The operation instruction management server 20 includes a communication unit 21, a control unit 22, and a storage unit 23.

  The communication unit 21 functions as a reception unit and a transmission unit, and executes a connection process between the operation instruction management server 20 and the mounting terminal 10 and a connection process between the operation instruction management server 20 and the management terminal 30 through the network NW. . The communication unit 21 transmits and receives data between the operation instruction management server 20 and the connection destination device.

The control unit 22 includes a CPU, and based on programs and data stored in the storage unit 23, controls communication in the communication unit 21, reads and writes information in the storage unit 23, and manages operation instructions such as various arithmetic processes. Controls each part of the server 20.
The control unit 22 includes an information specifying unit 22a and an information integration unit 22b as part of a configuration for realizing the operation instruction method by the operation instruction system.

  The information specifying unit 22 a specifies a basic image to be projected on the mounting terminal 10 based on the medium information received from the mounting terminal 10 by the communication unit 21 and the correspondence data 23 a stored in the storage unit 23. Then, the operation instruction management server 20 transmits the specified basic image data to the mounting terminal 10 via the communication unit 21 to instruct the mounting terminal 10 to project the basic image.

  The information integration unit 22 b transmits the shooting data received by the communication unit 21 from the mounting terminal 10 to the management terminal 30 via the communication unit 21. Further, the information integration unit 22b receives the auxiliary data from the management terminal 30 based on the input of the auxiliary instruction to the management terminal 30 when the basic image is projected by the mounting terminal 10. The basic image data and the auxiliary data are integrated to generate integrated image data in which an auxiliary instruction is added to the basic image. Then, the operation instruction management server 20 transmits the generated integrated image data to the mounting terminal 10 via the communication unit 21 and instructs the mounting terminal 10 to project the integrated image.

  Further, the information integration unit 22b allows the communication unit 21 to receive new auxiliary data from the management terminal 30 based on a further input of an auxiliary instruction to the management terminal 30 when an integrated image is projected by the mounting terminal 10. , The integrated image data projected by the mounting terminal 10 and the new auxiliary data are integrated to generate new integrated image data in which an auxiliary instruction is further added to the integrated image. Then, the operation instruction management server 20 transmits the generated new integrated image data to the mounting terminal 10 via the communication unit 21 and instructs the mounting terminal 10 to project the new integrated image.

  When there are a plurality of management terminals 30, the information integration unit 22 b transmits shooting data to each of the management terminals 30 via the communication unit 21. Then, the information integration unit 22b integrates the auxiliary data received from each of the management terminals 30 with the data of the basic image or the integrated image to generate new integrated image data.

  The storage unit 23 temporarily or permanently stores programs and data necessary for the processing of the control unit 22. As an example of such data, the storage unit 23 stores basic image data and the above-described correspondence data 23a. In the correspondence data 23a, the medium information and the image information indicating the basic image are associated with each other. The image information is information that defines an image to be projected on the mounting terminal 10 and is information assigned to each basic image, and functions as identification information for the basic image. As a result, in the operation instruction management server 20, the medium information and the basic image are associated with each other via the correspondence data 23a.

In addition, in the control part 22, the structure which has a function as the information specific | specification part 22a and the information integration part 22b may be actualized with various hardwares, such as several CPU and memory, and it is in one common hardware. It may be embodied by software that provides multiple functions.
A specific example of the data structure of the correspondence data 23a will be described with reference to FIG.
As shown in FIG. 5, in the correspondence data 23a, the medium information, that is, the identification information of the identification mark 40 and the image information are associated with each other.

  For example, in the example shown in FIG. 5, “001”, which is one of the medium information, is associated with image information indicating “basic image A”, and “002”, which is one of the medium information, It is associated with image information indicating “image B”. The basic image A is an image including an operation instruction of an operation to be performed by the subject at a place where the identification mark 40 that holds “001” as medium information is arranged, and the basic image B is “002” as medium information. "Is an image including an operation instruction of an operation to be performed by the subject at a place where the identification mark 40 holding"

  As described above, in the correspondence data 23a, one piece of image information is associated with one piece of medium information, and different pieces of image information are associated with different pieces of medium information.

[Operation instruction procedure]
With reference to FIG. 6 and FIG. 7, the procedure in which the instruction | indication of the operation | movement to a subject by an operation | movement instruction | indication system is performed is demonstrated. First, a procedure until a basic image is projected by the mounting terminal 10 will be described with reference to FIG.

  As shown in FIG. 6, when the identification mark 40 enters the shooting range of the shooting unit 12 of the mounting terminal 10, the control unit 15 of the mounting terminal 10 recognizes the identification mark 40 and acquires medium information (step S10). . Then, the mounting terminal 10 transmits the medium information to the operation instruction management server 20 (step S11).

  When the medium information is received from the mounting terminal 10, the information specifying unit 22a of the operation instruction management server 20 extracts the image information associated with the medium information in the correspondence data 23a based on the medium information and the correspondence data 23a. Specify (step S12). Then, the operation instruction management server 20 transmits the basic image data indicated by the specified image information to the mounting terminal 10 (step S13).

  When the basic image data is received from the operation instruction management server 20, the control unit 15 of the mounting terminal 10 causes the projection unit 13 to project the basic image based on the received data (step S14). At this time, the control unit 15 corrects the basic image based on the information acquired by the projection auxiliary unit 14 and causes the projection unit 13 to project the image as necessary.

As a result, a basic image including an operation instruction regarding an operation to be performed by the subject at that location is projected in front of the subject, and the subject can perform an operation according to the operation instruction by the image.
Next, a procedure from when the basic image is projected by the mounting terminal 10 to when the integrated image is projected will be described with reference to FIG.

  As shown in FIG. 7, shooting data generated by shooting by the shooting unit 12 of the mounting terminal 10 in a state where a basic image is projected is transmitted from the mounting terminal 10 to the operation instruction management server 20 (step). S15). The photographing data received by the operation instruction management server 20 is transmitted from the operation instruction management server 20 to the management terminal 30 (step S16). Note that the transmission of shooting data from the mounting terminal 10 to the management terminal 30 via the operation instruction management server 20 is not limited to after the basic image is projected, and the shooting is continued after the shooting unit 12 starts shooting. It may always be done while you are.

  When receiving the photographic data from the operation instruction management server 20, the management terminal 30 displays the photographic image on the display unit of the management terminal 30 based on the received data (step S17). As a result, an image that is almost the same as the scene that the subject is viewing is displayed on the management terminal 30.

  The administrator views the captured image displayed on the management terminal 30 and operates the operation unit of the management terminal 30 to input an auxiliary instruction to the management terminal 30 (step S18). The management terminal 30 generates auxiliary data indicating the input auxiliary instruction and transmits it to the operation instruction management server 20 (step S19).

  When the auxiliary data is received from the management terminal 30, the information integration unit 22 b of the operation instruction management server 20 integrates the basic image data and the auxiliary data projected by the projection unit 13 at that time, thereby integrating the integrated image data. Is generated (step S20). Then, the operation instruction management server 20 transmits the generated integrated image data to the mounting terminal 10 (step S21).

When the integrated image data is received from the operation instruction management server 20, the control unit 15 of the mounting terminal 10 causes the projection unit 13 to project the integrated image based on the received data (step S22). At this time, if necessary, the control unit 15 corrects the integrated image based on the information acquired by the projection assisting unit 14 and causes the projection unit 13 to project it.
Thereby, the integrated image in which the auxiliary instruction is added to the basic image is projected in front of the subject, and the subject can perform the operation according to the auxiliary instruction.

  In addition, in the state where the integrated image is projected, the integration image of the auxiliary data is used as the integrated image data that is projected by the projection unit 13 at that time, and the processing of step S15 to step S22 is repeated, so that the integrated image is obtained. A new integrated image to which a further auxiliary instruction is added is projected.

  When there are a plurality of management terminals 30, the operation instruction management server 20 receives auxiliary data indicating auxiliary instructions input to each of the management terminals 30 from each of the management terminals 30. Then, each time the operation instruction management server 20 receives the auxiliary data, the operation instruction management server 20 generates integrated image data and transmits it to the mounting terminal 10 to cause the mounting terminal 10 to project the integrated image. Thereby, since the instructions from the plurality of management terminals 30 are reflected in the projected image each time, an accurate instruction is given to the target person.

[Action]
The operation of this embodiment will be described with reference to FIG. FIG. 8 is a diagram illustrating an example of a process in which the subject performs an operation in response to an instruction from the operation instruction system.
As shown in FIG. 8A, when the subject Us wearing the wearing terminal 10 comes to the place where the work is performed and the identification mark 40 enters the shooting range of the shooting unit 12 of the wearing terminal 10, the wearing terminal 10 recognizes the identification mark 40. Based on the recognition of the identification mark 40, the medium information is transmitted from the mounting terminal 10 to the operation instruction management server 20, and the basic image data associated with the medium information in the operation instruction management server 20 is the operation instruction management. It is transmitted from the server 20 to the mounting terminal 10.

  As a result, as shown in FIG. 8B, the basic image Ia corresponding to the identification mark 40 is projected by the projection unit 13 of the wearing terminal 10. The subject Us looks at the projected basic image Ia and performs an operation according to an operation instruction included in the basic image Ia. At this time, in the configuration in which the identification mark 40 is an AR marker, the projection position of the basic image is defined on the basis of the position of the identification mark 40, so that an operation instruction can be easily displayed at a preset position. It is. Therefore, for example, an instruction for individually pointing to an object to be operated is possible, so that a more accurate operation instruction can be performed.

  As shown in FIG. 8C, an image captured by the imaging unit 12 of the mounting terminal 10 by transmitting the shooting data from the mounting terminal 10 to the management terminal 30 via the operation instruction management server 20. Are displayed on the management terminal 30 including the basic image Ia and the behavior of the subject Us.

  As shown in FIG. 8D, the administrator Um looks at the captured image displayed on the management terminal 30 and, for example, uses the management terminal 30 as a sub instruction with a finer operation instruction or an additional operation instruction. To enter. Thereby, auxiliary data indicating the auxiliary instruction is transmitted from the management terminal 30 to the operation instruction management server 20. Then, the operation instruction management server 20 integrates the basic image data and the auxiliary data, and transmits the integrated image data to the mounting terminal 10.

  As a result, as shown in FIG. 8E, the integrated image Ib is projected by the projection unit 13 of the mounting terminal 10. The subject Us sees the projected integrated image Ib and further performs an operation according to the auxiliary instruction included in the integrated image Ib. As shown in FIG. 8, the auxiliary instruction may be written at an arbitrary position of the image. For example, a display field for displaying a sentence is partitioned in the integrated image Ib, and the auxiliary field is displayed in this display field. Instructions may be displayed.

  In addition, when the integrated image Ib is projected, the projection state is also reflected in the captured image. Therefore, the auxiliary terminal included in the integrated image Ib is also used in the management terminal 30 other than the management terminal 30 to which the auxiliary instruction is input. The contents of the instruction can be confirmed. Therefore, the administrator who uses the other management terminal 30 confirms the content of the auxiliary instruction included in the integrated image Ib and the operation of the target person who receives the auxiliary instruction through the captured image, and sends further auxiliary instructions to the management terminal 30. By inputting, further instructions can be given to the subject. As described above, as a result of the captured image being shared by the plurality of management terminals 30, it is possible to instruct the subject to perform an accurate operation.

According to such a configuration, the effects listed below can be obtained.
(1) Since an image including an operation instruction is projected from the wearing terminal 10 worn by the object person, the object person moves to the terminal location where the instruction is displayed for confirmation of the operation instruction as in the past. There is no need to take out the terminal every time the operation is confirmed, or to operate while holding the terminal. Therefore, it is possible to suppress a decrease in the efficiency of the operation in a situation where the operation is performed while receiving an instruction.

  In addition, since the operation instruction is displayed as an image in the real environment by projection, for example, compared to the case where the operation instruction is displayed on a display unit such as a display, for example, a direct instruction to the operation target is given. Can be displayed, and more accurate operation instructions can be given. Further, the operation instruction can be easily switched by switching the projected image. Therefore, even when an operation instruction is performed in a plurality of steps, it is easy to display such operation instructions in order.

  (2) Shooting data shot by the mounting terminal 10 is transmitted to the management terminal 30 via the operation instruction management server 20, and a shot image is displayed on the management terminal 30. Therefore, even when the administrator is in a remote place, the state of the operation of the target person can be grasped through the captured image. Further, even when a plurality of managers are in different locations, each manager can grasp the behavior of the target person using the plurality of management terminals 30.

  (3) The auxiliary instruction input to the management terminal 30 is added to the basic image or the integrated image, and the integrated image is projected from the mounting terminal 10. According to this, even when the administrator is in a remote place, the administrator can instruct the subject according to the situation while viewing the captured image displayed on the management terminal 30.

  (4) Further, in the configuration in which the operation instruction system includes a plurality of management terminals 30, an auxiliary instruction is input to each of the plurality of management terminals 30, and an image including the plurality of auxiliary instructions is projected as an integrated image. A plurality of managers can give instructions according to the situation to the subject while viewing the captured image. As a result, opinions of a plurality of managers are gathered, and it is possible to instruct the target person more accurately.

  (5) Since the mounting terminal 10 includes the projection auxiliary unit 14 that collects information used for correcting the projection status of the basic image and the integrated image, the projection status of the image can be corrected using the collected information. It becomes. As a result, a more appropriate image can be projected. Specifically, in the configuration in which the projection auxiliary unit 14 is an infrared camera and the stereoscopic effect of the image is corrected according to the surface shape of the three-dimensional object arranged at the place where the image is projected, the projection unit 13 The projected image is easier to see. Further, in the configuration in which the projection assisting unit 14 is a gyro sensor and the projection position of the image is corrected according to the inclination or shaking of the wearing terminal 10, it is possible to suppress the occurrence of blurring in the image due to the movement of the target person. It is done.

  (6) In the configuration in which the mounting terminal 10 is mounted on the subject's head, it is easy to project an image at a position that is easy for the subject to view. Moreover, compared with the case where the mounting | wearing terminal 10 is mounted | worn with a trunk and limbs, a subject becomes easier to perform operation | movement.

  (7) Since the identification mark 40 functions as an AR marker, and the projection position of the basic image and the integrated image is defined with reference to the position of the identification mark 40, the position set in advance by setting the position of the identification mark 40 It is possible to easily display an operation instruction. Therefore, it is possible to instruct a more accurate operation.

[Modification]
The above embodiment can be implemented with the following modifications.
The generation of integrated image data may be performed by the mounting terminal 10. In this case, the operation instruction management server 20 transmits the auxiliary data received from the management terminal 30 to the mounting terminal 10, and the control unit 15 of the mounting terminal 10 displays the basic image or the integrated image projected by the projection unit 13 at that time. The image data and the auxiliary data are integrated to generate integrated image data. Then, the control unit 15 causes the projection unit 13 to project the integrated image based on the generated integrated image data.

  The basic image data is stored in the storage unit 16 of the wearing terminal 10 in advance, and the wearing terminal 10 projects a predetermined image among the basic images stored in the wearing terminal 10 in accordance with an instruction from the operation instruction management server 20. May be. In this case, the operation instruction management server 20 instructs the mounting terminal 10 to project the basic image indicated by the image information specified by the information specifying unit 22a, and the control unit 15 of the mounting terminal 10 is stored in the storage unit 16. The basic image data that matches the instruction from the operation instruction management server 20 is identified from the basic image data, and the basic image is projected on the projection unit 13 based on this data. In such a configuration, the generation of the integrated image data may be performed by the mounting terminal 10.

  The correction of the image based on the information collected by the projection auxiliary unit 14 may be performed by the operation instruction management server 20. In this case, information collected by the projection auxiliary unit 14 is transmitted from the mounting terminal 10 to the operation instruction management server 20, and based on this information, the control unit 22 of the operation instruction management server 20 corrects the basic image or the integrated image. Do. Then, the corrected basic image or integrated image data is transmitted from the operation instruction management server 20 to the mounting terminal 10, and the mounting terminal 10 projects the corrected basic image or integrated image based on this data.

  The medium information only needs to include the identification information of the identification mark 40 and may not be the identification information itself. For example, the mounting terminal 10 transmits the image captured by the imaging unit 12 to the operation instruction management server 20 as medium information, and the operation instruction management server 20 analyzes the image to obtain the medium information. Identification information may be extracted from the image. And in correspondence data 23a, identification information and image information should just be matched.

  The identification mark 40 may have a three-dimensional shape as a pattern for each identification mark 40, and the three-dimensional shape for each identification mark 40 may function as identification information. In this case, in addition to the configuration of the above-described embodiment, the mounting terminal 10 may further include a configuration for measuring the three-dimensional shape of the object as an acquisition unit, or a Web camera as the imaging unit 12 and the projection auxiliary unit 14. The infrared camera as a function as an acquisition unit, and the three-dimensional shape of the identification mark 40 may be measured by these cooperation.

  The medium having identification information is not limited to a medium that holds unique information as a pattern, such as the identification mark 40. For example, the medium having the identification information may be a medium that holds the identification information as digital data, such as an IC tag. In this case, the mounting terminal 10 only needs to have a configuration as an acquisition unit that acquires medium information including identification information from the medium in accordance with how the identification information is retained in the medium. For example, when the medium is an IC tag, the mounting terminal 10 may include a reader that reads information from an IC chip as the acquisition unit.

  Furthermore, the output mode of the identification information held by the medium is not limited to the output as a two-dimensional or three-dimensional pattern or digital data. For example, the medium may include a lighting device having an LED or the like, and output identification information as a pattern of blinking infrared light or visible light or a lighting color pattern by the lighting device. Specifically, it is possible to output identification information based on a light pattern by using a PPM (Pulse Position Modulation) signal that expresses a bit value by a combination of the length of the lighting time and the lighting time. In this case, the mounting terminal 10 only needs to have a configuration capable of recognizing the output as an acquisition unit according to the output mode of the identification information in the medium, and the mounting terminal 10 acquires in addition to the configuration of the above embodiment. A configuration as a unit may be provided, or a configuration such as a Web camera as the photographing unit 12 and an infrared camera as the projection auxiliary unit 14 may be functioned as the acquisition unit.

  -An auxiliary instruction | indication may not be input into the management terminal 30, but the captured image based on imaging | photography data may only be displayed on the management terminal 30. FIG. That is, the integrated image data is not generated, and the mounting terminal 10 may only project the basic image. Even with such a configuration, the effects (1) and (2) can be obtained. Furthermore, the transmission of the photographing data to the management terminal 30 may not be performed. In other words, the operation instruction system may not include the management terminal 30. Even with such a configuration, the effect (1) can be obtained.

  An image including only the auxiliary instruction may be projected by the mounting terminal 10 without adding the auxiliary instruction to the basic image. For example, if the projection of the basic image is stopped after a certain period of time or when a predetermined instruction is given to the wearing terminal 10, the management terminal 30 will have a surrounding area around the wearing terminal 10 that does not include the basic image as a captured image. Status is displayed. In this state, when an auxiliary instruction is input to the management terminal 30 and auxiliary data is transmitted from the management terminal 30 to the operation instruction management server 20, the operation instruction management server 20 includes the auxiliary instruction indicated by the auxiliary data. The wearing terminal 10 is instructed to project an image. Even with such a configuration, an administrator who uses the management terminal 30 can give an instruction according to the status of the operation of the target person.

  The operation that is the target of the instruction using the operation instruction system and the operation instruction method is not limited to the operation included in the work at the factory or the construction site, or the treatment at the medical site. The above operation may be an operation included in cooking, a game, or the like, and may be a human movement. In short, the application scenes of the motion instruction system and the motion instruction method are not limited, and the motion instruction included in the basic image to be projected may be a configuration that instructs a predetermined motion to the subject.

  DESCRIPTION OF SYMBOLS 10 ... Installation terminal, 11 ... Communication part, 12 ... Imaging | photography part, 13 ... Projection part, 14 ... Projection auxiliary | assistant part, 15 ... Control part, 16 ... Memory | storage part, 20 ... Operation instruction management server, 21 ... Communication part, 22 ... Control part, 22a ... Information specifying part, 22b ... Information integration part, 23 ... Storage part, 23a ... Corresponding data, 30 ... Management terminal, 40 ... Identification mark.

Claims (11)

A wearing terminal to be worn by a target person who receives an operation instruction;
An operation instruction management server connected to the wearing terminal via a network,
The wearing terminal is
An acquisition unit for acquiring medium information including the identification information from a medium holding the identification information;
A projection unit for projecting an image,
The operation instruction management server
A receiving unit for receiving the medium information from the wearing terminal;
A storage unit that stores correspondence data in which the identification information is associated with image information indicating an image including an operation instruction;
An information specifying unit for specifying the image information associated with the identification information included in the medium information using the medium information and the correspondence data received by the receiving unit;
A transmission unit that instructs the mounting terminal to project the image indicated by the image information specified by the information specifying unit. The operation instruction system is a management terminal connected to the operation instruction management server via a network, and further includes the management terminal having a display unit for displaying an image,
The wearing terminal is
Including a region where the projection unit projects an image, a photographing unit that photographs the surroundings of the wearing terminal,
In the operation instruction management server,
The receiving unit receives shooting data generated by shooting of the shooting unit of the wearing terminal from the wearing terminal,
The transmitting unit transmits the photographing data received by the receiving unit to the management terminal,
The operation instruction system according to claim 1, wherein the management terminal displays a captured image, which is an image based on the captured data received from the operation instruction management server, on the display unit. The management terminal receives input of an auxiliary instruction that is an instruction for a situation indicated by the captured image, and transmits auxiliary data indicating the input auxiliary instruction to the operation instruction management server;
In the operation instruction management server,
The receiving unit receives the auxiliary data from the management terminal;
The operation instruction system according to claim 2, wherein the transmission unit instructs the wearing terminal to project an image including the auxiliary instruction indicated by the auxiliary data. The operation instruction system includes a plurality of the management terminals,
In the operation instruction management server,
The receiving unit receives the auxiliary data from each of the plurality of management terminals,
The operation instruction system according to claim 3, wherein the transmission unit instructs the mounting terminal to project an image including a plurality of the auxiliary instructions indicated by the plurality of auxiliary data received from each of the plurality of management terminals. The operation instruction system according to claim 1, wherein the wearing terminal further includes a projection auxiliary unit that collects information used for correcting a projection state of the image indicated by the image information. The projection auxiliary unit includes an infrared camera that collects information for correcting the stereoscopic effect of the image according to the surface shape of a three-dimensional object arranged at a place where the image is projected, and a projection position of the image The operation instruction system according to claim 5, comprising at least one of gyro sensors that collect information for correction. The operation instruction system according to claim 1, wherein the wearing terminal is worn on a head of the subject. The medium functions as an AR marker,
The operation instruction system according to any one of claims 1 to 7, wherein a projection position of the image indicated by the image information by the mounting terminal is defined on the basis of the position of the medium. A wearing terminal equipped with a projection unit that projects an image and worn by a subject who receives an operation instruction;
An operation instruction management server connected to the wearing terminal via a network;
An operation instruction method performed using
The mounting terminal obtaining medium information including the identification information from a medium holding the identification information;
The operation instruction management server receiving the medium information from the mounting terminal;
The identification information included in the medium information received by the operation instruction management server using correspondence data in which the operation instruction management server associates the identification information with image information indicating an image including the operation instruction. Identifying the image information associated with
The operation instruction management server instructing the mounting terminal to project the image indicated by the identified image information;
In accordance with an instruction from the operation instruction management server, the mounting terminal projects the image indicated by the image information;
An operation instruction method including: An operation terminal that is connected to an operation instruction management server that manages operation instructions via a network and is attached to a target person who receives operation instructions;
An acquisition unit for acquiring medium information including the identification information from a medium holding the identification information;
The medium information acquired by the acquisition unit is transmitted to the operation instruction management server, and is an image associated with the identification information included in the medium information from the operation instruction management server and includes the operation instruction. A communication unit for receiving an image projection instruction;
A projection unit that projects the image according to the instruction from the operation instruction management server;
Wearing terminal equipped with. An operation instruction management server having a projection unit that projects an image and is attached to a target person who receives an operation instruction, and a management terminal having a display unit that displays an image via a network. And
A storage unit that stores correspondence data in which the identification information of the medium holding the identification information is associated with image information indicating an image including an operation instruction;
A receiving unit that receives medium information including the identification information of the medium and shooting data generated by shooting around the mounting terminal by the mounting terminal from the mounting terminal;
An information specifying unit for specifying the image information associated with the identification information included in the medium information using the medium information and the correspondence data received by the receiving unit;
A transmission unit that instructs the wearing terminal to project the image indicated by the image information specified by the information specifying unit, and transmits the photographing data received by the receiving unit to the management terminal;
Further, the receiving unit receives auxiliary data indicating an auxiliary instruction that is an instruction for a situation indicated by an image displayed on the management terminal based on the shooting data from the management terminal,
The transmission unit instructs the wearing terminal to project an image including the auxiliary instruction indicated by the auxiliary data.

Images