JP6515473B2 - Operation instruction system, operation instruction method, and operation instruction management server - Google Patents

Description

The present invention, operation instruction system utilized with an instruction operation for the subject, the operation instruction method, and to operate instruction management server that is used in these operation instruction system and operating instruction method.

  2. Description of the Related Art Conventionally, a system has been developed which is used to instruct work contents and work procedures to a worker who carries out work in a factory or the like. In such a system, for example, work information indicating work content and work procedure is displayed on a computer terminal assigned to each worker, and the worker performs work according to the work information (for example, see Patent Document 1).

JP, 2013-73511, A

  By the way, when the computer terminal is installed and used in the above system, there is a restriction on the place where the terminal can be installed at the work site, so the terminal is installed near the place where the worker is working It is not always possible. Then, when the installation location of the terminal is not secured near the worker, the worker must interrupt the work and move to the terminal every time confirmation of the work information.

  On the other hand, when the computer terminal is a terminal that is carried and used like a tablet terminal, the worker does not need to move the worker, but the worker takes out the terminal every time confirmation of work information, or , I have to work while holding the terminal.

  As a result, in any case, the act of confirming work information leads to a decrease in work efficiency. Such a problem is not limited to a system for instructing work in a factory, but is a system for instructing an operation of a target person, such as a system for instructing work at a construction site or a system for instructing treatment at a medical site. It is a task that is generally common.

The present invention provides an operation instruction system, an operation instruction method, and an operation instruction according to such an operation instruction system and operation instruction method capable of instructing the object person to operate while suppressing a decrease in the efficiency of the operation by the object person. an object of the present invention is to provide an operating instruction management server that can.

  The operation instruction system for solving the above problems comprises: an attached terminal attached to a target person who receives an instruction of operation; and an operation instruction management server connected to the attached terminal via a network, and the attached terminal includes: An acquisition unit for acquiring medium information including the identification information from a medium holding identification information, and a projection unit for projecting an image, the operation instruction management server receives the medium information from the mounting terminal A storage unit that stores correspondence data in which a reception unit, the identification information, and image information indicating an image including an operation instruction are associated, and the medium information received by the reception unit and the correspondence data An information identifying unit identifying the image information associated with the identification information included in the medium information; and a projection of the image indicated by the image information identified by the information identifying unit. And a transmission unit that instructs the.

  The operation instruction method for solving the above problems includes a mounting terminal mounted on a target person who receives an operation instruction, including a projection unit that projects an image, and an operation instruction management server connected to the mounting terminal via a network. , Wherein the mounting terminal acquires medium information including the identification information from a medium holding the identification information, and the operation instruction management server receives the information from the mounting terminal. The operation instruction management server receives the medium information, the operation instruction management server using correspondence data in which the identification information and the image information indicating the image including the operation instruction are associated with each other. Identifying the image information associated with the identification information included in the medium information; and the operation instruction management server indicating the identified image information A step of instructing the projection of serial images to the mounting terminal, according to an instruction of the operation instruction management server, comprising the steps of: projecting the image showing the mounting terminals of the image information.

  According to the above configuration, since the image including the operation instruction is projected from the wearing terminal worn by the object person, the location of the terminal where the object person displays the instruction for confirming the operation instruction as in the related art Moving to the terminal, removing the terminal every time the operation is confirmed, and suppressing the operation while holding the terminal. Therefore, in the situation where the subject performs the operation while receiving the instruction, it is possible to instruct the subject to perform the operation while suppressing the decrease in the efficiency of the motion by the subject.

  The operation instruction system is a management terminal connected to the operation instruction management server via a network, further comprising the management terminal having a display unit for displaying an image, and the projection unit is an image And an imaging unit for imaging the periphery of the mounting terminal including a region for projecting the image data, and in the operation instruction management server, the receiving unit is configured to generate the imaging data generated by the imaging of the imaging unit of the mounting terminal. The transmission unit receives from the attached terminal, and the transmission unit transmits the imaging data received by the reception unit to the management terminal, and the management terminal is an imaging based on the imaging data received from the operation instruction management server It is preferable to display an image on the display unit.

  According to the above configuration, the imaging data generated by imaging with the mounting terminal is transmitted to the management terminal via the operation instruction management server, and the photographed image is displayed on the management terminal. Therefore, even when the user of the management terminal is at a remote location, the user of the management terminal can grasp the state of the operation of the subject through the photographed image.

  In the above operation instruction system, the management terminal receives an input of an auxiliary instruction which is an instruction for a condition indicated by the photographed image, and transmits auxiliary data indicating the input auxiliary instruction to the operation instruction management server, In the operation instruction management server, preferably, the reception unit receives the auxiliary data from the management terminal, and the transmission unit instructs the mounting terminal to project an image including the auxiliary instruction indicated by the auxiliary data. .

  According to the above configuration, the image including the auxiliary instruction input to the management terminal is projected from the mounting terminal. Therefore, even when the user of the management terminal is at a remote place, it is possible to give an instruction according to the situation to the target person while looking at the photographed image displayed on the management terminal.

  The operation instruction system includes a plurality of management terminals, and in the operation instruction management server, the receiving unit receives the auxiliary data from each of the plurality of management terminals, and the transmission unit is configured to perform the plurality of managements. It is preferable to instruct the mounting terminal to project an image including a plurality of the auxiliary instructions indicated by the plurality of auxiliary data received from each of the terminals.

  According to the above configuration, the auxiliary instruction is input to each of the plurality of management terminals, and the image including the plurality of auxiliary instructions is projected from the mounted terminal. Therefore, a plurality of users using the plurality of management terminals are targeted Can give instructions depending on the situation. As a result, the opinions of a plurality of users gather, and it is possible to instruct the target person to perform a more accurate operation.

In the operation instruction system, the mounting terminal preferably further includes a projection assistance unit that collects information used for correcting the projection situation of the image indicated by the image information.
According to the above configuration, it is possible to collect information for correcting the projection condition of the image. Therefore, it is possible to correct the projection situation of the image.

  In the above operation instruction system, the projection assisting unit is an infrared camera that collects information for correcting the three-dimensional effect of the image according to the surface shape of the three-dimensional object arranged at the place where the image is projected; It is preferable to include at least one of gyro sensors that collect information for correcting the projection position of the image.

  According to the above configuration, in the configuration in which the projection auxiliary unit includes the infrared camera, the projected image is corrected by correcting the stereoscopic effect of the image according to the surface shape of the three-dimensional object arranged at the position where the image is projected. Can be made more visible. Further, in the configuration in which the projection assistance unit includes the gyro sensor, the projection position of the image is corrected according to the tilt or shaking of the attached terminal to suppress occurrence of blurring in the image due to the movement of the object person. it can.

In the operation instruction system, it is preferable that the mounting terminal be mounted on the head of the subject.
According to the above configuration, it is easy to project the image to a position easy to see by the target person. In addition, compared with the case where the wearing terminal is worn on the trunk or limbs, the subject person can operate more easily.

  In the motion indication stem, preferably, the medium functions as an AR marker, and a projection position of the image indicated by the image information by the mounting terminal is defined based on a position of the medium.

  According to the above configuration, by setting the position of the medium, it is possible to easily display the operation instruction at the position set in advance. Therefore, more accurate operation instructions can be given.

  The mounting terminal for solving the above problems is a mounting terminal which is connected to the operation instruction management server for managing the operation instruction via the network and is mounted on the target person who receives the operation instruction, and which holds identification information An acquisition unit for acquiring medium information including the identification information, and the medium information acquired by the acquisition unit is transmitted to the operation instruction management server, and the identification included in the medium information from the operation instruction management server A communication unit configured to receive an instruction to project the image including an operation instruction, the communication unit including the operation instruction, and a projection unit to project the image according to the instruction from the operation instruction management server .

  By using the mounting terminal configured as described above, an image including an operation instruction is projected from the mounting terminal worn by the target person, so in the operation instructing system or the operation instructing method using such a mounting terminal, It is possible to instruct the target person to operate while suppressing the decrease in efficiency.

  The operation instruction management server which solves the above-mentioned subject has a projection unit which projects an image, and a management terminal and a network which have an attached terminal attached to a target person who receives an operation instruction, and a display unit which displays an image. An operation instruction management server connected via the storage unit storing correspondence data in which the identification information of a medium holding identification information and image information indicating an image including the operation instruction are associated; A receiving unit for receiving medium information including the identification information of the medium and shooting data generated by shooting around the mounting terminal by the mounting terminal from the mounting terminal; and the medium information received by the receiving unit An information identification unit that identifies the image information associated with the identification information included in the medium information using the correspondence data and the correspondence data; and the image identified by the information identification unit A transmitting unit for instructing the mounting terminal to project the image represented by the information, and transmitting the imaging data received by the receiving unit to the management terminal, and the receiving unit further includes: Receiving auxiliary data indicating an auxiliary instruction which is an instruction for a condition indicated by the image displayed on the management terminal based on the photographing data, the transmitting unit projects a projection of the image including the auxiliary instruction indicated by the auxiliary data. An instruction is given to the attached terminal.

  By using the operation instruction management server configured as described above, an image including an operation instruction is projected from the attached terminal worn by the object person, so in the operation instruction system or the operation instruction method using such an operation instruction management server It is possible to instruct the target person to perform the operation while suppressing a decrease in the efficiency of the operation by the person. In addition, since the image including the auxiliary instruction input to the management terminal is projected from the mounting terminal, the user of the management terminal can view the photographed image displayed on the management terminal even when the user is at a remote location. And can give instructions to the target person according to the situation.

  According to the present invention, it is possible to instruct the target person to operate while suppressing a decrease in the efficiency of the operation by the target person.

It is a figure which shows the whole structure of the operation | movement instruction | command system of one Embodiment. It is a figure which shows the structure of the mounting | wearing terminal of one Embodiment. It is a figure which shows an example of the mounting | wearing terminal of one Embodiment. It is a figure which shows the structure of the operation | movement instruction | command management server of one Embodiment. It is a figure showing an example of data composition of corresponding data in one embodiment. It is a sequence diagram which shows the procedure of the instruction | indication of the operation | movement by the operation | movement instruction | indication system of one Embodiment, Comprising: It is a figure which shows the procedure until a basic image is projected. It is a sequence diagram which shows the procedure of the instruction | indication of the operation | movement by the operation instruction | command system of one Embodiment, Comprising: It is a figure which shows the procedure until an integrated image is projected. (A)-(e) is a figure which shows an example of a mode that a subject person receives the instruction | indication of the operation | movement by the operation | movement instruction | indication system of one Embodiment.

  An embodiment of an operation instruction system, an operation instruction method, a mounting terminal, and an operation instruction management server will be described with reference to FIGS. 1 to 8.

[Configuration of operation instruction system]
The entire configuration of the operation instruction system will be described with reference to FIG.
As shown in FIG. 1, the operation instruction system includes a mounting terminal 10, an operation instruction management server 20, and one or more management terminals 30. The mounting terminal 10, the operation instruction management server 20, and one or more management terminals 30 are connected to a network NW such as the Internet. The mounting terminal 10 and the operation instruction management server 20 mutually transmit and receive data, and the management terminal 30 and the operation instruction management server 20 mutually transmit and receive data.

  The network to which the mounting terminal 10 and the operation instruction management server 20 are connected and the network to which the management terminal 30 and the operation instruction management server 20 are connected may be mutually different networks.

  The mounting terminal 10 is a terminal having, as main functions, a recognition function of the identification mark 40, a projection function of an image, and a photographing function of an image, and is mounted on a target person who receives an operation instruction. The mounting mode is not particularly limited, and the mounting terminal 10 may be mounted on the head of a subject having a helmet-like mounting portion, or may have a ring-shaped mounting portion through which a neck passes. It may be hung on the neck of a person, or may be worn on the face of a subject with a glasses-like attachment. Note that the mounting terminal 10 is preferably mounted on the head in terms of enhancing the ease of operation and the visibility of the projected image.

  The identification mark 40 has a pattern for each identification mark 40, thereby holding information unique to each identification mark 40. The identification mark 40 may be a barcode or a two-dimensional code, or the pattern of the identification mark 40 may constitute a picture, a pattern or a character. Information unique to the identification mark 40 held by the identification mark 40 functions as identification information of the identification mark 40.

  The mounting terminal 10 transmits, to the operation instruction management server 20, medium information which is identification information of the identification mark 40 acquired based on the recognition of the identification mark 40. Then, the mounting terminal 10 projects an image including the operation instruction based on the data received from the operation instruction management server 20. It is preferable that the identification mark 40 be an AR marker and function as a mark that designates the projection position of the image by the mounting terminal 10.

  The projected image includes a basic image that is first projected after the identification mark 40 is recognized, and an integrated image on which the instruction from the management terminal 30 is reflected. The basic image and the integrated image may be a still image or a moving image. Also, the basic image and the integrated image may be a planar image or a stereoscopic image. The operation instruction included in the basic image and the integrated image is an operation instruction for the target person, includes the content and procedure of the operation, and is embodied in an operation instruction with characters and an operation instruction with a figure such as an arrow. . The basic image or integrated image to be projected may be content to which sound is added.

  Further, the mounting terminal 10 captures the periphery of the mounting terminal 10 including the basic image and the integrated image projected by the mounting terminal 10, and transmits the captured data to the operation instruction management server 20. It is preferable that the imaging data is data of a moving image.

  The operation instruction management server 20 transmits the data of the basic image associated with the medium information by the operation instruction management server 20 to the mounting terminal 10 based on the medium information received from the mounting terminal 10. In addition, the operation instruction management server 20 transmits the imaging data received from the mounting terminal 10 to the management terminal 30, receives the auxiliary data received from the management terminal 30, and the data of the basic image transmitted to the mounting terminal 10. Are generated and transmitted to the mounting terminal 10.

  The management terminal 30 is a computer terminal used by a manager who manages the progress of the operation of the object person. The management terminal 30 includes a display unit on which an image is displayed, and an operation unit that receives an operation of a manager, and is embodied as, for example, a personal computer, a tablet terminal, or a smartphone. For example, the management terminal 30 is used by the administrator at a place different from the place where the operation by the subject is performed. When there are a plurality of management terminals 30, each of a plurality of administrators may use the management terminal 30 at a mutually different place.

When the imaging data is received from the operation instruction management server 20, the management terminal 30 causes the display unit to display a captured image which is an image based on the imaging data, and receives an input of an auxiliary instruction for the displayed captured image. The auxiliary instruction is a more detailed instruction or an additional instruction for the operation of the object person, and is embodied in, for example, characters or a figure, and is input to the management terminal 30 by the operation of the operation unit by the administrator. The auxiliary instruction is preferably input using, for example, a mouse or a touch panel as an operation unit, and is preferably written over the captured image displayed on the display unit. When the auxiliary instruction is input, the management terminal 30 generates auxiliary data indicating the content of the auxiliary instruction, the input position of the auxiliary instruction to the photographed image, and the like, and transmits the auxiliary data to the operation instruction management server 20.
[Configuration of attached terminal]
The configuration of the mounting terminal 10 will be described in detail with reference to FIGS. 2 and 3.
As shown in FIG. 2, the mounting terminal 10 includes a communication unit 11, a photographing unit 12, a projection unit 13, a projection assistance unit 14, a control unit 15, and a storage unit 16.

  The communication unit 11 executes connection processing between the mounting terminal 10 and the operation instruction management server 20 through the network NW. Further, the communication unit 11 transmits and receives data between the mounting terminal 10 and the operation instruction management server 20.

  The imaging unit 12 is embodied, for example, as a web camera, and images a portion of the periphery of the mounting terminal 10 that mainly falls within the field of view of the subject wearing the mounting terminal 10. Photographed data generated by photographing of the photographing unit 12 is transmitted to the operation instruction management server 20 via the communication unit 11 in real time or with a time difference close to real time.

  The imaging unit 12 also serves as an acquisition unit for acquiring medium information from the identification mark 40, and images the identification mark 40 within the imaging range of the imaging unit 12. The image taken by the imaging unit 12 is analyzed by the control unit 15, and based on such analysis, the identification mark 40 is recognized and medium information is extracted from the identification mark 40. The medium information thus acquired by the mounting terminal 10 is transmitted to the operation instruction management server 20 via the communication unit 11.

The projection unit 13 is embodied, for example, in a projector, and displays an image by projecting the image on a wall or the like. As described above, the image projected by the projection unit 13 includes the basic image and the integrated image.
The projection assistance unit 14 collects information for correcting the projection condition of the image by the projection unit 13.

  For example, the projection assistance unit 14 is an infrared camera, and collects information for correcting the stereoscopic effect of an image. Specifically, the projection assisting unit 14 performs three-dimensional measurement of a three-dimensional object present in the range where the image is projected, and based on the data obtained from the three-dimensional measurement, the three-dimensional effect of the projected image Is corrected. That is, when the number of dimensions of the space on which the image is projected is different from the number of dimensions of the space required for display of the image, the space projected on the surface is determined as if it were determined for display of the image The image data is corrected at the mounting terminal 10 so that the image is displayed as if it were projected onto the image.

  For example, in the case where the basic image is a two-dimensional image, the image projected on a portion including a three-dimensional object is corrected to be visible in a plane, and when the two-dimensional image is a three-dimensional image, The projected image is corrected so as to appear with a predetermined three-dimensional effect. In a work site or the like where an operation is performed by a target person, it may be difficult to secure a plane as an image projection location, but even in such a case, according to the above configuration, the projection unit 13 projects Images are easier to see. In addition, when the information of the unevenness of the projection place of the image can be obtained in advance, it is preferable that the data of the basic image is created so that the operation instruction is displayed in an easy-to-see manner in consideration of such unevenness.

Also, for example, the projection assistance unit 14 is a gyro sensor, and collects information for correcting the projection position of the image. Specifically, the projection assisting unit 14 detects the angle of inclination and the angular velocity of the mounting terminal 10, and the position of the image on which the image is projected is corrected based on the detection result. As a result, blurring of the image due to the movement of the subject is suppressed.
In addition, the mounting terminal 10 may be equipped with the several projection assistance part 14 which has a mutually different function.

  The control unit 15 includes a CPU, and controls communication in the communication unit 11, reads and writes information in the storage unit 16, and includes various arithmetic processing and the like based on programs and data stored in the storage unit 16. Control of each part of the terminal 10 is performed.

  For example, the control unit 15 analyzes the image captured by the imaging unit 12, recognizes the identification mark 40, and acquires medium information. Further, in response to an instruction from the operation instruction management server 20, the control unit 15 causes the projection unit 13 to project a basic image and an integrated image. The control unit 15 also corrects the basic image and the integrated image to be projected by the projection unit 13 based on the information collected by the projection assistance unit 14.

  The storage unit 16 temporarily or permanently stores programs and data required for the processing of the control unit 15. For example, the storage unit 16 stores programs and data necessary for recognizing the identification mark 40, and programs and data necessary for correcting the basic image and the integrated image.

  The configuration for realizing the functions of the control unit 15 may be embodied by various hardware such as a plurality of CPUs and memories, and is embodied by software which gives a plurality of functions to one common hardware. It is also good.

An example of the mounting terminal 10 will be described with reference to FIG. In FIG. 3, as the mounting terminal 10, a terminal mounted on the head of a subject is illustrated.
The mounting terminal 10 has a helmet-like mounting unit 18, and the mounting unit 18 is mounted with a Web camera as the imaging unit 12, a projector as the projection unit 13, and an infrared camera as the projection assistance unit 14. . Further, a gyro sensor as the projection auxiliary unit 14 is also assembled to the projection unit 13.

  The projection unit 13 is disposed at a position where the image can be projected toward the front of the mounting unit 18, that is, a part that enters the field of view of the subject. The imaging unit 12 and the infrared camera as the projection auxiliary unit 14 are disposed at positions where an image of a part including the area onto which the image is projected by the projection unit 13 can be captured.

The imaging unit 12, the projection unit 13, and the projection assistance unit 14 are connected to the control device 19. The control device 19 functions as the communication unit 11, the control unit 15, and the storage unit 16. The control device 19 may be, for example, a tablet terminal. The photographing unit 12, the projection unit 13, and the projection assisting unit 14 may be connected to the control device 19 by wire or wirelessly, but when the wireless connection is performed, the arrangement position of the control device 19 or It is preferable because the degree of freedom in size is increased.
The target person wears the mounting terminal 10 by putting the mounting unit 18 on his head and putting the control device 19 in a pocket or the like of the clothes of the target person or in the vicinity of the target person.

In addition, arrangement | positioning of each part in the mounting terminal 10 is not restricted to the above-mentioned example, For example, when the control apparatus 19 is small, the control apparatus 19 may also be attached to the mounting part 18. FIG. In addition, the shape of the mounting portion 18 when the mounting terminal 10 is mounted on the head of the subject is not limited to the helmet shape as described above, and the mounting portion 18 has a headband shape and the head of the subject It may be wound around a part.
[Configuration of operation instruction management server]
The configuration of the operation instruction management server 20 will be described in detail with reference to FIGS. 4 and 5.
The operation instruction management server 20 includes a communication unit 21, a control unit 22, and a storage unit 23.

  The communication unit 21 functions as a reception unit and a transmission unit, and executes connection processing between the operation instruction management server 20 and the mounting terminal 10 and connection processing between the operation instruction management server 20 and the management terminal 30 through the network NW. . Further, the communication unit 21 transmits and receives data between the operation instruction management server 20 and the device of the connection destination.

Control unit 22 includes a CPU, and controls communication in communication unit 21, reads and writes information in storage unit 23, and manages operation instructions such as various arithmetic processing based on programs and data stored in storage unit 23. Control of each part of the server 20 is performed.
The control unit 22 includes an information specifying unit 22a and an information integration unit 22b as a part of the configuration for realizing the operation instruction method by the operation instruction system.

  The information specifying unit 22 a specifies the basic image to be projected on the mounting terminal 10 based on the medium information received by the communication unit 21 from the mounting terminal 10 and the correspondence data 23 a stored in the storage unit 23. Then, the operation instruction management server 20 transmits the data of the specified basic image to the mounting terminal 10 via the communication unit 21 and instructs the mounting terminal 10 to project the basic image.

  The information integration unit 22 b transmits the imaging data received by the communication unit 21 from the attachment terminal 10 to the management terminal 30 via the communication unit 21. Further, when the communication unit 21 receives auxiliary data from the management terminal 30 based on the fact that the auxiliary instruction is input to the management terminal 30 when the basic image is projected by the mounting terminal 10, the information integration unit 22b The basic image data and the auxiliary data are integrated to generate integrated image data in which an auxiliary instruction is added to the basic image. Then, the operation instruction management server 20 transmits the generated integrated image data to the mounting terminal 10 via the communication unit 21 and instructs the mounting terminal 10 to project the integrated image.

  In addition, when the integrated image is projected by the mounting terminal 10, the information integration unit 22b causes the communication unit 21 to generate new auxiliary data from the management terminal 30 based on the fact that the auxiliary instruction is further input to the management terminal 30. Is received, the data of the integrated image projected by the mounting terminal 10 and the new auxiliary data are integrated to generate data of a new integrated image in which an auxiliary instruction is further added to the integrated image. Then, the operation instruction management server 20 transmits the generated data of the new integrated image to the mounting terminal 10 via the communication unit 21 and instructs the mounting terminal 10 to project a new integrated image.

  When there are a plurality of management terminals 30, the information integration unit 22b transmits the imaging data to each of the management terminals 30 via the communication unit 21. Then, the information integration unit 22b integrates the auxiliary data received from each of the management terminals 30 with the data of the basic image or the integrated image to generate data of a new integrated image.

  The storage unit 23 temporarily or permanently stores programs and data required for the processing of the control unit 22. As an example of such data, the storage unit 23 stores data of a basic image and the correspondence data 23a described above. In the correspondence data 23a, medium information and image information indicating a basic image are associated. The image information is information that defines an image to be projected on the mounting terminal 10, is information assigned to each basic image, and functions as identification information of the basic image. As a result, in the operation instruction management server 20, the medium information and the basic image are associated via the correspondence data 23a.

In the control unit 22, the configuration having the functions as the information specifying unit 22a and the information integration unit 22b may be embodied by various hardware such as a plurality of CPUs and memories, or in one common hardware. It may be embodied by software that provides multiple functions.
A specific example of the data configuration of the correspondence data 23a will be described with reference to FIG.
As shown in FIG. 5, in the correspondence data 23a, medium information, that is, identification information of the identification mark 40 and image information are associated with each other.

  For example, in the example shown in FIG. 5, one of the medium information “001” is associated with the image information indicating “basic image A”, and one of the medium information “002” is “basic”. It is matched with the image information which shows "image B". The basic image A is an image including an operation instruction of an operation to be performed by the subject at the place where the identification mark 40 holding “001” as the medium information is arranged, and the basic image B is “002” as the medium information. "" Is an image including an operation instruction of an operation to be performed by the subject at the place where the identification mark 40 holding "is placed.

  As described above, in the correspondence data 23a, one piece of medium information is associated with one piece of image information, and different pieces of medium information are associated with different pieces of image information.

[Operation instruction procedure]
With reference to FIG. 6 and FIG. 7, the procedure by which the instruction | indication of the operation | movement to a subject by the operation instruction | command system is performed is demonstrated. First, with reference to FIG. 6, the procedure until the basic image is projected by the mounting terminal 10 will be described.

  As shown in FIG. 6, when the identification mark 40 is in the imaging range of the imaging unit 12 of the mounting terminal 10, the control unit 15 of the mounting terminal 10 recognizes the identification mark 40 and acquires medium information (step S10). . Then, the mounting terminal 10 transmits the medium information to the operation instruction management server 20 (step S11).

  When the medium information is received from the mounting terminal 10, the information specifying unit 22a of the operation instruction management server 20, based on the medium information and the correspondence data 23a, the image information associated with the medium information in the correspondence data 23a. It identifies (step S12). Then, the operation instruction management server 20 transmits data of a basic image indicated by the specified image information to the mounting terminal 10 (step S13).

  When the data of the basic image is received from the operation instruction management server 20, the control unit 15 of the mounting terminal 10 causes the projection unit 13 to project the basic image based on the received data (step S14). At this time, the control unit 15 corrects the basic image and causes the projection unit 13 to project the basic image based on the information acquired by the projection assistance unit 14 as necessary.

Thereby, the basic image including the action instruction for the action that the subject should perform at the place is projected in front of the subject, and the subject can perform the action according to the action instruction by the image.
Subsequently, with reference to FIG. 7, a procedure from the projection of the basic image by the mounting terminal 10 to the projection of the integrated image will be described.

  As shown in FIG. 7, in the state where the basic image is projected, the photographing data generated by the photographing of the photographing unit 12 of the mounting terminal 10 is transmitted from the mounting terminal 10 to the operation instruction management server 20 (step S15). Then, the shooting data received by the operation instruction management server 20 is transmitted from the operation instruction management server 20 to the management terminal 30 (step S16). Note that transmission of shooting data from the mounting terminal 10 to the management terminal 30 via the operation instruction management server 20 is not limited to after the basic image is projected, and shooting is continued after the shooting unit 12 starts shooting. It may always be done while

  When the imaging data is received from the operation instruction management server 20, the management terminal 30 displays the photographed image on the display unit of the management terminal 30 based on the received data (step S17). As a result, an image substantially the same as the scene viewed by the subject is displayed on the management terminal 30.

  The administrator looks at the photographed image displayed on the management terminal 30, and operates the operation unit of the management terminal 30 to input an auxiliary instruction to the management terminal 30 (step S18). The management terminal 30 generates auxiliary data indicating the input auxiliary instruction, and transmits the auxiliary data to the operation instruction management server 20 (step S19).

  When the auxiliary data is received from the management terminal 30, the information integration unit 22b of the operation instruction management server 20 integrates the data of the basic image projected by the projection unit 13 at that time and the auxiliary data, and the data of the integrated image Are generated (step S20). Then, the operation instruction management server 20 transmits data of the generated integrated image to the mounting terminal 10 (step S21).

When data of the integrated image is received from the operation instruction management server 20, the control unit 15 of the mounting terminal 10 causes the projection unit 13 to project the integrated image based on the received data (step S22). At this time, the control unit 15 corrects the integrated image and causes the projection unit 13 to project the integrated image based on the information acquired by the projection assistance unit 14 as necessary.
As a result, the integrated image in which the assistance instruction is added to the basic image is projected in front of the subject, and the subject can perform an operation in accordance with the assistance instruction.

  Further, in a state where the integrated image is projected, the integrated image is obtained by repeating the processing of step S15 to step S22 as data of the integrated image that the projection unit 13 is projecting at that time, the integration target of the auxiliary data. The new integrated image is projected with additional assistance instructions added to.

  When there are a plurality of management terminals 30, the operation instruction management server 20 receives, from each of the management terminals 30, auxiliary data indicating an auxiliary instruction input to each of the management terminals 30. Then, every time the operation instruction management server 20 receives the auxiliary data, the operation instruction management server 20 generates integrated image data and transmits it to the mounting terminal 10, and causes the mounting terminal 10 to project the integrated image. As a result, since the instructions from the plurality of management terminals 30 are reflected in the image being projected each time, an accurate instruction is given to the target person.

[Effect]
The operation of the present embodiment will be described with reference to FIG. FIG. 8 is a diagram showing an example of a process in which the subject person performs an operation in response to an instruction from the operation instruction system.
As shown in FIG. 8A, when the target person Us who wears the wearing terminal 10 comes to the place where the operation is performed and the identification mark 40 falls within the imaging range of the photographing unit 12 of the wearing terminal 10, the wearing terminal 10 recognizes the identification mark 40. Based on the recognition of the identification mark 40, the medium information is transmitted from the mounting terminal 10 to the operation instruction management server 20, and the data of the basic image associated with the medium information in the operation instruction management server 20 is the operation instruction management. It is transmitted from the server 20 to the mounting terminal 10.

  As shown in FIG. 8B, as a result, the basic image Ia corresponding to the identification mark 40 is projected by the projection unit 13 of the mounting terminal 10. The subject Us looks at the projected basic image Ia and performs an operation according to the operation instruction included in the basic image Ia. At this time, in the configuration in which the identification mark 40 is an AR marker, the projection position of the basic image is defined on the basis of the position of the identification mark 40. Therefore, the operation instruction can be easily displayed at a preset position. It is. Therefore, for example, since an instruction to individually indicate the target of the operation is also possible, it is possible to perform the more accurate instruction of the operation.

  As illustrated in FIG. 8C, an image captured by the imaging unit 12 of the mounting terminal 10 by transmitting the imaging data from the mounting terminal 10 to the management terminal 30 via the operation instruction management server 20. Is displayed on the management terminal 30 including the basic image Ia and the behavior of the target person Us.

  As shown in FIG. 8D, the administrator Um looks at the photographed image displayed on the management terminal 30, and, for example, the management terminal 30 uses an instruction of a finer operation or an instruction of an additional operation as an auxiliary instruction. Enter in Thereby, the auxiliary data indicating the auxiliary instruction is transmitted from the management terminal 30 to the operation instruction management server 20. Then, in the operation instruction management server 20, the data of the basic image and the auxiliary data are integrated, and the data of the integrated image is transmitted to the mounting terminal 10.

  As shown in FIG. 8E, as a result, the integrated image Ib is projected by the projection unit 13 of the mounting terminal 10. The subject Us looks at the projected integrated image Ib and performs further operation in accordance with the assistance instruction included in the integrated image Ib. The assistance instruction may be written at an arbitrary position of the image as shown in FIG. 8. For example, in the integrated image Ib, a display field for displaying a sentence is partitioned, and the display field is assisted. Instructions may be displayed.

  In addition, when the integrated image Ib is projected, the projection status is also reflected in the photographed image, so that the assistance included in the integrated image Ib also in the management terminal 30 different from the management terminal 30 to which the assistance instruction is input. It becomes possible to confirm the content of the instruction. Therefore, the administrator using the other management terminal 30 confirms the content of the assistance instruction included in the integrated image Ib and the operation of the subject who received it through the captured image, and sends the assistance instruction to the management terminal 30 By inputting, the subject can be given further instructions. As described above, as a result of the photographed image being shared by the plurality of management terminals 30, it is possible to instruct the target person of an appropriate operation.

According to such a configuration, the effects listed below can be obtained.
(1) Since the image including the operation instruction is projected from the mounting terminal 10 worn by the object person, the object person moves to the location of the terminal for displaying the instruction for confirmation of the operation instruction as in the prior art It is not necessary to take out the terminal every time the operation is confirmed or to carry out the operation while holding the terminal. Therefore, in the situation where the operation is performed while receiving the instruction, the decrease in the operation efficiency can be suppressed.

  In addition, since the instruction of the operation is displayed as an image in the real environment by projection, for example, a direct instruction to the target of the operation as compared with the case where the instruction of the operation is displayed on a display unit such as a display Can also be displayed, and more accurate operation instructions can be given. Furthermore, switching of the image to be projected can easily be performed switching of the operation instruction. Therefore, even when the operation instruction is performed step by step a plurality of times, it is easy to sequentially display such operation instructions.

  (2) The photographed data photographed by the mounting terminal 10 is transmitted to the management terminal 30 via the operation instruction management server 20, and the photographed image is displayed on the management terminal 30. Therefore, even when the manager is at a remote place, the manager can grasp the behavior of the target person through the photographed image. In addition, even when a plurality of administrators are in different places, each administrator can grasp the behavior of the target person using the plurality of management terminals 30.

  (3) The auxiliary instruction input to the management terminal 30 is added to the basic image or the integrated image, and the integrated image is projected from the mounting terminal 10. According to this, even when the administrator is at a remote place, the administrator can give an instruction according to the situation to the target person while viewing the photographed image displayed on the management terminal 30.

  (4) Further, in the configuration in which the operation instruction system includes the plurality of management terminals 30, the auxiliary instruction is input to each of the plurality of management terminals 30, and an image including the plurality of auxiliary instructions is projected as an integrated image. A plurality of managers can give instructions according to the situation to the subject while viewing the photographed image. As a result, the opinions of a plurality of managers gather, and it is possible to give more accurate instructions to the target person.

  (5) The mounting terminal 10 includes the projection assisting unit 14 that collects information used to correct the projection condition of the basic image and the integrated image, so correction of the image projection condition is possible using the collected information. It becomes. As a result, it is possible to project an image more appropriately. Specifically, in a configuration in which the projection assisting unit 14 is an infrared camera and the stereoscopic effect of the image is corrected according to the surface shape of the three-dimensional object arranged at the place where the image is projected, the projection unit 13 The projected image is easier to see. Further, in the configuration in which the projection assisting unit 14 is a gyro sensor and the projection position of the image is corrected according to the tilt or shaking of the wearing terminal 10, the occurrence of blurring in the image due to the movement of the object person is suppressed. Be

  (6) In the configuration in which the mounting terminal 10 is mounted on the head of the subject, it is easy to project the image to a position where the target is easy to see. Moreover, compared with the case where the wearing terminal 10 is worn on the trunk or limbs, the subject person can operate more easily.

  (7) The identification mark 40 functions as an AR marker, and the projection position of the basic image or the integrated image is defined with reference to the position of the identification mark 40. Therefore, the position set in advance by setting the position of the identification mark 40 It is easily possible to display an operation instruction on Therefore, more accurate operation instructions can be given.

[Modification]
The above embodiment can be modified as follows.
The generation of data of the integrated image may be performed at the mounting terminal 10. In this case, the operation instruction management server 20 transmits the auxiliary data received from the management terminal 30 to the mounting terminal 10, and the control unit 15 of the mounting terminal 10 combines the basic image projected by the projection unit 13 at that time or integration. The image data and the auxiliary data are integrated to generate integrated image data. Then, the control unit 15 causes the projection unit 13 to project the integrated image based on the generated data of the integrated image.

  The data of the basic image is stored in advance in the storage unit 16 of the mounting terminal 10, and the mounting terminal 10 projects a predetermined image of the basic images stored in the mounting terminal 10 according to the instruction of the operation instruction management server 20. You may In this case, the operation instruction management server 20 instructs the mounting terminal 10 to project the basic image indicated by the image information specified by the information specifying unit 22a, and the control unit 15 of the mounting terminal 10 stores the storage unit 16 From the data of the basic image, the data of the basic image that matches the instruction from the operation instruction management server 20 is specified, and the basic image is projected on the projection unit 13 based on this data. In such a configuration, generation of data of the integrated image may be performed by the mounting terminal 10.

  The correction of the image based on the information collected by the projection assistance unit 14 may be performed by the operation instruction management server 20. In this case, the information collected by the projection assistance unit 14 is transmitted from the mounting terminal 10 to the operation instruction management server 20, and based on this information, the control unit 22 of the operation instruction management server 20 corrects the basic image or the integrated image. Do. Then, the corrected basic image or integrated image data is transmitted from the operation instruction management server 20 to the mounting terminal 10, and based on this data, the mounting terminal 10 projects the corrected basic image or integrated image.

  The medium information may include the identification information of the identification mark 40 and may not be the identification information itself. For example, the mounting terminal 10 transmits the image captured by the imaging unit 12 to the operation instruction management server 20 as the medium information, and the image is analyzed by the operation instruction management server 20 to obtain the medium information. Identification information may be extracted from the image. Then, in the correspondence data 23a, the identification information and the image information may be associated with each other.

  The identification mark 40 may have a three-dimensional shape as a pattern for each identification mark 40, and the three-dimensional shape for each identification mark 40 may function as identification information. In this case, the mounting terminal 10 may further include a configuration for measuring the three-dimensional shape of the object as an acquisition unit in addition to the configuration of the above embodiment, and the Web camera as the imaging unit 12 and the projection assistance unit 14 The infrared camera as a may function as an acquisition unit, and the three-dimensional shape of the identification mark 40 may be measured by the cooperation of these.

  The medium having identification information is not limited to a medium that holds information unique as a pattern, such as the identification mark 40. For example, the medium having identification information may be a medium that holds identification information as digital data, such as an IC tag. In this case, the mounting terminal 10 may have a configuration as an acquisition unit that acquires medium information including identification information from the medium according to the manner of holding identification information in the medium. For example, when the medium is an IC tag, the mounting terminal 10 may be provided with a reader that reads information from the IC chip as an acquisition unit.

  Furthermore, the aspect of the output of the identification information held by the medium is not limited to the output as a two-dimensional or three-dimensional pattern or digital data. For example, the medium may include a lighting device having an LED or the like, and may output identification information as a blinking pattern or a lighting color pattern of infrared light or visible light by the lighting device. Specifically, by using a PPM (Pulse Position Modulation) signal representing a bit value by a combination of lighting time and lighting off time, it is possible to output identification information based on a light pattern. In this case, the mounting terminal 10 may have a configuration capable of recognizing the output as an acquisition unit according to the output mode of the identification information in the medium, and the mounting terminal 10 is acquired in addition to the configuration of the above embodiment. A configuration as a unit may be provided, or a configuration such as a Web camera as the imaging unit 12 or an infrared camera as the projection auxiliary unit 14 may function as an acquisition unit.

  The auxiliary instruction may not be input to the management terminal 30, and only the photographed image based on the photographed data may be displayed on the management terminal 30. That is, data of the integrated image may not be generated, and the mounting terminal 10 may only project the basic image. The effects of the above (1) and (2) can be obtained also by this configuration. Furthermore, transmission of imaging data to the management terminal 30 may not be performed. That is, the operation instruction system may not include the management terminal 30. Also by such a configuration, the effect of the above (1) can be obtained.

  The auxiliary instruction may not be added to the basic image, and an image including only the auxiliary instruction may be projected by the mounting terminal 10. For example, when the projection of the basic image is stopped after a certain period of time has passed or when the predetermined instruction to the mounting terminal 10 is given, the management terminal 30 displays the periphery of the mounting terminal 10 not including the basic image as a photographed image The status of is displayed. In this state, when the auxiliary instruction is input to the management terminal 30 and the auxiliary data is transmitted from the management terminal 30 to the operation instruction management server 20, the operation instruction management server 20 is configured from the auxiliary instruction indicated by the auxiliary data. It instructs the mounting terminal 10 to project an image. With this configuration as well, a manager using the management terminal 30 can issue an instruction according to the status of the operation of the target person.

  The operation targeted for the instruction using the operation instruction system and the operation instruction method is not limited to the operation at the factory or the construction site, or the operation included in the treatment at the medical site or the like. The above operation may be an operation included in cooking, a game or the like, and may be a human movement. The point is that the application scene of the motion instruction system and the motion instruction method is not limited, as long as the motion instruction included in the basic image to be projected indicates a predetermined motion to the object person.

  DESCRIPTION OF REFERENCE NUMERALS 10 mounting terminal 11 communication unit 12 imaging unit 13 projection unit 14 projection auxiliary unit 15 control unit 16 storage unit 20 operation instruction management server 21 communication unit 22 Control part, 22a ... Information identification part, 22b ... Information integration part, 23 ... Storage part, 23a ... Correspondence data, 30 ... Management terminal, 40 ... Identification mark.

Claims (7)

An attached terminal attached to a target person who receives an operation instruction;
An operation instruction management server connected to the attached terminal via a network;
A management terminal connected to the operation instruction management server via a network, the plurality of management terminals having a display unit for displaying an image ;
The attached terminal is
An acquisition unit that acquires medium information including the identification information from a medium that holds the identification information;
A projection unit that projects an image;
And a photographing unit that photographs the periphery of the mounting terminal including the image projected by the projection unit .
The operation instruction management server is
A receiving unit that receives the medium information from the attached terminal;
A storage unit that stores correspondence data in which the identification information is associated with image information indicating an image including an operation instruction;
An information identification unit that identifies the image information associated with the identification information included in the medium information using the medium information received by the reception unit and the correspondence data;
And a transmitter configured to instruct the mounting terminal to project the image indicated by the image information identified by the information identifier .
The receiving unit further receives, from the mounting terminal, photographing data generated by the photographing of the photographing unit of the mounting terminal.
The transmitting unit further transmits the imaging data received by the receiving unit to each of the plurality of management terminals.
Each management terminal displays a photographed image, which is an image based on the photographed data received from the operation instruction management server, on the display unit, and receives an input of an auxiliary instruction, which is an instruction for a condition indicated by the photographed image. Transmitting auxiliary data indicating the input auxiliary instruction to the operation instruction management server;
The receiving unit further receives the auxiliary data from each of the plurality of management terminals,
The operation instruction system , wherein the transmitting unit further instructs the attached terminal to project an image including a plurality of the auxiliary instructions indicated by a plurality of the auxiliary data received from each of the plurality of management terminals . The operation instruction system according to claim 1 , wherein the mounting terminal further includes a projection assistance unit configured to collect information used for correcting a projection situation of the image indicated by the image information. The projection auxiliary unit is an infrared camera that collects information for correcting the stereoscopic effect of the image according to the surface shape of the three-dimensional object arranged at the place where the image is projected, and the projection position of the image The operation instruction system according to claim 2 , comprising at least one of a gyro sensor that collects information for correction. The operation instruction system according to any one of claims 1 to 3 , wherein the attachment terminal is attached to a head of the subject. The medium functions as an AR marker,
The operation instruction system according to any one of claims 1 to 4 , wherein a projection position of the image indicated by the image information by the mounting terminal is defined based on the position of the medium. A mounting terminal including a projection unit that projects an image and mounted on a target person who receives an operation instruction;
An operation instruction management server connected to the attached terminal via a network;
A management terminal connected to the operation instruction management server via a network, the plurality of management terminals having a display unit for displaying an image;
Operation instruction method performed using
Acquiring the medium information including the identification information from the medium holding the identification information;
The operation instruction management server receiving the medium information from the attached terminal;
The identification information included in the medium information received by the operation instruction management server by using the correspondence data in which the operation instruction management server associates the identification information with the image information indicating the image including the operation instruction. Identifying the image information associated with
The operation instruction management server instructing the mounting terminal to project the image indicated by the specified image information;
The mounting terminal projects the image indicated by the image information according to an instruction of the operation instruction management server;
The mounting terminal photographing the periphery of the mounting terminal including the image projected by the mounting terminal;
The operation instruction management server receiving, from the mounting terminal, photographing data generated by photographing of the mounting terminal, and transmitting the photographing data to each of the plurality of management terminals;
Each of the plurality of management terminals displaying a photographed image which is an image based on the photographed data received from the operation instruction management server;
Each of the plurality of management terminals receives an input of an auxiliary instruction which is an instruction for a condition indicated by the photographed image, and transmits auxiliary data indicating the input auxiliary instruction to the operation instruction management server;
The operation instruction management server receives the auxiliary data from each of the plurality of management terminals, and instructs the mounting terminal to project an image including the plurality of auxiliary instructions indicated by the plurality of auxiliary data received; ,
Projecting the image including the plurality of auxiliary instructions according to an instruction of the operation instruction management server;
Operation instruction method including. An operation instruction management server connected via a network to a plurality of management terminals having a projection unit that projects an image and that is attached to a target person who receives an operation instruction, and a display unit that displays an image And
A storage unit storing correspondence data in which the identification information of a medium holding identification information and image information indicating an image including an operation instruction are associated;
A receiving unit that receives, from the mounting terminal, medium information including the identification information of the medium and shooting data generated by shooting around the mounting terminal by the mounting terminal;
An information identification unit that identifies the image information associated with the identification information included in the medium information using the medium information received by the reception unit and the correspondence data;
The projection of the image indicated by the image information identified by the information identifying unit is instructed to the mounting terminal, and the photographing data received by the receiving unit , including the image projected by the projection unit, is included. And a transmitter configured to transmit the shooting data generated by shooting around the attached terminal to each of the plurality of management terminals.
Furthermore, the receiving unit may receive from each of a plurality of the management terminal, the auxiliary data indicating an instruction supplementary instructions for the situation indicated by the image displayed on the management terminal based on the imaging data,
And the transmission unit, the operation instruction management server which instructs a projection of the image including a plurality of said auxiliary indication plurality of said auxiliary data received from each of a plurality of the management terminal to the mounting terminal.

Images