JP2017049680A - Object indication system, object indication method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To allow an indication image to be displayed and superposed adequately on an object image even if the object image in a photographic image moves in a system that gives an instruction on work or the like by displaying and superposing the indication image to give an instruction on the work or the like on the object image in the photographic image obtained by photographing an object with a camera or the like.SOLUTION: An object indication system 1 comprises: a photographing unit 2 for photographing an object and outputting an object image; an image display unit 7 for displaying the object image output by the photographing unit 2; an indication information creation unit 4 for creating indication information to give an instruction to the object image; an image superposition unit 5 for making display by superposing an indication image based on the indication information created by the indication information creation unit 4 on the object image displayed on the image display unit 7; and an indication image moving unit 6 for moving the indication image in response to the movement of the object image displayed on the image display unit 7.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an object instruction system, an object instruction method, and a program capable of giving an instruction to an image obtained by photographing an object.

When giving an instruction from a management center or the like to an operator at a work site, the instruction is generally given by voice using a mobile phone or the like.
However, due to recent developments in image processing technology, computer / communication technology, etc., an object such as work is photographed with a camera at the work site, and an image of the obtained object is transmitted to a management center or the like. An instruction is also given by giving an instruction to the received image and transmitting the instructed image to an operator's portable terminal or the like.
For example, in Japanese Patent Application Laid-Open No. 2007-72733 (Patent Document 1), a work target is photographed by a video camera, information on the photographed photographed image is transmitted to a remote terminal, and an instruction based on the photographed image made by the remote terminal is disclosed. An annotation image corresponding to the image is projected from the projector onto the work target, and the information of the photographed image is also transmitted to the portable terminal owned by the worker, so that remote support for supporting work on a large apparatus or production facility from a remote location A system is disclosed.
Recently, computers that can be worn and carried, so-called wearable computers, have begun to spread. Among these wearable computers, there are head mounted displays equipped with cameras, or eyeglass computers called smart glasses. is there.
Work instructions using wearable computers such as head-mounted displays and smart glasses have also been proposed.
For example, in JP2013-16020A (Patent Document 2), an image of a worker's line-of-sight direction captured by a camera mounted on a head-mounted display device and photographed by a camera mounted on the head-mounted display device is disclosed. A work support system that displays an instruction image on which an instruction pointer image is superimposed on a head-mounted display device and gives an instruction to an operator is disclosed.

However, in the remote support system of Patent Document 1, when the work target moves after the annotation image according to the instruction based on the photographed image taken by the camera is projected on the work target, the relative position between the work target and the annotation image is determined. There is a problem that the instruction by the annotation image cannot be accurately performed.
Also in the work support system of Patent Document 2, when an instruction image in which an instruction pointer image is superimposed on a captured image is displayed on the head mounted display device, an object in the captured image moves or the camera itself moves. Since only the object in the instruction image moves and the instruction pointer image does not move, there is a problem in that the relative position of the two shifts and the instruction by the instruction pointer image cannot be performed accurately.

JP 2007-72733 A JP2013-16020A

  The problem to be solved by the present invention is to superimpose and display on the object image an instruction image that gives an instruction for work or the like on the object image in the captured image obtained by photographing the object with a camera or the like. In a system for giving an instruction for work or the like, even if an object image in a photographed image moves, the instruction image can be accurately displayed superimposed on the object image.

  The present invention includes a photographing unit that photographs a target and outputs a target image, an image display unit that displays a target image output by the photographing unit, and instruction information for giving an instruction to the target image. Instruction information creating means for creating, image superimposing means for superimposing and displaying an instruction image based on the instruction information created by the instruction information creating means on the object image displayed on the image display means, and the image An object instruction system including instruction image moving means for moving the instruction image in response to movement of the object image displayed on the display means is provided to solve the above-described problem.

  According to the present invention, an imaging unit that captures an object and outputs an object image, an image display unit that displays the object image output by the imaging unit, and the object image output by the imaging unit are transmitted. Image transmitting means, image receiving means for receiving the object image transmitted by the image transmitting means, and instruction information creating means for creating instruction information for giving an instruction to the object image received by the image receiving means An instruction information transmitting unit that transmits the instruction information created by the instruction information generating unit, an instruction information receiving unit that receives the instruction information transmitted by the instruction information transmitting unit, and an image display unit An image superimposing means for superimposing and displaying an instruction image based on the instruction information received by the instruction information receiving means on the object image; and the object image displayed on the image display means. Providing an object instruction system including a designation image moving means for moving the indication image corresponding to the mobile, it is to solve the above problems.

  According to the present invention, an imaging unit that captures an object and outputs an object image, a first image transmission unit that transmits the object image output by the imaging unit, and a transmission that is transmitted by the first image transmission unit. First image receiving means for receiving the object image, instruction information creating means for creating instruction information for giving an instruction to the object image received by the first image receiving means, and the first An image superimposing means for superimposing an instruction image based on the instruction information created by the instruction information generating means on the object image received by the image receiving means, and the object image received by the first image receiving means. Instruction image moving means for moving the instruction image in response to movement of the image, second image transmission means for transmitting the object image and the instruction image superimposed by the image superimposing means, and the second image Sending means A second image receiving means for receiving the received object image and the instruction image; and an image display means for displaying the object image and the instruction image received by the second image receiving means. An object instruction system is provided to solve the above problems.

  In the present invention, the instruction image moving means includes a feature point extracting means for extracting a feature point of the object image, a feature point before the movement of the object image extracted by the feature point extracting means, and a feature after the movement. An object indication system comprising: a feature point comparison unit that compares points; and a movement amount calculation unit that calculates a movement amount of the instruction image from feature points before and after the movement of the object image compared by the feature point comparison unit To solve the above problems.

  The present invention solves the above problems by providing an object instruction system in which the photographing means and the image display means are attached to a wearable computer attached to the face or head of the wearer.

  The present invention shoots an object by an imaging unit and outputs an object image, displays the object image output by the imaging unit on an image display unit, and instructs the object image output by the imaging unit The object image displayed on the image display means is displayed by superimposing the instruction image based on the instruction information on the object image displayed on the image display means. An object instruction method including a step of moving the instruction image in response to movement of the object is provided to solve the above-described problem.

  In the present invention, an object is photographed by a photographing unit to output a target image, the target image output by the photographing unit is displayed on an image display unit, and the target image output by the photographing unit is displayed as an image. An image receiving unit receives the object image transmitted by the transmitting unit and transmitted by the image transmitting unit, creates instruction information for giving an instruction to the object image received by the image receiving unit, and Information is transmitted by the instruction information transmitting means, the instruction information receiving means receives the instruction information transmitted by the instruction information transmitting means, and the instruction information receiving means is added to the object image displayed on the image display means. An object comprising a step of superimposing and displaying an instruction image based on the received instruction information and moving the instruction image in response to movement of the object image displayed on the image display means Providing How to Display, it is to solve the above problems.

  In the present invention, an object is imaged by an imaging unit to output an object image, the object image output by the imaging unit is transmitted by a first image transmission unit, and the first image transmission unit transmits The object image to be received is received by the first image receiving means, the instruction information for giving an instruction to the object image received by the first image receiving means is created, and the image superimposing means An instruction image based on the instruction information is superimposed on the object image received by the image receiving means, and the instruction image is moved in response to the movement of the object image received by the first image receiving means. The second image transmitting unit transmits the object image and the instruction image superimposed by the image superimposing unit, and the second image transmitting unit transmits the object image and the instruction image transmitted by the second image transmitting unit. Receiving means receives and Said object image and said instruction image image received by the receiving means to provide an object instruction method comprising the step of displaying on the image display means, it is to solve the above problems.

  According to the present invention, in the step of moving the instruction image, the feature point before the movement of the object image and the feature point after the movement are extracted, the extracted feature points are compared, and the instruction point is based on the compared feature points. An object instruction method comprising a step of calculating a moving amount of an image is provided to solve the above problem.

  The present invention solves the above-mentioned problems by providing an object indication method in which the photographing means and the image display means are attached to a wearable computer attached to the face or head of the wearer.

  This invention solves the said subject by providing the program which makes a computer perform the said object instruction | indication method.

  In the object instruction system of the present invention, the object is photographed by the photographing means and the object image is output, the output object image is displayed on the image display means, and the object information is created by the instruction information creating means. Instruction information for giving an instruction is created, and an instruction image based on the instruction information created by the instruction information creating unit is superimposed and displayed on the object image displayed on the image display unit by an image superimposing unit. The instruction image is moved in response to the movement of the object image displayed on the image display means by the instruction image moving means. Therefore, even if the object image is moved, the instruction image is exactly the object. Overlaid on the image, it is possible to give an accurate instruction.

  In the object instruction system of the present invention, the object is photographed by the photographing means and the object image is output, and the outputted object image is displayed on the image display means and transmitted by the image transmitting means. Instruction information for giving an instruction to the object image received by the receiving means and received by the instruction information creating means is created, and the created instruction information is transmitted by the instruction information transmitting means and received by the instruction information receiving means. The instruction image based on the instruction information received by the instruction information receiving unit is superimposed and displayed on the object image displayed on the image display unit by the image superimposing unit, and the image display is performed by the instruction image moving unit. Since the instruction image is moved in response to the movement of the object image displayed on the means, even if the object image is moved, the instruction image is accurately It is superimposed on the image, an effect that it is accurate instructions.

  In the object instruction system of the present invention, the object is photographed by the photographing means and the object image is output, and the outputted object image is transmitted by the first image transmitting means and sent to the first image receiving means. Instruction information for receiving an instruction to the received object image is generated by the instruction information creating means, and the instruction is applied to the object image received by the first image receiving means by the image superimposing means. An instruction image based on the instruction information created by the information generating unit is superimposed, and the instruction image moving unit moves the instruction image in response to the movement of the object image received by the first image receiving unit, The object image and the instruction image superimposed by the image superimposing unit are transmitted by the second image transmitting unit, received by the second image receiving unit, and received by the image display unit. Since the object image and the instruction image being is displayed, even if the moving object image, an instruction image is displayed superimposed precisely the object image, an effect that it is accurate instructions.

  In the instruction image moving means of the object instruction system of the present invention, the feature points of the object image are extracted by the feature point extracting means, and the extracted feature points before and after the movement of the object image are extracted. Since the points are compared by the feature point comparison means, and the movement amount of the instruction image is calculated by the movement amount calculation means from the feature points before and after the movement of the compared object image, the instruction image is more accurately targeted. There is an effect that the object image is superimposed and displayed.

  In the object indication system of the present invention, the photographing means and the image display means are attached to a wearable computer attached to the wearer's face or head, so that the object image is displayed more accurately. There is an effect that can be done.

  In the object instruction method of the present invention, an object is imaged by an imaging unit, an object image is output, the output object image is displayed on an image display unit, and the object image output by the imaging unit is displayed. Instruction information for giving an instruction is created, the instruction image based on the instruction information is superimposed and displayed on the object image displayed on the image display means, and the object displayed on the image display means Since the instruction image is moved in response to the movement of the image, even if the object image is moved, the instruction image is accurately superimposed and displayed on the object image, and an accurate instruction can be obtained.

In the object instruction method of the present invention, an object is imaged by an imaging unit, an object image is output, the output object image is displayed on an image display unit, and the object image output by the imaging unit is displayed. Transmitted by the image transmitting means, the image receiving means receives this, creates instruction information for giving an instruction to the received object image, transmits the created instruction information by the instruction information transmitting means, An instruction image based on the instruction information received by the instruction information receiving means is superimposed on the object image received by the instruction information receiving means and displayed on the image display means, and the displayed object Since the instruction image is moved in response to the movement of the image, even if the object image is moved, the instruction image is accurately superimposed and displayed on the object image, and an accurate instruction can be obtained.
There is an effect.

In the object instruction method of the present invention, an object is imaged by an imaging unit, an object image is output, the output object image is transmitted by a first image transmission unit, and this is transmitted to the first image receiving unit. Is generated, and instruction information for giving an instruction to the received object image is generated, and an image superimposing unit adds an instruction image based on the instruction information to the object image received by the first image receiving unit. And the instruction image is moved in response to the movement of the object image received by the first image receiving means, and the object image and the instruction image superimposed by the image superimposing means are The image transmitting means transmits the received image, the second image receiving means receives it, and the received object image and the instruction image are displayed on the image display means. Accurately displayed superimposed on the object image An effect that it is accurate instructions.
There is an effect.

  The step of moving the instruction image in the object instruction method of the present invention includes extracting the feature point before the movement of the object image and the feature point after the movement, comparing the extracted feature points, and comparing the extracted feature points. Since the step of calculating the movement amount of the instruction image based on the above is provided, there is an effect that the instruction image is more accurately superimposed on the object image.

  In the object indication method of the present invention, since it is attached to a wearable computer attached to the wearer's face or head, the object image can be displayed more accurately.

  The program of the present invention has the same effect as the object indication method.

It is a block diagram which shows schematic structure of one Embodiment of the target object instruction | indication system of this invention. It is a block diagram which shows an example of the specific structure of the target object instruction | indication system shown in FIG. It is an external view which shows the external appearance of the smart glass 20 of FIG. It is a block diagram which shows the other example of the specific structure of the target object instruction | indication system shown in FIG. It is explanatory drawing for demonstrating the specific example of the target image which the imaging | photography part 2 (camera 22, 62) image | photographed, and was taken in by the image acquisition part 3 (23, 63), and the instruction image superimposed on it. It is a block diagram which shows the main structures of the instruction | indication image moving part 6 (26,76). FIG. 6 is an explanatory diagram for explaining the feature points in the instruction image SG extracted by the feature point extraction unit 90 at the current time t and the outer peripheral region of the target image TG shown in FIG. 5. It is explanatory drawing explaining the feature point in the instruction | indication image SG which the feature point extraction part 90 extracted at the time (t + dt) after fixed time (dt) progress from a present point, and its external peripheral area | region. It is explanatory drawing which showed the feature-value and its coordinate about the feature points PA1-PA8 which the feature point extraction part 90 extracted. It is explanatory drawing which showed the feature-value and its coordinate about the feature points PB1-PB8 which the feature point extraction part 90 extracted. Explanatory drawing showing the object image TG and the instruction image SG at a certain time t before the object image TG moves, and at a time (t + dt) after the object image TG moves after a certain time (dt) has passed. It is. In the object instruction | indication system 10 (50), it demonstrates the example which utilized the object image and the instruction | indication image for place guidance, and is explanatory drawing showing the state before the movement of an object image and an instruction | indication image. In the object instruction | indication system 10 (50), it demonstrates the example which utilized the object image and the instruction | indication image for location guidance, and is explanatory drawing showing the state after the movement of an object image and an instruction | indication image. 3 is a flowchart showing the operation of the object instruction system 1. 4 is a flowchart showing an operation of an instruction image moving unit 6 of the object instruction system 1. 3 is a flowchart showing the operation of the object instruction system 10. 5 is a flowchart showing the operation of the object instruction system 50.

[Schematic configuration of object indication system]
FIG. 1 is a block diagram showing a schematic configuration of an embodiment of an object instruction system of the present invention. In the figure, 1 is an object instruction system, 2 is a photographing unit, 3 is an image acquisition unit, 4 is an instruction information creation unit, 5 is an image superimposing unit, 6 is an instruction image moving unit, and 7 is an image display unit.
As shown in the figure, the object instruction system 1 includes an imaging unit 2, an image acquisition unit 3, an instruction information creation unit 4, an image superimposing unit 5, an instruction image moving unit 6, an image display unit 7, and the like.
The photographing unit 2 is composed of a camera or the like attached to a wearable computer such as a smart glass worn by an operator (wearer), and takes an image of the object in the operator's field of view. Output an image.
The image acquisition unit 3 captures the object imaged and output by the image capturing unit 2 and the surrounding image, temporarily stores it as an object image (including a background image of the object), and then stores the object image Is output.
The instruction information creation unit 4 performs, for the target image sent from the image acquisition unit 3, instruction information performed by an operator other than the operator at a management center or the like located away from the work place, for example, Information on an instruction image such as a circular frame surrounding the work portion of the object is created, and the created instruction information is output.
The image superimposing unit 5 is an instruction image based on the instruction information sent from the instruction information creating unit 4 with respect to the object image sent from the image acquisition unit 3, for example, a circular frame surrounding the work location of the object The instruction image is superimposed and the superimposed image is output, but if there is no instruction image from the instruction information creation unit 4, only the object image is output.
The instruction image moving unit 6 moves the instruction image corresponding to the movement of the object image with respect to the object image and the instruction image superimposed by the image superimposing unit 5.
The image display unit 7 displays the superimposed object image and the instruction image sent from the image superimposing unit 5, and displays only the target image when only the target image is sent from the image superimposing unit 5. To do.

[Specific configuration of object indication system]
2 is a block diagram showing an example of a specific configuration of the object instruction system shown in FIG. 1. In FIG. 2, 10 is an object instruction system, 20 is a smart glass which is a kind of wearable computer, and 21 is a control unit. , 22 is a camera, 23 is an image acquisition unit, 24 is an instruction information acquisition unit, 25 is an image superimposition unit, 26 is an instruction image moving unit, 27 is an image display unit, 28 is a transmission unit, 29 is a reception unit, and 30 is a server. 33 is an image acquisition unit, 34 is an instruction information creation unit, 38 is a transmission unit, 39 is a reception unit, and 40 and 41 are communication lines such as the Internet.
As shown in FIG. 2, the object instruction system 10 includes a smart glass 20 and a server 30 connected to the smart glass 20 via communication lines 40 and 41.
Hereinafter, a specific configuration of the object instruction system will be described.

[Smart Glass 20]
FIG. 3 is an external view showing the external appearance of the smart glass 20 of FIG. In the figure, 201 is a mounting portion, 202 is a frame, and 203 is a lens portion. The smart glass 20 includes a glasses-type mounting unit 201 and a controller (not shown) built therein, and a controller 21 is provided in the controller.
The mounting unit 201 includes a frame 202 and a pair of lens units 203, the camera 22 is mounted on the frame 202, and the image display unit 27 is mounted on the lens unit 203. This wearing part 201 is worn in the same manner as glasses on the face of the worker who is the wearer.
The camera 22 is attached in the vicinity of the lens unit 203 on the right side of the frame 202 of the smart glass 20, and photographs and photographs the work object existing in front of the worker wearing the smart glass 20 (mounting unit 201). Output the object image.
The image display unit 27 is provided on the upper portion of the lens unit 203 of the smart glass 20, and the object image photographed and output by the camera 22, the superimposed object image and instruction image sent from the image superimposing unit 25, and the like. Is projected and displayed.
Note that the controller may not be built in the glasses-type mounting unit 201 but may be a separate one.

[Control unit 21]
The control unit 21 includes an image acquisition unit 23, an instruction information acquisition unit 24, an image superimposition unit 25, an instruction image movement unit 26, a transmission unit 28, a reception unit 29, and the like.
The image acquisition unit 23 captures an object captured and output by the camera 22 and its surrounding image, temporarily stores it as an object image (including a background image of the object), and then stores the stored object image. Output.
The instruction information acquisition unit 24 temporarily stores the instruction information received by the reception unit 29 (the instruction information created by the instruction information creation unit 34 of the server 30), and then outputs the saved instruction information.
The image superimposing unit 25 has the same function as the image superimposing unit 5 of the object instruction system 1. That is, the image superimposing unit 25 sends an instruction image based on the instruction information sent from the instruction information obtaining unit 24 to the object image sent from the image obtaining unit 23, for example, a circle surrounding the work portion of the object. An instruction image such as a frame is superimposed and the superimposed image is output, but if there is no instruction image from the instruction information acquisition unit 24, only the object image is output.
The instruction image moving unit 26 has the same function as the instruction image moving unit 6 of the object instruction system 1. That is, the instruction image moving unit 26 moves the instruction image corresponding to the movement of the object image with respect to the object image and the instruction image superimposed by the image superimposing unit 25.
The transmission unit 28 transmits the object image output from the image acquisition unit 23 to the server 30 via the communication line 40.
The receiving unit 29 receives instruction information transmitted from the server 30 via the communication line 41.

[Server 30]
The server 30 is installed in a management center or the like far away from the work site where the worker works. The server 30 includes an image acquisition unit 33, an instruction information creation unit 34, a transmission unit 38, a reception unit 39, and the like.
The receiving unit 39 receives an object image transmitted from the transmitting unit 28 of the smart glass 20 via the communication line 40.
The image acquisition unit 33 temporarily stores the object image received by the reception unit 39, and then outputs the stored object image.
The instruction image creation unit 34 has the same function as the instruction image creation unit 4 of the object instruction system 1. That is, for the object image sent from the image acquisition unit 33, instruction information given by an operator such as a management center, for example, information of an instruction image such as a circular frame surrounding the work portion of the object, Output the created instruction information.
The transmission unit 38 transmits the instruction information output from the instruction image creation unit 34 to the smart glass 20 via the communication line 41.

[Other examples of specific configuration of the object indication system]
FIG. 4 is a block diagram showing another example of the specific configuration of the object instruction system shown in FIG. 1, in which 50 is an object instruction system, 60 is a smart glass, 61 is a control unit, and 62 is a camera. 63 and 65 are image acquisition units, 67 is an image display unit, 68 is a transmission unit, 69 is a reception unit, 70 is a server, 73 is an image acquisition unit, 74 is an instruction information creation unit, 75 is an image superimposition unit, and 76 is An instruction image moving unit, 78 is a transmitting unit, 79 is a receiving unit, and 80 and 81 are communication lines such as the Internet.
As shown in FIG. 4, the object indication system 50 includes a smart glass 60 and a server 70 connected to the smart glass 60 via communication lines 80 and 81.

[Smart Glass 60]
The smart glass 60 has the same external configuration as the smart glass 20 of FIG. 3, and the camera 62 and the image display unit 67 have the same functions as the camera 22 and the image display unit 27 of the smart glass 20, but the controller The configuration of the provided control unit 61 is different from the configuration of the control unit 21 of the smart glass 20.
That is, the control unit 61 includes only image acquisition units 63 and 65, a transmission unit 68, a reception unit 69, and the like.
The image acquisition unit 63 has the same function as the image acquisition unit 23 of the smart glass 20. That is, the image acquisition unit 63 captures an object captured and output by the camera 62 and its surrounding image, temporarily stores it as an object image (including a background image of the object), and then stores the object Output an image.
The image acquisition unit 63 temporarily stores an image (object image, an image obtained by superimposing the object image and the instruction image) sent from the server 70 via the communication line 81 and performed by the reception unit 69, and then the stored image. Is output.
The transmission unit 68 transmits the object image output from the image acquisition unit 63 to the server 70 via the communication line 80.
The receiving unit 69 receives an image (an object image, an image obtained by superimposing the object image and an instruction image) transmitted from the server 70 via the communication line 81.

[Server 70]
The server 70 is installed in a management center or the like far away from the work site where the worker works, similar to the server 30 of the object instruction system 10, but the configuration is different from the configuration of the server 30. That is, similarly to the server 30, the server 70 includes an image acquisition unit 73, an instruction information creation unit 74, a transmission unit 78, a reception unit 79, and the like, and further includes an image superimposing unit 75 and an instruction image moving unit 76. .
The receiving unit 79 receives the object image transmitted from the transmitting unit 68 of the smart glass 60 via the communication line 80.
The image acquisition unit 73 temporarily stores the object image received by the reception unit 79 and then outputs the stored object image.
The instruction image creation unit 74 has the same function as the instruction image creation unit 4 of the object instruction system 1. That is, for the object image sent from the image acquisition unit 73, instruction information given by an operator such as a management center, for example, information of an instruction image such as a circular frame surrounding the work portion of the object is created. Output the created instruction information.
The image superimposing unit 75 has a function similar to that of the image superimposing unit 5 of the target object indicating system 1 and the image superimposing unit 25 of the target object indicating system 10. In other words, the image superimposing unit 75 sends an instruction image based on the instruction information sent from the instruction information obtaining unit 74 to the object image sent from the image obtaining unit 73, for example, a circle surrounding the work location of the object. An instruction image such as a frame is superimposed and the superimposed image is output, but if there is no instruction image from the instruction information acquisition unit 74, only the object image is output.
The transmission unit 78 transmits the object image output from the image superimposing unit 75 and an image obtained by superimposing the object image and the instruction image to the smart glass 60 via the communication line 81.

[Instruction information creating unit 4 (34, 74) and image superimposing unit 5 (25, 75)]
FIG. 5 is an explanatory diagram for describing a specific example of an object image captured by the image capturing unit 2 (cameras 22 and 62) and captured by the image acquisition unit 3 (23 and 63) and an instruction image superimposed thereon. In the figure, TG is an object image, SG is an instruction image, and CG is a superimposed image in which the instruction image SG is superimposed on the object image TG.
Hereinafter, the functions of the instruction information creating unit 4 (34, 74) and the image superimposing unit 5 (25, 75) will be described in detail with reference to FIG.
If the object image TG taken into the image acquisition unit 3 (23, 63) is an image of a car (including a background image) as shown in FIG. 5A, the object image TG is displayed on a display (not shown). The operator inputs information indicating the location of the work (for example, information on a circular frame surrounding the work location). The instruction information creating unit 4 (34, 74) creates instruction information (for example, the center and diameter of a circular frame surrounding the work location) based on information input by the operator, and based on this instruction information, the information shown in FIG. ) Is generated, and the instruction image SG is sent to the image superimposing unit 5 (25, 75).
The image superimposing unit 5 (25, 75) creates a superimposed image CG in which the instruction image SG is superimposed on the object image TG sent from the image acquisition unit 3 (23, 63), and outputs the created superimposed image CG. To do. In this case, when only the target image TG is sent to the image superimposing unit 5 (25, 75) and the instruction image SG is not sent, the image superimposing unit 5 (25, 75) outputs only the target image TG. .

[Instruction image moving unit 6 (26, 76)]
FIG. 6 is a block diagram showing the main configuration of the instruction image moving unit 6 (26, 76). In the figure, 90 is a feature point extracting unit, 91 is a feature point comparing unit, and 92 is a moving amount calculating unit. .
Hereinafter, the configuration and function of the instruction image moving unit 6 (26, 76) will be described.
As shown in FIG. 6, the instruction image moving unit 6 (26, 76) includes a feature point extraction unit 90, a feature point comparison unit 91, and a movement amount calculation unit 92.
The feature point extraction unit 90 extracts feature points in the instruction image and in the outer peripheral area of the target image at regular intervals.
The feature point comparison unit 91 calculates the feature amount of each feature point at a certain time point (time t) extracted by the feature point extraction unit 90 and each feature point at a time point (time t + dt) after a certain time (dt) has elapsed. The feature points are compared, and feature points are associated.
If there is a change in the position of the feature point associated with the feature point comparison unit 91, the movement amount calculation unit 92 calculates the movement amount (including the movement direction) of the target image, assuming that the target image has moved. .
Then, the instruction image moving unit 6 (26, 76) moves the instruction image based on the movement amount calculated by the movement amount calculating unit 92.
Note that the feature point extraction by the feature point extraction unit 90 may be performed not on a part of the object image but on the entire object image.

[Feature point extraction unit 90]
FIG. 7 is an explanatory diagram for explaining the feature points in the instruction image SG extracted by the feature point extraction unit 90 at the current time t with respect to the object image TG shown in FIG. FIG. 6 is an explanatory diagram for explaining the feature points in the instruction image SG extracted by the feature point extraction unit 90 and its outer peripheral region at a time (t + dt) after a lapse of a fixed time (dt) from the current point. ˜PA8, PB1 to PB8 are feature points, GM is the screen of the image display unit, and Q is the point at the upper left corner of the screen GM that is the origin.
The feature point extraction performed by the feature point extraction unit 90 will be described below with reference to the drawings.
The feature point extraction unit 90 includes feature quantities such as Scale-Invariant Feature Transform (SIFT) and Speeded Up Robust Features (SURF) with respect to the inside of the instruction image SG and the external peripheral area of the target image TG. By using a feature point detection method based on a feature amount, a feature point having a large change in shading in an image is detected.
7 shows the feature points PA1 to PA8 extracted by the feature point extraction unit 90 at the present time, and FIG. 8 shows the feature points PB1 to PB8 extracted by the feature point extraction unit 90 after a certain period of time has elapsed. It is shown.

[Feature Point Comparison Unit 91]
FIG. 9 is an explanatory diagram showing the feature amounts and the coordinates of the feature points PA1 to PA8 extracted by the feature point extraction unit 90, and FIG. 10 shows the feature points PB1 to PB8 extracted by the feature point extraction unit 90. It is explanatory drawing which showed the feature-value and its coordinate.
The feature point correlation performed by the feature point comparison unit 91 will be described below with reference to the drawings.
As shown in FIG. 9, the feature amount (local feature amount) for the feature points PA1 to PA8 extracted by the feature point extraction unit 90 is, for example, from information on the luminance gradient of the 4 × 4 rectangular area around the feature point. It is represented by a generated 128-dimensional histogram.
In this case, the coordinates of the feature points PA1 to PA8 are represented by (XA1, YA1) to (XA8, YA8) with the point at the upper left corner of the screen GM as the origin Q.
Further, as shown in FIG. 10, the feature amounts (local feature amounts) of the feature points PB1 to PB8 extracted by the feature point extraction unit 90 are also represented by a 128-dimensional histogram similar to that in FIG. (XB1, YB1) to (XB8, YB8).
Then, the feature point comparison unit 91 compares the feature amounts of the feature points PA1 to PA8 and the feature amounts of the feature points PA1 to PA8, and associates feature points having the same or most similar feature amounts.
In this example (FIGS. 7 to 10), the feature point comparison unit 91 includes a feature point PA1 and a feature point PB1, a feature point PA2 and a feature point PB2, a feature point PA3 and a feature point PB3, a feature point PA4 and a feature point PB4, The feature point PA5 and the feature point PB5, the feature point PA6 and the feature point PB6, the feature point PA7 and the feature point PB7, and the feature point PA8 and the feature point PB8 are associated with each other.

[Movement amount calculation unit 92]
The movement amount calculation unit 92 calculates the movement amount in the X-axis direction and the Y-axis direction of the object image TG from the feature points associated with the feature point comparison unit 91.
Specifically, if there is a change in the coordinates of each associated feature point, the movement amount calculation unit 92 obtains the difference between the X coordinates and the Y coordinates, and takes the average thereof to obtain the X axis. The movement amount DX in the direction and the movement amount DY in the Y-axis direction are calculated. For example, the X coordinate difference between the feature point PA1 and the corresponding feature point PB1 is (XB1-XA1), and the Y coordinate difference is (YB1-YA1).
The instruction image moving unit 6 (26, 76) moves the instruction image SG based on the X-axis direction movement amount DX and the Y-axis direction movement amount DY calculated by the movement amount calculation unit 92.

[Move instruction image SG]
FIG. 11 shows the object image TG and the instruction image SG at a certain time t before the object image TG moves, and at a time (t + dt) after the object image TG moves after a certain time (dt) has elapsed. It is explanatory drawing shown, the target object image TG and the instruction | indication image SG in a certain time t (before movement) are represented with a broken line, and then the target object image TG in the time (t + dt) (after movement) after a fixed time (dt) progresses. The instruction image SG is represented by a solid line.
Hereinafter, the movement of the instruction image SG by the instruction image moving unit 6 (26, 76) will be described with reference to the drawings.
First, the instruction image moving unit 6 (26, 76) obtains coordinates (XRA, YRA) of the center point RA of the instruction image SG at a certain time t (before movement of the object image TG). Next, the instruction image moving unit 6 (26, 76) uses the movement amount DX in the X-axis direction and the movement amount DY in the Y-axis direction calculated by the movement amount calculating unit 92 as the coordinates (XRA, YRA) of the center point RA. The coordinates (XRB, YRB) of the added point RB are obtained.
In this case, XRB = XRA + DX and YRB = YRA + DY.
Then, the instruction image moving unit 6 (26, 76) moves the instruction image SG with the point RB as the center of the instruction image SG at the time (t + dt) (after movement of the object image TG).
Thereby, the relative position of the object image TG and the instruction image SG at a certain time t before the object image TG moves, and the time after the object image TG moves after a certain time (dt) (t + dt) The relative position of the object image TG and the instruction image SG in () is the same, and the instruction image SG has moved corresponding to the movement of the object image TG.

[Other examples of object images]
12 and 13 are explanatory diagrams for explaining an example in which the object image and the instruction image are used for location guidance in the object instruction system 10 (50). In the figure, tg1 is a building in the object image TG. It is an image.
As shown in FIG. 12, the object image TG captured by the camera 22 (62) attached to the smart glass 20 (60) worn by the wearer (user) is displayed on the screen GM of the image display unit 27 (67). It is displayed. The object image TG is an image in a state in which buildings are lined up on both sides across a central road where a car is running.
Now, the instruction image SG is superimposed on the building image tg1, assuming that the building image tg1 is a building image guided by the object image TG displayed on the screen GM.
FIG. 13 shows a state in which the object image TG has moved relative to the screen GM due to the movement of the camera 22 (62) caused by the wearer moving his / her face after a predetermined time has elapsed. In the figure, the object image TG and the instruction image SG before movement are indicated by broken lines.
At this time, the instruction image moving unit 26 (76) extracts the feature points specified by the feature amount and the position (coordinates) of the peripheral image of the instruction image SG in the object image TG before and after the elapse of a predetermined time. Then, the extracted feature points are compared and associated with each other, the movement amount of the object image TG is calculated, and the instruction image SG is moved based on the calculated movement amount.
As a result, as shown in FIG. 13, the instruction image SG also moves corresponding to the movement of the building image tg1, and the building indicated by the building image tg1 can be guided to the wearer (user).

Next, the operation of the object instruction systems 1, 10, and 50 will be described.
[Operation of Object Instruction System 1]
FIG. 14 is a flowchart showing the operation of the object instruction system 1. Hereinafter, the operation of the object instruction system 1 will be described with reference to FIG.
First, the imaging unit 2 captures an object of work that is in the field of view of the worker (wearer), outputs the captured object image, and the image acquisition unit 3 acquires the output object image ( S1).
Next, the object image acquired by the image acquisition unit 3 is sent to the instruction information creation unit 4, and the instruction information creation unit 4 determines whether or not instruction information is input to the object image by the operator (S 2). When the instruction information is input, the instruction information, for example, instruction image information such as a circular frame surrounding the work portion of the object is created (S3).
Next, the image superimposing unit 5 generates an instruction image based on the instruction information sent from the instruction information creating unit 4, for example, an instruction image such as a circular frame surrounding the work portion of the target object, and sends it from the image acquisition unit 3. The instruction image thus generated is superimposed on the object image, and the superimposed object image and instruction image are sent to the image display unit 7 for display (S4).
Next, the instruction image moving unit 6 determines whether or not the object image has moved (S5). When the object image has moved, the instruction image is moved in response to the movement of the object image (S6). ), The process is terminated. On the other hand, if the object image does not move, the process ends without moving the instruction image.
If no instruction information is input in S2, the image superimposing unit 5 sends only the object image sent from the image acquisition unit 3 to the image display unit 7 and displays it (S7), and the process ends.

[Operation of Instruction Image Moving Unit 6]
FIG. 15 is a flowchart showing the operation of the instruction image moving unit 6 of the object instruction system 1, and the operation of the instruction image moving unit 6 will be described below with reference to FIG.
First, the feature point extraction unit 90 extracts feature points of the object image at the current time (time t) (S10).
This feature point extraction is performed by using a feature point detection method based on a feature amount (local feature amount) such as SIFT or SURF for the inside of the instruction image and the image of the peripheral area outside the instruction image or the entire object image. This is performed by detecting a feature point having a large change in shading.
Next, the feature point extraction unit 90 extracts the feature points of the object image by the same method at a time (time t + dt) after a lapse of a certain time (dt) (S11).
Next, the feature point comparison unit 91 calculates the feature quantity of each feature point extracted at the present time (time t) and the feature quantity of each feature point extracted at the time (time t + dt) after a predetermined time (dt) has elapsed. In contrast, each feature point is associated (S12).
This association of feature points is performed by connecting feature points having the same or most similar feature quantities.
Next, the movement amount calculation unit 92 determines whether or not there is a change in the position of the associated feature point (S13).
If it is determined that there is a change in the position of the associated feature point, the movement amount (including the movement direction) of the object image is calculated assuming that the object image has moved (S13).
The movement amount of the object image is calculated by taking the average of the difference between the X coordinate and the Y coordinate of the associated feature points and using the difference as the movement amount in the X direction and the movement amount in the Y direction.
Finally, the instruction image moving unit 6 moves the instruction image based on the movement amount calculated by the movement amount calculating unit 92 (S15), and the process ends.
If it is determined in S13 that there is no change in the position of the feature point, it is determined that the object image has not moved, and the process returns to S10 to extract the feature point of the object image at the current time.

[Operation of Object Instruction System 10]
FIG. 16 is a flowchart showing the operation of the object instruction system 10, and the operation of the object instruction system 10 will be described below with reference to FIG.
First, in the smart glass 20, the camera 22 captures a work object that is in the field of view of the worker (wearer), outputs the captured object image, and outputs the output object image to the image acquisition unit 23. Is acquired (S20).
Next, the object image acquired by the image acquisition unit 23 is sent to the transmission unit 28, and the transmission unit 28 sends the object image to the server 30 via the communication line 40 (S21).
Next, in the server 30, the reception unit 39 receives the object image (S22), and the image acquisition unit 33 once acquires the object image.
Next, the object image is sent to the instruction information creating unit 34, where it is determined whether or not the operator has input instruction information for the object image (S23). If so, the instruction information, for example, information on the instruction image such as a circular frame surrounding the work location of the object is created (S24).
Next, the instruction information is sent to the transmission unit 38, and the transmission unit 38 transmits the instruction information to the smart glass 20 via the communication line 41 (S25).
In the smart glass 20, the receiving unit 29 receives the instruction information (S26), and sends the received instruction information to the instruction information acquiring unit 24.
Next, the image superimposing unit 25 generates an instruction image based on the instruction information sent from the instruction information acquiring unit 24, for example, an instruction image such as a circular frame surrounding the work portion of the target object, and sends it from the image acquiring unit 23. The instruction image thus generated is superimposed on the object image, and the superimposed object image and instruction image are sent to the image display unit 27 for display (S27).
Next, the instruction image moving unit 26 determines whether or not the object image has moved (S28). When the object image has moved, the instruction image is moved in response to the movement of the object image (S29). ), The process is terminated. On the other hand, if the object image does not move, the process ends without moving the instruction image.
If no instruction information is input in S23, the image superimposing unit 25 sends only the object image sent from the image acquiring unit 23 to the image display unit 27 and displays it (S30), and the process ends.

[Operation of Object Instruction System 50]
FIG. 17 is a flowchart showing the operation of the object instruction system 50. Hereinafter, the operation of the object instruction system 50 will be described with reference to FIG.
First, in the smart glass 60, the camera 62 images a work object that is in the field of view of the worker (wearer), outputs the photographed object image, and outputs the output object image to the image acquisition unit 63. Is acquired (S40).
Next, the object image acquired by the image acquisition unit 63 is sent to the transmission unit 68, and the transmission unit 68 sends the object image to the server 70 via the communication line 80 (S41).
Next, in the server 70, the receiving unit 79 receives the object image (S42), and the image acquiring unit 73 once acquires the object image.
Next, the object image is sent to the instruction information creating unit 74, where it is determined whether or not the operator has input instruction information for the object image (S43). If so, the instruction information, for example, information of the instruction image such as a circular frame surrounding the work part of the object is created (S44).
Next, the image superimposing unit 75 generates an instruction image based on the instruction information sent from the instruction information obtaining unit 74, for example, an instruction image such as a circular frame surrounding the work location of the object, and sends it from the image obtaining unit 73. The generated instruction image is superimposed on the object image (S45).
Next, the instruction image moving unit 76 determines whether or not the object image has moved (S46). When the object image has moved, the instruction image is moved in response to the movement of the object image (S47). ), The superimposed object image and instruction image are sent to the transmission unit 78.
On the other hand, when the object image does not move, the object image and the instruction image superimposed without moving the instruction image are sent to the transmission unit 78.
Next, the transmission unit 78 transmits the target object image and the instruction image superimposed via the communication line 81 to the smart glass 50 (S48).
In the smart glass 50, the reception unit 69 receives the superimposed object image and instruction image (S49), and sends the received superimposed object image and instruction image to the image acquisition unit 65.
Finally, the image acquisition unit 65 sends the superimposed object image and instruction image to the image display unit 67 for display (S50), and ends the process.
If no instruction information is input in S43, the image acquisition unit 65 sends only the object image sent from the receiving unit 69 to the image display unit 67 and displays it (S51), and the process ends.

  The object instruction system, the object instruction method, and the program superimpose an instruction image that gives an instruction on work on the object image in the captured image obtained by photographing the object with a camera or the like. In a system that displays and gives an instruction for work or the like, even if the object image in the photographed image moves, the instruction image can be accurately superimposed on the object image.

DESCRIPTION OF SYMBOLS 1 Object instruction | indication system 2 Image pick-up part 3 Image acquisition part 4 Instruction information creation part 5 Image superimposition part 6 Instruction image movement part 7 Image display part 10 Object instruction | indication system 20 Smart glass 21 Control part 22 Camera 23 Image acquisition part 24 Instruction information Acquisition unit 25 Image superposition unit 26 Instruction image moving unit 27 Image display unit 28 Transmission unit 29 Reception unit 30 Server 33 Image acquisition unit 34 Instruction information creation unit 38 Transmission unit 39 Reception unit 40, 41 Communication line 50 Object indication system 60 Smart Glass 61 Control unit 62 Camera 63, 65 Image acquisition unit 67 Image display unit 68 Transmission unit 69 Reception unit 70 Server 73 Image acquisition unit 74 Instruction information creation unit 75 Image superimposition unit 76 Instruction image movement unit 78 Transmission unit 79 Reception unit 80, 81 communication line 90 feature point extraction unit 91 feature point comparison unit 92 movement amount calculation unit 201 mounting unit 202 frame 203 lens unit TG object image SG instruction image CG superimposed image PA1~PA8 feature points PB1~PB8 feature point GM screen Q origin tg1 building image

Claims (11)

Photographing means for photographing the object and outputting the object image;
Image display means for displaying an object image output by the photographing means;
Instruction information creating means for creating instruction information for giving an instruction to the object image;
Image superimposing means for superimposing and displaying an instruction image based on the instruction information created by the instruction information creating means on the object image displayed on the image display means;
An object instruction system comprising: instruction image moving means for moving the instruction image in response to movement of the object image displayed on the image display means. Photographing means for photographing the object and outputting the object image;
Image display means for displaying an object image output by the photographing means;
An image transmitting means for transmitting the object image output by the photographing means; an image receiving means for receiving the object image transmitted by the image transmitting means;
Instruction information creating means for creating instruction information for giving an instruction to the object image received by the image receiving means;
Instruction information transmitting means for transmitting the instruction information created by the instruction information generating means;
Instruction information receiving means for receiving the instruction information transmitted by the instruction information transmitting means;
Image superimposing means for superimposing and displaying an instruction image based on the instruction information received by the instruction information receiving means on the object image displayed on the image display means;
An object instruction system comprising: instruction image moving means for moving the instruction image in response to movement of the object image displayed on the image display means. Photographing means for photographing the object and outputting the object image;
A first image transmitting means for transmitting the object image output by the photographing means; a first image receiving means for receiving the object image transmitted by the first image transmitting means;
Instruction information creating means for creating instruction information for giving an instruction to the object image received by the first image receiving means;
Image superimposing means for superimposing an instruction image based on the instruction information generated by the instruction information generating means on the object image received by the first image receiving means;
The instruction image moving means for moving the instruction image in response to the movement of the object image received by the first image receiving means, and the object image and the instruction image superimposed by the image superimposing means are transmitted. A second image transmission means;
Second image receiving means for receiving the object image and the instruction image transmitted by the second image transmitting means;
Image display means for displaying the object image and the instruction image received by the second image receiving means;
An object indication system characterized by comprising:   The instruction image moving unit compares the feature point extracting unit that extracts the feature point of the object image, and the feature point before the movement of the object image extracted by the feature point extracting unit and the feature point after the movement. 2. A feature point comparison means, and a movement amount calculation means for calculating a movement amount of the instruction image from feature points before and after movement of the object image compared with the feature point comparison means. The object indicating system according to any one of claims 3 to 4.   5. The object indication system according to claim 1, wherein the photographing unit and the image display unit are attached to a wearable computer attached to a user's face or head. . Shoot the object with the shooting means and output the object image,
Displaying the object image output by the photographing means on an image display means;
Creating instruction information for giving an instruction to the object image output by the photographing means;
An instruction image based on the instruction information is superimposed and displayed on the object image displayed on the image display means;
An object instruction method comprising the step of moving the instruction image in response to the movement of the object image displayed on the image display means. Shoot the object with the shooting means and output the object image,
Displaying the object image output by the photographing means on an image display means;
The object image output by the photographing unit is transmitted by an image transmission unit,
An image receiving means receives the object image transmitted by the image transmitting means,
Creating instruction information for giving an instruction to the object image received by the image receiving means;
The instruction information is transmitted by instruction information transmission means,
The instruction information receiving means receives the instruction information transmitted by the instruction information transmitting means,
An instruction image based on the instruction information received by the instruction information receiving unit is superimposed on the object image displayed on the image display unit and displayed.
An object instruction method comprising the step of moving the instruction image in response to the movement of the object image displayed on the image display means. Shoot the object with the shooting means and output the object image,
The object image output by the photographing unit is transmitted by a first image transmission unit,
The first image receiving means receives the object image transmitted by the first image transmitting means,
Creating instruction information for giving an instruction to the object image received by the first image receiving means;
An image superimposing unit superimposes an instruction image based on the instruction information on the object image received by the first image receiving unit,
Moving the instruction image in response to the movement of the object image received by the first image receiving means;
A second image transmitting unit transmits the object image and the instruction image superimposed by the image superimposing unit;
The second image receiving means receives the object image and the instruction image transmitted by the second image transmitting means,
An object instruction method comprising: displaying the object image and the instruction image received by the second image receiving unit on an image display unit.   The step of moving the instruction image includes extracting a feature point before the movement of the object image and a feature point after the movement, comparing the extracted feature points, and moving the instruction image based on the compared feature points. The object indicating method according to any one of claims 6 to 8, further comprising:   10. The object indicating method according to claim 6, wherein the photographing unit and the image display unit are attached to a wearable computer attached to a user's face or head. . A program for causing a computer to execute the object instruction method according to any one of claims 6 to 10.

Images