JP6146869B2 - Video instruction display method, system, terminal, and program for superimposing instruction image on photographing moving image synchronously - Google Patents

Description

  The present invention relates to a technique of online video service between terminals.

  In recent years, with the spread of terminals such as smartphones and tablets, online video services via a network have been provided between geographically distant terminals (see, for example, Non-Patent Document 1). According to this service, for example, as an application for field work, an image captured by a terminal of a field worker can be transmitted to a remote work manager in real time. On the other hand, the work manager can recognize the situation at the work site by video and can give an instruction by voice.

  FIG. 1 is a system configuration diagram of an online video service.

  According to the system of FIG. 1, a terminal such as a mobile phone or a smartphone transmits captured video data to the other terminal in a streaming manner via a network. In recent years, HD (High-Definition) class images can be taken even by portable devices such as portable terminals.

  According to FIG. 1, the terminal 1 is carried by a field worker (instructed person), and the video is photographed by a mounted camera. On the other hand, the terminal 2 is possessed by a work manager (instructor). Then, the terminal 1 transmits the video data to the terminal 2 in real time via the access network and the Internet. The terminal 2 can make the work manager visually recognize the situation of the field worker by reproducing the received video data on the display.

  However, there are many cases where the work manager cannot clearly give instructions to the field worker by voice alone. For example, it is desirable for a work manager to be able to instruct on-site workers with images of the positions of various devices and operation parts on the site.

  Conventionally, a field worker sends a still image captured by a terminal that he / she owns to a work manager's terminal, and a work manager superimposes a still image on which instruction information is superimposed to the field worker's terminal. There is a technique for transmitting (see, for example, Non-Patent Document 2). As a result, the work manager can instruct the field worker using a still image other than sound.

  Also, augmented reality (AR) technology can be applied to specify a predetermined position on the video (see, for example, Non-Patent Documents 3 and 4). The position of the AR marker is identified by recognizing the AR marker from the video. Further, it is possible to use a markerless type / object recognition method for detecting an object appearing in the video from a large number of object images without using an AR marker. However, according to the technique of the markerless type / object recognition method, it is necessary to register object images in advance. Here, when the shape of the object in the video and the object image are similar, an incorrect object image may be associated.

  By using AR technology, it is possible to intuitively understand the operation target location, etc. Compared to the case where the manual or the help text of the target device is referred to or the case where the remote operator receives only voice support, It is expected to solve the problem smoothly.

  Furthermore, as another conventional technique using a marker, there is a technique in which a two-dimensional marker is used to superimpose and display an instruction image acquired from a help server on a video of a supported device photographed by a user (see, for example, Patent Document 1). ). In addition, there is a technique of superimposing and displaying an instruction image input by a remote operator on a video of a support target device photographed by an operator using a two-dimensional marker (see, for example, Patent Document 2). Further, a technology for recognizing a target device by comparing a video image of a supported device photographed by an operator in advance with the image of the supported device registered as a marker and displaying an instruction image input by a remote operator in a superimposed manner (For example, refer to Patent Document 3).

JP 2010-219879 A JP 2013-115578 A JP 2011-034315 A

“Skype”, [online], [March 11, 2014 search], Internet <URL: http://www.skype.com/en/> Structural Planning Laboratory, “Remote Guideware”, [online], [March 11, 2014 search], Internet <http://www4.kke.co.jp/guideware/> Fujitsu, “Work Support Technology Using AR”, [online], [Search on March 11, 2014], Internet <http://jp.fujitsu.com/solutions/industry/nextvalue/technology/tec_ar.html > NTT Giken, "Equipment management business system using AR", [online], [Search on March 11, 2014], Internet <http://www.ntt.co.jp/journal/1302/files/jn201302042 .pdf> "Image processing solution", [online], [March 11, 2014 search], Internet <http://imagingsolution.blog107.fc2.com/blog-entry-86.html> "Matrix calculation", [online], [March 11, 2014 search], Internet <http://www.cg.info.hiroshima-cu.ac.jp/~miyazaki/knowledge/tech07.html> “Camera calibration and 3D reconstruction”, [online], [March 11, 2014 search], Internet <http://opencv.jp/opencv-2svn/cpp/camera_calibration_and_3d_reconstruction.html> “3D geometric analysis”, [online], [March 11, 2014 search], Internet <http://www.ieice-hbkb.org/files/02/02gun_02hen_03.pdf> "AutoStitch: a new dimension in automatic image stitching", [online], [March 11, 2014 search], Internet <http://www.cs.bath.ac.uk/brown/autostitch/autostitch.html > “The World of Computer Vision-Mirai Now”, [online], [Search March 11, 2014], Internet <http://news.mynavi.jp/series/computer_vision/027/>

  According to the technique described in Patent Document 3, when the difference in the viewpoint between the target equipment and the reference marker (the image of the target equipment registered in advance) increases due to a change in viewpoint, the instruction image cannot be displayed or is erroneous. It will be displayed at the position.

  On the other hand, how should the inventors of this application set the marker image used as a reference | standard? Was a problem. According to the prior art, unless the marker image is reflected in the captured moving image, the position of the instruction image cannot be specified in that portion. In other words, the farther away from the marker image, the more difficult it is to display the instruction image in a superimposed manner.

  In addition, according to the technique described in Patent Document 1, there is a problem that only a predetermined type of instruction image recorded in advance in the help server can be provided, and the support range is limited. Further, according to the technique described in Patent Document 2 or 3, there is a problem that the operator cannot confirm the superimposed display of the instruction image, and issues an operation instruction due to an erroneous superimposed display. Even with the technique described in Patent Document 3, the failure of the superimposed display cannot be grasped from the operator side.

  On the other hand, the inventors of the present application have a problem that the display position calculated for the superimposed display of the instruction image on the site worker side is not shared with the operator side. I thought. According to the AR technology, it is necessary to detect a marker image from a captured moving image and display an instruction image in a superimposed manner at that position. In general, such image processing cannot be processed on the GPU (Graphic Processing Unit) side despite the heavy processing load on the arithmetic processor. Therefore, processing is performed on the CPU (Central Processing Unit) side.

  The captured moving image needs to be transmitted in real time as much as possible, and there is a problem that it takes a long time to perform the calculation process when transmitting the superimposed moving image with the instruction image. In addition, it is difficult to embed a sequence for superimposing an instruction image on a captured moving image in real time in a video transmission technique using an existing online service. Therefore, the captured moving image, the instruction image, and the display position are generally transmitted separately. As a result, the instruction-side terminal 2 receives and displays the captured moving image, and only transmits the instruction image and the display position unilaterally to the instructed-side terminal 1. For this reason, the operator of the instruction-side terminal 2 cannot visually recognize how the instruction image is displayed on the captured moving image on the display of the instructed-side terminal 1.

  Therefore, the present invention provides a video instruction display method, system, terminal, and program that enables an operator of an instruction side terminal to select an appropriate reference marker for an instruction image displayed on a display of an instructed terminal. The purpose is to provide.

According to the present invention, in a video instruction display method in a system in which a first terminal having a camera and a display and a second terminal having a display are connected via a network,
The first terminal and the second terminal store a reference marker image that will appear in the captured moving image,
The second terminal sequentially receives the captured moving images from the first terminal and displays the captured moving images on the display.
(Reference 1) When a reference marker image is detected in a captured moving image, the reference marker image is used, or
(Criteria 2) a first step for allowing a user to select whether to use all or part of a captured image as a reference marker image;
The second terminal transmits to the first terminal the instruction image to be superimposed and displayed on the captured moving image, the relative position of the instruction image with respect to the reference marker image, and the reference marker image in the case of reference 2. And the steps
The first terminal transmits a moving image captured by the camera to the second terminal, detects a reference marker image reflected in the captured moving image, and places an instruction image at a position corresponding to a relative position from the reference marker image. And a third step of superimposing and displaying on the display.

According to another embodiment of the video instruction display method of the present invention,
For the first step, the second terminal
(Criteria 3) A plurality of captured images continuously received from the first terminal within a predetermined time range may be generated as one image, and the user may further select whether to use all of the images as the reference marker image. preferable.

According to another embodiment of the video instruction display method of the present invention,
For the second step, the second terminal
A reference marker image (0th predetermined range) reflected in a captured moving image that matches a prestored reference marker image is detected;
It is also preferable to calculate the coordinate conversion between the reference marker image (0th predetermined range) and the instruction image (first predetermined range) as a relative position.

According to another embodiment of the video instruction display method of the present invention,
The relative position is also preferably a translation matrix, a combination of a translation matrix and an enlargement / reduction matrix, an Euclidean transformation matrix, an affine transformation matrix, a projective transformation matrix, or an attitude transformation matrix.

According to another embodiment of the video instruction display method of the present invention,
The display mounted on the second terminal is a touch panel display,
About a 2nd step, it is also preferable that a 2nd terminal makes an instruction | indication image the image drawn with the finger | toe to the user on the touch panel display.

According to another embodiment of the video instruction display method of the present invention,
The second terminal is further connected to a touch pen input device that allows the user to draw the captured moving image displayed on the display,
About a 2nd step, it is also preferable that a 2nd terminal makes the instruction | indication image the image drawn by the user with the touch pen.

According to another embodiment of the video instruction display method of the present invention,
A fourth step in which the first terminal calculates a display position of the instruction image on the screen of the captured moving image and transmits the display position to the second terminal;
The second terminal further includes a fifth step of superimposing the instruction image at a position corresponding to the display position of the captured moving image on the screen and displaying the instruction image on the display. The third step to the fifth step include It is also preferable that it is always repeated.

According to another embodiment of the video instruction display method of the present invention,
Regarding the fourth step, when the first terminal calculates the display position, the first terminal further transmits the frame ID (IDentifier) of the captured moving image to the second terminal,
Regarding the fifth step, it is also preferable that the second terminal superimposes and displays the instruction image on the frame of the captured moving image that matches the frame ID.

According to another embodiment of the video instruction display method of the present invention,
For the fourth step, the first terminal
A reference marker image (0th predetermined range) reflected in a captured moving image that matches a prestored reference marker image is detected;
The instruction image (second predetermined range) is superimposed on the position corresponding to the relative position from the reference marker image (0th predetermined range) and displayed on the display.
The coordinate position of the instruction image (second predetermined range) is also preferably set as the display position.

According to another embodiment of the video instruction display method of the present invention,
The system has a plurality of second terminals,
For the second step, one second terminal transmits the instruction image to the other second terminal in advance,
About a 4th step, it is also preferable that a 1st terminal broadcasts a picked-up moving image and a display position to several 2nd terminal.

According to another embodiment of the video instruction display method of the present invention,
Regarding the second step, each of the plurality of second terminals has transmitted the instruction image and the relative position to the first terminal in advance,
About a 3rd step, it is also preferable that a 1st terminal superimposes each instruction | indication image on the position corresponded to each relative position from a reference | standard marker image, and displays it on a display.

According to another embodiment of the video instruction display method of the present invention,
The second terminal transmits a superimposed display stop notification to the first terminal in response to a user operation,
It is also preferable that the first terminal stops the superimposed display of the instruction image in the third step when receiving the superimposed display stop notification from the second terminal.

According to the present invention, in a system in which a first terminal having a camera and a display and a second terminal having a display are connected via a network,
The first terminal is
Reference marker storage means for storing a reference marker image that will appear in the captured moving image;
Shooting moving image transmitting means for transmitting a moving image captured by the camera to the second terminal;
Instruction image receiving means for receiving an instruction image to be superimposed on the captured moving image and a relative position of the instruction image with respect to the reference marker image from the second terminal, and receiving the reference marker image in accordance with a user selection When,
Reference marker detection means for detecting a reference marker image reflected in a moving image captured by the camera;
Display control means for superimposing the instruction image on the display at a position corresponding to the relative position from the reference marker image,
The second terminal
Reference marker storage means for storing a reference marker image that will appear in the captured moving image;
(Reference 1) When a reference marker image is detected in a captured moving image, the user selects whether to use the reference marker image or (Reference 2) to use all or part of the captured image as a reference marker image A reference marker selection means;
An instruction image transmitting means for transmitting the instruction image to be superimposed on the captured moving image, the relative position of the instruction image with respect to the reference marker image, and the reference marker image in the case of reference 2 to the first terminal;
It has a photographing moving image receiving means for receiving a photographing moving image from the first terminal.

According to the present invention, in a terminal equipped with a display and a camera,
Reference marker storage means for storing a reference marker image that will appear in the captured moving image;
As the commandee function,
Shooting moving image transmission means for transmitting a moving image captured by the camera to the counterpart terminal;
Reference marker detection means for detecting a reference marker image reflected in a moving image captured by the camera;
An instruction image receiving means for receiving an instruction image to be superimposed on the captured moving image and a relative position of the instruction image with respect to the reference marker image, and receiving a reference marker image in accordance with a user selection;
Display control means for superimposing the instruction image on the display at a position corresponding to the relative position from the reference marker image,
As an instruction side function,
(Reference 1) When a reference marker image is detected in a captured moving image, the user selects whether to use the reference marker image or (Reference 2) to use all or part of the captured image as a reference marker image A reference marker selection means;
An instruction image transmitting means for transmitting an instruction image to be superimposed on the captured moving image, a relative position of the instruction image with respect to the reference marker image, and a reference marker image in the case of reference 2 to the counterpart terminal;
It has a photographing moving image receiving means for receiving a photographing moving image from a counterpart terminal.

According to the present invention, in a program for causing a computer installed in a terminal equipped with a display and a camera to function,
Reference marker storage means for storing a reference marker image that will appear in the captured moving image;
As the commandee function,
Shooting moving image transmission means for transmitting a moving image captured by the camera to the counterpart terminal;
Reference marker detection means for detecting a reference marker image reflected in a moving image captured by the camera;
An instruction image receiving means for receiving an instruction image to be superimposed on the captured moving image and a relative position of the instruction image with respect to the reference marker image, and receiving a reference marker image in accordance with a user selection;
Display control means for superimposing the instruction image on the display at a position corresponding to the relative position from the reference marker image,
As an instruction side function,
(Reference 1) When a reference marker image is detected in a captured moving image, the user selects whether to use the reference marker image or (Reference 2) to use all or part of the captured image as a reference marker image A reference marker selection means;
An instruction image transmitting means for transmitting an instruction image to be superimposed on the captured moving image, a relative position of the instruction image with respect to the reference marker image, and a reference marker image in the case of reference 2 to the counterpart terminal;
A computer is caused to function as a shooting moving image receiving means for receiving a shooting moving image from a counterpart terminal.

  According to the video instruction display method, system, terminal, and program of the present invention, the operator of the instructing terminal can select an appropriate reference marker for the instruction image displayed on the display of the instructed terminal. .

1 is a system configuration diagram of an online video service. FIG. It is a sequence diagram in the present invention. It is a screen figure showing selection of "existing marker". It is a screen figure showing selection of a "photographed image marker". It is a screen figure showing selection of "panoramic image marker". It is the 1st screen figure in which the instructor has written the instruction into the 2nd terminal. It is a 2nd screen figure in which the instructor has written the instruction into the 2nd terminal. It is explanatory drawing showing the relative position of the instruction | indication image in case a panoramic image is made into a reference | standard marker image. It is the 1st explanatory view showing the relative position between the standard marker image in the indication side terminal, and the indication image. It is the 1st explanatory view showing the relative position between the reference marker image in the instructed side terminal, and an indication image. It is a screen figure reflected on the 1st terminal of the pointed side. It is a screen figure of the 2nd terminal when the imaging position with respect to a to-be-photographed object carries out projective movement. It is a sequence diagram which consists of one to-be-designated terminal and a plurality of instruction side terminals. It is a functional block diagram of a 1st terminal and a 2nd terminal.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 2 is a sequence diagram in the present invention.

  According to FIG. 2, a terminal 1 (instructed terminal) having a display and a camera and a terminal 2 (indicating terminal) having at least a display are connected via a network. The display and camera may be preinstalled in the terminal or may be connected to the outside. The camera continues to acquire video like a Web camera, for example.

  The terminal 1 and the terminal 2 store in advance an existing marker image that will appear in the captured moving image as a “reference marker image”. The “existing marker image” is used as a “key image” for image matching, and is used as a reference position for displaying an “instruction image” described later at an appropriate position. For example, it may be a two-dimensional marker (for example, a QR (Quick Response) (registered trademark) code, see Patent Documents 1 and 2) attached to a photographing object, or a pre-registered object image ( For example, it may be a markerless object image (see Patent Document 3).

  Note that the “existing marker image” stored in advance need not be the image itself, but may be a feature amount image for matching. The feature amount image is a feature amount calculated from a local region of the image, and is extracted from a local region such as an edge or a corner in the image, for example. Typically, for example, SIFT (Scale-Invariant Feature Transform) or SURF (Speeded Up Robust Features) is used. In addition, a binary feature amount that is excellent in calculation cost can also be used. Further, it may be a locally cut out image (patch) for performing matching by SSD (Sum of Squared Difference) or normalized cross correlation (NCC).

[First Step S1]
(S11) The terminal 1 captures an object with a camera. For example, the terminal 1 operated by a field worker (instructed person) captures a work situation (object) as a moving image (video). Then, the terminal 1 sequentially transmits the captured moving image to the terminal 2 operated by the work manager (instructor). The terminal 2 displays the received captured moving image on the display.

(S12) The terminal 2 detects an existing marker image as a “reference marker image” that appears in the captured moving image received from the terminal 1. A QR code as an existing marker image is affixed to the object to be imaged, and the reference marker image is detected by applying an existing two-dimensional code detection technique. Of course, the reference marker image may be detected or may not be detected.

(S13) Here, the terminal 2 causes the user (operator) to select whether the reference marker image is any of the following. The user can select any one of the following three buttons.
[Existing marker]: Reference marker image registered in advance [Captured image marker]: All or part of the captured image as a reference marker image [Panorama image marker]: Within a predetermined time range from the terminal 1 on the instructed person side A plurality of captured images that have been continuously received are generated as one image, and all of the images are used as reference marker images. Here, the reference marker image is selected by the user of the terminal 2 serving as the instruction side. Yes. The instructor gives the right to select more smoothly the work support than the instructed person.

  FIG. 3 is a screen diagram showing selection of “existing marker”.

  According to FIG. 3, an existing marker image stored in advance as a reference marker image is detected from the captured moving image (image matching). In this case, the “existing marker” button can be tapped. On the other hand, when the existing marker image is not detected, the “existing marker” button is not displayed.

  FIG. 4 is a screen diagram showing selection of a “photographed image marker”.

  According to FIG. 4, an existing marker image stored in advance as a reference marker image is not detected from the captured moving image. In this case, the “existing marker” button is not displayed. According to FIG. 4, the “captured image marker” is tapped by the user, and all or part of the captured image is used as the reference marker image.

  A captured image marker (still image) cut out from the entire display of the terminal 2 is used as a planar “reference marker image”. Of course, the captured image marker may also be the above-described feature amount image.

  Since the photographed image marker is planar, it is preferable that the photographed image marker is photographed from the front of the subject as much as possible so that the subject and the background are photographed substantially planarly. However, the depth difference between the subject and the background is large, and it may be difficult to recognize the entire plane. That is, there is a case where occlusion occurs due to movement of the photographing viewpoint, and it is difficult to use the entire still image as the reference marker image.

  In this case, it is also preferable that the terminal 2 trims a partial area from the cut out still image and creates reference marker information only from the partial area. This partial area is preferably a subject area, but may be a background area as long as it is a part having substantially the same depth as the subject. On the other hand, when the depth of a part of the subject area is different, it is also preferable to exclude it from the partial area. That is, even if the captured image marker is a partial region, the captured image marker is less affected by the movement of the capturing viewpoint as it is generally planar and is captured almost directly from the camera.

  In addition, the partial area | region does not necessarily need to be a rectangular shape about a picked-up image marker, The convex hull area | region designated by the user freehand may be sufficient. It is also possible to specify that the captured image includes partial areas of a plurality of captured image markers.

  FIG. 5 is a screen diagram showing selection of “panoramic image marker”.

  According to FIG. 5, the “panoramic image marker” is tapped by the user, and a plurality of captured images continuously received within a predetermined time range from the terminal 1 are generated as one image, and all of the images are used as the reference marker. Used as an image. For example, the terminal 1 is moved for about 5 seconds so that the instructed person can perform a planar examination. A captured image captured in that time range is generated by the terminal 2 into a single panoramic image. A technique for generating a panoramic image is generally referred to as “Image Stitching”. According to this technique, it is possible to input a plurality of images with overlapping portions, calculate the camera movement amount of these images, and combine them on one coordinate system based on the movement amount. Specifically, there is an application “AutoStich” published by the University of British Columbia (for example, see Non-Patent Documents 9 and 10).

  According to FIG. 5, only a part of the subject is captured in each captured image, but the panoramic image of the entire subject is generated by moving the terminal 1. In particular, it is effective to generate a panoramic image of the entire subject when it is difficult for the instructed person to capture the entire subject (for example, the target device) at a wide angle. Of course, as with the captured image marker, a partial area can be trimmed by a user operation from the panoramic image, and the partial area can be used as a reference marker image.

[Second Step S2]
FIG. 6 is a first screen diagram in which the instructor writes an instruction on the second terminal.

(S21) According to FIG. 6, the “instruction writing” button is clearly shown in the upper right of the display of the terminal 2. The instructor operating the terminal 2 can stop a single image as a target by pressing the “instruction writing” button while the captured moving image is being displayed. The instructor can write an instruction to the stopped image.

  According to FIG. 6, the instructor can write the “instruction image” for the captured moving image with a finger on the touch panel display of the terminal 2. Here, the instructor draws “← here” by inserting the key [R] portion of the subject keyboard.

  As another embodiment, the terminal 2 may be further connected to a touch pen input device that allows a user to draw a captured moving image displayed on a display. In this case, the terminal 2 can set an image drawn by the user with the touch pen as an “instruction image”.

  The “reference marker image” and the “instruction image” described above may be resolution-compressed images for a low data amount. These images do not need to be bitmap format images, and may be compressed images such as JPEG.

  FIG. 7 is a second screen diagram in which the instructor writes an instruction on the second terminal.

  According to FIG. 7, the instructor operating the terminal 2 writes an instruction after stopping a single image by pressing the “instruction writing” button. Here, the instructor draws “← Soko” by inserting the upper left portion of the display of the subject.

(S22) The terminal 2 calculates the “relative position” of the instruction image with respect to the reference marker image.

  FIG. 8 is an explanatory diagram showing the relative position of the instruction image when the panoramic image is the reference marker image. According to FIG. 8, since the panoramic image is wider than the photographed image, the relative position with respect to the reference marker image can be specified even for an instruction image that is not shown as a subject.

  FIG. 9 is a first explanatory diagram illustrating a relative position between the reference marker image and the instruction image in the instruction-side terminal.

  A reference marker image (0th predetermined range) reflected in the captured moving image on the display of the instruction side terminal 2 is detected. Here, the reference marker image is matched with the captured moving image while projecting / posture conversion. Since the captured moving image is constantly moving, the tracking process for matching with the reference marker image is always executed.

According to FIG. 9, the terminal 2 calculates the coordinate transformation between the reference marker image (0th predetermined range) and the instruction image (first predetermined range) as “relative position” (matrix in FIG. 9). Hp).
0th predetermined range: range of reference marker image 1st predetermined range: range including an instruction image written by the user on the instructing terminal Here, “relative position” is a translation matrix, a translation matrix, and an enlargement / reduction matrix Of Euclidean transformation matrix, affine transformation matrix, projective transformation matrix, or attitude transformation matrix (for example, see Non-Patent Documents 5 to 8). For example, in the case of a rectangular range, this means a movement difference between the four vertex coordinates of the zero predetermined range and the four vertex coordinates of the first predetermined range. The positional relationship can be converted only by multiplication of these matrices.

ｘ，ｙ：第１の所定範囲におけるｘ座標及びｙ座標
ｘ'，ｙ'：第０の所定範囲におけるｘ座標及びｙ座標
ｈ₁₁，ｈ₁₂，ｈ₂₁，ｈ₂₂要素：線形変換要素
ｈ₁₃，ｈ₂₃要素：平行移動要素 In the case of the affine transformation, between the 0th predetermined range and the first predetermined range, the linear transformation is represented by multiplying the 2 × 2 matrix with respect to the coordinates, and the translation is represented by the addition of a two-dimensional vector. .

x, y: x-coordinate and y-coordinate in the first predetermined range x ′, y ′: x-coordinate and y-coordinate in the zeroth predetermined range h ₁₁ , h ₁₂ , h ₂₁ , h ₂₂ elements: linear transformation element h ₁₃ , H ₂₃ element: translation element

ｘ，ｙ：第１の所定範囲におけるｘ座標及びｙ座標
ｘ'，ｙ'：第０の所定範囲におけるｘ座標及びｙ座標
θ：回転変換要素
tx，ty：平行移動要素 In the case of Euclidean transformation, it is expressed as follows.

x, y: x-coordinate and y-coordinate in the first predetermined range x ′, y ′: x-coordinate and y-coordinate in the zeroth predetermined range θ: rotation conversion element
tx, ty: translation element

ｘ，ｙ：基準マーカ画像におけるｘ座標及びｙ座標
ｘ'，ｙ'：マッチング先のｘ座標及びｙ座標
ｈ₁₁〜ｈ₃₃：パラメータ Furthermore, in the case of “projective transformation”, a projection expressing the perspective of a plane can be further added to the parallel rotational movement, and is expressed as follows, for example.

x, y: x-coordinate and y-coordinate in reference marker image x ′, y ′: x-coordinate and y-coordinate of matching destination h _{11 to} h ₃₃ : parameter

Ａ：カメラの内部パラメータ
予めカメラキャリブレーションによって求めておくことが望ましい。
しかしながら、実際の値とずれた場合でも最終的に姿勢行列と打ち消し合う
ために、重畳表示の位置には影響しない。そのため、本発明の利用用途の場合、
一般的なカメラの値で代用することができる。
Ｒ（r11〜r33）：３次元空間内の回転を表すパラメータ
各パラメータはオイラー角等の表現によって３パラメータで表現可能
ｔ（t1〜t3）：３次元空間内の平行移動を表すパラメータ。
ｘ，ｙ：基準マーカ画像におけるｘ座標及びｙ座標
ｘ'，ｙ'：マッチング先のｘ座標及びｙ座標
姿勢変換によれば、カメラの視点（撮影角度や撮影距離の変化を含む）が大きく変わった場合でも、その位置に正確に指示画像を表示することができる。 Furthermore, in the case of “posture conversion”, it is a rigid body motion in a three-dimensional space, and can be expressed by a posture matrix of 6 degrees of freedom. Here, the “posture matrix” belongs to the three-dimensional special Euclidean group SE (reference 3), and is represented by a three-degree-of-freedom three-dimensional rotation matrix and a three-dimensional translation vector. For example, it is represented by the following determinant.

A: Camera internal parameters
It is desirable to obtain in advance by camera calibration.
However, even if it deviates from the actual value, it will eventually cancel out with the attitude matrix
Therefore, the superimposed display position is not affected. Therefore, in the use application of the present invention,
A common camera value can be used instead.
R (r11 to r33): parameter representing rotation in the three-dimensional space
Each parameter can be expressed by three parameters by expressing Euler angles etc. t (t1 to t3): A parameter representing a parallel movement in a three-dimensional space.
x, y: x-coordinate and y-coordinate in the reference marker image x ′, y ′: x-coordinate and y-coordinate of matching destination According to the orientation conversion, the viewpoint of the camera (including changes in shooting angle and shooting distance) changes greatly. Even in the case, the instruction image can be accurately displayed at the position.

(S23) Then, the user operates “instruction transmission” on the terminal 2, for example (for example, tap of a transmission button). As a result, the terminal 2 displays the “instruction image” to be superimposed on the captured moving image and the “relative position” (parameter element of the matrix Hp in FIG. 9) of the instruction image with respect to the reference marker image. 1 in advance. In addition, “captured image marker” and “panoramic image marker” are also transmitted to the instructed terminal 1 in advance. In the case of “existing marker”, since the existing marker is stored in both the terminal 1 and the terminal 2, it is not necessary to transmit it.

[Third Step S3]
(S31) The instructee terminal 1 captures the object as a moving image by the camera. Then, the terminal 1 sequentially transmits the captured moving image to the terminal 2. The terminal 2 displays the received captured moving image on the display.

  FIG. 10 is a first explanatory diagram illustrating a relative position between the reference marker image and the instruction image in the instructed terminal.

(S32) The terminal 1 matches the reference marker image stored in advance for each frame of the captured moving image continuously input from the camera, and displays the reference marker image (the 0th predetermined range) displayed in the captured moving image. ) Is detected. This is the same process as S12 of the terminal 2. The reference marker image is matched with the captured moving image while projecting / posture changing. Since the captured moving image is constantly moving, the tracking process for matching with the reference marker image is always executed.

(S33) The terminal 1 then superimposes the “instruction image” on the second predetermined range reflecting the relative position (see the matrix Hp in FIG. 10) on the detected reference marker image (0th predetermined range). On the display.
0th predetermined range: range of the reference marker image reflected in the captured moving image at the commanded terminal 2nd predetermined range: a range including the command image reflected in the captured moving image at the commanded terminal The position also has a conversion relationship based on a translation matrix, a combination of a translation matrix and an enlargement / reduction matrix, an Euclidean transformation matrix, an affine transformation matrix, a projective transformation matrix, or an attitude transformation matrix (see, for example, Non-Patent Documents 5 to 9).

  FIG. 11 is a screen image displayed on the first terminal on the instructed side.

  According to FIG. 11, the “instruction image” written by the instructor of the terminal 2 is displayed superimposed on the captured moving image displayed on the terminal 1. Here, the part of the key [R] on the keyboard is inserted and “← here” is displayed.

  When the terminal 1 is a device such as a head mounted display (HMD), it is not necessary to display even a captured moving image on the display. In the case of such a see-through type HMD, it is only necessary to superimpose and display only the instruction image in the field of view.

[Fourth Step S4]
(S41) The terminal 1 calculates the “display position” of the instruction image with respect to the captured moving image screen. The “display position” may be a vertex coordinate (for example, four vertices in the case of a rectangular shape) with respect to the screen of the captured moving image or a parameter of a conversion matrix from a predetermined position (for example, a screen frame). Such a “display position” does not require the detection process of the reference marker image, and can directly determine the superimposed position of the instruction image.

  The display position calculation process can be executed at the camera frame rate (30 frames / sec for most cameras). On the other hand, the transmission process of a captured moving image can be transmitted only at a frame rate (for example, flame / sec) that is equal to or less than the frame rate of the camera due to encoding processing, communication band restrictions, and the like. In other words, a delay occurs due to the communication buffer and encoding / decoding processing for moving images. Although it cannot be expressed sufficiently according to FIG. 2, for one captured moving image, the terminal 1 transmits the “display position” (+ frame ID) to the terminal 2 and then transmits the captured moving image to the terminal 2. It will be. After receiving “display position + frame ID”, the terminal 2 can wait for a captured moving image of the frame and display the instruction image in a superimposed manner.

(S42) The terminal 1 sequentially transmits the “display position” to the terminal 2. Here, it is preferable that the terminal 1 further transmits the frame ID (IDentifier) of the captured moving image to the second terminal when the display position is calculated. The frame ID is an index for identifying a moving image frame in MPEG, for example. By further transmitting the frame ID, the superimposed display of the captured moving image and the instruction image can be finally completely synchronized at the second terminal.

[Fifth Step S5]
The terminal 2 superimposes its instruction image on a position corresponding to the display position of the captured moving image on the screen and displays it on the display.

  FIG. 12 is a screen diagram of the second terminal when the shooting position with respect to the shooting target has been projected and moved.

  As a result, the terminal 2 can directly determine the superimposed position of the instruction image without requiring the detection process of the reference marker image. Further, the terminal 2 can superimpose and display the instruction image on the frame of the captured moving image that matches the frame ID by receiving the frame ID to be superimposed together with the display position from the terminal 1.

  And the sequence of S3-S5 will always be repeated along the flow of the sequence of FIG. Thereby, the operator of the instruction side terminal 2 can visually recognize how the instruction image is superimposed on the captured moving image on the display of the instructed side terminal 1, and supports remote work by AR. be able to.

  As another embodiment, according to FIG. 12, a “stop superimposed display” button is displayed on the display of the terminal 2. When the sequence of S3 to S5 is always repeated, there may be a deviation in the position of the instruction image superimposed on the captured moving image when viewed from the operator of the instruction side terminal 2. In this case, the terminal 2 can transmit a superimposed display stop notification to the terminal 1 by touching the “stop superimposed display” button by the operator. On the other hand, when receiving the superimposed display stop notification from the terminal 2, the terminal 1 stops the superimposing display of the instruction image in S3 (third step). It is possible to prevent an erroneous superimposed display from being visually recognized by an operator operation. Thereafter, in this case, the terminal 2 can newly adjust the position of the superimposed display of the instruction image on the captured moving image by executing S1 (first step) again.

  FIG. 13 is a sequence diagram including one instructed terminal and a plurality of instructing terminals.

  According to FIG. 13, for example, different instruction images are created by a plurality of remote operators. In this case, the terminal 1 simultaneously superimposes and displays a plurality of instruction images received from the plurality of terminals 2. Of course, it is assumed that the terminal 1 and the plurality of terminals 2 recognize each other's destination address.

According to FIG. 13, the sequence numbers are skipped as appropriate, but it is assumed that undescribed sequence numbers are executed in the same manner as the processing of FIG.
(S11) The terminal 1 broadcasts a moving image captured by the camera to the plurality of terminals 21 and 22.
(S23) Regarding the terminal 21, it is assumed that an instruction image has been written by the operator. The terminal 21 transmits the instruction image and the relative position to the terminal 1 and also to the other terminal 22. The instruction image and the relative position are shared between the plurality of terminals 2 on the instruction side.
(S31) Thereafter, the terminal 1 broadcasts a moving image captured by the camera to the plurality of terminals 21 and 22.
(S32) The terminal 1 superimposes and displays the instruction image received from the terminal 21 on the moving image captured by the camera.
(S4) The terminal 1 broadcasts the display position of the instruction image on the captured moving image screen to the plurality of terminals 21 and 22.
(S5) The terminals 21 and 22 superimpose and display the instruction image at the display position on the screen of the captured moving image received from the terminal 1. Thereby, even the instruction side terminal 22 that does not create the instruction image can visually recognize the superimposed display of the captured moving image and the instruction image on the terminal 1.

Next, it is assumed that an instruction image is written on the terminal 22 by the operator.
(S 23) The terminal 22 transmits the instruction image and the relative position to the terminal 1 and also to the other terminal 21. The instruction image and the relative position are shared between the plurality of terminals 2 on the instruction side.
(S31) Thereafter, the terminal 1 broadcasts a moving image captured by the camera to the plurality of terminals 21 and 22.
(S32) The terminal 1 superimposes and displays the instruction image received from the terminal 22 on the moving image captured by the camera.
(S4) The terminal 1 broadcasts the display position of the instruction image on the captured moving image screen to the plurality of terminals 21 and 22.
(S5) The terminals 21 and 22 superimpose and display the instruction image at the display position on the screen of the captured moving image received from the terminal 1. Thereby, even the instruction-side terminal 21 that does not create the instruction image can visually recognize the superimposed display of the captured moving image and the instruction image on the terminal 1.

  As another embodiment, a system including a plurality of terminals 1 and one terminal 2 may be used. In this case, the terminal 2 needs to receive the captured moving images from the plurality of terminals 1 and may display them simultaneously or may be displayed by switching with a selection button based on a user operation. Of course, the present invention can also be applied to a system including a plurality of terminals 1 and a plurality of terminals 2.

  FIG. 14 is a functional configuration diagram of the first terminal and the second terminal.

  A terminal 1 as an instructed terminal has a display 17 and a camera 18 as well as connected to a network. In addition, the terminal 1 includes a reference marker storage unit 10, a captured moving image transmission unit 11, a reference marker detection unit 12, an instruction image reception unit 13, a display control unit 14, a display position calculation unit 15, and a display position. And a transmission unit 16. These functional components are realized by executing a program that causes a computer mounted on the terminal 1 to function.

The reference marker storage unit 10 stores a reference marker image that will appear in the captured moving image. This reference marker image is referred to by the reference marker detection unit 12.
The captured moving image transmission unit 11 transmits a captured moving image from the camera to the terminal 2 (similar to the processing of S11 and S31 in FIG. 2).
The reference marker detection unit 12 detects a reference marker image shown in a moving image captured by the camera (similar to the process of S32 in FIG. 2). The position range (0th predetermined range) of the detected reference marker image is output to the display control unit 14.
The instruction image receiving unit 13 receives in advance from the terminal 2 the “instruction image” to be superimposed on the captured moving image and the “relative position” of the instruction image with respect to the reference marker image (the processing of S23 in FIG. 2). The same). The received instruction image and relative position are output to the display control unit 14.
The display control unit 14 superimposes the instruction image on the position corresponding to the relative position from the reference marker image and displays it on the display (similar to the process of S33 in FIG. 2).
The display position calculation unit 15 calculates the display position of the instruction image on the captured moving image screen (similar to the process of S41 in FIG. 2). The calculated display position is output to the display position transmission unit 16.
The display position transmission unit 16 transmits the display position to the terminal 2 (similar to the process of S42 in FIG. 2).

  The terminal 2 as the instruction side terminal is connected to the network and has a touch panel display 27. The terminal 2 also includes a reference marker storage unit 20, a captured moving image reception unit 21, a reference marker selection unit 22, an instruction image transmission unit 23, a display control unit 24, and a display position reception unit 26. These functional components are realized by executing a program that causes a computer mounted on the terminal 2 to function.

The reference marker storage unit 20 stores a reference marker image that will appear in the captured moving image. This reference marker image is referred to by the reference marker selection unit 22.
The captured moving image receiving unit 21 receives a captured moving image from the terminal 1 (similar to the processing of S11 and S31 in FIG. 2).
The reference marker selection unit 22 detects a reference marker image shown in the captured moving image received from the terminal 1 (similar to the processing in S12 and S13 in FIG. 2).
The instruction image transmission unit 23 transmits in advance the instruction image to be superimposed on the captured moving image and the relative position of the instruction image with respect to the reference marker image to the terminal 1 (similar to the process of S23 in FIG. 2).
The display position receiving unit 26 receives the display position from the terminal 1 (similar to the process of S42 in FIG. 2).
The display control unit 24 superimposes the instruction image at a position corresponding to the display position on the screen of the captured moving image and displays it on the display (similar to the process of S5 in FIG. 2).

  Note that the pointing-side function and the pointed-side function in FIG. 14 can be combined into a dual-purpose terminal. These functional components are realized by executing a program that causes a computer mounted on one terminal to function.

  As described above in detail, according to the video instruction display method, system, terminal, and program of the present invention, the operator of the instruction-side terminal is appropriate for the instruction image displayed on the display of the instruction-side terminal. A simple reference marker can be selected. Further, the operator of the instruction side terminal can visually recognize how the instruction image is superimposed on the captured moving image on the display of the instruction side terminal. Smooth remote work is realized by making the operator of the instruction side terminal recognize the displacement of the position of the instruction image with respect to the captured moving image.

  Various changes, modifications, and omissions of the above-described various embodiments of the present invention can be easily made by those skilled in the art. The above description is merely an example, and is not intended to be restrictive. The invention is limited only as defined in the following claims and the equivalents thereto.

DESCRIPTION OF SYMBOLS 1 Instructed side terminal 10 Reference marker storage part 11 Shooting moving image transmission part 12 Reference marker detection part 13 Instruction image receiving part 14 Display control part 15 Display position calculation part 16 Display position transmission part 17 Display 18 Camera 2 Directed side terminal 20 Reference marker storage unit 21 Shooting moving image reception unit 22 Reference marker selection unit 23 Instruction image transmission unit 24 Display control unit 26 Display position reception unit 27 Touch panel display

Claims (15)

In a video instruction display method in a system in which a first terminal having a camera and a display and a second terminal having a display are connected via a network,
The first terminal and the second terminal store a reference marker image that will appear in the captured moving image,
The second terminal sequentially receives the moving image captured by the camera from the first terminal, displays the captured moving image on the display,
(Reference 1) When the reference marker image is detected in the captured moving image, the reference marker image is used, or
(Criteria 2) a first step for allowing the user to select whether to use all or part of the captured image as a reference marker image;
The second terminal transmits an instruction image to be superimposed on the captured moving image, a relative position of the instruction image with respect to the reference marker image, and a reference marker image in the case of reference 2 to the first terminal. A second step;
The first terminal transmits a moving image captured by the camera to the second terminal, detects the reference marker image reflected in the captured moving image, and at a position corresponding to the relative position from the reference marker image, And a third step of displaying the instruction image on a display in a superimposed manner. For the first step, the second terminal
(Criteria 3) Generating a plurality of captured images continuously received from the first terminal within a predetermined time range as a single image and allowing the user to further select whether to use all of the images as the reference marker image. The video instruction display method according to claim 1, wherein: For the second step, the second terminal
Detecting a reference marker image (0th predetermined range) reflected in the captured moving image that matches the reference marker image stored in advance;
The video according to claim 1 or 2, wherein coordinate conversion between the reference marker image (0th predetermined range) and the instruction image (first predetermined range) is calculated as the relative position. Instruction display method. The relative position is a translation matrix, a combination of a translation matrix and a scaling matrix, an Euclidean transformation matrix, an affine transformation matrix, a projective transformation matrix, or an attitude transformation matrix. The video instruction display method according to claim 1. The display mounted on the second terminal is a touch panel display,
5. The video instruction according to claim 1, wherein the second terminal uses, as the instruction image, an image drawn by a user's finger on the touch panel display. Display method. The second terminal is further connected to a touch pen input device that allows the user to draw the captured moving image displayed on the display,
5. The video instruction display method according to claim 1, wherein in the second step, the second terminal uses an image drawn by the user with the touch pen as an instruction image. 6. A fourth step in which the first terminal calculates a display position of the instruction image with respect to the screen of the captured moving image and transmits the display position to the second terminal;
The second terminal further includes a fifth step of superimposing the instruction image on a display at a position corresponding to the display position with respect to the screen of the captured moving image, and displaying from the third step to the fifth step. The video instruction display method according to any one of claims 1 to 6, characterized in that the above steps are always repeated. Regarding the fourth step, when the first terminal calculates the display position, the first terminal further transmits a frame ID (IDentifier) of the captured moving image to the second terminal,
8. The video instruction display method according to claim 7, wherein the second terminal displays the instruction image superimposed on a frame of the captured moving image that matches the frame ID. For the fourth step, the first terminal
A reference marker image (0th predetermined range) reflected in the captured moving image that matches a pre-stored reference marker image is detected,
The instruction image (second predetermined range) is superimposed on a position corresponding to the relative position from the reference marker image (0th predetermined range), and is displayed on the display.
The video instruction display method according to claim 7 or 8, wherein a coordinate position of the instruction image (second predetermined range) is set as the display position. The system has a plurality of second terminals,
For the second step, one second terminal transmits the instruction image to the other second terminal in advance,
10. The fourth step, wherein the first terminal transmits the captured moving image and the display position to a plurality of second terminals in a broadcast manner. The video instruction display method described in 1. For the second step, each of a plurality of second terminals has transmitted the instruction image and the relative position to the first terminal in advance,
11. The video according to claim 10, wherein, in the third step, the first terminal superimposes and displays each instruction image on a position corresponding to each relative position from the reference marker image. Instruction display method. The second terminal transmits a superimposed display stop notification to the first terminal in response to a user operation,
The first terminal stops the superimposed display of the instruction image in the third step when receiving the superimposed display cancellation notification from the second terminal. The video instruction display method described in 1. In a system in which a first terminal having a camera and a display and a second terminal having a display are connected via a network,
The first terminal is
Reference marker storage means for storing a reference marker image that will appear in the captured moving image;
Shooting moving image transmitting means for transmitting a moving image captured by the camera to the second terminal;
An instruction image for receiving an instruction image to be superimposed on the captured moving image and a relative position of the instruction image with respect to the reference marker image from a second terminal, and receiving a reference marker image according to a user's selection Receiving means;
Reference marker detection means for detecting the reference marker image reflected in the moving image captured by the camera;
Display control means for superimposing the instruction image on the display at a position corresponding to the relative position from the reference marker image,
The second terminal
Reference marker storage means for storing a reference marker image that will appear in the captured moving image;
(Reference 1) Whether the reference marker image is used when the reference marker image is detected in the captured moving image, or (Reference 2) whether all or part of the captured image is used as the reference marker image A reference marker selection means to be selected,
An instruction image transmission means for transmitting an instruction image to be superimposed on the captured moving image, a relative position of the instruction image with respect to the reference marker image, and a reference marker image in the case of reference 2 to the first terminal;
A system comprising: a captured moving image receiving unit configured to receive a captured moving image from a first terminal. In a terminal equipped with a display and a camera,
Reference marker storage means for storing a reference marker image that will appear in the captured moving image;
As the commandee function,
Shooting moving image transmission means for transmitting a moving image captured by the camera to a counterpart terminal;
Reference marker detection means for detecting the reference marker image reflected in the moving image captured by the camera;
Instruction image receiving means for receiving an instruction image to be superimposed on the captured moving image and a relative position of the instruction image with respect to the reference marker image from a counterpart terminal and receiving a reference marker image according to a user's selection When,
Display control means for superimposing the instruction image on the display at a position corresponding to the relative position from the reference marker image,
As an instruction side function,
(Reference 1) Whether the reference marker image is used when the reference marker image is detected in the captured moving image, or (Reference 2) whether all or part of the captured image is used as the reference marker image A reference marker selection means to be selected,
An instruction image transmitting means for transmitting an instruction image to be superimposed on the captured moving image, a relative position of the instruction image with respect to the reference marker image, and a reference marker image in the case of reference 2 to a counterpart terminal;
A terminal comprising: a captured moving image receiving means for receiving a captured moving image from a counterpart terminal. In a program for causing a computer mounted on a terminal equipped with a display and a camera to function,
Reference marker storage means for storing a reference marker image that will appear in the captured moving image;
As the commandee function,
Shooting moving image transmission means for transmitting a moving image captured by the camera to a counterpart terminal;
Reference marker detection means for detecting the reference marker image reflected in the moving image captured by the camera;
Instruction image receiving means for receiving an instruction image to be superimposed on the captured moving image and a relative position of the instruction image with respect to the reference marker image from a counterpart terminal and receiving a reference marker image according to a user's selection When,
Display control means for superimposing the instruction image on the display at a position corresponding to the relative position from the reference marker image,
As an instruction side function,
(Reference 1) Whether the reference marker image is used when the reference marker image is detected in the captured moving image, or (Reference 2) whether all or part of the captured image is used as the reference marker image A reference marker selection means to be selected,
An instruction image transmitting means for transmitting an instruction image to be superimposed on the captured moving image, a relative position of the instruction image with respect to the reference marker image, and a reference marker image in the case of reference 2 to a counterpart terminal;
A program for a terminal, which causes a computer to function as a shooting moving image receiving means for receiving a shooting moving image from a counterpart terminal.

Images