JP2015115724A - Video instruction method capable of superposing instruction picture on imaged moving picture, system, terminal, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a video instruction method and the like for performing an instruction on video imaged in a camera in one terminal, from another terminal by using a picture without using a marker or a registered object picture.SOLUTION: A first terminal sequentially transmits an imaged moving picture. Next, a second terminal has a user designate a second predetermined range for a received imaged moving picture and has the user write an instruction still picture. Next, the second terminal calculates a coordinate relative value between a first predetermined range including the instruction still picture written by the user and the second predetermined range, and transmits the coordinate relative value, the instruction still picture, and an imaged still picture obtained by trimming the imaged moving picture as a still picture in the second predetermined range to the first terminal. The first terminal detects a third predetermined range in which an imaged imaged moving picture and the imaged still picture match with each other, detects a fourth predetermined range from the third predetermined range by using the coordinate relative value, and superposes the instruction still picture on the fourth predetermined range to display it on the display.

Description

  The present invention relates to a technique of online video service between terminals.

  In recent years, with the spread of terminals such as smartphones and tablets, online video services via a network have been provided between geographically distant terminals (see, for example, Non-Patent Document 1). According to this service, for example, as an application for field work, an image captured by a terminal of a field worker can be transmitted to a remote work manager in real time. On the other hand, the work manager can recognize the situation at the work site by video and can give an instruction by voice.

  FIG. 1 is a system configuration diagram of an online video service.

  According to the system of FIG. 1, a terminal such as a mobile phone or a smartphone transmits captured video data to the other terminal in a streaming manner via a network. In recent years, HD (High-Definition) class images can be taken even by portable devices such as portable terminals.

  According to FIG. 1, the terminal 1 is carried by a field worker (instructed person), and the video is photographed by a mounted camera. On the other hand, the terminal 2 is possessed by a work manager (instructor). Then, the terminal 1 transmits the video data to the terminal 2 in real time via the access network and the Internet. The terminal 2 can make the work manager visually recognize the situation of the field worker by reproducing the received video data on the display.

  However, there are many cases where the work manager cannot clearly give instructions to the field worker by voice alone. For example, it is desirable for a work manager to be able to instruct the on-site worker with the position of various equipment and operation parts on the site.

  Conventionally, a field worker sends a still image captured by a terminal that he / she owns to the work manager's terminal, and on the other hand, a still image on which the work manager superimposes instruction information is transmitted to the field worker's terminal. There is a technique for transmitting (see, for example, Non-Patent Document 2). As a result, the work manager can instruct the field worker using a still image other than sound.

  Also, augmented reality (AR) technology can be applied to specify a predetermined position on the video (see, for example, Non-Patent Documents 3 and 4). The position of the AR marker is identified by recognizing the AR marker from the video. Further, it is possible to use a markerless type / object recognition method for detecting an object appearing in the video from a large number of object images without using an AR marker.

"Skype", [online], [searched November 13, 2013], Internet <URL: http://www.skype.com/en/> Structural Planning Laboratory, “Remote Guideware”, [online], [searched on November 13, 2013], Internet <http://www4.kke.co.jp/guideware/> Fujitsu, "work support technology using AR", [online], [searched on November 13, 2013], Internet <http://jp.fujitsu.com/solutions/industry/nextvalue/technology/tec_ar.html > NTT Giken, "Equipment management business system using AR", [online], [searched on November 13, 2013], Internet <http://www.ntt.co.jp/journal/1302/files/jn201302042 .pdf> “Affine transformation and projective transformation”, [online], [December 10, 2013 search], Internet <http://www58.atwiki.jp/dooooornob/pages/43.html> “Image Processing Solution”, [online], [Searched on December 10, 2013], Internet <http://imagingsolution.blog107.fc2.com/blog-entry-86.html> "Matrix calculation", [online], [December 10, 2013 search], Internet <http://www.cg.info.hiroshima-cu.ac.jp/~miyazaki/knowledge/tech07.html> “Camera calibration and 3D reconstruction”, [online], [Search on December 10, 2013], Internet <http://opencv.jp/opencv-2svn/cpp/camera_calibration_and_3d_reconstruction.html> "3D Geometric Analysis", [online], [December 10, 2013 search], Internet <http://www.ieice-hbkb.org/files/02/02gun_02hen_03.pdf>

  However, according to the technique described in Non-Patent Document 2, it is necessary to fix the camera mounted on the terminal of the field worker. If the shooting position moves, the position may be different from that of the still image sent from the work manager, and the site worker may not be able to recognize the specified position for the dense equipment or operation part. .

  According to the techniques described in Non-Patent Documents 3 and 4, an AR marker on which a special pattern is printed is required to specify the position on the video on which the instruction image is superimposed. Pasting the AR marker in advance on the device or the operation part is extremely time-consuming.

  Further, according to the technique of the markerless type / object recognition method, it is necessary to register a large number of object images in advance. Of course, in the case where the object image and the object image are similar in shape, the wrong object image may be associated.

  Therefore, the present invention provides a video instruction method, system, terminal, and program capable of instructing an image captured from the camera of one terminal imagewise from the other terminal without using a marker or a registered object image. The purpose is to provide.

According to the present invention, in a video instruction method in a system in which a first terminal having a display and a camera and a second terminal having a display are connected via a network,
A first step in which a first terminal sequentially transmits a moving image captured by a camera to a second terminal;
The second terminal displays the received captured moving image on the display, causes the user to specify a second predetermined range for the captured moving image, and causes the user to write an instruction still image for the captured moving image. Steps,
The second terminal calculates a first coordinate relative value between the first predetermined range including the instruction still image written by the user and the second predetermined range, and the first coordinate relative value A third step of transmitting the instruction still image and the captured still image obtained by trimming the captured moving image as a still image in the second predetermined range to the first terminal;
The first terminal detects a third predetermined range in which the captured moving image captured by the camera matches the captured still image (second predetermined range), and the second predetermined range and the third predetermined range are detected. A second coordinate relative value is calculated, and the designated still image is in the fourth predetermined range reflecting the first coordinate relative value and the second coordinate relative value in the captured still image (second predetermined range). And a fourth step of displaying on the display in a superimposed manner.

According to another embodiment of the video instruction method of the present invention,
The first coordinate relative value is a translation matrix, a combination of a translation matrix and a scaling matrix, a Euclidean transformation matrix or an affine transformation matrix,
The second coordinate relative value is preferably a projective transformation matrix or an attitude transformation matrix.

According to another embodiment of the video instruction method of the present invention,
Regarding the first step, it is also preferable that the first terminal transmits to the second terminal only a frame obtained by thinning the captured moving image by a predetermined time width.

According to another embodiment of the video instruction method of the present invention,
Regarding the first step, it is also preferable that the captured moving image transmits only an I (Intra-picture) frame, which is a reference for the motion compensation interframe prediction method, to the second terminal.

According to another embodiment of the video instruction method of the present invention,
Regarding the first step, it is also preferable that the first terminal sets the data rate of the I frame to a relatively high rate that is equal to or lower than the data rate of one GOP (Group Of Pictures).

According to another embodiment of the video instruction method of the present invention,
The captured still image is a feature amount image for matching or a resolution-compressed image for low data amount,
The instruction still image is also preferably a resolution-compressed image for a low data amount.

According to another embodiment of the video instruction method of the present invention,
The display mounted on the second terminal is a touch panel display,
About a 2nd step, it is also preferable that a 2nd terminal makes an instruction | indication still image the image drawn with the finger | toe to the user on the touch panel display.

According to another embodiment of the video instruction method of the present invention,
The second terminal is further connected to a touch pen input device that allows the user to draw the captured moving image displayed on the display,
About a 2nd step, it is also preferable that a 2nd terminal makes an instruction | indication still image the image drawn by the user with the touch pen.

According to another embodiment of the video instruction method of the present invention,
Regarding the fourth step, it is also preferable that the first terminal applies an AR (Augmented Reality) markerless type object recognition method.

According to the present invention, in a video instruction system in which a first terminal having a display and a camera and a second terminal having a display are connected via a network,
The first terminal is
Shooting moving image transmitting means for sequentially transmitting a moving image captured by the camera to the second terminal;
A third predetermined range in which a captured moving image captured by the camera matches a captured still image (second predetermined range) received from the second terminal is detected, and the second predetermined range and the third predetermined range are detected. The second coordinate relative value between the range and the fourth predetermined range reflecting the first coordinate relative value and the second coordinate relative value in the captured still image (second predetermined range) is indicated. Video display control means for displaying a still image superimposed on the display,
The second terminal
An instruction still image input means for displaying the received captured moving image on a display, causing the user to specify a second predetermined range for the captured moving image, and causing the user to write an instruction still image for the captured moving image;
Calculating the first coordinate relative value between the first predetermined range including the instruction still image written by the user and the second predetermined range; the first coordinate relative value; the instruction still image; It has an instruction still image transmission means for transmitting to a first terminal a captured still image obtained by trimming a captured moving image as a still image within a second predetermined range.

According to the present invention, in a terminal equipped with a display and a camera,
Shooting moving image transmission means for sequentially transmitting a moving image captured by the camera to the counterpart terminal;
Instructed still image input means for displaying a captured moving image received from the counterpart terminal on a display, causing the user to specify a second predetermined range for the captured moving image, and for allowing the user to write an instruction still image for the captured moving image. ,
Calculating the first coordinate relative value between the first predetermined range including the instruction still image written by the user and the second predetermined range; the first coordinate relative value; the instruction still image; An instruction still image transmitting means for transmitting a captured still image obtained by trimming a captured moving image as a still image in a second predetermined range to a counterpart terminal;
A third predetermined range in which a captured moving image captured by the camera matches a captured still image (second predetermined range) received from the counterpart terminal is detected, and the second predetermined range and the third predetermined range are detected. A second coordinate relative value is calculated, and the designated still image is in the fourth predetermined range reflecting the first coordinate relative value and the second coordinate relative value in the captured still image (second predetermined range). And video display control means for superimposing and displaying on the display.

According to the present invention, in a program for causing a computer installed in a terminal equipped with a display and a camera to function,
Shooting moving image transmission means for sequentially transmitting a moving image captured by the camera to the counterpart terminal;
Instructed still image input means for displaying a captured moving image received from the counterpart terminal on a display, causing the user to specify a second predetermined range for the captured moving image, and for allowing the user to write an instruction still image for the captured moving image. ,
Calculating the first coordinate relative value between the first predetermined range including the instruction still image written by the user and the second predetermined range; the first coordinate relative value; the instruction still image; An instruction still image transmitting means for transmitting a captured still image obtained by trimming a captured moving image as a still image in a second predetermined range to a counterpart terminal;
A third predetermined range in which a captured moving image captured by the camera matches a captured still image (second predetermined range) received from the counterpart terminal is detected, and the second predetermined range and the third predetermined range are detected. A second coordinate relative value is calculated, and the designated still image is in the fourth predetermined range reflecting the first coordinate relative value and the second coordinate relative value in the captured still image (second predetermined range). The computer is caused to function as video display control means for superimposing and displaying on the display.

  According to the video instruction method, system, terminal, and program of the present invention, an image instruction is given from the other terminal to the video shown in the camera of one terminal without using a marker or a registered object image. Can do.

1 is a system configuration diagram of an online video service. FIG. It is a sequence diagram in the present invention. It is explanatory drawing showing the flame | frame of the picked-up moving image in this invention. It is the screen figure which displayed the image image | photographed by the 1st terminal on the display of the 2nd terminal. It is a screen figure in which the instructor has written the instruction into the second terminal. It is explanatory drawing showing an instruction | indication still image and a picked-up still image. It is a screen figure of the 1st terminal by which the instruction | indication still image was superimposed and displayed on the part of the picked-up still image. FIG. 8 is a screen diagram of the first terminal when the shooting position with respect to the shooting target is projected and moved with respect to FIG. 7. It is a functional block diagram of a 1st terminal and a 2nd terminal. FIG. 3 is a functional configuration diagram of a dual-purpose terminal equipped with both functions of a transmission side and a reception side.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  According to the present invention, regarding the application of the AR markerless type / object recognition method, a user arbitrarily sets a “captured still image” (second predetermined range) as a matching key.

  FIG. 2 is a sequence diagram in the present invention.

  According to FIG. 2, a terminal 1 (instructed terminal) having a display and a camera and a terminal 2 (indicating terminal) having at least a display are connected via a network. The display and camera may be preinstalled in the terminal or may be connected to the outside.

[First Step S1]
The terminal 1 sequentially transmits a moving image captured by the camera to the terminal 2. For example, the terminal 1 operated by a work site worker (instructed person) captures a work situation (object) as a moving image (video). Here, it is preferable to transmit only the frames that are thinned out by a predetermined time width as the “captured moving image”. In other words, by making the moving image “a flip image”, the instructor operating the terminal 2 can easily recognize the captured moving image.

  FIG. 3 is an explanatory diagram showing a frame of a captured moving image according to the present invention.

  According to FIG. 3A, for example, in the case of Motion JPEG, the captured moving image is obtained by JPEG compression of all the frames, and is simply thinned out with a predetermined time width.

  According to FIG. 3B, for example, in the case of a motion compensation inter-frame prediction method, a plurality of frames are configured in units of GOP (Group Of Pictures). A GOP is generally composed of one I (Intra-picture) frame, a plurality of P (Predictive-picture) frames, and a B (Bidirectionally-picture) frame. According to the present invention, only an I (Intra-picture) frame is extracted as a captured moving image. That is, only a frame in which the entire image is encoded is transmitted as a flip image.

  It is also preferable to set the data rate of the I frame to a relatively high rate that is equal to or lower than the data rate of one GOP. For example, the data rate of one I frame can be made the same as the GOP data rate. Thereby, it is possible to increase the resolution of one flip image in the captured moving image, and to easily recognize the captured moving image in detail for the instructor who operates the terminal 2.

[Second Step S2]
(S21) The terminal 2 displays the received captured moving image on the display, and instructs the user to specify a second predetermined range for the captured moving image.

  FIG. 4 is a screen diagram in which an image captured by the first terminal is displayed on the display of the second terminal.

  According to FIG. 4, on the terminal 2 possessed by the work manager (instructor), the scene situation photographed by the terminal 1 operated by the work site worker (instructed person) is displayed as a moving image (para image). Is done. In addition, a “write instruction” button is clearly shown in the upper right of the display of the terminal 2. The instructor can stop a single image as a target by pressing the “instruction writing” button while the captured moving image sequentially proceeds as a flip image.

  FIG. 5 is a screen diagram in which the instructor writes an instruction on the second terminal.

  According to FIG. 5, first, the user designates the “second predetermined range” with a finger on the touch panel display of the terminal 2. Here, not the keyboard but the part in which the smartphone is shown is designated as the second predetermined range. The “second predetermined range” is a range of “captured still image” for matching. Of course, the second predetermined range may be designated as a frame of the designated still image that the user intends to draw with a finger, or designates a part completely different from the designated still image to be drawn. There may be.

(S22) The terminal 2 causes the user to write an instruction still image for the captured moving image. Here, the user draws “← here” by inserting the key [R] portion of the keyboard.

  As another embodiment, the terminal 2 may be further connected to a touch pen input device that allows a user to draw a captured moving image displayed on a display. In this case, the terminal 2 can use the image drawn by the user with the touch pen as the designated still image.

[Third Step S3]
The terminal 2 transmits the following two still images and the first coordinate relative value to the terminal 1.
“Instructed still image”: Still image of a first predetermined range written by the user “Captured still image”: Still image obtained by trimming a captured moving image within a second predetermined range “First coordinate relative value”: First Relative coordinate value between the predetermined range and the second predetermined range

(S31) The terminal 2 determines a first predetermined range including the instruction still image written by the user. The “first predetermined range” is automatically set to a rectangular predetermined range so as to include the designated still image.

  As will be described later, the “photographed still image” is used as a “key image” for image matching. Therefore, the captured still image does not need to be the image itself, and may be a feature amount image for matching. The feature amount image is a feature amount calculated from a local region of the image, and is extracted from a local region such as an edge or a corner in the image, for example. Typically, for example, SIFT (Scale-Invariant Feature Transform) or SURF (Speeded Up Robust Features) is used. In addition, a binary feature amount that is excellent in calculation cost can also be used. Further, it may be a locally cut out image (patch) for performing matching by SSD (Sum of Squared Difference) or normalized cross correlation (NCC).

  Furthermore, the “photographed still image” and the “instructed still image” may be resolution-compressed images for a low data amount. These images do not need to be bitmap format images, and may be compressed images such as JPEG.

  FIG. 6 is an explanatory diagram showing an instruction still image and a captured still image.

FIG. 6 shows the relationship between the first predetermined range and the second predetermined range.
First predetermined range: a range including an instruction still image written by the user at the instruction side terminal. Second predetermined range: a range including a matching still image specified by the user at the instruction side terminal. Have the following conversion relationship.
Between the second predetermined range and the first predetermined range-> Euclidean transformation

(S32) The terminal 2 calculates a first coordinate relative value between the “first predetermined range” and the “second predetermined range” of the instruction still image (see the matrix Ha in FIG. 6). The first coordinate relative value may be, for example, a translation matrix, a combination of a translation matrix and an enlargement / reduction matrix, an Euclidean transformation matrix, or an affine transformation matrix (see, for example, Non-Patent Documents 5 to 7). These refer to a translation only or a combination of translation and a linear transformation (linear transformation such as scaling, shearing, rotation, etc.). For example, a rectangular range means a movement difference between the four vertex coordinates of the first predetermined range and the four vertex coordinates of the second predetermined range.

ｘ，ｙ：第２の所定範囲におけるｘ座標及びｙ座標
ｘ'，ｙ'：第１の所定範囲におけるｘ座標及びｙ座標
ｈ₁₁，ｈ₁₂，ｈ₂₁，ｈ₂₂要素：線形変換要素
ｈ₁₃，ｈ₂₃要素：平行移動要素 In the case of affine transformation, between the first predetermined range and the second predetermined range, the linear transformation is represented by multiplying the 2 × 2 matrix with respect to the coordinates, and the translation is represented by the addition of a two-dimensional vector. .

x, y: x-coordinate and y-coordinate in the second predetermined range x ′, y ′: x-coordinate and y-coordinate in the first predetermined range h ₁₁ , h ₁₂ , h ₂₁ , h ₂₂ element: linear transformation element h ₁₃ , H ₂₃ element: translation element

ｘ，ｙ：第２の所定範囲におけるｘ座標及びｙ座標
ｘ'，ｙ'：第１の所定範囲におけるｘ座標及びｙ座標
θ：回転変換要素
tx，ty：平行移動要素 In the case of Euclidean transformation, it is expressed as follows.

x, y: x-coordinate and y-coordinate in the second predetermined range x ′, y ′: x-coordinate and y-coordinate in the first predetermined range θ: rotation conversion element
tx, ty: translation element

(S33) According to FIG. 5, a “command transmission” button is clearly shown in the upper right of the display of the terminal 2. After writing the instruction still image, the user presses the “instruction transmission” button. As a result, the terminal 2 obtains the first coordinate relative value (the parameter element of the matrix Ha in FIG. 6), the designated still image, and the captured still image obtained by trimming the captured moving image as a still image within the second predetermined range. And transmitted to the instructed terminal 1.

[Fourth Step S4]
FIG. 6 further shows the relationship between the third predetermined range and the fourth predetermined range.
Third predetermined range: a range matched with a captured still image at the instructed side terminal Fourth predetermined range: a range in which the instructed still image is to be superimposed at the instructed side terminal Here, the following conversion is performed between the predetermined ranges: Have a relationship.
Between the second predetermined range and the third predetermined range-> Projection conversion / attitude conversion Between the first predetermined range and the fourth predetermined range-> Projection conversion / attitude conversion

The terminal 1 displays the instruction still image superimposed on the captured moving image in the following steps.
(S41) The terminal 1 matches the “captured moving image” (captured preview video) captured by the camera with the “captured still image” (second predetermined range) received from the terminal 2 in the third predetermined range. Is detected. Here, it is also preferable to match the photographed still image with the photographed moving image while performing projective transformation or posture transformation. Since the captured moving image is constantly moving, the tracking process for matching with the captured still image is always executed.

(S42) A second coordinate relative value between the second predetermined range and the third predetermined range is calculated (see matrix Hp in FIG. 6). The second coordinate relative value is, for example, either a projective transformation matrix or a posture transformation matrix (see, for example, Non-Patent Documents 8 and 9).

ｘ，ｙ：撮影静止画像におけるｘ座標及びｙ座標
ｘ'，ｙ'：マッチング先のｘ座標及びｙ座標
ｈ₁₁〜ｈ₃₃：パラメータ “Projective transformation” is obtained by further adding a projection expressing the perspective of a plane to a parallel rotational movement. For example, it is represented by the following determinant.

x, y: x-coordinate and y-coordinate in the photographed still image x ′, y ′: x-coordinate and y-coordinate of matching destination h _{11 to} h ₃₃ : parameter

Ａ：カメラの内部パラメータ
予めカメラキャリブレーションによって求めておくことが望ましい。
しかしながら、実際の値とずれた場合でも最終的に姿勢行列と打ち消し合う
ために、重畳表示の位置には影響しない。そのため本発明の利用用途の場合、
一般的なカメラの値で代用することができる。
Ｒ（r11〜r33）：３次元空間内の回転を表すパラメータ
各パラメータはオイラー角等の表現によって３パラメータで表現可能
ｔ（t1〜t3）：３次元空間内の平行移動を表すパラメータ。
ｘ，ｙ：撮影静止画像におけるｘ座標及びｙ座標
ｘ'，ｙ'：マッチング先のｘ座標及びｙ座標 “Posture transformation” is a rigid body motion in a three-dimensional space and is expressed by a posture matrix of 6 degrees of freedom. Here, the “attitude matrix” belongs to the three-dimensional special Euclidean group SE (3) and is represented by a three-degree-of-freedom three-dimensional rotation matrix and a three-dimensional translation vector. For example, it is represented by the following determinant.

A: Camera internal parameters
It is desirable to obtain in advance by camera calibration.
However, even if it deviates from the actual value, it will eventually cancel out with the attitude matrix
Therefore, the superimposed display position is not affected. Therefore, in the use application of the present invention,
A common camera value can be used instead.
R (r11 to r33): parameter representing rotation in the three-dimensional space
Each parameter can be expressed by three parameters by expressing Euler angles etc. t (t1 to t3): A parameter representing a parallel movement in a three-dimensional space.
x, y: x-coordinate and y-coordinate in the captured still image x ′, y ′: x-coordinate and y-coordinate of matching destination

(S43) The “designated still image” is superimposed on the fourth predetermined range reflecting the first coordinate relative value and the second coordinate relative value on the captured still image (second predetermined range) and displayed on the display. (Refer to Hp · Ha in FIG. 6). Specifically, an AR markerless type / object recognition method is applied.

FIG. 7 is a screen diagram of the first terminal when the shooting position with respect to the shooting target object has been translated and rotated.
FIG. 8 is a screen diagram of the first terminal when the shooting position with respect to the shooting target is moved in a projective manner in FIG.

  FIG. 9 is a functional configuration diagram of the first terminal and the second terminal.

A terminal 1 as an instructed terminal has a display 13 and a camera 14 as well as connected to a network. Further, the terminal 1 includes a captured moving image transmission unit 11 and a video display control unit 12. These functional components are realized by executing a program that causes a computer mounted on the terminal 1 to function.
The captured moving image transmission unit 11 sequentially transmits captured moving images from the camera 14 to the counterpart terminal 2 (similar to S1 in FIG. 2).
The video display control unit 12 detects a third predetermined range in which the captured moving image captured by the camera 14 matches the captured still image (second predetermined range) received from the counterpart terminal 2. Next, a second coordinate relative value between the second predetermined range and the third predetermined range is calculated. Then, the instruction still image is superimposed on the fourth predetermined range reflecting the first coordinate relative value and the second coordinate relative value on the captured still image (second predetermined range) and displayed on the display 13 (FIG. 2 S41 to S43).

The terminal 2 as the instruction side terminal is connected to the network and has a touch panel display 23. The terminal 2 includes an instruction still image input unit 21 and an instruction still image transmission unit 22. These functional components are realized by executing a program that causes a computer mounted on the terminal 2 to function.
According to the second embodiment, the instruction still image input unit 21 allows the user to specify a second predetermined range for the captured moving image and causes the user to write the instruction still image for the captured moving image (FIG. 2). Of S21 and S22 in FIG.
According to the second embodiment, the instruction still image transmission unit 22 calculates the first coordinate relative value between the first predetermined range including the instruction still image written by the user and the second predetermined range. calculate. Then, the first coordinate relative value, the instruction still image, and the captured still image obtained by trimming the captured moving image as a still image within the second predetermined range are transmitted to the terminal 1 (similar to S31 to S33 in FIG. 2). ).

  FIG. 10 is a functional configuration diagram of a dual-purpose terminal equipped with both functions on the transmission side and reception side.

  According to FIG. 10, each functional component in the dual-purpose terminal 3 is exactly the same as the functional component in the instructed terminal 1 and the instructing terminal 2 in FIG. 9. Further, these functional components are realized by executing a program that causes a computer mounted on the terminal 3 to function.

  As described above in detail, according to the video instruction method, system, terminal, and program of the present invention, without using a marker or a registered object image, a video captured by the camera of one terminal is Image instructions can be given from the terminal.

  Various changes, modifications, and omissions of the above-described various embodiments of the present invention can be easily made by those skilled in the art. The above description is merely an example, and is not intended to be restrictive. The invention is limited only as defined in the following claims and the equivalents thereto.

DESCRIPTION OF SYMBOLS 1 Commanded side terminal 11 Shooting moving image transmission part 12 Image | video display control part 13 Display 14 Camera 2 Instruction side terminal 21 Instruction still image input part 22 Instruction still image transmission part 23 Touch panel display 3 Dual-use terminal

Claims (12)

In a video instruction method in a system in which a first terminal having a display and a camera and a second terminal having a display are connected via a network,
A first step in which a first terminal sequentially transmits a moving image captured by the camera to a second terminal;
The second terminal displays the received captured moving image on the display, causes the user to specify a second predetermined range for the captured moving image, and causes the user to write an instruction still image for the captured moving image. Two steps,
The second terminal calculates a first coordinate relative value between the first predetermined range including the instruction still image written by the user and the second predetermined range, and the first coordinate relative value A third step of transmitting the instruction still image and a captured still image obtained by trimming the captured moving image as a still image within a second predetermined range to the first terminal;
The first terminal detects a third predetermined range in which the captured moving image captured by the camera matches the captured still image (second predetermined range), and the second predetermined range and the third predetermined range are detected. Calculating a second relative coordinate value between the range and a fourth predetermined range reflecting the first coordinate relative value and the second coordinate relative value in the captured still image (second predetermined range); And a fourth step of displaying the instruction still image on the display in a superimposed manner. The first coordinate relative value is a translation matrix, a combination of a translation matrix and a scaling matrix, a Euclidean transformation matrix or an affine transformation matrix,
The video instruction method according to claim 1, wherein the second coordinate relative value is a projective transformation matrix or a posture transformation matrix.   3. The video instruction according to claim 1, wherein the first terminal transmits to the second terminal only a frame obtained by thinning the captured moving image by a predetermined time width. Method.   4. The first step according to claim 3, wherein the captured moving image transmits only an I (Intra-picture) frame, which is a reference for a motion compensation inter-frame prediction method, to the second terminal. 5. Video instruction method.   5. The first terminal sets the data rate of the I frame to a relatively high rate that is equal to or lower than the data rate of one GOP (Group Of Pictures) with respect to the first step. The video instruction method described in 1. The photographed still image is a feature amount image for the matching or a resolution compressed image for a low data amount,
The video instruction method according to claim 1, wherein the instruction still image is a resolution-compressed image for a low data amount. The display mounted on the second terminal is a touch panel display,
The video according to any one of claims 1 to 6, wherein in the second step, the second terminal uses the image drawn by the finger of the user on the touch panel display as an instruction still image. Instruction method. The second terminal is further connected to a touch pen input device that allows the user to draw the captured moving image displayed on the display,
7. The video instruction method according to claim 1, wherein the second terminal uses the image drawn by the user with the touch pen as an instruction still image. 8.   9. The fourth step according to claim 1, wherein the first terminal applies an AR (Augmented Reality) markerless type / object recognition method. 10. Video instruction method. In a video instruction system in which a first terminal having a display and a camera and a second terminal having a display are connected via a network,
The first terminal is
Shooting moving image transmitting means for sequentially transmitting a moving image captured by the camera to the second terminal;
A third predetermined range in which a captured moving image captured by the camera matches a captured still image (second predetermined range) received from the second terminal is detected, and the second predetermined range and the third predetermined range are detected. A second coordinate relative value between the predetermined range is calculated, and a fourth predetermined range reflecting the first coordinate relative value and the second coordinate relative value in the captured still image (second predetermined range) is calculated. Video display control means for displaying the instruction still image on the display in a superimposed manner,
The second terminal
Instructed still image input means for displaying the received captured moving image on the display, causing the user to specify a second predetermined range for the captured moving image, and for allowing the user to write an instruction still image for the captured moving image;
The first coordinate relative value between the first predetermined range including the designated still image written by the user and the second predetermined range is calculated, and the first coordinate relative value, the designated still image, A video instruction system comprising: an instruction still image transmission unit that transmits a captured still image obtained by trimming the captured moving image as a still image within a second predetermined range to the first terminal. In a terminal equipped with a display and a camera,
Shooting moving image transmitting means for sequentially transmitting a moving image captured by the camera to the counterpart terminal;
Instructed still image input means for displaying a captured moving image received from the counterpart terminal on the display, causing the user to specify a second predetermined range for the captured moving image, and for the user to write an instruction still image for the captured moving image. When,
The first coordinate relative value between the first predetermined range including the designated still image written by the user and the second predetermined range is calculated, and the first coordinate relative value, the designated still image, An instruction still image transmitting means for transmitting a captured still image obtained by trimming the captured moving image as a still image within a second predetermined range to a counterpart terminal;
A third predetermined range in which a captured moving image captured by the camera matches a captured still image (second predetermined range) received from the counterpart terminal is detected, and the second predetermined range and the third predetermined range are detected. The second coordinate relative value between the first coordinate relative value and the second coordinate relative value is reflected in the captured still image (second predetermined range). A terminal comprising: video display control means for displaying an instruction still image superimposed on the display. In a program for causing a computer mounted on a terminal equipped with a display and a camera to function,
Shooting moving image transmitting means for sequentially transmitting a moving image captured by the camera to the counterpart terminal;
Instructed still image input means for displaying a captured moving image received from the counterpart terminal on the display, causing the user to specify a second predetermined range for the captured moving image, and for the user to write an instruction still image for the captured moving image. When,
The first coordinate relative value between the first predetermined range including the designated still image written by the user and the second predetermined range is calculated, and the first coordinate relative value, the designated still image, An instruction still image transmitting means for transmitting a captured still image obtained by trimming the captured moving image as a still image within a second predetermined range to a counterpart terminal;
A third predetermined range in which a captured moving image captured by the camera matches a captured still image (second predetermined range) received from the counterpart terminal is detected, and the second predetermined range and the third predetermined range are detected. The second coordinate relative value between the first coordinate relative value and the second coordinate relative value is reflected in the captured still image (second predetermined range). A program for a terminal, which causes a computer to function as video display control means for displaying an instruction still image superimposed on the display.

Images