JP2012156820A - Video communication system, and operation method of the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a video communication system consisting of two or more video communication terminals and a video relay server.SOLUTION: In a video communication system, a first video communication terminal 20 has an original video transmission part 201 transmitting an original video to a video relay server. A second video communication terminal 30 has: an attention object image generation part 301 generating an attention object image including all or a part of an attention object designated in the original video; and an attention object image transmission part 302 transmitting the attention object image to a video relay server 10. The video relay server 10 has: an attention object detector 101 collating the original video received from the first video communication terminal 20 with the attention object image to detect an attention object coordinate in the original video; a composite video generation part 102 compositing a predetermined diagram at a position of the attention object in the original video based on the attention object coordinate to generate a composite video; and a composite video transmission part 103 transmitting the composite video to the first video communication terminal 20 or the second video communication terminal 30.

Description

  The present invention relates to a video communication system that transmits and receives video between video communication terminals, and more particularly to a video communication system that facilitates communication between users who use each video communication terminal and an operation method thereof.

  Video communication systems are already in widespread use and have been implemented not only in expensive business videoconferencing systems but also in videoconferencing systems using a general-purpose mobile phone as a video communication terminal.

  As a general problem of video communication systems, it is difficult to talk to objects shown in video by pointing to them with directives and gestures such as “that” and “it” like normal face-to-face conversations. There are some. In order to solve this problem, the same video is shared between video communication terminals, a graphic is written in the video that the user of the video communication terminal communicates in the shared video, and the video and the graphic are superimposed. A displayable system has also been put into practical use (for example, see Non-Patent Document 1). In many of these systems, screens such as whiteboards, electronic files, and web pages are shared and written between video communication terminals separately from video conference video, but graphics are superimposed on video conference video as well. Technically possible.

“Real Collaboration”, NTT Software Corporation, [online], [searched on January 11, 2011], Internet <http://www.ntts.co.jp/products/realcollabo/index.html>

  By applying the graphic superimposition display function to the videophone of a mobile terminal such as a mobile phone, it is thought that it will be useful for facilitating the conversation in various scenes, not only in the place of the conventional videoconference. The following issues exist.

  Many mobile terminals have a large amount of camera shake because they are used by hand, and there is a problem that the position of the superimposed graphic in the video is shifted from the intended location. In the video conference system for enterprises such as the technology of Non-Patent Document 1 described above, the camera used for the video is basically fixed, and the subject (target object) captured in the video also moves. There are few. On the other hand, considering the use of a mobile terminal with a camera as a video communication terminal, not only does the camera move frequently, but also the movement of objects in the video increases in scenes such as outdoors. Thus, for example, even if an arrow is written on an object that was reflected in the center of the image at a certain moment, it can be sufficiently that the object is displayed at a different location in the image at the next moment. . In such a situation, if the position of the arrow is fixed at the center of the original image, the object pointed to by the arrow may change, and the meaning of the arrow may be lost.

  Furthermore, although the video conference system of Non-Patent Document 1 can be provided with sufficient processing capability to use a business fixed terminal as a video communication terminal, mobile terminals are often relatively limited in processing capability. For this reason, the problem that a high load process, such as a superimposition of a figure, cannot be performed with a mobile terminal is also assumed.

  Also, in order to realize high-load processing such as superimposing graphics, special dedicated software is required, so it is assumed that a commercially available mobile terminal cannot be used as it is, Providing dedicated software for each video communication terminal causes problems such as an increase in system cost and user convenience.

  Accordingly, the present invention has been made in view of the above-described problems, and it is an object of the present invention to provide a video communication system that facilitates communication between users who use each video communication terminal and an operating method thereof.

  The video communication system according to the present invention is different from the conventional video conference system in that a predetermined figure is added to the original video provided from the video communication terminal in order to facilitate communication between users who use each video communication terminal. A video relay server that generates and provides a composite video on which is superimposed is provided, and each video communication terminal is configured to realize video communication via the video relay server. Therefore, in the video communication system according to the present invention, the subject of each video communication terminal is not the method of designating the graphic to be superimposed and its coordinate position, but the graphic to be superimposed by each video communication terminal and its coordinate position. It should be noted that the video relay server, when specified, is technically distinct from conventional video conferencing systems due to the subject matter of generating and providing composite video.

  That is, the video communication system according to the first aspect of the present invention is a video communication system including two or more video communication terminals and a video relay server, and the first video communication terminal transmits the original video to the video relay server. The second video communication terminal includes an attention object image generation unit that generates an attention object image including all or a part of the attention object specified in the original image; and A target object image transmission unit for transmitting to the video relay server, and the video relay server collates the target image with the original video received from the first video communication terminal, and the target object is reflected in the original video. A target object detection unit for detecting a target object coordinate indicating a position of the target object, and a composite image by combining a predetermined figure with the position of the target object in the original video based on the target object coordinate And generating synthesized image generating unit, the combined image, characterized in that it comprises a combined image transmission unit to be transmitted to the first video communication terminal or the second video communication terminal. Thereby, the video communication between each video communication terminal is realizable, reducing the processing burden in a 1st video communication terminal.

  In the video communication system according to the first aspect of the present invention, the composite video transmission unit includes means for transmitting the composite video to both the first video communication terminal and the second video communication terminal. To do. Thereby, the second video communication terminal can share the composite video with the first video communication terminal.

  In the video communication system according to the first aspect of the present invention, the target object image generation unit of the second video communication terminal generates a target object image including all or part of the target object specified in the composite video. It has the means to do. As a result, the second video communication terminal can further specify the object of interest for the synthesized video.

  Furthermore, the video communication system according to the second aspect of the present invention is a video communication system including two or more video communication terminals and a video relay server, and the first video communication terminal transmits the original video to the video relay server. The second video communication terminal includes an attention area coordinate designation processing section for generating attention area coordinate information indicating an attention area designated in the original image; and the information on the attention area coordinates. A region-of-interest coordinate transmission unit that transmits the image to the video relay server, and the video relay server includes all or one of the objects of interest in the original video based on the information of the region-of-interest coordinates received from the second video communication terminal. A target object image extraction unit that extracts a partial image including a part and generates a target object image, and collates the target image with the original image received from the first video communication terminal. A target object detection unit that detects a target object coordinate indicating a position where the target object is reflected in the original video, and a predetermined figure is synthesized at the position of the target object in the original video based on the target object coordinate. A composite video generation unit that generates a composite video, and a composite video transmission unit that transmits the composite video to the first video communication terminal or the second video communication terminal. Thereby, the video communication between each video communication terminal is realizable, reducing the processing burden in a 1st video communication terminal.

  In the video communication system according to the second aspect of the present invention, the composite video transmission unit includes means for transmitting the composite video to both the first video communication terminal and the second video communication terminal. To do. Thereby, the second video communication terminal can share the composite video with the first video communication terminal.

  Further, in the video communication system according to the second aspect of the present invention, the attention area coordinate designation processing unit of the second video communication terminal generates information of attention area coordinates indicating the attention area designated in the synthesized video. It is characterized by having. As a result, the second video communication terminal can further specify the object of interest for the synthesized video.

  The operating method in the video communication system according to the first aspect of the present invention is an operating method of the video relay server in the video communication system comprising two or more video communication terminals and a video relay server, wherein the first video communication terminal is The second video communication terminal includes an attention object image that includes all or part of the attention object specified in the original image, and includes an original image transmission unit that transmits the original image to the image relay server. An object image generation unit, and a target object image transmission unit that transmits the target object image to the video relay server. The source image received from the first video communication terminal is collated with the target object image, and Detecting a target object coordinate indicating a position where the target object is reflected in the original video; and a position of the target object in the original video based on the target object coordinate. Generating a synthesized and combined image a predetermined shape to, the synthetic video image, characterized in that it comprises the steps of: transmitting to the first video communication terminal or the second video communication terminal.

  The operating method in the video communication system according to the second aspect of the present invention is an operating method of a video relay server in a video communication system comprising two or more video communication terminals and a video relay server, wherein the first video communication terminal The second video communication terminal includes an original video transmission unit that transmits the original video to the video relay server, and the second video communication terminal generates attention region coordinate designation that generates attention region coordinate information indicating the attention region designated in the original video A processing unit, and a region-of-interest coordinate transmission unit that transmits information on the region-of-interest coordinates to the video relay server, and based on the information on the region-of-interest coordinates received from the second video communication terminal, the original video Extracting a partial image including all or part of the object of interest in the image and generating it as the object of interest image; and an original image received from the first video communication terminal Collating the target object image and detecting a target object coordinate indicating a position where the target object is reflected in the original video; and a predetermined position at a position of the target object in the original video based on the target object coordinate. The method includes a step of generating a composite video by combining figures and a step of transmitting the composite video to the first video communication terminal or the second video communication terminal.

  According to the present invention, when a graphic is superimposed on a video by the video relay server, a predetermined graphic is superimposed on the target object in the video specified by the video communication terminal. Even if is moved, it is possible to provide a composite image in which the figure follows the target object.

  In addition, since the process of generating and providing the composite video that superimposes the figure to follow the target object is performed not by each video communication terminal but by the video relay server on the network, the video communication terminal with low processing capability is also used in this case. It can be used for the video communication system of the invention.

  In addition, an existing standard method can be used as a video transmission / reception method from the video communication terminal, and the standard video communication terminal and its program can be used without being dedicated.

  Therefore, according to the present invention, the video relay server and the video composition are performed by the video relay server, so that an existing videophone system using a video communication terminal with low performance and only a standard video communication function is used. A video communication system with improved convenience can be provided.

It is a figure which shows the structural example of the video communication system which concerns on this invention. It is a figure which shows the structure of the video communication system of 1st Embodiment by this invention. 1 is a block diagram of a video communication system according to a first embodiment of the present invention. It is an operation | movement flowchart of the video communication system of 1st Embodiment by this invention. It is a figure which shows the structure of the video communication system of 2nd Embodiment by this invention. It is a block diagram of the video communication system of 2nd Embodiment by this invention. It is an operation | movement flowchart of the video communication system of 2nd Embodiment by this invention. It is a figure which illustrates the synthetic | combination image | video in the image | video communication system which concerns on this invention.

  The video communication system of each embodiment according to the present invention will be described below with reference to the drawings. First, a comprehensive configuration of the video communication system according to the present invention will be described, and more specific embodiments will be described later in detail.

  FIG. 1 is a diagram showing a configuration example of a video communication system according to the present invention. The video communication system according to the present invention includes two or more video communication terminals and a video relay server that relays video and audio of each video communication terminal through a network. In the following description, as shown in FIG. 1, the video relay server 10 is typically set between the first video communication terminal 20 used by the user A and the second video communication terminal 30 used by the user B. An example of realizing video communication via the network will be described. Although the video relay server 10 can be realized by a single computer, the video composition unit will be described for convenience of explanation in order to explain an example in which the first video communication terminal 20 and the second video communication terminal 30 communicate remotely via a network. Explanation will be made on the assumption that it is composed of 10a and a multipoint connection unit 10b.

  The first video communication terminal 20 is a terminal used by the user A as an existing mobile terminal with a camera, for example. The user A has a function of transmitting an original image captured by the built-in camera using such a camera-equipped mobile terminal to the second video communication terminal 30 via the video relay server 10. On the other hand, the first video communication terminal 20 receives a composite video obtained by superimposing a predetermined figure on the original video from the video relay server 10 and can be viewed by the user A on the monitor screen of its own display. It has a display / playback function.

  The second video communication terminal 30 is a terminal used by the user B as an existing personal computer, for example. This personal computer may use a mouse, a touch panel, or a portable terminal. The second video communication terminal 30 receives a composite video viewed by the first video communication terminal 20 from the video relay server 10 in the same manner, and has a display reproduction function that allows the user B to view on the monitor screen of its own display. Have. Further, the second video communication terminal 30 designates an object to which the user B pays attention in the original video or the synthesized video, and the designated target object image (first embodiment) or attention area coordinates (second embodiment). ) Information to the video relay server 10, and a composite video in which a predetermined graphic from the original video is superimposed on the video relay server 10 (or a predetermined graphic from the received composite video is superimposed) (Composite video) can be generated. For example, a mouse or touch panel type is preferable as the user interface for designating an object of interest of the user B in the original video or the synthesized video.

  Here, both the 1st video communication terminal 20 and the 2nd video communication terminal 30 have the function which implement | achieves video communication via the multipoint connection unit 10b similarly to the existing video conference system. And For example, the multipoint connection unit 10b can be configured by a server for RTP (Real-time Transport Protocol) communication.

  The video relay server 10 is interconnected with the multipoint connection unit 10b, and is a composite video in which a predetermined figure is superimposed on the original video (or a composite video generated and designated last time from the second video communication terminal 30). A video synthesis unit 10a is provided for generating and providing a further synthesized video on which a predetermined figure is superimposed.

  That is, the video relay server 10 receives the original video from the first video communication terminal 20, and when the target object is specified from the second video communication terminal 30, the target object for the original video from the second video communication terminal 30. Receiving information on the region of interest coordinates indicating the region of the image or its target object, extracting the image region of the specified target object, generating a composite image in which a predetermined figure is assigned and superimposed on each target object, Transmit to the first video communication terminal 20 and / or the second video communication terminal 30. From the viewpoint of sharing the composite video, the composite video is preferably transmitted to both the first video communication terminal 20 and the second video communication terminal 30, and the video relay server 10 is connected to the first video communication terminal 20. The original video is received and transmitted to both the first video communication terminal 20 and the second video communication terminal 30 as a composite video until the target object is designated from the second video communication terminal 30.

  Here, the predetermined graphic to be superimposed is prepared in advance by the video composition unit 10a, and is, for example, a graphic such as “circle” or “arrow”. In the aspect in which the superimposed graphic is sequentially changed according to the designated number of times from the second video communication terminal 30, or the attention object image information indicating the attention object image or the region of the attention object with respect to the original image from the second video communication terminal 30. On the other hand, auxiliary information for designating a figure (for example, flag 1 for “circle”, flag 2 for “arrow”, flag 3 for “balloon”), etc. may be considered. . In this case, it is preferable to provide the second video communication terminal 30 with tablet-type application software for performing graphic selection.

  In addition, the video composition unit 10a receives from the original video or synthesized video having the designated target object with respect to the target object image for the original video from the second video communication terminal 30 or the information of the target area coordinates indicating the area of the target object. Even when an existing object extraction process (for example, MPEG-4 object extraction technique is known) and the original video is received from the first video communication terminal 20 repeatedly after the composite video is transmitted, the second video is also received. Until the target object is further specified from the communication terminal 30, a composite image in which a predetermined figure is superimposed on a position following the specified target object in the original image is generated and provided. When the designated target object disappears or hides from the repeatedly obtained original image, it is possible to provide a composite image in which the figure is not superimposed.

Accordingly, in the video communication system according to the present invention, the following procedure usage modes are assumed.
(1) User A and User B perform video communication via the multipoint connection unit 10b using the video communication terminals 20 and 30, respectively.
(2) User A captures the surrounding situation with the built-in camera of the first video communication terminal 20 and transmits it to the multipoint connection unit 10b as an original video.
(3) While viewing the synthesized video on the second video communication terminal 30 and viewing a noticeable object in the conversation, the user B designates the object on the screen of the second video communication terminal 30. .
(4) The second video communication terminal 30 transmits, to the video composition unit 10a, the target object image cut out from the composite video specified by the user B or information on the target region coordinates indicating the location specified on the screen. .
(5) The video composition unit 10a compares the target object image (the one received from the second video communication terminal 30 or the partial image cut out from the original video or the like based on the received attention area coordinates) with the original video. The location where the target object is present in the original video is detected.
(6) The video composition unit 10a synthesizes a predetermined figure at the detected location, and transmits it as a composite image to the video communication terminals 20 and 30 via the multipoint connection unit.
Accordingly, each video communication terminal 20 and 30 displays the received composite video, and each user A and B can view the composite video on which the graphic associated with the object designated by the user B is superimposed.

  Hereinafter, the video communication system according to the first embodiment of the present invention will be described more specifically with reference to FIGS. FIG. 2 is a diagram showing the configuration of the video communication system according to the first embodiment of the present invention. FIG. 3 is a block diagram of the video communication system according to the first embodiment of the present invention. FIG. 4 is an operation flowchart of the video communication system according to the first embodiment of the present invention.

[First Embodiment]
Referring to FIG. 2, in the video communication system according to the first embodiment of the present invention, an original video is generated by the first video communication terminal 20 and transmitted to the video relay server 10, and the video relay server 10 The target object image sent from the video communication terminal 30 and the original video sequentially transmitted from the first video communication terminal 20 are collated to detect the position of the target object in the original video, and the figure is placed at this detected position. Configured to overlap. More specifically, the video communication system of the embodiment includes a video relay server 10, a first video communication terminal 20, and a second video communication terminal 30. The video relay server 10 includes a target object detection unit 101, a composite video generation unit 102, and a composite video transmission unit 103. The first video communication terminal 20 has an original video transmission unit 201. The second video communication terminal 30 includes a target object image generation unit 301 and a target object image transmission unit 302. It should be noted that only main parts according to the present invention are shown in FIG. 2 and do not exclude functions provided in an existing video communication terminal such as an image display / playback function, a communication function, and a user interface function. .

  Note that the video relay server 10 according to the present embodiment can be configured as one or more computers, and a program describing processing contents for realizing each function of the video relay server 10 is stored in a predetermined storage unit of the computer. It can be realized by storing the program in a computer (not shown) and reading and executing the program by a central processing unit (CPU) of the computer.

  The original video transmission unit 201 in the first video communication terminal 20 is a functional unit that operates the built-in camera by the user A and transmits the original video obtained by capturing the current situation to the video relay server 10. The original image may be a moving image or a still image, but in the following description, a moving image will be described as an example.

  An attention object image generation unit 301 in the second video communication terminal 30 generates an attention object image by extracting a partial image designated by a method such as a mouse click of an area of interest in the original image viewed by the user B. It is a functional part to do. That is, the target object image generation unit 301 acquires a frame image at the moment when the user B designates from the original video, and cuts out a part of the frame image as the target object image based on the target region coordinates. This partial image does not need to cut out an accurate object of interest, and is cut out with a specified size (for example, 1/50 size of a frame image) necessary for specifying the object of interest. Of course, the entire frame image may be designated as the target object image.

  The target object image transmission unit 302 in the second video communication terminal 30 is a functional unit that transmits the target object image generated by the target object image generation unit 301 to the video relay server 10.

  The target object detection unit 101 in the video relay server 10 collates the target object image received from the second video communication terminal 30 with the received and retained original video, and identifies the location where the target object exists in the original video as “ This is a functional unit that is detected as “target object coordinates”. Here, as a technique for collating and detecting a place where the target object exists in the original video, an existing object extraction technique can be used, but it may be specified by simple pixel value matching.

  The synthesized video generation unit 102 in the video relay server 10 is a functional unit that synthesizes a predetermined figure at a location detected by the target object detection unit 101 and generates a synthesized video.

  The composite video transmission unit 103 in the video relay server 10 is a functional unit that transmits the composite video generated by the composite video generation unit 102 to the video communication terminals 20 and 30.

  3 and 4 show more specific examples in which the video relay server 10 is configured as a video composition unit 10a and a multipoint connection unit 10b.

  4 will be described with reference to FIG. 3. First, the multipoint connection unit 10 b of the video relay server 10 is used for video / audio communication between the first video terminal 20 and the second video terminal 30. In addition, RTP communication is established (S1).

  The first video terminal 20 acquires the original video by the camera photographing unit 2011 by the operation of the user A, encodes the original video by a predetermined encoding method (for example, MPEG-4) by the encoding unit 2012, and transmits the RTP transmission unit. By 2013, it transmits to the multipoint connection unit 10b by RTP communication (S2). Here, the second video communication terminal 30 is in a standby state for the original video (S3).

  The multipoint connection unit 10b receives the original video encoded from the first video terminal 20 by the RTP receiving unit 1013, converts it to a predetermined bit rate suitable for video / audio communication by the transcoding unit 1016, The transmission unit 1017 transfers the data to the second video communication terminal 30 (S4). The multipoint connection unit 10b converts the encoded original video received from the first video terminal 20 into a predetermined bit rate suitable for RTP communication with the video synthesis unit 10a by the transcoding unit 1014, and RTP The data is transferred to the video composition unit 10a by the transmission unit 1015. The encoded original video transferred to the video synthesizing unit 10 a is received and decoded by the RTP reception / decoding unit 1018 and sent to the target object detection unit 101.

  The second video communication terminal 30 receives and decodes the encoded original video from the multipoint connection unit 10b by the RTP receiving / decoding unit 3011 and displays the original video on the monitor screen by the display display unit 3014. At the same time (S5), the image is sent to the target object image generation unit 301. The target object image generation unit 301 includes a partial image extraction unit 3012 that extracts a partial image corresponding to one video frame in the original video specified by the user B via the user input unit 3013. This partial image extraction The partial image is generated as a target object image by the unit 3012 and sent to the target object image transmission unit 302 (S6). The target object image transmission unit 302 includes a frame image transmission unit 3021 that transmits the target object image to the video composition unit 10a through the network. The frame image transmission unit 3021 is one video in the original video specified by the user B. The target object image corresponding to the frame is transmitted to the frame image receiving unit 1012 of the video composition unit 10a (S7).

  The target object detection unit 101 of the video composition unit 10 a includes a video collation recognition processing unit 1011 that collates the target object image received via the frame image reception unit 1012 and the original video received from the first video terminal 20. Then, the image verification recognition processing unit 1011 identifies the image area of the object of interest in the original image (S8). Further, the video composition unit 10a includes a composite video generation unit 102 that generates a composite video by superimposing a predetermined figure on the position of the target image coordinate of the identified target object on the original video, and the composite video generation unit 102 is a video composition in which information of a predetermined graphic assigned to the object of interest is superimposed on an original video obtained from the RTP receiving / decoding unit 1018 from a storage unit (not shown) storing a predetermined graphic. It has a video composition processing unit 1021 for executing processing (S9). Note that the second video communication terminal 30 designates one or more target objects, an identifier for identifying each target object, and a flag for selecting and specifying a graphic corresponding to the identifier. In this case, the target object detection unit 101 and the composite video generation unit 102 of the video composition unit 10a can display the graphic specified by the flag on the target object identified by the identifier. It can also be configured to obtain and superimpose each object of interest from a storage unit (not shown).

  The encoding / RTP transmission unit 1031 of the video composition unit 10a encodes the composite video obtained from the composite video generation unit 102 and sends it to the multipoint connection unit 10b by RTP communication. The multipoint connection unit 10b receives the composite video at the RTP receiver 1032 and converts it into a predetermined bit rate suitable for video / audio communication by the transcode unit 1033, and the first video communication terminal 20 by the RTP transmitter 1034. And it transfers to the 2nd video communication terminal 30 (S10). Therefore, the encoding / RTP transmission unit 1031, the RTP reception unit 1032, the transcoding unit 1033, and the RTP transmission unit 1034 function as the composite video transmission unit 103 that transmits the composite video to the video communication terminal 20. Note that the composite video transmission unit 103 can also transmit the composite video to the video communication terminal 30 by the transcode unit 1016 and the RTP transmission unit 1017.

  The first video communication terminal 20 receives and decodes the composite video by the RTP receiving / decoding unit 2014, and displays the composite video on the monitor screen by the display display unit 2015 (S11). Similarly, the second video communication terminal 30 receives and decodes the composite video by the RTP receiving / decoding unit 3011, and displays the composite video on the monitor screen by the display display unit 3014 (S12).

  As described above, according to the video communication system of the present embodiment, the user B of the second video communication terminal 30 displays a figure in the video exchanged between the first video communication terminal 20 and the second video communication terminal 30. Since the image area of the object of interest specified as the object to be superimposed is identified by matching and the figure is superimposed, the drawing position was specified when drawing the figure, so it was superimposed due to camera shake or object movement. The problem that the figure is displaced can be solved.

  Further, according to the video communication system of the present embodiment, the first video communication terminal 20 is configured to transmit the original video to the video relay server 10 and execute the superimposition of the figure on the video relay server 20. The problem relating to the lack of processing capability of the mobile terminal used as the video communication terminal 20 can be solved.

  Further, according to the video communication system of the present embodiment, it is not necessary to use special dedicated hardware or software for the first video communication terminal 20, that is, the first video communication terminal 20 (or the second video communication terminal). 30) and the video transmission / reception server 20 can be implemented without causing a problem such as an excessive increase in cost.

  Next, a video communication system according to a second embodiment of the present invention will be described with reference to FIGS. FIG. 5 is a diagram showing a configuration of a video communication system according to the second embodiment of the present invention. FIG. 6 is a block diagram of a video communication system according to the second embodiment of the present invention. FIG. 7 is an operation flowchart of the video communication system according to the second embodiment of the present invention. In addition, the same reference number is attached | subjected to the component similar to 1st Embodiment.

[Second Embodiment]
Referring to FIG. 5, the video communication system according to the second embodiment of the present invention generates an original video at the first video communication terminal 20 and transmits it to the video relay server 10 to superimpose a figure. Is specified by the user B at the second video communication terminal 30 and the information of the attention area coordinates indicating the specified location is transmitted to the video relay server 10. The video relay server 10 extracts a target object image from the original video received from the first video communication terminal 30 based on the information on the target region coordinates, and collates the extracted target object image with the original video to check the original video. The position of the target object is detected as target object coordinate information, and a figure is superimposed on the detected position. More specifically, the video communication system of this embodiment includes a video relay server 10, a first video communication terminal 20, and a second video communication terminal 30. The video relay server 10 includes a target object detection unit 101, a composite video generation unit 102, a composite video transmission unit 103, and a target object image extraction unit 104. The first video communication terminal 20 has an original video transmission unit 201. The second video communication terminal 30 includes a region of interest coordinate designation processing unit 303 and a region of interest coordinate transmission unit 304. It should be noted that only main parts according to the present invention are shown in FIG. 5 and do not exclude functions of existing video communication terminals such as an image display / playback function, a communication function, and a user interface function. .

  In the second embodiment, the second video communication terminal 30 includes an attention area coordinate designation processing unit 303 and an attention area coordinate transmission unit 304, and the video relay server 10 includes an attention object image extraction unit 104. Is different.

  Note that the video relay server 10 according to the present embodiment can be configured as one or more computers, and a program describing processing contents for realizing each function of the video relay server 10 is stored in a predetermined storage unit of the computer. It can be realized by storing the program in a computer (not shown) and reading and executing the program by a central processing unit (CPU) of the computer.

  The original video transmission unit 201 in the first video communication terminal 20 is a functional unit that operates the built-in camera by the user A and transmits the original video obtained by capturing the current situation to the video relay server 10. The original image may be a moving image or a still image, but in the following description, a moving image will be described as an example.

  The attention area coordinate designation processing unit 303 in the second video communication terminal 30 specifies the coordinates of the attention area designated by a method such as mouse click on the area of interest when the original video is browsed by the user B, and It is a functional part generated as area coordinate information. In other words, the attention area coordinate designation processing unit 303 acquires a frame image at the moment when the designation is made by the user B from the original video, and designates attention area coordinates that can cut out a part of the frame image as the attention object image. . The attention area coordinates may be a coordinate group surrounding the attention object, or may be a coordinate designating one point. In this case, a specified size (for example, necessary for specifying the attention object on the video relay server 10 side) , 1/50 size of the frame image).

  The attention area coordinate transmission unit 304 in the second video communication terminal 30 is a functional unit that transmits information on the attention area coordinates generated by the attention area coordinate designation processing unit 303 to the video relay server 10.

  The target object image extraction unit 104 in the video relay server 10 extracts the target object image from the original video based on the information of the “target region coordinates” acquired from the second video communication terminal 30 and sends it to the target object detection unit 101. To do. The extraction of the target object image may be performed with a specified size (for example, 1/50 size of the frame image) necessary for specifying the target object with the information of the target region coordinates as the center of gravity.

  The target object detection unit 101 in the video relay server 10 collates the target object image extracted by the target object image extraction unit 104 with the received and stored original video, and determines the location where the target object exists in the original video. This is a functional unit that is detected as “target object coordinates”. Here, as a technique for collating and detecting a place where the target object exists in the original video, an existing object extraction technique can be used, but it may be specified by simple pixel value matching.

  The synthesized video generation unit 102 in the video relay server 10 is a functional unit that synthesizes a predetermined figure at a location detected by the target object detection unit 101 and generates a synthesized video.

  The composite video transmission unit 103 in the video relay server 10 is a functional unit that transmits the composite video generated by the composite video generation unit 102 to the video communication terminals 20 and 30.

  6 and 7 show more specific examples in which the video relay server 10 is configured as a video composition unit 10a and a multipoint connection unit 10b.

  7 will be described with reference to FIG. 6. First, the multipoint connection unit 10 b of the video relay server 10 is used for video / audio communication between the first video terminal 20 and the second video terminal 30. In addition, RTP communication is established (S21).

  The first video terminal 20 acquires the original video by the camera photographing unit 2011 by the operation of the user A, encodes the original video by a predetermined encoding method (for example, MPEG-4) by the encoding unit 2012, and transmits the RTP transmission unit. 201 transmits to the multipoint connection unit 10b by RTP communication (S22). Here, the second video communication terminal 30 is in a standby state for the original video (S23).

  The multipoint connection unit 10b receives the original video encoded from the first video terminal 20 by the RTP receiver 1013 and transmits it to the video composition unit 10a via the transcode unit 1014 and the RTP transmitter 1015. Since there is no designated attention area coordinate for the captured original video, the original video is acquired as a synthesized video by the video synthesizing unit 10a via the RTP receiving unit 1032 and is transmitted via the transcoding unit 1033 and the RTP transmitting unit 1034. Transfer to the 2 video communication terminal 30 (S24).

  The second video communication terminal 30 receives and decodes the encoded original video from the multipoint connection unit 10b by the RTP receiving / decoding unit 3011 and displays the original video on the monitor screen by the display display unit 3014. (S25). The attention area coordinate designation processing unit 303 generates attention area coordinate information that identifies a partial image corresponding to one video frame in the original image designated by the user B via the user input unit 3031 (S26). The information is sent to the attention area coordinate transmission unit 304. The attention area coordinate transmission section 304 has a coordinate transmission section 3021b that transmits information on the attention area coordinates to the video composition unit 10a through the network, and the coordinate transmission section 3021b includes one image in the original image designated by the user B. Information on the attention area coordinates for specifying the attention object corresponding to the frame is transmitted to the coordinate receiving unit 1012b of the video composition unit 10a (S27).

  The target object image extraction unit 104 of the video composition unit 10a includes a partial image extraction unit 1041 that extracts a partial image from the original video received from the first video terminal 20 based on the information of the target region coordinates. The unit 1041 sends the extracted partial image to the target object detection unit 101 as the target object image. The target object detection unit 101 receives the target object image extracted by the target object image extraction unit 104 and the first video terminal 20. The video collation recognition processing unit 1011 for collating with the original video is specified. The video collation recognition processing unit 1011 identifies the target image coordinates of the target object in the original video (S28).

  The video composition unit 10a includes a composite video generation unit 102 that generates a composite video by superimposing a predetermined figure on the position of the target image coordinate of the identified target object on the original video, and the composite video generation unit 102 A video composition process for acquiring and superimposing information on a predetermined graphic assigned to the object of interest from a storage unit (not shown) storing a predetermined graphic with respect to the original video obtained from the RTP receiving / decoding unit 1018 The video composition processing unit 1021 to be executed is included (S29). When the second video communication terminal 30 specifies one or more attention area coordinates that designate one or more attention objects, the second video communication terminal 30 selects an identifier for identifying each attention object and a graphic corresponding to the identifier. In this case, the target object detection unit 101 and the composite video generation unit 102 of the video composition unit 10a use the identifiers. A graphic specified by the flag may be acquired for each target object from the storage unit (not shown) and superimposed on the identified target object.

  The encoding / RTP transmission unit 1031 of the video composition unit 10a encodes the composite video obtained from the composite video generation unit 102 and sends it to the multipoint connection unit 10b by RTP communication. The multipoint connection unit 10b receives the composite video by the RTP receiver 1032 and converts it into a predetermined bit rate suitable for video / audio communication by the transcode unit 1033, and the second video communication terminal 30 by the RTP transmitter 1034. And the transcoding unit 1016 converts the bit rate to a predetermined bit rate suitable for video / audio communication, and the RTP transmission unit 1017 transfers the bit rate to the first video communication terminal 20 (S30). Therefore, the encoded / RTP transmitting unit 1031, the RTP receiving unit 1032, the transcoding unit 1033, the RTP transmitting unit 1034, the transcoding unit 1033, and the RTP transmitting unit 1034 transmit the synthesized video to the video communication terminals 20 and 30, respectively. It functions as the transmission unit 103.

  The first video communication terminal 20 receives and decodes the composite video by the RTP receiving / decoding unit 2014, and displays the composite video on the monitor screen by the display display unit 2015 (S31). Similarly, the second video communication terminal 30 receives and decodes the composite video by the RTP receiving / decoding unit 3011, and displays the composite video on the monitor screen by the display display unit 3014 (S32).

  Thus, the video communication system of the present embodiment can also be configured to include all the advantages of the first embodiment.

  FIG. 8 shows an example in which a predetermined figure is superimposed on the original video. “Circle” can be added to the region of interest in the captured image (original video), “arrow” can be added, and “balloon” can be added. It is also possible to input text by transmitting text information by RTP communication in response to an instruction from the first video communication terminal 20 or the second video communication terminal 30 with respect to the composite video provided with this “speech balloon”.

  According to the present invention, when a graphic is superimposed on a video by the video relay server, a predetermined graphic is superimposed on the target object in the video specified by the video communication terminal. Even if the image moves, it is possible to provide a composite image in which the figure follows the target object, which is useful for video communication using a mobile terminal.

DESCRIPTION OF SYMBOLS 10 Video relay server 10a Video composition unit 10b Multipoint connection unit 20 1st video communication terminal 30 2nd video communication terminal 101 Object-of-interest detection part 102 Composite image generation part 103 Composite image transmission part 104 Object-of-interest image extraction part 201 Original image transmission Unit 301 attention object image generation unit 302 attention object image transmission unit 303 attention region coordinate designation processing unit 304 attention region coordinate transmission unit

Claims (8)

A video communication system comprising two or more video communication terminals and a video relay server,
The first video communication terminal includes an original video transmission unit that transmits the original video to the video relay server,
The second video communication terminal includes a target object image generation unit that generates a target object image including all or part of a target object specified in the original video, and a target for transmitting the target object image to the video relay server. An object image transmission unit,
The video relay server is
An attention object detection unit that compares the original image received from the first video communication terminal with the attention object image and detects an attention object coordinate indicating a position where the attention object is reflected in the original image;
A synthesized video generating unit that generates a synthesized video by synthesizing a predetermined figure at the position of the target object in the original video based on the target object coordinates;
A composite video transmission unit for transmitting the composite video to the first video communication terminal or the second video communication terminal;
A video communication system comprising:   The video communication system according to claim 1, wherein the composite video transmission unit includes means for transmitting the composite video to both the first video communication terminal and the second video communication terminal.   The object-of-interest image generation unit of the second video communication terminal includes means for generating an object-of-interest image including all or part of the object of interest specified in the composite image. The video communication system described. A video communication system comprising two or more video communication terminals and a video relay server,
The first video communication terminal includes an original video transmission unit that transmits the original video to the video relay server,
The second video communication terminal generates an attention area coordinate specification processing unit that generates attention area coordinate information indicating the attention area specified in the original video, and transmits the attention area coordinate information to the video relay server. An area coordinate transmitter,
The video relay server is
An attention object image extraction unit that extracts a partial image including all or part of the attention object in the original image based on the information of the attention area coordinates received from the second video communication terminal; ,
An attention object detection unit that compares the original image received from the first video communication terminal with the attention object image and detects an attention object coordinate indicating a position where the attention object is reflected in the original image;
A synthesized video generating unit that generates a synthesized video by synthesizing a predetermined figure at the position of the target object in the original video based on the target object coordinates;
A composite video transmission unit for transmitting the composite video to the first video communication terminal or the second video communication terminal;
A video communication system comprising:   The video communication system according to claim 4, wherein the composite video transmission unit includes means for transmitting the composite video to both the first video communication terminal and the second video communication terminal.   The attention area coordinate specification processing unit of the second video communication terminal includes means for generating attention area coordinate information indicating an attention area specified in the composite video. Video communication system. An operation method of a video relay server in a video communication system including two or more video communication terminals and a video relay server,
The first video communication terminal includes an original video transmission unit that transmits the original video to the video relay server,
The second video communication terminal includes a target object image generation unit that generates a target object image including all or part of a target object specified in the original video, and a target for transmitting the target object image to the video relay server. An object image transmission unit,
Collating the original image received from the first video communication terminal with the object-of-interest image and detecting object-of-interest coordinates indicating a position where the object of interest is reflected in the original image;
Generating a synthesized image by synthesizing a predetermined figure at the position of the object of interest in the original image based on the object of interest coordinates;
Transmitting the composite video to the first video communication terminal or the second video communication terminal;
A method for operating a video relay server, comprising: An operation method of a video relay server in a video communication system including two or more video communication terminals and a video relay server,
The first video communication terminal includes an original video transmission unit that transmits the original video to the video relay server,
The second video communication terminal generates an attention area coordinate specification processing unit that generates attention area coordinate information indicating the attention area specified in the original video, and transmits the attention area coordinate information to the video relay server. An area coordinate transmitter,
Extracting a partial image including all or part of the target object in the original video based on the information of the target area coordinates received from the second video communication terminal, and generating the target object image;
Collating the original image received from the first video communication terminal with the object-of-interest image and detecting object-of-interest coordinates indicating a position where the object of interest is reflected in the original image;
Generating a synthesized image by synthesizing a predetermined figure at the position of the object of interest in the original image based on the object of interest coordinates;
Transmitting the composite video to the first video communication terminal or the second video communication terminal;
A method for operating a video relay server, comprising:

Images