JP5813542B2 - Image communication system, AR (Augmented Reality) video generation device, and program - Google Patents

Description

  The present invention relates to communication using augmented reality (AR).

  There is a technique for distributing a composite image by an AR technique via a communication network (see, for example, Patent Document 1). As a method for obtaining a composite image, a method using a background of a specific color, such as chroma key composition (blue back composition), or a non-existent image (such as an avatar) obtained by CG (Computer Graphics) The one that synthesizes is known.

JP 2004-341642 A

By the way, a conventional composite image using the AR technology is generally a superposition of a previously prepared image on a video shot by a user. Therefore, it cannot be applied as it is to real-time communication using actual videos (that is, images not prepared in advance) between users such as so-called videophones.
Therefore, the present invention provides a technique that enables real-time communication using a moving image synthesized based on videos taken by mutual users.

An image communication system according to an aspect of the present invention includes a connection management device that manages a connection between a first terminal and a second terminal, and the first terminal and the second terminal that are managed by the connection management device. An AR (Augmented Reality) video generation device that transmits composite video data obtained by synthesizing videos captured by both sides to at least one, wherein the connection management device includes the first terminal and the second terminal. Is connected to the AR video generation device, and the AR video generation device captures a subject including a predetermined object from one of the first terminal and the second terminal. A first receiver for receiving first moving image data including distance information for each pixel of the subject, and a marker of a predetermined shape from the other of the first terminal and the second terminal. A second receiving unit that receives second moving image data that captures a background including the image, and a pixel corresponding to the predetermined object from the subject based on the distance information of the first moving image data received by the first receiving unit And a three-dimensional coordinate system of the background defined based on the marker in the background represented by the second moving image data received by the second receiving unit, and extracted by the extracting unit Generating a combined moving image data generated by combining the image corresponding to the selected pixel with the background so as to correspond to the coordinate system, and transmitting the combined moving image data generated by the generating unit to the other terminal and a transmission unit, according to the channel information supplied by the connection management apparatus, Ru with different image represented by the composite video data.
In a preferred aspect, the AR video generation device includes a determination unit that determines whether communication using the composite video data is possible for each of the first terminal and the second terminal based on the line information. The synthesized moving image data is generated and transmitted for a terminal that is determined by the determination unit to be able to communicate with the synthesized moving image data.
An image communication system according to an aspect of the present invention includes a connection management device that manages a connection between a first terminal and a second terminal, the first terminal whose connection is managed by the connection management device, and the second terminal. An AR (Augmented Reality) video generation device that transmits composite video data obtained by synthesizing videos captured by both sides to at least one of the terminals, wherein the connection management device includes the first terminal and the second terminal. Resource information relating to hardware or software resources in the terminal is supplied to the AR video generation device, and the AR video generation device photographs a subject including a predetermined object from one of the first terminal and the second terminal. A first receiving unit that receives first moving image data including distance information for each pixel of the subject, the first terminal, and the front Based on the distance information of the 1st animation data received by the 2nd receiving part which receives the 2nd animation data which photoed the background containing the marker of the predetermined shape from the other of the 2nd terminal, and the 1st receiving part A three-dimensional background defined based on the marker in the background represented by the second moving image data received by the second receiver and an extraction unit that extracts pixels corresponding to the predetermined object from the subject A generating unit that generates a composite moving image data in which an image corresponding to the pixel extracted by the extracting unit is combined with the background so as to correspond to the coordinate system, and generated by the generating unit A transmission unit that transmits the combined moving image data to the other terminal, and changes the video represented by the combined moving image data according to the resource information supplied by the connection management device.
In a preferred aspect, the AR video generation device includes a determination unit that determines whether communication using the composite video data is possible for each of the first terminal and the second terminal based on the resource information. The synthesized moving image data is generated and transmitted for a terminal that is determined by the determination unit to be able to communicate with the synthesized moving image data.
An image communication system according to an aspect of the present invention includes a connection management device that manages a connection between a first terminal and a second terminal, the first terminal whose connection is managed by the connection management device, and the second terminal. An AR (Augmented Reality) video generation device that transmits composite video data obtained by synthesizing videos captured by both sides to at least one of the terminals, wherein the connection management device includes the first terminal and the second terminal. A transmission / reception unit that transmits / receives a voice message to / from a terminal, and supplies information representing a change in volume or sound quality of the voice message transmitted / received by the transmission / reception unit to the AR video generation device; First moving image data obtained by photographing a subject including a predetermined object from one of the first terminal and the second terminal, and distance information for each pixel of the subject. And a second receiving unit for receiving second moving image data obtained by capturing a background including a marker having a predetermined shape from the other of the first terminal and the second terminal. Based on the distance information of the first moving image data received by the first receiving unit, an extraction unit for extracting pixels corresponding to the predetermined object from the subject, and the second receiving unit A three-dimensional coordinate system of the background defined based on the marker in the background represented by the second moving image data is specified, and an image corresponding to the pixel extracted by the extraction unit is made to correspond to the coordinate system. A volume that is provided by the connection management device, and includes a generation unit that generates the combined moving image data combined with the background and a transmission unit that transmits the combined moving image data generated by the generation unit to the other terminal. Alternatively, the video represented by the synthetic moving image data is made different according to the information representing the change in sound quality.
An image communication system according to an aspect of the present invention includes a connection management device that manages a connection between a first terminal and a second terminal, the first terminal whose connection is managed by the connection management device, and the second terminal. An AR (Augmented Reality) moving image generating device that transmits combined moving image data obtained by synthesizing videos taken by both the terminals to at least one of the terminals, wherein the AR moving image generating device includes the first terminal and the first A first receiver for receiving first moving image data obtained by photographing a subject including a predetermined object from one of two terminals, the first moving image data including distance information for each pixel of the subject; A second receiving unit that receives second moving image data obtained by capturing a background including a marker having a predetermined shape from the other of the terminal and the second terminal; and a front of the first moving image data received by the first receiving unit. Based on the distance information, an extraction unit that extracts a pixel corresponding to the predetermined object from the subject, and the marker that is defined based on the marker in the background represented by the second moving image data received by the second reception unit A generating unit that identifies a three-dimensional coordinate system of a background and generates synthetic moving image data synthesized with the background so that an image corresponding to the pixel extracted by the extracting unit corresponds to the coordinate system; A transmitting unit that transmits the combined moving image data generated by the unit to the other terminal, and an analyzing unit that analyzes the change in the subject in the first moving image data received by the first receiving unit. The analyzed information representing the change in the subject is supplied to the connection management apparatus, and the connection management apparatus transmits and receives a voice message between the first terminal and the second terminal. A receiving unit, comprising: a transmitting / receiving unit that varies the volume or sound quality of the voice message to be transmitted / received in accordance with the information representing the change of the subject supplied by the AR video generation device.

In favorable preferable embodiment, the AR video production apparatus analyzes the second moving image data received by the second receiving unit includes a calculation unit for calculating the coordinate system, the generating unit, by the calculation unit Based on the calculated coordinate system, the synthesized moving image data is generated.

An AR moving image generating apparatus according to another aspect of the present invention receives first moving image data including distance information for each pixel of a subject, the first moving image data capturing a subject including a predetermined object. A second receiving unit that receives second moving image data in which a background including a marker having a predetermined shape is captured, wherein a three-dimensional coordinate system is defined based on the marker; An extraction unit for extracting pixels corresponding to the predetermined object from the subject based on the distance information of the first moving image data received by the first reception unit, and a pixel corresponding to the pixels extracted by the extraction unit A generating unit that generates synthesized moving image data in which the image is combined with the background so as to correspond to the coordinate system defined in the second moving image data received by the second receiving unit; And a transmission unit for transmitting the synthesized video data, the connection management apparatus for managing a connection with the transmitted terminal the second moving image data and transmitted by the terminal to the first video data, transmitting the first video data the terminal and the second transmit the moving picture data line information about the communication line for connecting the terminal is supplied in accordance with a supplied the line information, Ru with different image represented by the composite video data.
The AR video generation device according to another aspect of the present invention receives first video data including distance information for each pixel of the subject, the first video data capturing a subject including a predetermined object. A second receiving unit that receives second moving image data in which a three-dimensional coordinate system is defined based on the first receiving unit that captures a background including a marker having a predetermined shape. An extraction unit that extracts pixels corresponding to the predetermined object from the subject based on the distance information of the first moving image data received by the first reception unit, and a pixel extracted by the extraction unit A generating unit that generates synthesized moving image data synthesized with the background so as to correspond to the coordinate system defined in the second moving image data received by the second receiving unit, and the generating unit A connection unit that connects the terminal that has transmitted the first moving image data and the terminal that has transmitted the second moving image data to a hardware or software in the terminal Resource information related to these resources is supplied, and the video represented by the composite video data is made different according to the supplied resource information.
The AR video generation device according to another aspect of the present invention receives first video data including distance information for each pixel of the subject, the first video data capturing a subject including a predetermined object. A second receiving unit that receives second moving image data in which a three-dimensional coordinate system is defined based on the first receiving unit that captures a background including a marker having a predetermined shape. An extraction unit that extracts pixels corresponding to the predetermined object from the subject based on the distance information of the first moving image data received by the first reception unit, and a pixel extracted by the extraction unit A generating unit that generates synthesized moving image data synthesized with the background so as to correspond to the coordinate system defined in the second moving image data received by the second receiving unit, and the generating unit A transmission unit that transmits the synthesized video data formed, and a transmission / reception unit that transmits and receives a voice message between the terminal that transmitted the first video data and the terminal that transmitted the second video data, Information indicating a change in volume or sound quality of a voice message transmitted / received by the transmitter / receiver is supplied and supplied from a connection management device that supplies information indicating a change in volume or sound quality of a voice message transmitted / received by the transmitter / receiver. In accordance with the information, the video represented by the synthesized video data is made different.
The AR video generation device according to another aspect of the present invention receives first video data including distance information for each pixel of the subject, the first video data capturing a subject including a predetermined object. A second receiving unit that receives second moving image data in which a three-dimensional coordinate system is defined based on the first receiving unit that captures a background including a marker having a predetermined shape. An extraction unit that extracts pixels corresponding to the predetermined object from the subject based on the distance information of the first moving image data received by the first reception unit, and a pixel extracted by the extraction unit A generating unit that generates synthesized moving image data synthesized with the background so as to correspond to the coordinate system defined in the second moving image data received by the second receiving unit, and the generating unit A transmission unit that transmits the synthesized moving image data formed; and an analysis unit that analyzes a change in the subject in the first moving image data received by the first reception unit; the terminal that transmitted the first moving image data; A transmission / reception unit that transmits / receives a voice message to / from a terminal that has transmitted the second moving image data, wherein transmission / reception varies in volume or sound quality of the voice message to be transmitted / received according to supplied information representing the change in the subject. Information representing a change in the subject analyzed by the analysis unit is supplied to a connection management device including the unit.

According to another aspect of the present invention, there is provided a program for receiving, on a computer, first moving image data obtained by photographing a subject including a predetermined object, the first moving image data including distance information for each pixel of the subject. A second step of receiving second moving image data in which a background including a marker having a predetermined shape is captured, wherein the second moving image data defines a three-dimensional coordinate system based on the marker; A third step of extracting pixels corresponding to the predetermined object from the subject based on the distance information of the first moving image data received in the first step, and an image corresponding to the pixels extracted in the third step Is generated on the background so as to correspond to the coordinate system defined in the second moving image data received in the second step. A connection management device for managing a connection between the terminal that has transmitted the first moving image data and the terminal that has transmitted the second moving image data, and a fifth step of transmitting the combined moving image data generated by the step 4 To, line information relating to a communication line connecting the terminal that transmitted the first moving image data and the terminal that transmitted the second moving image data is supplied, and the combined moving image data is represented according to the supplied line information. This is a program for executing the sixth step of making a video different.
In addition, a program according to another aspect of the present invention receives first moving image data including distance information for each pixel of a subject, the first moving image data capturing a subject including a predetermined object, on a computer. A second step of receiving second moving image data in which a background including a marker having a predetermined shape is captured, wherein the second moving image data defines a three-dimensional coordinate system based on the marker; A third step of extracting a pixel corresponding to the predetermined object from the subject based on the distance information of the first moving image data received in the first step, and according to the pixel extracted in the third step The synthesized video data is synthesized with the background corresponding to the coordinate system defined in the second video data received in the second step. Connection management for connecting the terminal that has transmitted the first moving picture data and the terminal that has transmitted the second moving picture data to the fourth step that transmits the synthetic moving picture data generated by the fourth step. A program for executing resource information on hardware or software resources in the terminal from a device, and executing a sixth step of differentiating the video represented by the composite video data according to the supplied resource information. is there.
In addition, a program according to another aspect of the present invention receives first moving image data including distance information for each pixel of a subject, the first moving image data capturing a subject including a predetermined object, on a computer. A second step of receiving second moving image data in which a background including a marker having a predetermined shape is captured, wherein the second moving image data defines a three-dimensional coordinate system based on the marker; A third step of extracting a pixel corresponding to the predetermined object from the subject based on the distance information of the first moving image data received in the first step, and according to the pixel extracted in the third step The synthesized video data is synthesized with the background corresponding to the coordinate system defined in the second video data received in the second step. A voice message between the terminal that has transmitted the first moving image data and the terminal that has transmitted the second moving image data. And a change in volume or sound quality of the voice message transmitted / received by the transmitter / receiver from a connection management device that supplies information indicating a change in volume or sound quality of the voice message transmitted / received by the transmitter / receiver. Is provided, and a sixth step is executed to execute the sixth step of making the video represented by the synthesized moving image data different in accordance with the supplied information.
In addition, a program according to another aspect of the present invention receives first moving image data including distance information for each pixel of a subject, the first moving image data capturing a subject including a predetermined object, on a computer. A second step of receiving second moving image data in which a background including a marker having a predetermined shape is captured, wherein the second moving image data defines a three-dimensional coordinate system based on the marker; A third step of extracting a pixel corresponding to the predetermined object from the subject based on the distance information of the first moving image data received in the first step, and according to the pixel extracted in the third step The synthesized video data is synthesized with the background corresponding to the coordinate system defined in the second video data received in the second step. A fourth step, a fifth step for transmitting the composite moving image data generated by the fourth step, a sixth step for analyzing a change in the subject in the first moving image data received by the first step, A transmission / reception unit that transmits and receives a voice message between a terminal that has transmitted the first moving image data and a terminal that has transmitted the second moving image data, and transmits and receives the message according to information representing a change in the supplied subject. A program for executing a seventh step of supplying information representing a change in the subject analyzed in the sixth step to a connection management device including a transmission / reception unit that varies the volume or sound quality of the voice message.

  ADVANTAGE OF THE INVENTION According to this invention, real-time communication using the moving image synthesize | combined based on the image | video image | photographed by the mutual user is possible.

Block diagram showing overall configuration of communication system Block diagram showing the hardware configuration of the user terminal The block diagram which shows the hardware constitutions of AR animation production | generation apparatus and a connection management apparatus The block diagram which shows the function structure of AR moving image production | generation apparatus and a connection management apparatus Schematic diagram for explaining the video represented by video data Diagram for explaining the object extraction principle The flowchart which shows the process which AR animation production | generation apparatus performs Sequence chart showing processing in each device of the communication system

[Embodiment]
FIG. 1 is a block diagram showing an overall configuration of a communication system 10 according to an embodiment of the present invention. The communication system 10 is an information processing system for realizing real-time communication using voice and AR video between users, and is an example of an image communication system according to the present invention. Here, the AR moving image refers to a moving image obtained by synthesizing videos taken by a plurality of users, and is generated using the AR technology. Note that the real time here means that the target processing is completed by a predetermined time, and that the time is relatively short. The communication system 10 includes an AR video generation device 100, a connection management device 200, a user terminal 300, and a network 400.

  The AR moving image generating device 100 is a server device that receives moving image data from a plurality of user terminals 300, and generates and transmits synthesized moving image data obtained by combining these moving image data. The connection management device 200 is a server device that manages the connection (here, call connection) between the user terminals 300 and controls the utterance and the call termination. Management by the connection management device 200 includes processing for ensuring real-time communication. Note that the AR video generation device 100 and the connection management device 200 may be connected to each other via a communication line different from the network 400, without using the network 400.

  The user terminal 300 is a communication terminal used by a user who communicates with the communication system 10. The user terminal 300 may be a dedicated communication terminal for the communication system 10 or may be a device in which peripheral devices necessary for performing communication according to the present embodiment are attached to a smartphone or a personal computer. Note that a large number of user terminals 300 may actually be connected to the network 400, but only two terminals that perform communication are illustrated in FIG. In the following, when it is necessary to distinguish these terminals, they will be referred to as “user terminal 300a” and “user terminal 300b”.

  The network 400 is a network for transmitting and receiving data such as voice and moving images. The network 400 may be configured such that a communication line for transmitting and receiving audio and a communication line for transmitting and receiving a moving image are different from each other, but this need not be the case. Further, the network 400 may use any of the communication lines of the packet switching system and the circuit switching system.

  FIG. 2 is a block diagram illustrating a hardware configuration of the user terminal 300. The user terminal 300 includes a control unit 310, a storage unit 320, a communication unit 330, a first imaging unit 340, a second imaging unit 350, a display unit 360, a voice input / output unit 370, and an operation unit 380. Is provided. The configuration shown in FIG. 2 is hereinafter referred to as “standard configuration”.

  The control unit 310 is means for controlling the operation of the user terminal 300. The control unit 310 includes an arithmetic processing device such as a CPU (Central Processing Unit) and a memory, and controls shooting of moving images, data communication, and the like by executing a predetermined program. In addition, the control unit 310 includes a calculation unit 311 that calculates a three-dimensional coordinate system to be described later. Note that the calculation method of the coordinate system by the calculation unit 311 may be the same as a known method. For example, the calculation unit 311 recognizes a marker having a predetermined shape from the captured video, specifies inclination based on a difference (that is, distortion) between the recognized shape and the actual shape, and calculates inclination information. In addition, a three-dimensional orthogonal coordinate system is defined based on the specified inclination and the position of the marker, and coordinate information representing the coordinates is calculated.

  The storage unit 320 is means for storing data. The storage unit 320 corresponds to an auxiliary storage device and includes, for example, a hard disk or a flash memory. The memory | storage part 320 has memorize | stored the resource information regarding the resource of the user terminal 300 (own apparatus). Here, the resource information refers to data indicating what resources the user terminal 300 has. The resource here may include both a hardware resource and a software resource. The resource information includes, for example, the identification information (model name, etc.) of the user terminal 300, the processing capability of the CPU, the performance (number of pixels, etc.) of the first imaging unit 340 and the second imaging unit 350, and the performance (display resolution) of the display unit 360. Etc.), the corresponding codec, software version information necessary for communication, etc.

  The communication unit 330 is means for transmitting / receiving data via the network 400. Communication by the communication unit 330 may be either wireless or wired. Further, the communication unit 330 may include a configuration for communicating with a peripheral device when a part of the user terminal 300 is configured with an external peripheral device.

  The first photographing unit 340 and the second photographing unit 350 are both means for photographing a subject and outputting moving image data. The moving image data output by the first photographing unit 340 and the second photographing unit 350 includes at least color information representing the color of each pixel of the subject. The first photographing unit 340 is used for photographing a subject including a user, and the second photographing unit 350 is used for photographing a subject including a marker (AR marker). Furthermore, the first imaging unit 340 includes a distance image sensor. On the other hand, the second imaging unit 350 may be a normal image sensor, that is, an image sensor that does not output distance information.

  The distance image sensor is an image sensor that can output distance information of each pixel. The distance image sensor irradiates light with a predetermined pattern, measures the distance between the sensor and the subject by distortion of the reflected light, and irradiates the subject with light. There is a “TOF (Time of Flight) method” in which the distance between the sensor and the subject is measured according to the time required for the light to reflect back to the subject, and any method is used for the first photographing unit 340. Also good.

  The first photographing unit 340 includes such a distance image sensor and a normal image sensor, and outputs color information and distance information. It is assumed that the distance image sensor and the image sensor photograph the same direction, and the correspondence between the pixels of the distance image sensor and the pixels of the image sensor is obtained in advance.

  The display unit 360 is a means for displaying an image. The display unit 360 may be a display such as a liquid crystal provided in the user terminal 300, or may be a display such as a television separate from the user terminal 300. Further, the display unit 360 may be configured to be worn by the user like an HMD (Head Mounted Display).

The voice input / output unit 370 is means for inputting and outputting voice. The voice input / output unit 370 includes a speaker and a microphone. The audio input / output unit 370 may perform audio input / output in either stereo or monaural.
The operation unit 380 is a unit that receives a user operation. The operation unit 380 includes a keypad (keyboard) and a touch screen provided to overlap the display of the display unit 360.

  Note that the user terminal 300 includes all of these configurations as a standard configuration. However, as will be described later, even in a terminal that does not have a part of the configuration, the communication of this embodiment (with restrictions) ) Is possible. In addition, the user terminal 300 does not have to match individual resources such as display resolution.

  FIG. 3 is a block diagram showing the hardware configuration of the AR video generation device 100 and the connection management device 200. The AR video generation device 100 includes a control unit 110, a storage unit 120, and a communication unit 130. The connection management apparatus 200 includes a control unit 210, a storage unit 220, and a communication unit 130. The control units 110 and 120 are means for controlling the operation of the own device (the AR video generation device 100 or the connection management device 200). The storage units 120 and 220 are units for storing data, and the communication units 130 and 230 are units for transmitting and receiving data. The communication units 130 and 230 can communicate with each other without going through the network 400 in addition to communicating with the network 400.

  FIG. 4 is a block diagram illustrating functional configurations of the AR moving image generating apparatus 100 and the connection management apparatus 200. The AR video generation device 100 and the connection management device 200 implement the functions shown in FIG. 4 by executing a predetermined program. The control unit 210 of the connection management apparatus 200 functions as a transmission / reception unit 211 and a data transmission / reception unit 212. In addition, the control unit 110 of the AR video generation device 100 functions as a synthesis control unit 110a and a synchronization control unit 110b.

  The transmission / reception unit 211 is means for transmitting / receiving a voice message. The transmission / reception unit 211 transmits the voice message transmitted from the user terminal 300a to the user terminal 300b, and transmits the voice message transmitted from the user terminal 300b to the user terminal 300a. In addition, the transmission / reception unit 211 performs connection management necessary for transmitting / receiving voice messages, and controls volume and sound quality as necessary.

  The data exchanging unit 212 is a means for exchanging data with the AR moving image generating apparatus 100. The data transmission / reception unit 212 acquires resource information from the AR video generation device 100 and supplies line information to the AR video generation device 100. The line information is information relating to the communication line used by each of the user terminals 300a and 300b, and indicates the transmission capability (transfer bandwidth, etc.) of the communication line. The line information may differ depending on the contents of the contract with the user's telecommunications carrier, and may differ depending on the actual usage status (degree of congestion) of the communication line.

  The composition control unit 110a is means for controlling the composition of video. More specifically, the composition control unit 110a includes first reception units 111a and 111b, second reception units 112a and 112b, extraction units 113a and 113b, generation units 114a and 114b, and transmission units 115a and 115b. Prepare.

  In addition, the 1st receiving part 111a, the 2nd receiving part 112a, the extraction part 113a, the production | generation part 114a, and the transmission part 115a are the structures for transmitting synthetic | combination moving image data to the user terminal 300a, respectively. On the other hand, the 1st receiving part 111b, the 2nd receiving part 112b, the extraction part 113b, the production | generation part 114b, and the transmission part 115b are the structures for transmitting synthetic | combination moving image data to the user terminal 300b, respectively. For example, the first receiving unit 111a and the first receiving unit 111b differ only in the terminal that receives the moving image data, and there is no difference in the operation itself. Similarly, the combinations of the second receiving units 112a and 112b, the extracting units 113a and 113b, the generating units 114a and 114b, and the transmitting units 115a and 115b differ only in the data to be processed and are different in the executable processing. There is no.

  The first receiving units 111a and 111b are means for receiving moving image data shot by the first shooting unit 340. That is, the first receiving units 111a and 111b selectively acquire the moving image data shot by the first shooting unit 340 among the data received via the communication unit 130. The first receiving unit 111a acquires the moving image data from the user terminal 300b, and the first receiving unit 111b acquires the moving image data from the user terminal 300a. The moving image data received by the first receiving units 111a and 111b is moving image data including distance information.

  The second receiving units 112 a and 112 b are means for receiving the moving image data shot by the second shooting unit 350. That is, the second receiving units 112a and 112b selectively acquire the moving image data shot by the second shooting unit 350 among the data received via the communication unit 130. The second receiving unit 112a acquires the moving image data from the user terminal 300a, and the second receiving unit 112b acquires the moving image data from the user terminal 300b. The moving image data received by the second receiving units 112a and 112b is moving image data including at least a marker in the video and including information on a three-dimensional coordinate system defined by the marker.

  In the following, for convenience of explanation, the moving image data shot by the first shooting unit 340 is referred to as “first moving image data”, and the moving image data shot by the second shooting unit 350 is referred to as “second moving image data”. That is, the first moving image data includes distance information, and the second moving image data includes an image of a marker (which can specify a coordinate system).

  The extraction units 113a and 113b are means for extracting pixels corresponding to a predetermined object from the first moving image data received by the first reception units 111a and 111b. Based on the distance information of the first moving image data, the extraction units 113a and 113b extract a pixel whose distance from the sensor satisfies a predetermined condition, and specifies the color information and distance information of the pixel. Here, the predetermined condition is determined by a threshold set for the distance from the sensor. As the threshold value, only the upper limit of the distance may be set, or both the upper limit and the lower limit may be set.

  The generating units 114a and 114b generate synthetic moving image data based on the first moving image data received by the first receiving units 111a and 111b and the second moving image data received by the second receiving units 112a and 112b. It is. The generation units 114a and 114b detect a marker in the background from the video represented by the second moving image data received by the second reception units 112a and 112b, and based on the detected marker, the three-dimensional coordinate system of the background is detected. The image corresponding to the pixel identified and extracted by the extraction units 113a and 113b is combined with the background so as to correspond to the coordinate system. In the present embodiment, the generation units 114 a and 114 b generate synthetic moving image data using the coordinate system calculated by the calculation unit 311 of the user terminal 300.

  The generation units 114a and 114b generate the combined moving image data so that the coordinate system of the video represented by the first moving image data corresponds to the coordinate system of the video represented by the second moving image data. Specifically, when the generation units 114a and 114b define the x-axis and the y-axis based on the pixel arrangement and the z-axis in the direction of the distance information for the first moving image data, the three-dimensional orthogonal coordinate system Can be defined. Then, the generation units 114a and 114b synthesize these videos so as to associate the coordinate system defined by the marker with the coordinate system defined in the first moving image data for the second moving image data. Once the generating units 114a and 114b associate the two coordinate systems once, the generating units 114a and 114b continue to synthesize the video so that the association is maintained. At this time, the generation units 114a and 114b align these images by performing synthesis so that a certain coordinate in one coordinate system matches a certain coordinate in the other coordinate system.

  The generation units 114a and 114b may perform image processing such as timing adjustment and effects when combining these videos. For example, when there is a time difference between the reception timings of the first moving image data and the second moving image data, the generation units 114a and 114b adjust the timing at the time of synthesis so that the time difference is reduced. The image processing here may be processing for superimposing some kind of image on the video, or processing for matching the brightness and color of the two videos to be combined.

  The transmission units 115a and 115b are means for transmitting the combined moving image data generated by the generation units 114a and 114b to the user terminals 300a and 300b. The transmission unit 115a transmits the composite video data to the user terminal 300a, and the transmission unit 115b transmits the composite video data to the user terminal 300b. By doing in this way, the user of user terminal 300a, 300b synthesize | combined the other user's object (a face, upper body, etc.) with the 2nd moving image data (moving image data which image | photographed the marker) which he transmitted. You will be able to watch videos.

  The synchronization control unit 110b is a means for executing processing for synchronizing the reproduction of the synthesized moving image data and the reproduction of the voice message. For example, the synchronization control unit 110b can change the compression method of the combined moving image data so as to follow the change in the line information. The synchronization control unit 110b can also supply the connection management device 200 with data necessary for the transmission / reception unit 211 to reproduce the voice message in accordance with the synthesized moving image data. More specifically, the synchronization control unit 110b includes a data exchange unit 116, a determination unit 117, and an analysis unit 118.

  The data transmission / reception unit 116 is a means for exchanging data with the connection management device 200. The data transfer unit 116 acquires line information from the connection management apparatus 200 and supplies resource information to the connection management apparatus 200.

  The determining unit 117 is a unit that determines, for each terminal, whether or not the user terminal 300a and the user terminal 300b can perform communication using the composite moving image data based on the line information or the resource information. For example, when one communication line of the user terminal 300a and the user terminal 300b does not satisfy the quality necessary for real-time reproduction of the composite video data, the synthesis control unit 110a generates the composite video data for the one. Instead, the process of generating and transmitting only the other combined moving image data can be performed based on the determination result by the determination unit 117. Further, based on the determination result based on the resource information, the composition control unit 110a can also prevent the composite moving image data from being generated when the user terminal 300 does not include the first photographing unit 340.

  The analysis unit 118 is a unit that analyzes changes in the subject in the first moving image data received by the first reception units 111a and 111b and generates information representing the analysis result. The synchronization control unit 110b supplies the information generated by the analysis unit 118 to the connection management apparatus 200. For example, the analysis unit 118 analyzes the movement of the user who is the subject. Alternatively, the analysis unit 118 may analyze changes in the facial expression of the user who is the subject using a known face recognition technique (for example, a technique for detecting a smile). Further, the analysis unit 118 may obtain a change in the entire subject (for example, a change in brightness) by analysis, not limited to a change in the user.

  The configuration of the communication system 10 is as described above. Under this configuration, the user communicates with other users in real time using the user terminal 300. The communication here uses audio and video, and is, for example, a so-called videophone. However, the communication of the present embodiment is different from the conventional videophone in that an image synthesized by the AR technology can be used.

  In order to realize such communication, the user terminal 300 captures and transmits two types of moving image data with different subjects. One of the moving image data is the first moving image data described above, and is obtained by the user photographing himself / herself. Another moving image data is the above-described second moving image data, which is obtained by a user shooting a marker.

  The AR moving image generating apparatus 100 receives these moving image data, and generates and transmits composite moving image data. The AR moving image generating apparatus 100 is synthesized using the second moving image data transmitted from one user terminal 300 and including the user as a subject, and the first moving image data transmitted from the other user terminal 300 and including the marker as the subject. Video data is generated and transmitted to the one user terminal 300. That is, the AR moving image generating apparatus 100 receives a moving image showing a marker from a certain user, and uses this as a background to synthesize an image showing another user here. At this time, the AR moving image generating apparatus 100 extracts a predetermined object corresponding to the user from the video of the second moving image data, and superimposes it on the background.

  FIG. 5 is a schematic diagram for explaining a video represented by moving image data transmitted and received in the present embodiment. Here, the videos V1a, V2a, and V3a are videos represented by moving picture data transmitted and received by the user terminal 300a, and the videos V1b, V2b, and V3b are videos represented by moving picture data transmitted and received by the user terminal 300b, respectively. Suppose that The videos V1a and V1b correspond to the first moving image data, the videos V2a and V2b correspond to the second moving image data, and the videos V3a and V3b correspond to the combined moving image data, respectively.

  In FIG. 5, Ma and Mb indicate markers. The markers Ma and Mb are small pieces formed by printing an image of a predetermined shape by printing or the like, and can be arranged at a place the user likes. The image formed on the markers Ma and Mb is any pattern image as long as it can be easily distinguished from other background parts and can be easily specified in the coordinate system and the inclination. May be.

  As shown in FIG. 5, the video V3a displayed on the user terminal 300a is obtained by synthesizing the first moving image data (V1b) from the user terminal 300b and the second moving image data (V2a) from the user terminal 300a. It is a picture. On the other hand, the video V3b displayed on the user terminal 300b is a video obtained by combining the first video data (V1a) from the user terminal 300a and the second video data (V2b) from the user terminal 300b.

  As shown in FIG. 5, the object (in this case, the user's upper body) to be combined with the videos V3a and V3b does not need to have the same magnification as the objects included in the videos V1a and V1b. It may be done as appropriate. In addition, the object combined with the videos V3a and V3b is a three-dimensional image (polygon) in which unevenness is expressed based on the distance information and colored based on the color information. May be.

  FIG. 6 is a diagram for explaining the principle of object extraction. The distance information obtained by the distance image sensor is obtained for each pixel, as indicated by the dashed arrows in the figure. Here, if a threshold indicated by Th in the figure is set and pixels whose distance represented by the distance information is smaller than this threshold are extracted, the portion of the subject corresponding to the user is selected. Thus, an object excluding a part other than the user (such as a wall behind the user) is obtained.

  This threshold value is set in advance. For example, the user may have a conversation within a place within 1 m from the distance image sensor, or the user may be able to select from a plurality of options. . Alternatively, the threshold value may be dynamically set by the user terminal 300. For example, the user terminal 300 can estimate the user's position (such as a face) based on the color information and set a threshold based on the estimation result.

  The user can communicate with a partner as if he / she is beside him / herself (position taken as a background) by talking with the partner while watching the synthesized video. Thereby, the user can perform realistic communication in real time.

  In the communication system 10, the AR video generation device 100 generates and transmits composite video data so that such video can be displayed on the user terminal 300. The AR moving image generating apparatus 100 refers to resource information and line information and generates necessary synthesized moving image data. For example, the AR moving image generating apparatus 100 determines the compression method and transfer rate of the combined moving image data according to the resource of the user terminal 300 and the state of the communication line. Further, the AR moving image generating apparatus 100 can determine whether or not to generate the combined moving image data based on the resource information and the line information. For example, the AR video generation device 100 may generate the composite video data when a transfer band sufficient for transmission of the composite video data is not secured or when the user terminal 300 does not have a function of reproducing the composite video data in the first place. Can be transmitted, or alternative data having a data amount smaller than that of the combined moving image data can be transmitted.

  FIG. 7 is a flowchart showing processing executed when the AR moving image generating apparatus 100 generates composite moving image data. First, the control unit 110 of the AR moving image generating apparatus 100 determines the generation mode of the combined moving image data using at least one of the resource information and the line information (Step Sa1). At this time, the control unit 110 determines whether or not the composite moving image data can be generated, the compression method and the transfer rate when generating the combined moving image data, for each user terminal 300 that is the transmission target of the combined moving image data.

  Then, the control part 110 performs the process according to the determination result of step Sa1. The control unit 110 determines whether or not to generate the combined moving image data for the user terminal 300a that is one user terminal 300 (step Sa2), and generates the combined moving image data when it is determined to generate the combined moving image data. On the other hand, generation is not performed otherwise (step Sa3). Further, when generating the composite moving image data, the control unit 110 varies the compression method, the transfer rate, and the like of the composite moving image data according to the determination result of Step Sa1 (that is, according to the line information and resource information).

  Subsequently, similarly to the user terminal 300b, which is the other user terminal 300, the control unit 110 determines whether or not to generate the combined moving image data (Step Sa4) and generates the combined moving image data (Step Sa5). Run. And the control part 110 transmits synthetic | combination moving image data to each of user terminal 300a, 300b by supplying the produced | generated synthetic | combination moving image data to the communication part 130 (step Sa6).

  The AR moving image generating apparatus 100 can generate composite moving image data having a quality corresponding to each of the user terminal 300a and the user terminal 300b by executing such processing. Further, the AR moving image generating apparatus 100 can omit the generation of the composite moving image data that does not need to be transmitted, and can proceed with the processing by efficiently using the resource of the own apparatus.

  FIG. 8 is a sequence chart showing processing in each device of the communication system 10. The example shown in FIG. 8 is a case where the composite moving image data is transmitted to both the user terminals 300a and 300b, that is, the case where both determinations in steps Sa2 and Sa4 in FIG. 7 are “YES”. Also, in FIG. 8, for convenience of illustration, two AR video generation devices 100 are shown, and a configuration for transmitting the composite video data to the user terminal 300a and a configuration for transmitting the composite video data to the user terminal 300b. In practice, however, the same apparatus may actually perform processing in parallel.

  First, the user terminal 300a and the user terminal 300b establish a call connection via the connection management device 20 (step Sb1). For example, the user can designate a communication partner using a telephone number or an alternative (such as a user account) and start speaking.

  When the call connection is established, the connection management device 20 transmits connection information to the user terminals 300a and 300b (step Sb2). The connection information is information that is transmitted when the user terminal 300a and the user terminal 300b are connected, and includes information related to a communication partner. For example, the connection information includes call identification information used for identifying a call within the system after the call connection. Further, the connection information may include the above-described line information and resource information. When line information or resource information is included in the connection information, the user terminal 300 may determine whether to transmit the first moving image data or the second moving image data based on the connection information. In this way, a part of the determination shown in FIG.

  In addition, when the call connection is established, the user terminals 300a and 300b start capturing video (step Sb3). That is, the user terminals 300a and 300b activate the first photographing unit 340 and the second photographing unit 350 so that moving image data can be transmitted. Then, the user terminals 300a and 300b transmit the first moving image data and the second moving image data to the AR moving image generating device 100, respectively (steps Sb4 and Sb5). When receiving the first moving image data and the second moving image data, the AR moving image generating apparatus 100 generates the combined moving image data (step Sb6) and transmits it to the user terminals 300a and 300b (step Sb7).

  As described above, according to the present embodiment, it is possible to synthesize the video of the other user with the background photographed by one user and exchange voice messages using the synthesized video. In the present embodiment, since the AR moving image generation apparatus 100 performs combining of moving image data, it is not necessary to execute a relatively high load process such as polygon data rendering in the user terminal 300.

  In the present embodiment, the AR moving image generating device 100 can change the generation mode of the combined moving image data according to the resource information and the line information. As a result, the AR moving image generating apparatus 100 can perform data transfer in a mode in which there is less risk of delay, or can omit performing unnecessary processing. In addition, in the AR video generation device 100, if only one user terminal 300 includes a distance image sensor and the other user terminal 300 does not include a distance image sensor, the AR moving image generation apparatus 100 transmits the other user terminal 300 to the other user terminal 300. Only synthetic video data can be generated, and communication in such a manner can be realized.

[Modification]
The present invention is not limited to the aspect of the embodiment described above, and can be implemented in other aspects. The following is an example of another embodiment of the present invention. In addition, you may combine these modifications suitably as needed.

(1) The present invention is applicable not only to two parties but also to communication of three or more parties. In this case, the user terminal 300 of each user may be provided with the distance image sensor, but a mode in which only one specific user photographs itself with the distance image sensor is also possible. For example, when three users, a first user, a second user, and a third user, communicate with each other, only the first user takes a picture with the distance image sensor, and the second user and the third user only take the background. It is also possible to communicate in a manner such as shooting. In this case, the AR moving image generating apparatus 100 extracts an image corresponding to the photographed first user, combines this with the second moving image data photographed on the second user side, and photographed on the third user side. A process of synthesizing the generated second moving image data is executed to generate combined moving image data corresponding to each of the second user and the third user.

In this way, the second user can view the video in which the first user's video is superimposed on the background he / she is shooting, while the third user can take the video. A video in which the video of the first user is superimposed on the background (that is, a video in which the video of the first user is superimposed but is different from the video that the second user is viewing) can be viewed.
In such an aspect, the user terminal 300 of the first user may not include the second photographing unit 350, and the user terminal 300 of the second user and the third user may include the first photographing unit 340. It does not have to be provided.

(2) As described above, in the communication system 10, the video of the synthesized video data is changed according to the change in the volume or the sound quality of the voice message, or the voice message is changed according to the change in the video of the first video data. Volume or sound quality can be changed. Specific examples include the following.

  For example, when the AR video generation device 100 combines the extracted objects with respect to the composite video data to be transmitted to the user terminal 300a, when the volume of the voice message transmitted from the user terminal 300b increases, While the object is enlarged, the object may be reduced when the volume of the voice message decreases. In addition, when the sound quality deteriorates due to noise generated in the voice message, the AR moving image generating apparatus 100 adds (intentional) noise to the video represented by the combined moving image data or deteriorates the image quality. May be. In this way, it is possible to give a sensory link between the synthesized video and audio.

  In addition, the connection management device 200 changes the size of the extracted object, changes in the distance information, or changes in the brightness of the entire subject based on the information representing the change in the subject from the AR video generation device 100. Can be recognized. The connection management device 200 changes the sound volume according to the change in the size of the object and the distance, for example, according to the change in the image recognized in this way, and the sound quality depending on whether the subject is bright or dark. It may be different. Further, the connection management apparatus 200 may vary the sound quality and volume according to changes in the facial expression of the user who is the subject. Further, if the voice message is multi-channel (for example, stereo) data, the connection management apparatus 200 controls the sound image localization according to the change in the position where the predetermined object is extracted, for example, the user who is the subject. The volume and delay may be adjusted so as to follow the movement.

(3) A function corresponding to the calculation unit 311 may be included in the AR moving image generation device 100 instead of the user terminal 300. That is, the AR moving image generating apparatus 100 may be configured to calculate a coordinate system based on moving image data transmitted from the user terminal 300. For example, the second reception units 112a and 112b may have a function corresponding to the calculation unit 311.

(4) The AR moving image generating apparatus 100 can synthesize the extracted object by rotating it according to the movement of the user terminal 300 (more specifically, the movement of the second photographing unit 350). However, the distance image sensor can shoot only the side (front side) where the user faces the sensor, and cannot shoot the opposite side (back side). Therefore, a portion of the extracted object that cannot be photographed by the distance image sensor may not be displayed or may be displayed as a suitable alternative video image that has been determined in advance.

  Note that the rotation of the object may be performed by a user operation. That is, the AR moving image generating apparatus 100 can receive a user operation, rotate an object in accordance with the received operation, and then synthesize it with a background image to generate combined moving image data.

(5) The AR moving image generating apparatus 100 can add a predetermined image to the extracted object and synthesize it with the background. The predetermined image referred to here is, for example, an image (sunglasses, clothes, etc.) imitating an ornament worn on the user's face or body. In this way, it is possible to display an image in which a part of the user's face or body is hidden. Alternatively, the AR moving image generating apparatus 100 may perform synthesis by replacing the extracted object with a predetermined image (such as an avatar). The AR video generation device 100 may determine when the second video data has not been transmitted or when there is a delay in the transfer of the second video data, and in such a case, replacement may be performed. . Note that image data for displaying these images may be stored in advance in the AR video generation device 100 or the connection management device 200, or transmitted to the connection management device 200 when the user terminal 300 starts communication. May be.

(6) The connection information may include the image data shown in the modified example (4) described above. In addition, the connection information may include attribute information (gender, hobby) set in advance by the user. In the modified example (4), the AR video generation device 100 determines an image to be displayed based on the attribute information when replacing the extracted object with a predetermined image or adding a predetermined image to the object. Also good.

(7) The size of the pixel of color information and the size of the pixel of distance information, that is, the resolutions of both do not necessarily match. For example, one pixel of distance information may correspond to four pixels of color information (vertical 2 pixels × horizontal 2 pixels). The distance information in this case may be the same value for the four pixels of the color information, but may be calculated by executing appropriate interpolation processing with reference to the distance information of adjacent pixels.

(8) The object extracted in the present invention may be any object as long as the distance information satisfies a predetermined condition, and is not necessarily a user. For example, the user can take a picture of a doll or a reared pet instead of himself and use it as a composition target video.

(9) The AR video generation device 100 and the connection management device 200 may be configured integrally instead of being configured as separate devices. Further, the present invention may replace text messages instead of voice messages. In this way, the present invention can be applied to communication such as chat. In this case, the AR moving image generating apparatus 100 replaces the volume and the sound quality, according to the character size and character decoration (font, underline, color, etc.), or according to the presence or absence of a specific character (exclamation mark, etc.). It is also possible to make the images different.

(10) The present invention can be understood not only as an AR moving image generating device and an image communication system including the AR moving image generating device, but also as a communication method using a program executed by the AR moving image generating device or a synthesized moving image. The program can also be provided in a form recorded on a recording medium such as an optical disk, or a form that is downloaded to a computer via a network such as the Internet, and can be installed and used.

DESCRIPTION OF SYMBOLS 10 ... Communication system, 100 ... AR animation production | generation apparatus, 110 ... Control part, 111a, 111b ... 1st reception part, 112a, 112b ... 2nd reception part, 113a, 113b ... Extraction part, 114a, 114b ... production | generation part, 115a , 115b ... transmission unit, 116 ... data exchange unit, 117 ... determination unit, 118 ... analysis unit, storage unit ... 120, communication unit ... 130, 200 ... connection management device, 210 ... control unit, 211 ... transmission / reception unit, 212 ... Data transmission / reception unit, 220 ... storage unit, 130 ... communication unit, 300, 300a, 300b ... user terminal, 310 ... control unit, 320 ... storage unit, 330 ... communication unit, 340 ... first imaging unit, 350 ... second imaging Part, 360 ... display part, 370 ... voice input / output part, 380 ... operation part, 400 ... network

Claims (15)

A connection management device for managing the connection between the first terminal and the second terminal;
An AR (Augmented Reality) video generation device that transmits composite video data obtained by synthesizing videos captured by both to the at least one of the first terminal and the second terminal whose connection is managed by the connection management device; Have
The connection management device includes:
Supplying line information regarding a communication line connecting the first terminal and the second terminal to the AR video generation device;
The AR video generation device
A first moving image data obtained by photographing a subject including a predetermined object from one of the first terminal and the second terminal, and receiving first moving image data including distance information for each pixel of the subject. A receiver,
A second receiving unit for receiving second moving image data obtained by capturing a background including a marker having a predetermined shape from the other of the first terminal and the second terminal;
An extraction unit that extracts pixels corresponding to the predetermined object from the subject based on the distance information of the first moving image data received by the first reception unit;
An image corresponding to the pixel extracted by the extraction unit by specifying a three-dimensional coordinate system of the background defined based on the marker in the background represented by the second moving image data received by the second reception unit A generating unit that generates synthesized moving image data synthesized with the background so as to correspond to the coordinate system;
A transmission unit that transmits the synthesized moving image data generated by the generation unit to the other terminal ,
The connection management apparatus according to channel information supplied by the image communication system Ru at different image represented by the composite video data. A connection management device for managing the connection between the first terminal and the second terminal;
An AR (Augmented Reality) video generation device that transmits composite video data obtained by synthesizing videos captured by both to the at least one of the first terminal and the second terminal whose connection is managed by the connection management device; Have
The connection management device includes:
Supplying resource information related to hardware or software resources in the first terminal and the second terminal to the AR video generation device;
The AR video generation device
A first moving image data obtained by photographing a subject including a predetermined object from one of the first terminal and the second terminal, and receiving first moving image data including distance information for each pixel of the subject. A receiver,
A second receiving unit for receiving second moving image data obtained by capturing a background including a marker having a predetermined shape from the other of the first terminal and the second terminal;
An extraction unit that extracts pixels corresponding to the predetermined object from the subject based on the distance information of the first moving image data received by the first reception unit;
An image corresponding to the pixel extracted by the extraction unit by specifying a three-dimensional coordinate system of the background defined based on the marker in the background represented by the second moving image data received by the second reception unit A generating unit that generates synthesized moving image data synthesized with the background so as to correspond to the coordinate system;
A transmission unit that transmits the synthesized moving image data generated by the generation unit to the other terminal ,
The connection management apparatus according to the resource information supplied by an image communication system in which Ru at different image represented by the composite video data. The AR video generation device
On the basis of the line information includes a determination unit for determining for each of the whether it is possible to communicate by said synthesized video data and the first terminal the second terminal,
Image communication system according to claim 1 wherein the terminal is determined to communications are possible by combining video data, generating and transmitting the composite video data by the determining unit. The AR video generation device
Before Based on cut source information includes a determination unit for determining for each of the whether it is possible to communicate by said synthesized video data and the first terminal the second terminal,
The image communication system according to claim 2 , wherein the composite moving image data is generated and transmitted with respect to a terminal that is determined by the determination unit to be able to communicate with the composite moving image data. A connection management device for managing the connection between the first terminal and the second terminal;
An AR (Augmented Reality) video generation device that transmits composite video data obtained by synthesizing videos captured by both to the at least one of the first terminal and the second terminal whose connection is managed by the connection management device; Have
The connection management device includes:
A transmission / reception unit that transmits / receives a voice message between the first terminal and the second terminal is provided, and information indicating a change in volume or sound quality of the voice message transmitted / received by the transmission / reception unit is supplied to the AR video generation device. And
The AR video generation device
A first moving image data obtained by photographing a subject including a predetermined object from one of the first terminal and the second terminal, and receiving first moving image data including distance information for each pixel of the subject. A receiver,
A second receiving unit for receiving second moving image data obtained by capturing a background including a marker having a predetermined shape from the other of the first terminal and the second terminal;
An extraction unit that extracts pixels corresponding to the predetermined object from the subject based on the distance information of the first moving image data received by the first reception unit;
An image corresponding to the pixel extracted by the extraction unit by specifying a three-dimensional coordinate system of the background defined based on the marker in the background represented by the second moving image data received by the second reception unit A generating unit that generates synthesized moving image data synthesized with the background so as to correspond to the coordinate system;
A transmission unit that transmits the synthesized moving image data generated by the generation unit to the other terminal ,
Image communication system in accordance with the information indicating the change of the supplied volume or sound quality, Ru with different image represented by the composite video data by the connection management apparatus. A connection management device for managing the connection between the first terminal and the second terminal;
An AR (Augmented Reality) video generation device that transmits composite video data obtained by synthesizing videos captured by both to the at least one of the first terminal and the second terminal whose connection is managed by the connection management device; Have
The AR video generation device
A first moving image data obtained by photographing a subject including a predetermined object from one of the first terminal and the second terminal, and receiving first moving image data including distance information for each pixel of the subject. A receiver,
A second receiving unit for receiving second moving image data obtained by capturing a background including a marker having a predetermined shape from the other of the first terminal and the second terminal;
An extraction unit that extracts pixels corresponding to the predetermined object from the subject based on the distance information of the first moving image data received by the first reception unit;
An image corresponding to the pixel extracted by the extraction unit by specifying a three-dimensional coordinate system of the background defined based on the marker in the background represented by the second moving image data received by the second reception unit A generating unit that generates synthesized moving image data synthesized with the background so as to correspond to the coordinate system;
A transmission unit for transmitting the composite video data generated by the generation unit to the other terminal ;
An analysis unit that analyzes changes in the subject in the first moving image data received by the first reception unit;
Supplying information representing the change in the subject analyzed by the analysis unit to the connection management device;
The connection management device includes:
A transmission / reception unit for transmitting / receiving a voice message between the first terminal and the second terminal, wherein the voice message to be transmitted / received according to information representing a change in the subject supplied by the AR video generation device; An image communication system including a transmission / reception unit that varies sound volume or sound quality . The AR video generation device
Analyzing a second moving image data received by the second receiving unit, and calculating a coordinate system;
The generator is
The image communication system according to any one of claims 1 to 6, wherein the synthesized moving image data is generated based on the coordinate system calculated by the calculation unit. A first receiver that receives first moving image data of a subject including a predetermined object, the first moving image data including distance information for each pixel of the subject;
A second receiving unit that receives second moving image data in which a background including a marker having a predetermined shape is captured, wherein the second moving image data defines a three-dimensional coordinate system based on the marker;
An extraction unit that extracts pixels corresponding to the predetermined object from the subject based on the distance information of the first moving image data received by the first reception unit;
Generates synthesized moving image data in which an image corresponding to the pixel extracted by the extracting unit is combined with the background so as to correspond to the coordinate system defined in the second moving image data received by the second receiving unit. A generator,
A transmission unit that transmits the composite video data generated by the generation unit ,
A terminal that transmits the first video data and a terminal that transmits the second video data from a connection management device that manages a connection between the terminal that transmits the first video data and the terminal that transmits the second video data; line information about the communication line for connecting is supplied, in accordance with a supplied the line information, the combined moving image data AR moving image generating apparatus of Ru with different image represents. A first receiver that receives first moving image data of a subject including a predetermined object, the first moving image data including distance information for each pixel of the subject;
A second receiving unit that receives second moving image data in which a background including a marker having a predetermined shape is captured, wherein the second moving image data defines a three-dimensional coordinate system based on the marker;
An extraction unit that extracts pixels corresponding to the predetermined object from the subject based on the distance information of the first moving image data received by the first reception unit;
Generates synthesized moving image data in which an image corresponding to the pixel extracted by the extracting unit is combined with the background so as to correspond to the coordinate system defined in the second moving image data received by the second receiving unit. A generator,
A transmission unit that transmits the composite video data generated by the generation unit ,
Resource information related to hardware or software resources in the terminal is supplied from a connection management device that connects the terminal that transmitted the first video data and the terminal that transmitted the second video data, and the supplied resource information Correspondingly, the combined moving image data AR moving image generating apparatus of Ru with different images represented by the. A first receiver that receives first moving image data of a subject including a predetermined object, the first moving image data including distance information for each pixel of the subject;
A second receiving unit that receives second moving image data in which a background including a marker having a predetermined shape is captured, wherein the second moving image data defines a three-dimensional coordinate system based on the marker;
An extraction unit that extracts pixels corresponding to the predetermined object from the subject based on the distance information of the first moving image data received by the first reception unit;
Generates synthesized moving image data in which an image corresponding to the pixel extracted by the extracting unit is combined with the background so as to correspond to the coordinate system defined in the second moving image data received by the second receiving unit. A generator,
A transmission unit that transmits the composite video data generated by the generation unit ,
A transmission / reception unit that transmits / receives a voice message between the terminal that has transmitted the first video data and the terminal that has transmitted the second video data, and changes in volume or sound quality of the voice message transmitted / received by the transmission / reception unit; Information representing a change in volume or sound quality of a voice message transmitted / received by the transmission / reception unit is supplied from a connection management device that supplies information to be represented, and the video represented by the composite video data is made different according to the supplied information. AR video generator. A first receiver that receives first moving image data of a subject including a predetermined object, the first moving image data including distance information for each pixel of the subject;
A second receiving unit that receives second moving image data in which a background including a marker having a predetermined shape is captured, wherein the second moving image data defines a three-dimensional coordinate system based on the marker;
An extraction unit that extracts pixels corresponding to the predetermined object from the subject based on the distance information of the first moving image data received by the first reception unit;
Generates synthesized moving image data in which an image corresponding to the pixel extracted by the extracting unit is combined with the background so as to correspond to the coordinate system defined in the second moving image data received by the second receiving unit. A generator,
A transmission unit for transmitting the composite video data generated by the generation unit ;
An analysis unit that analyzes changes in the subject in the first moving image data received by the first reception unit;
A transmission / reception unit that transmits and receives a voice message between a terminal that has transmitted the first moving image data and a terminal that has transmitted the second moving image data, and transmits and receives the message according to information representing a change in the supplied subject. An AR video generation device that supplies information representing a change in the subject analyzed by the analysis unit to a connection management device including a transmission / reception unit that varies the volume or sound quality of the voice message . On the computer,
A first step of receiving first moving image data obtained by photographing a subject including a predetermined object, the first moving image data including distance information for each pixel of the subject;
A second step of receiving second moving image data in which a background including a marker having a predetermined shape is captured, wherein the second moving image data defines a three-dimensional coordinate system based on the marker;
A third step of extracting a pixel corresponding to the predetermined object from the subject based on the distance information of the first moving image data received in the first step;
The synthesized moving image data is generated by combining the image corresponding to the pixel extracted in the third step with the background so as to correspond to the coordinate system defined in the second moving image data received in the second step. The fourth step;
A fifth step of transmitting the composite video data generated by the fourth step ;
A terminal that transmits the first video data and a terminal that transmits the second video data from a connection management device that manages a connection between the terminal that transmits the first video data and the terminal that transmits the second video data; A program for executing the sixth step of supplying line information related to the communication line connecting the video and differentiating the video represented by the composite video data according to the supplied line information . On the computer,
A first step of receiving first moving image data obtained by photographing a subject including a predetermined object, the first moving image data including distance information for each pixel of the subject;
A second step of receiving second moving image data in which a background including a marker having a predetermined shape is captured, wherein the second moving image data defines a three-dimensional coordinate system based on the marker;
A third step of extracting a pixel corresponding to the predetermined object from the subject based on the distance information of the first moving image data received in the first step;
The synthesized moving image data is generated by combining the image corresponding to the pixel extracted in the third step with the background so as to correspond to the coordinate system defined in the second moving image data received in the second step. The fourth step;
A fifth step of transmitting the composite video data generated by the fourth step ;
Resource information related to hardware or software resources in the terminal is supplied from a connection management device that connects the terminal that transmitted the first video data and the terminal that transmitted the second video data, and the supplied resource information And a sixth step of executing the sixth step of making the video represented by the combined video data different . On the computer,
A first step of receiving first moving image data obtained by photographing a subject including a predetermined object, the first moving image data including distance information for each pixel of the subject;
A second step of receiving second moving image data in which a background including a marker having a predetermined shape is captured, wherein the second moving image data defines a three-dimensional coordinate system based on the marker;
A third step of extracting a pixel corresponding to the predetermined object from the subject based on the distance information of the first moving image data received in the first step;
The synthesized moving image data is generated by combining the image corresponding to the pixel extracted in the third step with the background so as to correspond to the coordinate system defined in the second moving image data received in the second step. The fourth step;
A fifth step of transmitting the composite video data generated by the fourth step ;
A transmission / reception unit that transmits / receives a voice message between the terminal that has transmitted the first video data and the terminal that has transmitted the second video data, and changes in volume or sound quality of the voice message transmitted / received by the transmission / reception unit; Information representing a change in volume or sound quality of a voice message transmitted / received by the transmission / reception unit is supplied from a connection management device that supplies information to be represented, and the video represented by the composite video data is made different according to the supplied information. A program for executing the sixth step . On the computer,
A first step of receiving first moving image data obtained by photographing a subject including a predetermined object, the first moving image data including distance information for each pixel of the subject;
A second step of receiving second moving image data in which a background including a marker having a predetermined shape is captured, wherein the second moving image data defines a three-dimensional coordinate system based on the marker;
A third step of extracting a pixel corresponding to the predetermined object from the subject based on the distance information of the first moving image data received in the first step;
The synthesized moving image data is generated by combining the image corresponding to the pixel extracted in the third step with the background so as to correspond to the coordinate system defined in the second moving image data received in the second step. The fourth step;
A fifth step of transmitting the composite video data generated by the fourth step ;
A sixth step of analyzing a change in the subject in the first moving image data received in the first step;
A transmission / reception unit that transmits and receives a voice message between a terminal that has transmitted the first moving image data and a terminal that has transmitted the second moving image data, and transmits and receives the message according to information representing a change in the supplied subject. A program for executing a seventh step of supplying information representing a change in the subject analyzed in the sixth step to a connection management device including a transmission / reception unit that varies the volume or sound quality of the voice message .

Images