JP6697512B2 - Content distribution system, content distribution method, and computer program - Google Patents

Description

The present disclosure, a content distribution system in which the person to be photographed, which is the reality of the performers and the virtual character (avatar) is to deliver live content, such as a public program to participate, a content distribution method, and relates to a computer program.

  Conventionally, in terrestrial digital broadcasting, BS digital broadcasting, Internet live broadcasting, etc. (hereinafter collectively referred to as "broadcast"), programs in which a plurality of performers appear in a scene are often broadcast.

  For example, in FIG. 7A, which schematically shows one scene of a broadcast program according to the related art, an interviewer 101 who receives an interview is located close to an interviewer 102 who interviews, and the interviewer 101 is And an interviewer 102 are included in one scene, a camera (not shown) photographs them to generate an interview scene 100, which is broadcast and viewed by a viewer. If the camera is closer to them or the lens of the camera is zoomed up to a longer focus, then the same interview bust shot 103 is obtained, as seen in FIG. 7B.

  In such an interview in broadcasting, the interviewer 101 and the interviewer 102 may not be in the same place. In that case, FIG. 8A schematically showing another scene of the broadcast program according to the related art. ) Like this, a scene is shot by the camera alone and the interviewee 107 appears at the shooting site, and the interviewer talks with the interviewee 107 in the studio of the broadcasting station, for example. This interviewer does not appear in the broadcast scene 106, and only the voice uttered by the interviewer is broadcast.

  Further, as shown in FIG. 8B, the interviewer 111 is also in a place different from the interviewee 110, for example, in the studio of the broadcasting station, but the interviewer 111 is photographed by another camera and the interviewer 110 is displayed. In some cases, the interview scene 108 is created by a picture-in-picture method, which is set as a child screen 109 inside the photographed main screen.

  In this way, the scene where multiple performers appear in the broadcast is not limited to interviews, but for example, when a reporter or announcer reports from the scene where the incident occurred, a scene where a talker talks with another announcer in the broadcasting station studio. There can be many singing programs, entertainment shows such as Manzai, and dramas.

  By the way, a game in which a virtual character (hereinafter also referred to as an "avatar") created by using CG (computer graphic) technology is drawn in a virtual three-dimensional space created by CG and is moved or moved. , Broadcasting and content are being actively produced.

  For example, in Patent Document 1 below, a participant who intends to perform an operation of moving an avatar in a virtual space acquires position information of the participant in the actual space by using a GPS (global positioning system) technology, There is disclosed a configuration in which is converted into position information in the virtual space and used for position operation of the avatar in the virtual space.

Japanese Patent No. 4637889

  Virtual characters (avatars), which are becoming more and more widely used in this way, are drawn not only in a virtual space, but in a real space, and are used as the real performers. The actors (actors, performers, passers-by who are interviewed, etc.) and avatars draw on the same screen, which makes it surprising and interesting that there are artificial characters that cannot be found in a real scene. Alternatively, a proposal has recently been made to make it possible to enjoy the fun of adding a sense of reality to an avatar that is supposed to be virtual and artificial.

  However, in a broadcast or content in which an image in the real space and an artificial avatar are combined and rendered in one scene in this way, the following essential problems have existed and are unsolved in the past. It was

  As described above, a plurality of real performers are photographed as one scene. For example, in each scene shown in FIGS. 7A and 7B, two appearing performers 101, 102, 104, Since 105 visually recognize each other and have a direct conversation with each other, the conversation in the scene is naturally natural and comfortable, and is natural to the viewer.

  However, assuming that a real-life performer in the real space co-stars with an artificial avatar and fits into one scene, the avatar does not actually exist in front of the real-life performer. The performer does not have a conversation partner in front of him, and he is forced to perform an act as if he were talking alone, and the actual performer could not decide the direction to throw his eyes, and the whole scene was unnatural. turn into.

  Therefore, as shown in FIG. 8B shown above, if the camera shoots only the actual performer and the avatar appears on the inset screen (picture in picture), the avatar actually exists at the same place. Does not force the real performer and the avatar to be drawn on the same screen, so the real performer acts as if he / she were talking to him even though there was no one to talk to in front of him. There is no need for unnaturalness because it is not necessary, but the scene is different from the one in which an artificial avatar is drawn in the real space.

  Therefore, as described above, it is surprising or interesting that there is an artificial character that can not be present in the real scene, which is caused when the real performer and the avatar are drawn on the same screen, or it is virtual. Viewers may not be able to fully enjoy the fun of adding realism to an avatar that should be artificial.

Therefore, in the present disclosure, for the reality of the performers and the artificial and Avatar is a (virtual) character is to eliminate the unnaturalness in a scene that appear together, the content distribution system, a content distribution method, and computer program The purpose is to provide.

In order to solve the above problems, the present invention provides a content distribution system according, content distribution method, a computer program to 1) to 11) below.
1)
A content distribution system for displaying and distributing a photographed person in one scene together with a virtual character,
A scene shooting camera unit that shoots a subject and a dummy object having a display for displaying an image of a virtual character so that the dummy object fits in one scene,
An image replacement unit that replaces the image of the dummy object in the scene that was shot with the image of the virtual character and delivers it.
And
The content distribution system, wherein the image of the virtual character that has been replaced in the image distributed by the image replacement unit is in a different mode from the image of the virtual character displayed on the display included in the dummy object.
2)
The different aspect is based on different viewpoints, The content distribution system according to 1).
3)
The content further includes a broadcast transmitting unit that broadcasts the content including the distribution image of the image replacing unit via a communication path, and a receiving unit that receives user posted information to be displayed on the screen of the content being broadcast, and the user posted information is The content distribution system according to 1) or 2), characterized in that the display for displaying the image of the virtual character is also displayed.
4)
A content distribution system for displaying and distributing a photographed person in one scene together with a virtual character,
A scene shooting camera unit that shoots a subject and a dummy object having a display for displaying an image of a virtual character so that the dummy object fits in one scene,
An image replacement unit that replaces the image of the dummy object in the scene that was shot with the image of the virtual character and delivers it.
And
The content distribution system, wherein the dummy object has a camera for photographing the photographer from a position different from the position of the scene photographing camera unit and / or a microphone for picking up the voice of the photographed person.
5)
An operation unit for changing the appearance display mode of the virtual character, and a display unit for operating the appearance display mode of the virtual character that speaks information collected by the camera and / or the microphone. ) Content distribution system described in.
6)
The operation unit is configured to change the appearance display mode of the virtual character displayed on the display and / or the appearance display mode of the image of the virtual character in the image distributed by the image replacement unit. The content distribution system according to 5).
7)
The content further includes a broadcast transmitting unit that broadcasts the content including the distribution image of the image replacing unit via a communication path, and a receiving unit that receives user posted information to be displayed on the screen of the content being broadcast, and the user posted information is The content distribution system according to any one of 4) to 6), characterized in that the display is also displayed on a display for displaying an image of the virtual character.
8)
The content further includes a broadcast transmitting unit that broadcasts the content including the distribution image of the image replacing unit via a communication path, and a receiving unit that receives user posted information to be displayed on the screen of the content being broadcast, and the user posted information is The content distribution system according to 5) or 6), wherein the virtual character appearance display mode operation display section is also configured to display.
9)
A content distribution method for displaying and distributing a photographed person in one scene together with a virtual character,
The step of the scene photographing camera section performing photographing so that the person to be photographed and the dummy object having the display for displaying the image of the virtual character fit into one scene,
The image replacing unit replaces the image of the dummy object in the scene in which the image was taken with the image of the virtual character and delivers the image.
And
The content distribution method, wherein the image of the virtual character that has been replaced in the image distributed by the image replacement unit is in a different mode from the image of the virtual character displayed on the display included in the dummy object.
10)
The content distribution method according to 9), wherein different aspects are based on different viewpoints.
11)
A computer program including a code group for a computer to execute each step in the content distribution method according to 9) or 10).

  According to the present disclosure, it is possible to eliminate unnaturalness in a scene in which a subject and an avatar, which is an artificial (virtual) character, appear together.

It is the figure which showed typically the structure of the content delivery receiving system concerning this invention. FIG. 2 is a block diagram of a content distribution server included in the content distribution / reception system of FIG. 1. FIG. 2 is a block diagram of an avatar changing unit included in the content delivery / reception system of FIG. 1. 3 is a flowchart showing a process in which the content delivery / reception system of FIG. 1 transmits and displays an image in which a photographed person and an avatar are drawn. 3 is a flowchart showing a process in which the content delivery / reception system of FIG. 1 draws an avatar on a display for viewing a person to be photographed. FIG. 2A is a schematic diagram of an image captured by a scene capturing camera included in the content delivery / reception system of FIG. 1. FIG. 2B is a schematic diagram showing an avatar image generated by the avatar changing unit included in the content delivery / reception system of FIG. 1. (C) FIG. 2 is a schematic diagram of a scene generated by a content distribution server included in the content distribution / reception system of FIG. 1 in which a person to be photographed and an avatar are stored on one screen. It is a schematic diagram of the scene in the broadcasting concerning a prior art. It is a schematic diagram of the scene in the broadcasting concerning a prior art.

  Hereinafter, embodiments according to the present disclosure will be described. The embodiments described below are merely examples, and various modifications can be made without departing from the spirit of the invention, and these are also included in the invention.

[Configuration of Content Delivery / Reception System 1]
FIG. 1 is a diagram schematically showing the configuration of a content delivery / reception system 1 according to the present invention. The content delivery / reception system 1 is an avatar operation provided with a motion capture system or the like for an avatar operator to operate a studio where a real performer is filmed, or a performer shooting site 2 which is a broadcast relay site, and the movement of an avatar. The following configurations are arranged in a studio 3, a distribution environment 4 of a content distributor that broadcasts a program, and a browsing environment 5 of a content viewer who is an end user who views a broadcast program.

  Note that the above-mentioned “actual actor” refers to an actual photographed person who is not a virtual character (avatar), and is not limited to performers who perform acting, but also includes, for example, a pass-by interview in a street interview. ..

  It is not essential for implementing the present invention that the respective components are arranged at different places in this way, and any or all of the respective components may be arranged at the same place.

  A performer 20 is present at the performer shooting site 2 to perform an action for a program, and a performer display stand unit 21 is arranged near the performer. In this performer display stand unit 21, a performer photographing camera 21a for photographing the performer 20 for use by the avatar operator 30 for visual recognition, picking up utterances of the performer 20 and making the avatar operator 30 also listen, Alternatively, a speaker 21c for use as a voice of the program, a speaker 21c for listening to the speaker for uttering a voice uttered by the avatar 30 or a command voice from a broadcast director (not shown), and an avatar image 21e. 20 is provided, and as a result, a performer visual display 21d that provides an environment in which the performer 20 is facing an artificial avatar, and a stand 21f that supports each component of the performer display stand unit 21 are provided. I have it.

  Further, a scene photographing camera 22 is arranged at the performer photographing site 2 and is used for photographing so that the performer 20 and the performer display stand unit 21 are contained in one screen (scene photographing camera photographing zone 23).

  Each signal system of the performer display stand unit 21 and the scene photographing camera 22 is data-connected to the content distribution server 40 via the Internet communication networks 60, 61, 62 and 66.

  The description and illustration of the connection nodes between the Internet lines 60, 61, 62, 66 are omitted. In carrying out the present invention, various lines and communication networks can be used for transmission of signals and data to be explained at various points, and the present invention is not limited to the Internet lines 60, 61, 62 and 66. This point is the same in the following description.

  As described above, the avatar operation studio 3 is a place for the avatar operator 30 to operate the movement of the avatar by using the motion capture technology. In FIG. 1, the optical marker 31 is the body of the avatar operator 30 as an example. , And the motion capture camera 32 photographs each of the optical markers 31, and the avatar changing unit 34 described later detects the position coordinates of each optical marker 31 in the three-dimensional space using the principle of triangulation. is doing.

  In addition to performing the above-mentioned optical marker as a motion capture technology and a camera corresponding thereto, a method of detecting the movement of the avatar operator 30 using an acceleration sensor and a method of detecting high frequency electromagnetic waves in the avatar operator are also used. It is possible to use various methods such as a method of detecting the movement of the body of the avatar operator 30 by the Doppler phenomenon based on the analysis result of the electromagnetic waves that irradiate 30 and are reflected and returned.

  An avatar operator display stand unit 33 is arranged in the avatar operation studio 3, and the avatar operator 30 is informed that the avatar operator display stand unit 33 blinks as needed to indicate that the avatar operator is currently on air. A tally lamp 33a, an avatar operator photographing microphone 33b for picking up the utterance voice of the avatar operator, an avatar operator listening speaker 33c uttering the utterance voice of the performer 20 or a program director (not shown), a performer Each of the configuration of the avatar operator visual display 33d and the avatar operator display stand unit 33 for displaying the image 33e of FIG. Each of them is provided with a stand 33f for supporting. A camera (not shown) that captures the facial expression of the avatar operator 30 is provided near the tally lamp 33a or at another position, and the facial expression of the avatar operator 30 is used for rendering the avatar. It may be configured as follows.

  The signal systems of the motion capture camera 32 and the avatar operator display stand unit 33 are in data connection with the avatar changing unit 34 via the Internet communication networks 63 and 64. The configuration of the avatar changing unit 34 will be described separately. The avatar changing unit 34 is in data connection with the content distribution server 40 via the Internet communication networks 60, 65, 66.

  Note that a part or all of the configuration of the avatar changing unit 34 may be included in the content distribution server 40. In that case, the parts described as the configuration, function, and processing of the avatar changing unit 34 are the configuration, function, and processing of the content distribution server 40.

  Further, a content distribution server 40 is arranged in the distribution environment 4 of the content distributor, and its configuration will be described separately.

  A general-purpose personal computer 50 is provided in the browsing environment 5 of the content viewer, and the content viewer (not shown) can enjoy the program drawn on the personal computer display 50a. As will be described later, the image of the performer 50b and the image of the avatar 50c are configured to be drawn together on the screen of the program. As a result, the content viewer can see the same screen as the actual performer and the avatar. By being drawn in, the unexpectedness and fun of having an artificial character that can not be in a real scene, or the fun of adding a realistic feeling to an avatar that should be virtual and artificial, etc. You can fully enjoy it.

[Configuration of avatar changing unit 34]
As shown in the block diagram of FIG. 3, the avatar changing unit 34 includes a control unit 340, which is a CPU that controls the entire unit 34, a memory 341, and a motion tracker that converts the movement of the avatar operator 30 into data by a motion capture technique. 343, avatar data in the memory 341 The current avatar posture is determined using data (image data) such as the appearance of the avatar and motion data of the avatar operator 30 generated by the motion tracker 343, which is stored in the memory 341b. It is provided with a rendering unit 344 for generating a CG image corresponding thereto, a communication interface 345 for transmitting / receiving communication between the inside and outside of the unit 34, etc., and these components are connected to each other via a data bus line 342.

  The program memory 341a included in the memory 341 is executed by the avatar changing unit 34 including an execution code group for the control unit 340 and the like to execute each operation including the operation disclosed in the embodiment described in the present specification. The computer program to be stored is readablely stored, and the temporary memory 341c is also used as a so-called cache memory.

[Structure of Content Delivery Server 40]
As shown in the block diagram of FIG. 2, the content distribution server 40 is for a performer included in an image captured by a control unit 41, which is a CPU that controls the entire server 40, a memory 42, and a scene capturing camera 22. In order to specify which area the image of the display stand unit 21 is, a scene photographing camera using the image of the presenter display stand unit 21 stored in the template image memory 42b in the memory 42 as a template (template). The image of the present avatar sent from the template image tracker 44 and the avatar changing unit 34, which is photographed by 22 and identifies the area using a technique such as a fitting method (statistical correlation matching) on the image, is used to identify the area. A content image generation unit 45 for generating a broadcast content image in which the performer 20 and the avatar are drawn together by superimposing on or replacing the created area, a communication interface 46 for transmitting and receiving data between the inside and outside of the server 40, The respective components are connected to each other via a data bus line 43.

  Further, the program memory 42a included in the memory 42 is a computer to be executed by the server 40 including an execution code group for the control unit 41 and the like to execute each operation including the operations disclosed in the embodiments described in the present specification. The program is readablely stored, and the temporary memory 42c is also used as a so-called cache memory.

  Further, as will be described later, the content distribution server 40 is a server that displays and distributes the photographed person together with a virtual character (avatar) in one scene, and the photographed person and the dummy object are included in one scene. It is configured to replace the image of the dummy object in the scene captured so as to fit it with the image of the virtual character and deliver the image. Further, the signals distributed by the content distribution server 40 are carried by the Internet communication networks 60, 66, 67, etc., which are transmission lines, and are transmitted via these respective transmission lines.

[Generation and Display Process of Content Image 73]
Next, an image 73 in which the performer 20 and the avatar are drawn together, which should be used for the content (broadcast) (hereinafter also referred to as “content image”, is a schematic diagram of FIG. 6C or the personal image in FIG. 1). An example is also shown in the computer display 50a), and the process until the browsing user (not shown) can browse will be described with reference to the flowchart of FIG. 4 and the schematic diagram of FIG. To do.

  First, the control unit 340 of the avatar changing unit 34 reads out the data of the avatar to be drawn on the content image (image data representing the appearance of the virtual character) stored in the avatar data memory 341b in the memory 341 (step S101). ). Next, the control unit 340 controls the motion tracker 343 and obtains the movement of the avatar operator 30 by the motion capture technique described above, and converts it into data (step S102).

  Next, the control unit 340 causes the rendering unit 344 to perform rendering to generate a 3D drawn image of the avatar using the obtained motion capture data and the avatar data (step S103). In rendering, it is important from which position the avatar image viewed is generated. Therefore, the rendering unit 344 preliminarily informs the position of this viewpoint from the content distribution server 40, specifically, the performer shooting site in FIG. 2 is information on the position of the scene photographing camera 22, and information on the relative position of the performer display stand unit 21, which is an object to be replaced with the image of the avatar, is stored as described later. ing. There is no restriction as to which direction the avatar should be rendered from this point of view, but if the scene is where the performer 20 and the avatar are facing each other and the interview is performed, the avatar can be viewed from the side. Rendering a scenic scene is a good choice.

  In addition, it is possible to adopt another mode in which the configuration described above is partially different from the mode described above. In one mode, the avatar changing unit 34 performs only the operation related to motion capture, and the result is obtained. The data may be transmitted to the content distribution server 40, and the content distribution server 40 may render the avatar image described above. In this case, the configuration related to rendering is included in the content distribution server 40 instead of being included in the avatar changing unit 34 in the above description.

  Further, a part or all of the components arranged in the avatar operation studio 3 described above may be arranged in the distribution environment 4 of the content distributor or another place to have respective functions.

  Since the difference in the configuration and the difference in the processing in such another aspect can be easily understood from the description parts of this specification, the description thereof will be omitted.

  The image data of the avatar obtained by the rendering is transmitted to the content distribution server 40 via the communication interface 345 and the Internet communication networks 60, 65, 66 (step S104).

  The content distribution server 40 receives the transmitted data of the avatar image (step S204) and executes the following operation in advance.

  First, an image of an object to be replaced later with an image of an avatar, which is an image captured by the scene capturing camera 22 together with the performer 20 (the performer display stand unit 21 in FIG. 6A) is a template (template). The image is stored in advance in the template image memory 42b of the content distribution server 40 as an image, and the control unit 41 commands the template image tracker 44 to read the template image (step S201).

  Further, according to the control of the control unit 41, an image 70 (hereinafter also referred to as “scene shot image”) obtained by the scene shooting camera 22 shooting the performer display stand unit 21 together with the performer 20 is an internet communication network 60, 62. , 66, and is received via the communication interface 46 (step S202).

  Next, in accordance with the control of the control unit 41, the content image generation unit 45 specifies the template image in the received scene shot image 70, that is, the position where the performer display stand unit 21 is drawn (step S203). .. At the time of identification, there is a method of selecting the position having the highest correlation by a rubbing method of performing correlation matching with the template image while shifting the position little by little with respect to the captured image 70, and other methods may be used.

  For example, the position on the screen where the image 72 of the avatar should be drawn may be determined according to an operation of an operator (not shown).

  Next, the content image generation unit 45 uses the template image, that is, the image of the performer display stand unit 21 at the position of the identified template image, that is, the performer display stand unit 21, as the avatar image 72 transmitted in step S204. Replace (step S205). By the replacement, a content image 73 in which the performer 20 and the image 72 of the avatar are drawn together on the screen is generated, so that the content image 73 is sent to the personal computer via the communication interface 46 and the Internet communication networks 60, 66, 67. The data is transmitted to 50 (step S206). That is, the internet communication networks 60, 66 and 67 are the signals or the information generated by the content distribution server 40, and the performer 20 as the photographed person and the dummy object are included in one scene. The image of the dummy object is replaced with the image of the virtual character (avatar) to transmit or carry the delivered signal or information, and carry (carry), which is a reception terminal personal computer 50. Will be supplied to.

  As in the above-described aspect, an image of the dummy object (in the example, the display stand unit 21 for the performer) to be replaced with the avatar image is held as a template in advance, and the drawing position of the dummy object is determined by the statistical processing by the server 40. In addition to the mode in which the determination is made, it has been described above that the method may be such that the person who performs the operation manually specifies the drawing position of the avatar on the screen.

  On the other hand, the personal computer 50 used by the browsing user (not shown) displays the content image 73 transmitted by the content distribution server 40 on the personal computer display 50a (step S302).

  As shown in the flowchart of FIG. 4, the processes described above are repeated unless the end operation is detected in the determination steps S105, S207, and S303 that wait for the end operation, so that the content image 73 is repeatedly updated. As a result, the browsing user can browse the moving image in which the performer 20 and the image 73 of the avatar are drawn on the same screen. At this time, the performer 20 looks at the performer display stand unit 21, and the scene in which the actor is talking or gazes like a real interlocutor is the scene-captured image 70 before substitution. The browsing user who visually recognizes the subsequent content image 73 can have a feeling that the performer 20 is really talking with the avatar, and as a result, according to the present embodiment, it is possible to obtain a feeling that the performer 20 is a real performer. It is possible to eliminate unnaturalness in a scene in which an avatar, which is an artificial (virtual) character, appears together.

  Alternatively, even when the performer 20 does not directly face the avatar, the performer 20 can have a conversation or acting as if the avatar were present at the position of the performer visual display 21d. Also in, it is possible to eliminate the unnaturalness in the scene in which the real performer and the avatar, which is an artificial (virtual) character, appear together.

  Therefore, it is supposed that the real performer and the avatar draw on the same screen, and it is surprising or interesting that there is an artificial character that can not be in the real scene, or it should be virtual and artificial. Browsing users who are viewers can fully enjoy the fun of adding a sense of realism to the avatar.

[Processing of displaying avatar image 21e on performer visual display 21d]
In the above, the description has been centered on the process in which the image 50c of the avatar is drawn on the display screen of the personal computer display 50a that is browsed by the browsing user (not shown).

  Next, the process of displaying the image 21e of the avatar on the performer visual display 21d that the performer 20 browses is explained without the main purpose being the browsing by the browsing user.

  In the generated content image 73, it has been described above that it is preferable for the performer 20 to face the avatar, face the line, and draw the line of sight so as to achieve the effect of the present invention. ..

  Therefore, if a schematic front image in which the avatar is facing the performer 20 is drawn on the performer visual display 21d included in the performer display stand unit 21 facing the performer 20, the performer 20 is really relative to the avatar. As a result, the browsing user can also obtain a realistic sensation when browsing the content image 73.

  Therefore, as shown in the flowchart of FIG. 5, in the avatar changing unit 34, the control unit 340 of the avatar changing unit 34 first sets the avatar to be drawn on the content image stored in the avatar data memory 341b in the memory 341. Data (such as image data representing the appearance of the virtual character) is read (step S401). Next, the control unit 340 controls and controls the motion tracker 343 and obtains the movement of the avatar operator 30 by the motion capture technique described above, and converts it into data (step S402).

  Next, the control unit 340 causes the rendering unit 344 to perform rendering to generate a 3D drawn image of the avatar using the obtained motion capture data and the avatar data (step S403). When rendering, it is important from which position the image of the avatar viewed is generated. Therefore, the rendering unit 34 preliminarily informs the position of this viewpoint from the content distribution server 40, specifically, the performer shooting site in FIG. The information of the position of the camera 22 for shooting scenes in 2 is used as the information of the relative position of the performer display stand unit 21 which is the object to be replaced with the image of the avatar, as described later. .. There is no restriction as to which direction the avatar should be rendered from this point of view, but if the scene is where the performer 20 and the avatar face each other in an interview, the avatar can be viewed from the front. Rendering a scenic scene is a good choice.

  The image data of the avatar obtained by the rendering is transmitted to the content distribution server 40 via the communication interface 345 and the Internet communication networks 60, 65, 66 (steps S404, S501).

  The content distribution server 40 processes the transmitted image of the avatar as it is or with an appropriate background, and sends it to the performer visual display 21d via the Internet communication networks 60, 61 and 66 for drawing (step). S502).

  As shown in the flowchart of FIG. 5, each processing described above is repeated unless an end operation is detected in the determination steps S405 and S503 that wait for an end operation, so an image of the avatar drawn on the performer visual display 21d. Will be repeatedly updated, and as a result, the performer 20 will be able to view the moving image of the image of the avatar.

  At this time, the performer 20 looks at the performer display stand unit 21 and sees the scene as if he were talking or watching his eyes as if he were a real interlocutor. Since it is 70, the browsing user who visually recognizes the content image 73 after the replacement can feel as if the performer 20 is really talking with the avatar, and as a result, according to the present embodiment. For example, it is possible to eliminate unnaturalness in a scene in which a real performer and an avatar, which is an artificial (virtual) character, appear together.

  Alternatively, even when the performer 20 does not directly face the avatar, the performer 20 can have a conversation or acting as if the avatar were present at the position of the performer visual display 21d. Also in, it is possible to eliminate the unnaturalness in the scene in which the real performer and the avatar, which is an artificial (virtual) character, appear together.

  Therefore, it is supposed that the real performer and the avatar draw on the same screen, and it is surprising or interesting that there is an artificial character that can not be in the real scene, or it should be virtual and artificial. Browsing users who are viewers can fully enjoy the fun of adding a sense of realism to the avatar.

  In addition to the image 21e of the avatar as in the above-described embodiment, the performer visual display 21d displays instruction comments (so-called “cheat paper”, “campe”) created for the photographed person to read, and programs. An image (not shown) or sound of a staff member such as a director may be displayed, or the attached speaker may speak.

[Display of user posted information]
In addition to the embodiment described above or in addition to this configuration, the following configuration may be adopted.

  As a method for a viewer (not shown) to enjoy content that is broadcast live using an internet communication network, the impression of the content or the tweet of the viewer (hereinafter also referred to as “comment”) can be personalized by the viewer. By inputting and transmitting with the computer 50, and displaying these input comments in real time on a terminal (not shown) used by other viewers, the viewer who wrote the comment makes his / her own comment to other viewers. Viewers can be informed and read comments written by others, so that they can see and enjoy the relevant content as well as know the thoughts and impressions of other viewers. Then, you can get a feeling of solidarity (virtual) simultaneous participation and a sense of solidarity, such as enjoying a concert or event.

  At this time, in the present system 1, the comment posted by any of the viewers is displayed on at least one of the display 21d for viewing the actor and the display 33d for viewing the operator of the avatar via the content distribution server 40. Can be configured to be. As a result, the performer 20 or the avatar operator 30 who has read the comment can obtain the above-described feeling of simultaneous participation and a sense of solidarity, and can perform more realistic acting and response, and operate the avatar.

[Handling of audio in the content delivery / reception system 1]
As described above, the content delivery / reception system 1 according to the present embodiment can generate the content image 73 as if the performer 20 and the avatar exist together in the real space. .. In order to increase the realism, it is a natural idea to add the content image 73 and the speech of the performer 20 and the speech of the avatar to the speech of the avatar operator 30, for example, together with the content image 73. Yes, and an important point. Therefore, the present system 1 includes a performer utterance microphone 21b for picking up the utterance of the performer, an avatar operator photographing microphone 33b for picking up the utterance of the avatar operator 30, and the utterance of the performer 20. An avatar operator's listening speaker 33c for the avatar operator 30 to listen to, and a performer listening speaker 21c for the speaker 20 to hear the utterance of the avatar operator 30 are provided. The transmission of signals is clear from the embodiments relating to the transmission of each moving image described above, so repeated explanations will be avoided.

  Since the present embodiment relates to broadcast contents, when this broadcast is a live broadcast, a voice in which the voice of the performer 20 or the voice of the avatar is replaced by the voice of the avatar operator 30, for example, is Since a general-purpose personal computer that listens to a live broadcast can produce sound, a speaker attached to or connected to the general-purpose personal computer may be used instead of the speakers 21c and 33c described above.

  Similarly, if the image 33e obtained by shooting the performer 20 with the performer shooting camera 21a is drawn on the operator-visible display 33d of the avatar, the avatar operator 30 will also face the performer 20 in front of his / her eyes. Since the avatar can be operated as if he / she is operating, the effect of the invention can be further improved.

[Modification]
The content delivery / reception system 1 according to the present embodiment described above is not limited to the configuration described above, and various modifications can be made, and the present invention includes all of them.

  For example, the performer viewing display 21d may be omitted, and the performer 20 may listen to only the voice of the avatar operator 30 and perform a dialogue while not visually recognizing the image of the avatar operator 30. The system has a simpler configuration, which reduces costs and improves reliability.

  Alternatively, instead of using the display stand unit 21 for the performer, a mere figurine or a stuffed animal, that is, a dummy object is arranged, the performer 20 performs a dialogue operation, and the dummy object is processed by the same process as the replacement described above. The display portion may be replaced with the image of the avatar, and the content image 73 may be generated and used for browsing. In this case, the performer 20 can recognize the movement and utterance of the avatar by the image and sound of the live broadcast if it is a live broadcast, and thus the viewer can view the broadcast in which the avatar and the performer 20 are drawn on the same screen with a natural sense. Users will enjoy it, and such systems will also be simpler in configuration, reducing costs and improving reliability. In this way, the performer 20 performs while watching and listening to a personal computer (not shown) that outputs live broadcast video and audio, even if the personal computer is placed near a dummy object (a figurine, a stuffed animal, etc.). The personal computer may or may not be included in the scene shooting camera shooting zone 23. Further, the personal computer itself may be used as a dummy object whose drawing is replaced with the image of the avatar.

(Explanation of effects of the invention)
In content such as broadcasting, it is surprising or interesting that there is an artificial character that can not be in a real scene, or it is virtual and artificial so that the real actor and the avatar draw on the same screen. The viewer can fully enjoy the fun of adding realism to the supposed avatar, and it is unnatural in the scene where the real performer and the artificial (virtual) character avatar appear together. can be eliminated is, content distribution system, a content distribution method, and the present invention a computer program is provided.
The present invention can be implemented in various other modes, and various omissions, replacements, and changes can be made without departing from the spirit of the invention. These embodiments and modifications thereof are included in the scope of the invention and the scope of the invention, and are also included in the scope of the invention described in the claims and the scope of equivalents thereof.
The inventions described in the claims at the initial application of the present application will be additionally described below.
[C1] A content distribution system for displaying and distributing a photographed person in one scene together with a virtual character,
A scene photographing camera unit that photographs so that the person to be photographed and the dummy object are included in one scene,
An image replacement unit that replaces the image of the dummy object in the scene in which the shooting is performed with the image of the virtual character and delivers the image.
A content distribution system comprising:
[C2] The content distribution system according to [C1], wherein the dummy object includes a display that displays an image of the virtual character.
[C3] In the image distributed by the image replacement unit, the replaced virtual image of the virtual character is in a different mode from the virtual character image displayed on the display included in the dummy object. The content distribution system according to [C2].
[C4] The content distribution system according to [C3], wherein the different aspect is based on different viewpoints.
[C5] The dummy object has a camera for photographing the person to be photographed from a position different from the camera unit for photographing the scene and / or a microphone for picking up the voice of the person to be photographed, [C1] To the content distribution system according to any one of [C4].
[C6] An operation unit for changing the appearance display mode of the virtual character, and an appearance display mode operation display unit for displaying the information collected by the camera and / or the microphone. The content distribution system according to [C5].
[C7] The operation unit changes the appearance display mode of the virtual character displayed on the display and / or the appearance display mode of the image of the virtual character in the image distributed by the image replacement unit. The content distribution system according to [C6], which is configured to:
[C8] A broadcast transmission unit that broadcasts the content including the distribution image of the image replacement unit via a communication path, and a reception unit that receives user-posted information to be displayed on the screen of the content being broadcast, and The content distribution system according to any one of [C2] to [C7], wherein the user-posted information is also displayed on a display that displays an image of the virtual character. ..
[C9] A broadcast transmission unit that broadcasts the content including the distribution image of the image replacement unit via a communication path, and a reception unit that receives user-posted information to be displayed on the screen of the content being broadcast, and The content posted by [C6] to [C8], wherein the user-posted information is configured to be displayed also on the display unit for operating the appearance display mode of the virtual character. Delivery system.
[C10] A content distribution method of displaying and distributing a photographed person in a scene together with a virtual character,
The scene capturing camera unit captures the subject and the dummy object in a single scene.
The steps to do shadows,
An image replacement unit replacing the image of the dummy object in the scene in which the shooting is performed with the image of the virtual character and delivering the image.
A content distribution method comprising:
[C11] A computer program including a code group for a computer to execute each step in the content distribution method according to [C10].
[C12] A content distribution server for displaying and distributing a photographed person in one scene together with a virtual character,
The image of the dummy object in the scene photographed so that the person to be photographed and the dummy object are included in one scene is replaced with the image of the virtual character, and is delivered. A content distribution server.
[C13] A transmission path through which the content delivery server according to [C12] transmits the delivered signal.

1 Content Delivery / Reception System 2 Performer Shooting Site 3 Avatar Operation Studio 4 Content Distributor 5 Content Viewer Home 20 Performer 21 Performer Display Stand Unit 21a Performer Shooting Camera 21b Performer Speech Microphone 21c Performer Listening Speaker 21d Performer Visual Display 21e Avatar image 21f Stand 22 Scene shooting camera 23 Scene shooting camera Shooting zone 30 Avatar operator 31 Optical capture marker 32 Motion capture camera 33 Avatar operator display stand unit 33a Tally lamp 33b Avatar operation Microphone 33c for photographing a person Avatar operator listening speaker 33d Avatar operator visual recognition display 33e Actor image 33f Stand 34 Avatar changing unit 340 Control unit 341 Memory 341a Program memory 341b Avatar data memory 341c Temporary memory 342 Data bus line 343 Motion tracker 344 Rendering unit 345 Communication interface 40 Content distribution server 41 Control unit 42 Memory 42a Program memory 42b Template image memory 42c Temporary memory 43 Data bus line 44 Template image tracker 45 Content image generating unit 46 Communication interface 50 Personal computer 50a Personal computer display 50b Image of performer in scene 50c Image of avatar in scene 60, 61, 62, 63, 64, 65, 66, 67 Internet communication network

Claims (11)

A content distribution system for displaying and distributing a photographed person in one scene together with a virtual character,
A scene photographing camera unit that photographs so that the person to be photographed and a dummy object having a display for displaying an image of the virtual character fit into one scene,
An image replacement unit that replaces the image of the dummy object in the scene in which the shooting is performed with the image of the virtual character and delivers the image.
And
In the image delivered by the image replacement unit, the image of the virtual character that has been replaced is a mode different from the image of the virtual character displayed on the display included in the dummy object, Content distribution system.   The content distribution system according to claim 1, wherein the different aspects are based on different viewpoints. A broadcast transmitting unit that broadcasts content including the distribution image of the image replacing unit via a communication path,
A receiver for receiving user-posted information to be displayed on the screen of the content being broadcast, and the user-posted information is also displayed on a display for displaying an image of the virtual character. The content delivery system according to claim 1 or 2, wherein the content delivery system is provided. A content distribution system for displaying and distributing a photographed person in one scene together with a virtual character,
A scene photographing camera unit that photographs so that the person to be photographed and a dummy object having a display for displaying an image of the virtual character fit into one scene,
An image replacement unit that replaces the image of the dummy object in the scene in which the shooting is performed with the image of the virtual character and delivers the image.
And
The content distribution system, wherein the dummy object has a camera for photographing the person to be photographed from a position different from the scene photographing camera unit and / or a microphone for picking up the voice of the person to be photographed.   An operation unit for changing the appearance display mode of the virtual character, and a display unit for operating the appearance display mode of the virtual character that speaks information collected by the camera and / or the microphone. The content distribution system according to claim 4, wherein   The operation unit is configured to change the appearance display mode of the virtual character displayed on the display and / or the appearance display mode of the image of the virtual character in the image distributed by the image replacement unit. The content distribution system according to claim 5, wherein the content distribution system is performed. A broadcast transmitting unit that broadcasts content including the distribution image of the image replacing unit via a communication path,
A receiver for receiving user-posted information to be displayed on the screen of the content being broadcast, and the user-posted information is also displayed on a display for displaying an image of the virtual character. The content delivery system according to claim 4, wherein the content delivery system is provided. A broadcast transmitting unit that broadcasts content including the distribution image of the image replacing unit via a communication path,
A receiver for receiving user-posted information to be displayed on the screen of the content being broadcast is further provided, and the user-posted information is also displayed on the appearance display mode operation display unit of the virtual character. The content distribution system according to claim 5, wherein the content distribution system is configured as described above. A content distribution method for displaying and distributing a photographed person in one scene together with a virtual character,
A step in which the scene photographing camera section performs photographing so that the person to be photographed and a dummy object having a display for displaying an image of the virtual character are contained in one scene;
An image replacement unit replacing the image of the dummy object in the scene in which the shooting is performed with the image of the virtual character and delivering the image.
And
In the image delivered by the image replacement unit, the image of the virtual character that has been replaced is a mode different from the image of the virtual character displayed on the display included in the dummy object, Content delivery method.   The content distribution method according to claim 9, wherein the different aspects are based on different viewpoints.   A computer program comprising a code group for a computer to execute each step in the content distribution method according to claim 9.

Images