JP6853013B2 - Information processing systems, information processing methods, and information processing programs - Google Patents

Description

The present disclosure relates to information processing systems , information processing methods , and information processing programs .

In recent years, with the development of ICT (Information and Communication Technology), various contents using a virtual three-dimensional space drawn by CG (Computer Graphics) have been proposed. Examples of such contents include VR (Virtual Reality) contents, game contents, and e-sports (electronic sports) contents.

A user (hereinafter, also referred to as a user) who uses the above content is equipped with a device capable of viewing a three-dimensional image such as an HMD (Head-Mount Display), so that a virtual 3 drawn by CG is attached. It is possible to stereoscopically view the dimensional space from the user's point of view. Further, when a sensor or the like that detects the movement or position of the body such as the user's head or arm is attached, for example, it moves in the three-dimensional space according to the user's movement and is arranged in the three-dimensional space. Actions such as working on the created object become possible. Objects arranged in the three-dimensional space include, for example, objects (for example, characters and avatars) that can be operated by other users who use the above contents.

In contents such as games and e-sports, for example, it is possible to respond to an event caused by one's own action, have a conversation, compete, or compete with an object operated by another user. From each viewpoint, users who participate in the content can enjoy collisions and physical interactions in the three-dimensional space due to the events caused by the above actions according to predetermined rules defined by the content.

The following patent documents exist as prior art documents that describe the techniques related to the techniques described in the present specification.

Japanese Unexamined Patent Publication No. 2011-150402 Japanese Unexamined Patent Publication No. 2001-300131 Japanese Unexamined Patent Publication No. 11-12328

As a form of content, a three-dimensional space such as a conference hall, office, school, or shopping mall is drawn by CG, and a remote conference, VR office, e-learning, or e- in which multiple users participate in real time in the above three-dimensional space. Commerce etc. are assumed.

In the above form, for example, not only the interaction between users in the drawn three-dimensional space is participated in real time as a speaker, but also the distribution for users who watch the video after the fact or as a listener, and the viewing environment thereof. Demand for viewing formats and media conversion is expected.

If the view image of a three-dimensional space which is delivery, etc. after the fact, for example, according to the viewing environment improvement for the dedicated device purchase or 3D video reception to view three-dimensional images or an HMD burden Is assumed .

An object of the present disclosure is to provide a technique capable of reproducing a moving image from a received viewpoint based on multi-viewpoint 3D moving image data to be photographed.

One aspect of the disclosed technology is exemplified by an information processing system. This information processing system includes a moving image data generating means for generating multi-viewpoint 3D moving image data of a shooting target, a means for accepting a viewpoint setting for the multi-viewpoint 3D moving image data, and the shooting target from the received viewpoint. It is provided with a moving image generating means for generating a moving image when viewed, a means for providing the generated moving image to a viewer device, and a recording means for recording the set viewpoint .

According to one aspect of the disclosed technology, it is possible to provide a technology capable of reproducing a moving image from a received viewpoint based on multi-viewpoint 3D moving image data to be photographed.

It is a block diagram which shows an example of an information processing system. It is a block diagram which shows an example of the hardware configuration of a computer. It is a figure which shows an example of the data structure which is stored about the object in 3DCG space. It is a flowchart which illustrates the recording process of the event related to the event of a scene manager. It is a flowchart which illustrates the setting process of the virtual camera of a scene manager. It is explanatory drawing explaining the setting of the camera direction of a virtual camera. It is a block diagram which shows an example of an information processing system which includes a camera which takes a picture of a conference hall in a real space. It is a figure explaining the shooting of the subject in the real space when using chroma key composition. It is a figure explaining the shooting of the subject in the real space when using chroma key composition. It is a figure which shows an example of a composite video. It is a figure which shows an example of the synthetic image displayed on the 2D image device. It is explanatory drawing of the form in which a subject wearing an HMD participates in a remote conference. It is explanatory drawing of the form in which a subject who does not wear an HMD participates in a remote conference. It is explanatory drawing of the form which a subject who does not wear an HMD participates in a remote conference which is reproduced after the fact. It is a flowchart which illustrates the recording process of the event related to the event of the scene manager when the camera which shoots a real space is provided.

Hereinafter, the information processing system according to the embodiment will be described with reference to the drawings. The configuration of the following embodiment is an example, and the information processing system is not limited to the configuration of the following embodiment.

<Embodiment>
[1. System configuration〕
FIG. 1 is a configuration diagram showing an example of an information processing system according to the present embodiment. First, an outline of the information processing system 10 shown in FIG. 1 will be described. In the form of FIG. 1, for example, a conference hall is provided in a three-dimensional space created by CG (hereinafter, also referred to as “3DCG space”), and contents such as a remote conference are provided. Here, the "remote conference" means a conference in which a plurality of conference participants connected to the network share a conference hall in the 3DCG space and hold the conference in real time. A plurality of conference participants connected to the network perform actions such as speaking and dialogue in the conference, working on others, and moving through objects (avatars and the like) that can be operated in the 3DCG space. The state of the conference progress is provided to the HMD or the like provided by the conference participants as a 3DCG moving image viewed from the viewpoint of each avatar.
However, the content provided by the information processing system 10 may be a VR office, a game, an e-learning, or the like virtually formed in the 3DCG space. The content may be able to provide collisions and physical interactions due to events caused by user actions, movements, etc. in the 3DCG space in accordance with predetermined rules defined by the content.
In the following, the content provided by the information processing system 10 will be described as a remote conference using a conference hall provided in the 3DCG space. Further, a moving image of a three-dimensional space created by CG is also referred to as a "stereoscopic image".

In the 3DCG space, objects of various components such as desks, chairs, whiteboards, walls, floors, windows, doors, and ceilings that make up a conference hall for remote conferences are arranged. Similarly, avatars for a plurality of conference participants participating in a remote conference are arranged in the 3DCG space. An avatar is an object that can be manipulated by conference participants. The object arranged in the 3DCG space is, for example, an aggregate of polygons which are drawing elements. Each polygon that makes up an object has a common coordinate axis. The shape, arrangement position, orientation, etc. of an object in the world coordinate system in the 3DCG space are represented by three-dimensional coordinates (X, Y, Z) on the coordinate axes.

Conference participants are equipped with, for example, a device that can visually recognize a stereoscopic image such as an HMD (hereinafter, also referred to as a 3D image device), a sensor device that detects body movement and position, and an acoustic device such as a microphone or headphones, and are equipped with a remote conference. attend to. The state of the ongoing remote conference is provided to the conference participants in real time as a stereoscopic image through each avatar viewpoint. The conference participants watch the stereoscopic video of the process accompanying the progress of the conference from the viewpoint of the avatar, and operate the avatar while watching the stereoscopic video to move the conference hall and talk to and work with other conference participants (actions). I do. Speaking within a remote conference is made, for example, through an acoustic device such as a microphone or headphones. Events corresponding to the actions of each meeting participant (for example, actions such as conversation, proposal, and leaving) that occur in the process accompanying the progress of the meeting are reflected in the stereoscopic image being viewed.

The information processing system 10 according to the present embodiment has a function of relaying a 3DCG space in which a remote conference is proceeding as a stereoscopic image viewed from an arbitrary viewpoint position. The viewpoint position in the 3DCG space where the remote conference proceeds is specified by, for example, a virtual camera which is a virtual camera. The virtual camera can be moved to any position in the 3DCG space where the remote conference is going on. The virtual camera relays objects and avatars placed in the conference hall, and events such as events that occur in the process accompanying the progress of the conference as stereoscopic images viewed from an arbitrary viewpoint position. When there is a relay relaying in the 3DCG space where the remote conference is proceeding, the viewpoint position of the virtual camera is set according to the operation of the relay relay.

The information processing system 10 according to the present embodiment has a function of converting a stereoscopic image relayed by a virtual camera into a two-dimensional image. The converted two-dimensional image is delivered in real time or after the fact to the viewer who watches the state of the remote conference, for example. The viewer views the distribution video of the remote conference through a device (hereinafter, also referred to as a 2D video device) capable of viewing a two-dimensional video such as a smartphone, a mobile phone, or a tablet PC (Personal Computer).

The information processing system 10 according to the present embodiment has a function of receiving messages (reaction signals such as characters, voices, and actions) from a viewer who views a two-dimensional image. The information processing system 10 moves the viewpoint position of the virtual camera in the 3DCG space in which the remote conference proceeds, for example, based on the received message. The information processing system 10 notifies the relayer of the received message, and moves the viewpoint position of the virtual camera according to the operation of the relayer who received the notification. When the remote conference is in progress, the stereoscopic image relayed by the virtual camera is updated according to the movement of the viewpoint position, and the viewpoint position received from the viewer is reflected in the two-dimensional image being viewed.

The information processing system 10 according to the present embodiment has a function of recording an event in 3DCG space in which a remote conference proceeds and virtual camera information (viewpoint position, focal length, rotation, zoom parameter, etc.). The events in the 3DCG space include the coordinates of each object that moves in the 3DCG space in response to the events that accompany the actions of each conference participant, which occur in the process that accompanies the progress of the conference. Events in the 3DCG space include audio data of conference participants. For example, the minutes of a remote conference are created based on the recorded voice data and the like. The information processing system 10 generates a stereoscopic image based on the viewpoint position of the virtual camera based on the recorded information, and distributes the two-dimensional image based on the generated stereoscopic image to the viewer after the fact. A message from the viewer is also accepted in the subsequent viewing, and the viewpoint position by the subsequent viewer is reflected in the two-dimensional image being viewed.

Next, the configuration of the information processing system 10 shown in FIG. 1 will be described.
The information processing system 10 shown in FIG. 1 mainly includes a scene manager SM, a virtual camera VC, a camera processor C, a renderer R, and a distribution server CH. Each of the above configurations is connected via a network such as a LAN (Local Area Network). Each of the virtual camera VC, the camera processor C, and the renderer R may be connected to a plurality of networks. In FIG. 1, virtual cameras VC1, VC2, VC3, camera processors C1, C2, C3, renderers R1, R2, and R3 are exemplified. In the following description, the plurality of virtual cameras VC1, VC2, and VC3 are collectively referred to as a virtual camera VC. The same applies to the camera processor C and the renderer R.

Further, the scene manager SM and the distribution server CH are connected to an external network. The external network includes a public network such as the Internet, a mobile phone network, and a wired wireless LAN (Local Area Network). A plurality of 2D video devices of a viewer (Viewer) who view a two-dimensional video are connected to the external network. The information processing system 10 may be provided with a plurality of scene manager SMs and distribution server CHs. Further, the information processing system 10 may integrally include the functions of the virtual camera VC, the camera processor C, and the renderer R.

In FIG. 1, A1 represents a stereoscopic image being viewed by a distributor U1 who operates the viewpoint position of the virtual camera VC1. The stereoscopic image A1 being viewed by the distributor U1 includes the avatars of the conference participants (P1, P2, P3) of the remote conference as viewed from the viewpoint position of the virtual camera VC1. The distributor U1 who operates the viewpoint position of the virtual camera VC1 is not drawn in the stereoscopic image of the viewpoint position of the avatar.

The virtual cameras VC2 and VC3 in the stereoscopic image A1 are virtual cameras VC arranged as objects in the 3DCG space. The virtual camera VC2 follows, for example, the movement of the conference participant P2. The virtual camera VC3 follows, for example, the movement of the conference participant P3.

The distributor U1 wears the HMD1 for viewing the stereoscopic image of the virtual camera VC1 operated by the distributor U1. The HMD1 worn by the distributor U1 includes acoustic devices such as headphones and an intercom. Further, the distributor U1 includes, for example, a controller CN for designating the viewpoint position of the virtual camera VC1 in the 3DCG space. The controller CN is a user input device that operates two viewpoint positions for stereoscopic viewing of the 3DCG space. The controller CN includes a sensor (accelerometer, gyro sensor, geomagnetic sensor, image sensor, etc.) that detects the rotation angle of the virtual camera VC1. The virtual cameras VC1, VC2, and VC3 may have a function of switching the stereoscopic image being relayed. Further, there may be a plurality of distributors who operate the viewpoint position of the virtual camera VC1.

The scene manager SM is a platform on which an application program for creating a stereoscopic image in 3DCG space such as a remote conference operates. The platform is, for example, a computer system composed of information processing devices such as a server and WS (WorkStation).

The scene manager SM creates an image in 3DCG space by executing an application program, and provides a remote conference participant with a stereoscopic image from an avatar viewpoint in real time. Similarly, the scene manager SM creates a stereoscopic image of the 3DCG space viewed from the viewpoint position of the virtual camera VC by executing the application program, and transmits it to the renderer R. The scene manager SM outputs a stereoscopic image of the 3DCG space seen from each viewpoint position of the virtual cameras VC1, VC2, and VC3 to the distributor U1 who operates the viewpoint position of the virtual camera VC1. The distributor U1 views the stereoscopic image of the 3DCG space seen from the viewpoint position of the virtual cameras VC1, VC2, and VC3 via the HMD or the like. The stereoscopic image displayed on the HMD or the like is switched via the controller CN that receives the operation input of the distributor U1.

The scene manager SM stores and manages the data of objects such as structures and avatars constituting the conference room in the 3DCD space in association with the time code. Here, the "time code" is a counter that expresses the absolute time in the 3DCG space. All events related to the event that occurred in the process accompanying the progress of the remote conference are saved and managed by the time code.

The scene manager SM receives a message transmitted from a 2D video device such as a smartphone connected to an external network. The message includes reaction signals such as characters, voices, and actions from a viewer of a two-dimensional video delivered after the fact or in real time. The scene manager SM moves the viewpoint position of the virtual camera VC in the 3DCG space where the remote conference is proceeding based on the received message. Alternatively, the scene manager SM notifies the distributor U1 who operates the viewpoint position of the virtual camera VC1 in the 3DCG space of the viewpoint position based on the received message. The distributor receives an instruction from the scene manager SM via, for example, a notification of movement of the viewpoint position displayed on the HMD (characters, markers, etc.), a voice notification, or the like, and moves the viewpoint position of the virtual camera VC1. .. The message received by the scene manager SM is saved in association with the time code.

The function provided by the scene manager SM by executing a computer program is an example of "a moving image data generation means for generating multi-viewpoint three-dimensional moving image data obtained by shooting a shooting target from multiple viewpoints". Similarly, the function provided by the scene manager SM by executing a computer program is an example of "a means for receiving a reaction to a moving image from a viewer device". Similarly, the function provided by the scene manager SM by executing a computer program is an example of "a means for recording a set viewpoint, a time in a multi-view 3D moving image data, and a reaction".

The virtual camera VC is a virtual device that specifies a viewpoint position for relaying a stereoscopic image in 3DCG space in which a remote conference is proceeding. The virtual camera VC has two viewpoint positions and congestion points determined from the two viewpoint positions as camera information in order to relay the 3DCG space as a stereoscopic image. In the virtual camera VC that does not involve the operation of the distributor U1, the arrangement position in the 3DCG space is determined by the scene manager SM. The virtual camera VC may be drawn as a visible object in the 3DCG space, for example, or may not be visible to the conference participants. Further, the object of the virtual camera VC may be drawn only in the stereoscopic image viewed by the distributor U1 like the virtual cameras VC2 and VC3 in the stereoscopic image A1 so as not to be shown to the viewer. .. Virtual camera information (hereinafter, also simply referred to as “camera information”) such as focal length, zoom parameters, rotation, and viewpoint position of the virtual camera VC is controlled by the scene manager SM or the distributor U1 who operates the virtual camera VC. .. The virtual camera VC is an example of "means for accepting viewpoint settings for multi-view three-dimensional moving image data".

The camera processor C is a computer that records and stores camera information of the virtual camera VC in association with a time code. The camera processor C cooperates with the scene manager SM based on the camera information of the virtual camera VC to generate a stereoscopic image in the 3DCG space in which the remote conference proceeds. The camera processor C outputs the camera information of the virtual camera VC and the stereoscopic image in the 3DCG space seen from the viewpoint position of the virtual camera VC to the renderer R.

The renderer R is a computer that performs so-called rendering processing that converts a stereoscopic image in a 3DCG space whose viewpoint position is specified by a virtual camera VC into a two-dimensional image (video). In the rendering process, coordinate transformation such as affine transformation of an object in 3DCG space to a two-dimensional plane and drawing processing such as lighting processing are performed, and the polygon unit data arranged in 3DCG space is the viewpoint position of the virtual camera VC. It is converted to pixel-based data as seen from. The renderer R creates a two-dimensional image from a frame-based two-dimensional image drawn based on the converted pixel-unit data, and outputs the created two-dimensional image to the distribution server CH. The camera processor C and the renderer R are examples of "means for generating a moving image when a shooting target is viewed from a received viewpoint".

The distribution server CH is a computer that distributes the two-dimensional video (video) output from the renderer R to a 2D video device such as a smartphone connected to an external network. The distribution server CH switches the distribution of the two-dimensional video output from the plurality of renderers R1, R2, R3, etc. after the fact via the 2D video device or in response to a request from the viewer who is viewing in real time. The distribution server CH is an example of "means for providing the generated moving image to the viewer device".

[2. Device configuration〕
FIG. 2 is a configuration diagram showing an example of a computer hardware configuration. The scene manager SM, the camera processor C, the renderer R, and the distribution server CH shown in FIG. 1 are realized by using the computer 100 shown in FIG. The computer 100 illustrated in FIG. 2 is a connection bus 1.
It includes a CPU (Central Processing Unit) 101, a main storage device 102, an auxiliary storage device 103, a communication IF (Interface) 104, and an input / output IF 105 connected to each other by 06. The CPU 101 is also called a processor. However, the CPU 101 is not limited to a single processor, and may have a multiprocessor configuration. Further, a single CPU 101 connected by a single socket may have a multi-core configuration. Further, the above-mentioned components may be provided in a plurality of each, or some of the components may not be provided.

The CPU 101 is a central processing arithmetic unit that controls the entire computer 100. The CPU 101 executably deploys the program stored in the auxiliary storage device 103 in the work area of the main storage device 102, and controls peripheral devices through the execution of the program to provide a function that meets a predetermined purpose. The main storage device 102 is a storage medium in which the CPU 101 caches programs and data and expands a work area. The main storage device 102 includes, for example, a flash memory, a RAM (Random Access Memory), and a ROM (Read Only Memory).

The auxiliary storage device 103 is a storage medium that stores various programs executed by the OS (Operating System) and the CPU 101, operation setting information, and the like. The auxiliary storage device 103 is, for example, an HDD (Hard-disk Drive), an SSD (Solid State Drive), an EPROM (Erasable Programmable ROM), a flash memory, a USB (Universal Serial Bus) memory, or the like. The auxiliary storage device 103 includes a CD (Compact Disc) drive device, a DVD (Digital Versatile Disc) drive device, a BD (Blu-ray (registered trademark) Disc) drive device, and the like. Examples of the recording medium include a silicon disk including a non-volatile semiconductor memory (flash memory), a hard disk, a CD, a DVD, a BD, a USB (Universal Serial Bus) memory, an SD (Secure Digital) memory card, and the like.

The communication IF 104 is an interface with a network connected to the computer 100 and an external network. The communication IF 104 includes a wireless communication module that communicates based on a predetermined standard. The input / output IF 105 is an interface for inputting / outputting data to / from another device connected to the computer 100.

Execution of the program of the CPU 101 provides processing of the scene manager SM, the camera processor C, the renderer R, and the distribution server CH shown in FIG. However, at least a part of each of the above processes may be provided by a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a GPU (Graphics Processing Unit), or the like. Similarly, at least a part of each of the above processes is a dedicated LSI (large scale integration) such as an FPGA (Field-Programmable Gate Array), a numerical arithmetic processor, a vector processor, an image processor, and other digital circuits. May be good. Further, at least a part of the scene manager SM, the camera processor C, the renderer R, and the distribution server CH may include an analog circuit.

[3. Data structure]
FIG. 3 is a diagram showing an example of a data structure of an object in the 3DCG space recorded and saved by the scene manager SM. As shown in FIG. 3, the data about the objects in the 3DCG space recorded and saved in the scene manager SM is recorded as a record for each time code. The above records are "TC", "Object", "Pos (x, y, z)", "Rot (p, y, r)", "FOV (f)", "Scale (s)", "Option". It has a field such as "{Trigger, Target, ...}".

In the "TC" field, the counter value of the counter representing the absolute time in the 3DCG space is stored. The counter value is measured at a cycle of, for example, 0.1 ms. The time information calculated from the measured counter value may be stored in the "TC" field. In FIG. 3, time information in the form of time calendar, minute history, and second history such as “h: mm: ss” is illustrated.

The "Object" field stores identification information that uniquely identifies an object arranged in the 3DCG space. Further, as an example of the data stored in the "Object" field, there is identification information that uniquely identifies the virtual camera VC. In FIG. 3, avatars (P1, P2) that can be operated by the participants of the remote conference are illustrated. Similarly, the virtual camera VC1 (HID1, HID2) operated by the distributor U1 and the virtual camera VC2 (CAM1) that follows the avatar are exemplified. The virtual camera VC1 has two viewpoint positions (HID1 and HID2) for stereoscopically viewing the 3DCG space. The viewpoint positions (HID1, HID2) of the virtual camera VC1 are operated via the user input device included in the distributor U1.

The "Pos (x, y, z)" field stores the coordinates of the object stored in the "Object" field in the 3DCG space. If the object stored in the "Object" field is the avatar of a conference participant, the coordinates of the center of gravity of a plurality of polygons constituting the object are stored. If the object stored in the "Object" field is a virtual camera VC, the coordinates of the viewpoint position are stored. In the case of the virtual camera VC1 operated by the distributor U1, the coordinates of the viewpoint position for each of HID1 and HID2 are stored.

In the "Rot (p, y, r)" field, parameter coordinates representing the rotation of local coordinates due to the movement of the object stored in the "Object" field are stored. In the case of the virtual camera VC, the parameter coordinates representing the rotation direction of the pitch axis, yaw axis, and roll axis of the viewpoint position are stored. In the case of the virtual camera VC1 operated by the distributor U1, the above coordinates for each of HID1 and HID2 are stored.

In the "FOV (f)" field, a parameter representing the angle of view is stored as camera information of the virtual camera VC. In the "Scale (s)" field, the parameters of enlargement / reduction accompanying the movement of the object stored in the "Object" field in the 3DCG space are stored. In the "Option {Trigger, Target, ...}" Field, for example, a parameter for giving a texture to an object in the 3DCG space, a parameter indicating an operation, and the like are stored. In the case of the virtual camera VC, for example, information such as a zoom target mark and start / stop of the virtual camera VC1 operated by the distributor U1 is stored.

[4. Processing flow]
Hereinafter, the processing of the scene manager SM in the information processing system 10 will be mainly described with reference to the flowcharts shown in FIGS. The scene manager SM, the camera processor C, the renderer R, and the distribution server CH shown in FIG. 1 are executed by the CPU 101 reading various programs and various data stored in the auxiliary storage device 103 into the main storage device 102. , The process shown in FIG. 4 is performed. The same applies to the setting process of the virtual camera VC1 described later in FIG.

FIG. 4 is a flowchart showing a recording process of an event related to an event of the scene manager SM. The scene manager SM records the communication between the conference participants (avatars) generated in the process accompanying the progress of the remote conference provided via the information processing system 10 and the camera information of the virtual camera VC in association with the time code TC. .. For example, the scene manager SM records the camera information of the virtual camera VC, which is an object arranged in the 3DCG space, in the auxiliary storage device 103 according to the data structure described with reference to FIG. Similarly, the scene manager SM records a message from the viewer of the 2D video distributed from the distribution server CH in association with the time code TC. In the scene manager SM, the image in the 3DCG space created by executing the application program is provided to the remote conference participants in real time as a stereoscopic image via the avatar viewpoint.

In the flowchart of FIG. 4, the start of the process is exemplified when the application program that provides the content such as the remote conference is executed. The scene manager SM initializes (reset) the time code TC, which is a counter representing the absolute time in the 3DCG space (S1). The scene manager SM, for example, arranges various objects constituting the conference hall in the world coordinate system in the 3DCG space. Then, the scene manager SM waits until the login of the conference participant is accepted (S2). However, if the number of conference participants and their job titles participating in the remote conference are notified in advance, the avatars assigned to the conference participants in advance may be arranged in the 3DCG space. The object is registered in advance in the auxiliary storage device 103 referred to by the scene manager SM, for example. In the standby state (S2) of FIG. 4, the scene manager SM is in a state of waiting for a plurality of events exemplified by the processes of S3 to S7.

The scene manager SM accepts the login and logout of the conference participants (players) of the remote conference provided using the 3DCG space (S3). When the conference participant's login and logout are accepted (S3, accepted), the scene manager SM records the accepted conference participant's login and logout as a key record associated with the time code TC (S8). .. The key record includes, for example, identification information that uniquely identifies the conference participant, identification information of the avatar in the 3D space associated with the conference participant, and the like.

The scene manager SM provides the conference participants who have accepted the login with a stereoscopic image of the 3DCG space where the remote conference is performed from the viewpoint of the avatar. The logged-in conference participant operates the avatar while viewing the stereoscopic image via the HMD or the like, moves the conference hall, and interacts with and works with other conference participants. Events caused by the actions of conference participants in the 3DCG space are reflected in the stereoscopic image being viewed.

The scene manager SM accepts logins and logouts of a plurality of distributors Un that relay the 3DCG space in which the remote conference is proceeding (S4). When the scene manager SM accepts the login and logout of the distributor Un (S4, accept), the scene manager SM records the login and logout of the accepted distributor Un as a key record associated with the time code TC (S8). .. The key recording includes, for example, identification information that uniquely identifies the distributor Un, and identification information of the virtual camera VCn operated by the distributor Un. The initial placement position of the virtual camera VCn operated by the distributor Un in the 3DCG space is determined by the scene manager SM.

The distributor Un operates the viewpoint positions (HID1, HID2) of the virtual camera VCn, and relays the 3DCG space in which the remote conference is proceeding. The viewpoint positions (HID1, HID2) of the virtual camera VCn are set to arbitrary positions in the 3DCG space. The scene manager SM outputs a stereoscopic image of the 3DCG space seen from the viewpoint position of the virtual camera VCn to the distributor Un who has received the login. The distributor Un operates the viewpoint position of the virtual camera VCn while viewing the stereoscopic image displayed on the HMD or the like, and relays the 3DCG space in which the remote conference is proceeding. The stereoscopic image relayed through the operation of the virtual camera VCn reflects, for example, movement in the 3DCG space according to an event accompanying the progress of the remote conference, zooming up to a specific avatar, and the like.

The scene manager SM accepts operation input to the avatar by the conference participants (S5). When the received operation input causes the movement, rotation, enlargement / reduction of the object arranged in the 3DCG space where the remote conference proceeds (S5, Yes), the scene manager SM associates the object with the time code TC. Parameters representing object movement, rotation, and enlargement / reduction are recorded as key records (S9). The scene manager SM records various parameters for the target object using, for example, the data structure shown in FIG.

The scene manager SM accepts an operation input to the virtual camera VCn by the distributor Un (S6). When the scene manager SM accepts the operation input to the virtual camera VCn by the distributor Un (S6, Yes), the scene manager SM records the camera information of the virtual camera VCn as a key record in association with the time code TC (S10). .. The scene manager SM records camera information about the virtual camera VCn to be operated, for example, using the data structure shown in FIG.

The scene manager SM receives a message (viewer reaction) from the viewer of the 2D video distributed from the distribution server CH (S7). The message received from the viewer includes reaction signals such as characters, voices, and actions. When the scene manager SM receives a message from the viewer of the 2D video distributed from the distribution server CH (S7, Yes), the scene manager SM records the received message as a viewer reaction in association with the time code TC (S7, Yes). S11).

The content provider (service provider) reflects, for example, in the content or service content that provides the evaluation or request for the delivered 2D video included in the message recorded in the scene manager SM.

Next, the setting process of the virtual camera VC1 of the scene manager SM will be described with reference to the flowchart shown in FIG. FIG. 5 is a flowchart showing a setting process of the virtual camera VC1 of the scene manager SM. The start of the processing of the flowchart shown in FIG. 5 and the processing of S1 are the same as the processing shown in FIG.

The scene manager SM receives the operation input of the virtual camera VCn operated by the distributor Un via the controller CN (S21). The operation input includes camera information of the virtual camera VCn, for example, the viewpoint position in 3DCG space (each coordinate of HID1 and HID2), zoom-in / zoom-out (enlargement / reduction), camera orientation (rotation acquisition sensor), and angle of view. , A parameter indicating the start of video acquisition (HID trigger ON / OFF) and the like is included. The scene manager SM acquires the camera information of the received virtual camera VCn and temporarily stores it in a predetermined area of the main storage device 102. The processing of the scene manager SM shifts to the processing of S22.

When the operation input includes a parameter (HID trigger ON) indicating the start of video acquisition (S22, Yes), the scene manager SM shifts to the process of S23. If the operation input does not include the parameter (HID trigger ON) indicating the start of video acquisition, the scene manager SM returns to the process of S21 (S22, No).

The scene manager SM determines whether or not a virtual camera VCn that relays from an arbitrary viewpoint in the 3DCG space where the remote conference is provided is already arranged (S23). When the virtual camera VCn is already arranged (S23, Yes), the scene manager SM shifts to the process of S26. If the virtual camera VCn is not yet arranged (S23, No), the scene manager SM shifts to the process of S24.

In the process of S24, the scene manager SM sets the viewpoint positions (HID1, HID2) of the virtual camera VCn according to the camera direction setting algorithm described later with reference to FIG. The scene manager SM arranges the virtual camera VCn that relays in the 3DCG space from an arbitrary viewpoint at the viewpoint positions (HID1, HID2) set in the process of S24 (S25). The arrangement position of the virtual camera VCn in the 3DCG space is represented by the midpoint between the two viewpoint positions (HID1 and HID2). The processing of the scene manager SM shifts to the processing of S27.

In the process of S26, the scene manager SM updates the position, angle, and the like of the virtual camera VCn. The scene manager SM updates the position of the virtual camera VCn from the coordinates of HID1 and HID2 received in the process of S21, and updates the rotation angle of the camera from the direction of the camera. The processing of the scene manager SM shifts to the processing of S27.

In the process of S27, the scene manager SM records the camera information of the virtual camera VCn set in the processes of S25 and S26 in the auxiliary storage device 103. The camera information is recorded, for example, by the data structure described with reference to FIG. The processing of the scene manager SM shifts to the processing of S28.

In the process of S28, the scene manager SM determines whether the count value of the time code TC has reached the recordable time. Here, the recordable time is a predetermined set time, and can be obtained from, for example, the memory capacity of the auxiliary storage device 103 or the like. When the count value of the time code TC has reached the recordable time (S28, Yes), the scene manager SM ends the process shown in FIG. If the count value of the time code TC has not reached the recordable time (S28, No), the scene manager SM shifts to the process of S21 and continues the process shown in FIG.

Next, the camera direction setting algorithm will be described with reference to FIG. FIG. 6 is an explanatory diagram illustrating the setting of the camera direction of the virtual camera VCn. In FIG. 6, a top view is illustrated in which the rotation (clockwise) of the virtual camera VCn in the viewpoint direction is viewed from above. The state of the virtual camera VCn shown in FIG. 6 changes in the order of (1) → (2) → (3). The state shown in (1) represents an initial state.

In FIG. 6, HID1 and HID2 represent the viewpoint position of the virtual camera VCn in the 3DCG space, and the intermediate point P1 between HID1 and HID2 represents the arrangement position of the virtual camera VCn. The viewpoint position and the arrangement position are represented by the world coordinate system. The viewpoint positions of HID1 and HID2 are set by the distributor Un who operates the virtual camera VCn. The distance between HID1 and HID2 is associated with the angle of view (FOV) of the virtual camera VCn or the focus processing (f). The angle of view (FOV) of the virtual camera VCn is, for example, the angle of view (FOV) of the virtual camera VCn by increasing the distance between HID1 and HID2 by the operation input of the distributor Un or the control of the scene manager SM. ) Can be widened.

The gazing point in the 3DCG space of the virtual camera VCn arranged at the intermediate point P1 is represented by "VD", and the arrow from the intermediate point P1 to the gazing point VD indicates the visual axis direction of the virtual camera VCn in the 3DCG space. .. "VD1" represents the visual axis direction according to HID1, and "VD2" represents the visual axis direction according to HID2. Note that P2 and P3 shown in FIG. 6 represent the left and right viewpoints of the distributor Un.

In the camera direction setting algorithm shown in FIG. 6, the end point coordinate VP1 obtained by multiplying (h) the unit vector in the visual axis direction of HID1 and the end point coordinate VP2 obtained by multiplying (h) the unit vector in the visual axis direction of HID2. The gazing point VD of the virtual camera VCn that moves in the camera direction is set using the midpoint of.

As shown in FIG. 6 (1), in the initial state, the coordinates of the end point coordinates VP1, VP2, and the gazing point VD exist on the same plane orthogonal to the visual axis of the virtual camera VCn. In the state of FIG. 6 (1), the sensor value of the rotation angle of the virtual camera VCn is input by the operation input of the distributor Un. For example, the scene manager SM rotates the visual axis of HID2 to the right along the input rotation angle while maintaining the direction of the visual axis of HID1 (FIG. 6 (2)).

As shown in FIG. 6 (2), the scene manager SM sets the gazing point VD of the virtual camera VCn at the midpoint between the end point coordinate VP1 of the HID1 and the end point coordinate VP2 of the rotated HID2, for example. The visual axis direction of the virtual camera VCn in the 3DCG space is a linear direction connecting the intermediate point P1 between HID1 and HID2 and the gaze point VD after rotational movement.

From the state shown in FIG. 6 (2), the scene manager SM rotates the visual axis of HID2 to the right along the input rotation angle and rotates the visual axis of HID1 to the right (FIG. 6 (3). )). At this time, the amount of rotational movement of the visual axis of HID1 to the right corresponds to the amount of rotational movement of the visual axis of HID2 shown in FIG. 6 (2).

As shown in FIG. 6 (3), the scene manager SM sets the gazing point VD of the virtual camera VCn at the midpoint between the end point coordinate VP1 of HID1 and the end point coordinate VP2 of HID2 after rotational movement. The visual axis direction of the virtual camera VCn in the 3DCG space is a linear direction connecting the midpoint P1 between HID1 and HID2 and the gazing point VD updated by rotational movement.

The scene manager SM continuously executes the above-described processing until the input sensor value of the rotation angle is reflected, and rotates the gazing point VD and the visual axis direction of the virtual camera VCn. In the scene manager SM, by executing the camera direction setting algorithm described with reference to FIG. 6, a more natural rotation operation of the virtual camera VCn is reflected in the stereoscopic image.

As described above, the information processing system 10 can set the virtual camera at an arbitrary position in the 3DCG space where the remote conference proceeds. The information processing system 10 can relay an event such as an exchange between conference participants that occurs in the process of proceeding with a remote conference as a stereoscopic image viewed from an arbitrary viewpoint position via a virtual camera. The information processing system 10 can convert the relayed stereoscopic image into a two-dimensional image. The information processing system 10 can deliver in real time to a 2D video device such as a smartphone owned by a viewer who watches the state of a remote conference.

In the information processing system 10, the state of the remote conference in progress in the 3DCG space can be viewed in real time without burdening the viewer with the environment maintenance such as the dedicated device for viewing the 3D image and the bandwidth expansion of the communication line. ..

The information processing system 10 can receive messages (reaction signals such as characters, voices, and actions) from a viewer who views a two-dimensional image. For example, the information processing system 10 can move the viewpoint position relayed by the virtual camera in the 3DCG space where the remote conference is proceeding based on the received message. The stereoscopic image relayed by the virtual camera whose viewpoint position has been updated is converted into a two-dimensional image and distributed to the viewer. The information processing system 10 can reflect the viewer's intention in the two-dimensional video being distributed. In the information processing system 10, a two-dimensional image can be distributed in consideration of the reaction and viewpoint of the viewer. That is, for example, it is desired to distribute a video in consideration of the reaction and viewpoint of the viewer who watches the exchange between users who participate in the conference as a speaker after the fact or in real time, and this can be realized by the information processing system 10. ..

The information processing system 10 can record and store information indicating the state of an object in the 3DCG space in which a remote conference is proceeding and the state of a virtual camera in association with a time code. The information processing system 10 can reproduce a stereoscopic image of a remote conference based on the information recorded and stored in association with the time code. The information processing system 10 can deliver the reproduced two-dimensional image of the remote conference to the viewer's 2D image device to be viewed after the fact. The information processing system 10 can also receive a message from the viewer regarding the two-dimensional video to be delivered after the fact. In the information processing system 10, it is possible to deliver a two-dimensional image in consideration of the reaction and viewpoint of the viewer even after the fact.

According to the information processing system 10 according to the present embodiment, it is possible to provide a technique capable of real-time and asynchronous bidirectional participation of events in a three-dimensional space drawn by CG using a two-dimensional video device.

[Modification example]
In the embodiment, the content provided by the information processing system 10 has been described as the content using the 3DCG space. The information processing system 10 captures, for example, an actual conference hall with a camera equipped with an image sensor such as a CCD (Charge-Coupled Device) or CMOS (Complementary Metal-Oxide Semiconductor), and an object such as an avatar is captured in the captured conference hall. May be in the form of arranging. By using an actual conference hall, it is expected that the sense of presence and immersiveness in remote conferences will be improved.

FIG. 7 is a configuration diagram showing an example of an information processing system including a camera RC for photographing a conference hall A2 in a real space. The camera RC connects to an external network and acquires real-space images that serve as a conference hall for remote conferences. There may be a plurality of camera RCs that capture images of a conference hall in real space.

It is assumed that the camera RC that captures the image of the conference hall in the real space is fixed to the conference hall. The scene manager SM, for example, analyzes a moving image frame of an image captured using camera information such as the position, tilt, and angle of view of the camera RC, and estimates the visual axis of the camera that shoots the conference hall. Then, the scene manager SM specifies a world coordinate system in which an object such as an avatar drawn by 3DCG is arranged based on the fixed visual axis of the camera. The scene manager SM sets a marker or the like as a reference for arranging an object created by 3DCG in the world coordinate system. Then, the scene manager SM may arrange an object such as an avatar created by 3DCG at the marker position and provide the contents of the remote conference described in the embodiment. If the camera RC that shoots the real space is not fixed, a marker (real marker) that serves as an estimation reference for the shooting position may be installed in the real space.

To the conference participants of the remote conference, for example, a three-dimensional space in which the conference hall being photographed in the real space and the avatar created by 3DCG are combined is provided as a stereoscopic image viewed from the avatar viewpoint. Even in a conference hall photographed from a real space, conference participants can operate their own avatars to perform actions such as speaking and interacting in the conference, working with others, and moving within the conference hall.
The distributor Un, which relays the state of the remote conference, can operate the virtual camera VCn in the same manner as in the embodiment, and can relay the stereoscopic image viewed from an arbitrary viewpoint position in the synthesized three-dimensional space.

The information processing system 10 may photograph a subject such as a conference participant or a distributor, and synthesize an image of the photographed subject in the real space into the 3DCG space. Chroma key composition is exemplified as the composition of the image of the subject in the real space and the image of the 3DCG space. 8 and 9 are diagrams for explaining shooting of a subject in real space when chroma key composition is used.

In FIGS. 8 and 9, the background 20 is a background of a predetermined color such as a green background or a blue background. In chroma key compositing, a 3DCG image is composited in the area where the background 20 of a predetermined color is captured. The image of the subject 21 is photographed by the camera 22 against the background of a real space of a predetermined color designated by the background 20. The camera 22 captures an image of the subject 21 in the real space together with camera information such as the arrangement position, tilt, and angle of view of the camera, for example. The position of the subject 21 is specified by the sensor 23. The camera 22 may be, for example, a depth camera capable of acquiring the depth of the space to be photographed.

As shown in FIG. 9, the subject 21 may include an HMD 25 capable of detecting the movement of its own head and a motion sensor 24 capable of detecting the movement of its arm. When the subject 21 is a distributor of a stereoscopic image provided by the information processing system 10, it includes an operation device of a virtual camera VC that relays the synthesized 3DCG space from an arbitrary viewpoint position as described in the embodiment. .. The movement of the subject 21 detected via the HMD 25 and the motion sensor 24 is reflected in the movement and behavior in the synthesized 3DCG space.

FIG. 10 is a diagram showing an example of a composite video. In FIG. 10, a state in which the image 26 in the 3DCG space is synthesized with respect to the partial region in the image with respect to the background 20 of the predetermined color shown in FIG. 9 is illustrated. In the information processing system 10, when the subject 21 is a conference participant, a stereoscopic image obtained by synthesizing the subject 21 and the image 26 created in the 3DCG space is provided to the conference participant in the remote conference.

FIG. 11 is a diagram showing an example of a composite video displayed on a 2D video device. In FIG. 11, in the screen displayed on the 2D video device 27, the subject 21 equipped with the HMD or the like and the video 26 in the 3DCG space are displayed in a combined state. The 2D video device 27 in FIG. 11 may be a device that allows the naked eye to view a stereoscopic video in which the subject 21 equipped with the HMD or the like and the video 26 in the 3DCG space are combined. When the subject 21 is a distributor of a stereoscopic image provided by the information processing system 10, for example, an event in progress in the synthesized 3DCG space is relayed from an arbitrary viewpoint position, and an explanation of the event or an event is reached. The background can be notified to the viewer by voice.

Next, a mode in which a subject photographed by the camera participates in a remote conference will be described. FIG. 12 is an explanatory diagram of a mode in which a subject wearing an HMD participates in a remote conference. In FIG. 12, “C11” represents a camera that captures the subject Ua described with reference to FIGS. 8 and 9. Further, "R11" represents an image displayed in real time on the HMD worn by the subject Ua, and "R12" represents an image distributed to a viewer connected to an external network. “R13” represents a real-time image of a remote conference in which the subject Ua taken by the camera C11 is combined. Subject Ua participates remotely while watching the image displayed on the HMD. For example, the HMD displays an image of the viewpoint position corresponding to the movement of the head of the subject Ua detected via the HMD.

As described with reference to FIGS. 8 and 9, the subject Ua is provided with a background capable of chroma key composition such as a green background, and is a camera C11 capable of capturing RGB images, or a background from the captured image area. Take a picture of yourself with the depth camera C11 that can be cut out. The camera C11 records camera information including at least the camera position (Pos (x, y, z)) and rotation (Rot (p, y, r)) in association with the time code TC, and records the camera information and the subject. Output the Ua image to the scene manager SM.

The scene manager SM synthesizes the image of the camera C11 and the object of the 3DCG by matching the time code TC. As shown in the image R13, the image of the camera C11 synthesized by the scene manager SM and the object in the 3DCG space generate a chroma key-combined real-time image including the subject Ua. In the process of generating the image R13, for example, the color space of the RGB image for each frame taken by a green background or the like is rotated, and HSB (Hue) is performed.
, Saturation, Brightness) Convert to space. Then, in the above generation process, the background removal process for the converted image and the compositing process with the 3DCG image viewed from the visual axis direction of the camera C11 are performed in real time. The generated video R13 is distributed as a two-dimensional video to the viewer's 2D video device connected to the external network via the distribution server CH.

FIG. 13 is an explanatory diagram of a mode in which a subject without an HMD participates in a remote conference. Even when the subject does not wear the HMD, the same processing as in FIG. 12 is performed. However, the subject Ua may participate while viewing the image displayed on the display device capable of viewing the stereoscopic image of the remote conference progressing in real time with the naked eye, or the 2D image converted by the information processing system 10. For example, the scene manager SM may save the data of the remote conference in which the image of the camera C11 and the object in the 3DCG space are combined in the auxiliary storage device 103 as an offline moving image file (SV). The scene manager SM can store the data of the remote conference in the auxiliary storage device 103 in the same manner for the form shown in FIG.

The form shown in FIG. 13 can also be applied to, for example, a video of a remote conference to be reproduced after the fact. FIG. 14 is an explanatory diagram of a mode in which a subject without an HMD participates in a remote conference that is reproduced after the fact. In FIG. 14, an image stored as an offline moving image file (SV) in the auxiliary storage device 103 or the like is reproduced as a viewable image of the subject Ua. However, the post-reproduced video may be a compatible moving image such as another document video or a video or sequence in which a scene is recorded.

In FIG. 14, the scene manager SM of the information processing system 10 reads and reproduces an offline moving image file (SV) stored in the auxiliary storage device 103. The reproduced offline moving image file (SV) is displayed as a video R11 on a display device on which the subject Ua can be viewed. Here, the video R11 may be a stereoscopic video or a 2D video as long as the subject Ua can be viewed.

The subject Ua in FIG. 14 has a background capable of chroma key composition such as a green background, and uses a camera C11 capable of shooting RGB images or a depth camera C11 capable of cutting out the background from the captured video area. Take a picture. The camera C11 records camera information including at least the camera position (Pos (x, y, z)) and rotation (Rot (p, y, r)) in association with the time code TC, and records the camera information and the subject. Output the Ua image to the scene manager SM. The scene manager SM synthesizes the image of the camera C11 and the object of the 3DCG by matching the time code TC, and generates the image shown in the image R13 in which the subject Ua is chromakey-combined. The process of generating the video R13 has been described with reference to FIG.

The video R13 generated using the offline video file (SV) is distributed as a two-dimensional video to the viewer's 2D video device connected to the external network via the distribution server CH. A two-dimensional image including the subject Ua who participates in the remote conference as a subsequent participant is delivered to the viewer in the stereoscopic image reproduced using the offline moving image file (SV). The scene manager SM can save the video data in which the subject Ua is combined with the reproduced offline moving image file (SV) in the auxiliary storage device 103.

In the form shown in FIG. 14, for example, acting including actions such as gestures and gestures of the subject Ua is reflected in the reproduced offline moving image file (SV). Depending on the performance of the subject Ua, for example, it is possible to synthesize an image as if the subject Ua exists in a remote conference that has been conducted in the past, and record the image in association with the time code TC.
As a result, in the information processing system 10 of the form shown in FIG. 14, it becomes possible to share data such that a person who could not participate in the conference in real time participates in the conference without wearing the HMD and records his / her remarks. ..
Since the offline moving image file (SV) stored in the auxiliary storage device 103 can be used recursively, it is possible to perform a "meeting beyond time" in which a plurality of meeting participants join back and forth on the time code.

FIG. 15 is a flowchart showing a recording process of an event related to an event when a camera for photographing a subject in a real space is provided. When the information processing system 10 includes a camera that captures a subject in real space, for example, the scene manager SM executes the processes S31 to S33 shown in FIG. 15 before executing the process S1 shown in FIG. do it.

In the process of S31 of FIG. 15, the scene manager SM determines the existence of a camera (real camera) that captures a subject in the real space. If the actual camera exists (S31, Yes), the scene manager SM shifts to the process of S33. On the other hand, when the actual camera does not exist (S31, none), the scene manager SM shifts to the process of S32. In the process of S32, the scene manager SM generates the content image of the 3DCG space described in the embodiment. Further, in the processing of S33, the scene manager SM generates a content image (composite image) synthesized by matching the coordinate system of the image of the subject taken by the actual camera described in the modified example with the object in the 3DCG space. To do. In the scene manager SM that executes the process shown in FIG. 15, the event recording process related to the event shown in FIG. 4 is performed for the object of the video content generated in the process of S32 or S33.

The process shown in S31 of FIG. 15 may be, for example, an operation input of a content provider or a distributor. For example, when each of the content generated by the process of S32 and the content generated by the process of S33 can be provided, the scene manager SM can select the content to be generated according to the operation input.

As another modification, the information processing system 10 may collect and analyze messages from viewers and evaluate the viewpoint of the distributor. Since the distributors who are highly evaluated have a relative number of distribution applicants, for example, an advertising effect such as an advertisement included in the distribution video can be expected. Further, in the information processing system 10, it is possible to grasp the tendency of the viewer's interest by collecting and analyzing the message from the viewer. It is also possible to employ a viewer as a distributor as a service form of video distribution after the fact provided by the information processing system 10. It is possible to direct and edit the delivered video from the viewer's point of view.

10 Information processing system 20 Background 21 Subject 22 Camera 23 Sensor 24 Motion sensor 25 HMD
26 Video 27 2D Video Device 100 Computer 101 CPU
102 Main storage 103 Auxiliary storage 104 I / O IF
105 Communication IF
106 Connection bus C, C1, C2, C3 Camera processor C11 camera VC, VC1, VC2, VC3 Virtual camera (virtual camera)
P1, P2 Avatar R, R1, R2, R3 Renderer RC camera (actual camera)
SM Scene Manager SV Offline Video File

Claims (16)

A video data generation means for generating multi-view 3D video data of a shooting target including an avatar assigned to a person in real space, and
A means for accepting a viewpoint setting for the multi-viewpoint 3D moving image data in accordance with an operation on an input device by a person to which the avatar is assigned.
A moving image generation means for generating a moving image when the shooting target is viewed from the received viewpoint, and
A means for providing the generated moving image to the viewer device, and
A recording means for recording the set viewpoint and
With
The moving image data generating means operates the avatar in response to acquiring the detection result of the sensor.
Information processing system. The information processing system according to claim 1, wherein the moving image data generating means generates the multi-viewpoint three-dimensional moving image data including the viewpoint of the avatar. Video data generation means for generating multi-view 3D video data of the person to be photographed, and
A means for accepting a viewpoint setting for the multi-viewpoint 3D moving image data in accordance with an operation on an input device by the shooting target person,
A moving image generation means for generating a moving image when the shooting target is viewed from the received viewpoint, and
A means for providing the generated moving image to the viewer device, and
A recording means for recording the set viewpoint and
With
The moving image data generation means is an information processing system that generates the three-dimensional moving image data by synthesizing a real space image including the person to be photographed taken by a photographing device and a virtual three-dimensional space. The information processing system according to any one of claims 1 to 3 , wherein the moving image generating means generates the moving image by converting the multi-viewpoint three-dimensional moving image data into a two-dimensional image. The information processing system according to claim 3 or 4 , wherein the providing means provides the generated moving image to the viewer device for displaying a two-dimensional image. The information processing system according to claim 3 , wherein the shooting target person is a distributor who distributes a moving image. The information processing system according to any one of claims 1 to 6 , further comprising means for receiving a reaction to the moving image from the viewer device. The information processing system according to claim 7 , wherein the recording means further records the time in the multi-viewpoint three-dimensional moving image data and the reaction. The eighth aspect of the present invention, wherein the moving image generating means reproduces the moving image based on the viewpoint recorded in the recording means and the time in the multi-viewpoint three-dimensional moving image data recorded in the recording means. Information processing system. The information processing system according to claim 9, wherein the moving image generating means changes the viewpoint related to the moving image to a viewpoint different from the viewpoint recorded in the recording means based on the reaction. The information according to any one of claims 1 to 10 , further comprising means for synthesizing the multi-viewpoint three-dimensional moving image data with an image obtained after the time when the multi-viewpoint three-dimensional moving image data is generated. Processing system. The information processing system according to any one of claims 1 to 11 , wherein each means is realized by a computer connected to each other. Computers that connect to each other
A step to generate video data to generate multi-view 3D video data of the shooting target including an avatar assigned to a person in real space, and
A step of generating a moving image when the shooting target is viewed from the received viewpoint based on the viewpoint setting for the multi-viewpoint 3D moving image data in accordance with the operation of the input device by the person to whom the avatar is assigned. When,
The step of providing the generated video to the viewer device, and
The step of recording the set viewpoint and
And
The step of generating the moving image data operates the avatar in response to acquiring the detection result of the sensor.
Information processing method to execute. Computers that connect to each other
Taken by the imaging device, and a video of the real space including the imaging subject, and generating a multi-view three-dimensional moving image data of the imaging subject by synthesizing the virtual three-dimensional space,
A step of generating a moving image when the shooting target is viewed from the received viewpoint based on the viewpoint setting for the multi-viewpoint 3D moving image data in accordance with the operation of the shooting target person on the input device.
The step of providing the generated video to the viewer device, and
The step of recording the set viewpoint and
Information processing method to execute. Generate video data that generates multi-view 3D video data for shooting targets, including avatars assigned to people in real space.
Along with the operation of the input device by the person to whom the avatar is assigned, a moving image when the shooting target is viewed from the received viewpoint is generated based on the viewpoint setting for the multi-viewpoint 3D moving image data.
The generated video is provided to the viewer device,
Record the set viewpoint and
The means for generating the moving image data for generating the multi-viewpoint three-dimensional moving image data operates the avatar in response to acquiring the detection result of the sensor.
An information processing program that causes a computer to perform processing. Taken by the imaging device, and generates the image of the real space including the imaging subject, a multi-view three-dimensional moving image data of the imaging subject by synthesizing the virtual three-dimensional space,
Along with the operation of the input device by the shooting target person, a moving image when the shooting target is viewed from the received viewpoint is generated based on the viewpoint setting for the multi-viewpoint 3D moving image data.
The generated video is provided to the viewer device,
Record the set viewpoint,
An information processing program that causes a computer to perform processing.

Images