JP2022177053A - Information processing system, information processing method, and information processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technology that enables real-time and asynchronous bidirectional participation of an event in a 3D space drawn by CG using a 2D video device.

SOLUTION: An information processing system includes video data generation means that generates multi-viewpoint 3D video data obtained by shooting a shooting target from multiple viewpoints, means that accepts a viewpoint setting for multi-view 3D video data, means that generates a video when the shooting target is viewed from the received viewpoint, means that provides the generated video to a viewer device, means that receives reaction to the video from the viewer device, and means that records the set viewpoint, a time in the multi-view 3D video data, and the reaction.

SELECTED DRAWING: Figure 1

COPYRIGHT: (C)2023,JPO&INPIT

Description

Patent Law Article 30, Paragraph 2 application filed (Publication 1) Posting date June 20, 2016 http://cedec. cesa. or. jp/2016/whatsnew/session. html#session0620 http://cedec. cesa. or. jp/2016/session/AC/8656. html (Publication 2) Publication date August 22, 2016 https://www. youtube. com/watch? v=KwOZGsZN2A8 (Public 3) Date August 24, 2016 to August 26, 2016 Meeting Name Computer Entertainment Developers Venue Pacifico Yokohama Conference Center Conference (CEDEC2016) (1-1-1 Minatomirai, Nishi-ku, Yokohama) ) (Publication 4) Publication date September 13, 2016 http://blog. Shirai. la/blog/2016/09/tgs2016/ (Public 5) Exhibition date September 15, 2016 to September 18, 2016 Exhibition name Tokyo Game Show 2016 Venue Makuhari Messe (2-Nakase, Mihama-ku, Chiba-shi 1) (Publication 6) Posting date September 20, 2016 http://blog. Shirai. la/blog/2016/09/dcexpo2016/ (Publication 7) Posting date October 5, 2016 http://blog. Shirai. la/blog/2016/10/virtual-cafe/

The present disclosure relates to an information processing system, an information processing method, and an information processing program.

In recent years, with the development of ICT (Information and Communication Technology), various contents using a virtual three-dimensional space drawn by CG (Computer Graphics) have been proposed. Examples of such content include VR (Virtual Reality) content, game content, e-sports (electronic sports) content, and the like.

A user (hereinafter also referred to as a user) who uses the content described above can, for example, wear a device capable of viewing 3D video such as an HMD (Head-Mount Display), thereby creating a virtual 3D image rendered by CG. It becomes possible to stereoscopically view the dimensional space from the user's viewpoint. Furthermore, when a sensor or the like that detects the movement and position of the user's body such as the head and arms is worn, for example, it can move within the three-dimensional space according to the user's motion, and can be arranged within the three-dimensional space. It is possible to perform actions such as working on the object that has been touched. Objects arranged in the three-dimensional space include, for example, objects (such as characters and avatars) that can be operated by other users who use the content.

In content such as games and e-sports, for example, it is possible to respond to an event caused by one's own actions, have a conversation with an object operated by another user, have a competition, have a match, and the like. Users participating in the content can enjoy collisions and physical interactions in the three-dimensional space due to events caused by the above actions from their respective viewpoints according to predetermined rules defined by the content.

The following patent documents exist as prior art documents that describe technology related to the technology described in this specification.

JP 2011-150402 A Japanese Patent Application Laid-Open No. 2001-300131 JP-A-11-123280

As a form of content, three-dimensional spaces such as conference halls, offices, schools, and shopping malls are drawn with CG, and multiple users participate in the three-dimensional space in real time. Remote meetings, VR offices, e-learning, e- Commerce, etc. are assumed.

In the above form, for example, not only can users participate in real-time interaction as speakers in the drawn three-dimensional space, but also distribution for users who watch video after the fact or as listeners, and the viewing environment / Demand for viewing formats and media conversion is expected.

In the case of viewing distributed 3D space images after the fact, for example, there is a burden associated with purchasing a dedicated device for viewing 3D images such as an HMD and preparing a viewing environment for receiving 3D images. is assumed.

An object of the present disclosure is to provide technology capable of reproducing a moving image from an accepted viewpoint based on multi-viewpoint 3D moving image data to be captured.

One aspect of the disclosed technology is exemplified by an information processing system. This information processing system includes moving image data generating means for generating multi-viewpoint 3D moving image data of an object to be shot, means for receiving setting of viewpoints for the multi-viewpoint 3D moving image data, a moving image generating means for generating a moving image when viewing the video, means for providing the generated moving image to a viewer device, and recording means for recording the set viewpoint.

According to one aspect of the disclosed technique, it is possible to provide a technique capable of reproducing a moving image from a received viewpoint based on multi-viewpoint 3D moving image data of a shooting target.

1 is a configuration diagram showing an example of an information processing system; FIG. 1 is a configuration diagram showing an example of a hardware configuration of a computer; FIG. FIG. 4 is a diagram showing an example of a data structure saved for objects in a 3DCG space; FIG. 11 is a flow chart illustrating an event recording process related to an event of a scene manager; FIG. FIG. 11 is a flowchart illustrating setting processing of a virtual camera of a scene manager; FIG. FIG. 4 is an explanatory diagram for explaining setting of a camera direction of a virtual camera; 1 is a configuration diagram showing an example of an information processing system provided with a camera that captures images of a conference hall in real space; FIG. FIG. 10 is a diagram for explaining photographing of a subject in real space when using chromakey synthesis; FIG. 10 is a diagram for explaining photographing of a subject in real space when using chromakey synthesis; It is a figure which shows an example of a synthetic image. FIG. 10 is a diagram showing an example of a composite video displayed on a 2D video device; FIG. 4 is an explanatory diagram of a form in which a subject wearing an HMD participates in a remote conference; FIG. 10 is an explanatory diagram of a mode in which a subject not wearing an HMD participates in a remote conference; FIG. 10 is an explanatory diagram of a form in which a subject not wearing an HMD participates in a remote conference that is played back after the fact; 10 is a flow chart illustrating an event recording process related to an event of a scene manager when a camera that captures a real space is provided.

An information processing system according to an embodiment will be described below with reference to the drawings. The configurations of the following embodiments are examples, and the information processing system is not limited to the configurations of the following embodiments.

<Embodiment>
[1. System configuration〕
FIG. 1 is a configuration diagram showing an example of an information processing system according to this embodiment. First, an outline of the information processing system 10 shown in FIG. 1 will be described. In the form of FIG. 1, for example, a conference hall is provided in a three-dimensional space created by CG (hereinafter also referred to as "3DCG space"), and contents such as remote conferences are provided. Here, "remote conference" refers to a conference held in real time by a plurality of conference participants connected to a network, sharing a conference center in a 3DCG space. A plurality of conference participants connected to the network perform actions such as speaking and interacting in the conference, reaching out to others, and moving through objects (avatars, etc.) that can be manipulated in the 3DCG space. The proceedings of the conference are provided to the HMDs and the like of the conference participants as 3DCG moving images viewed from the viewpoint of each avatar.
However, the content provided by the information processing system 10 may be VR offices, games, e-learning, etc. that are virtually created in a 3DCG space. The content only needs to be able to provide collisions and physical interactions due to events caused by user actions and movements in the 3DCG space according to predetermined rules defined by the content.
Content provided by the information processing system 10 will be described below as a remote conference using a conference hall provided in a 3DCG space. In addition, moving images in three-dimensional space created by CG are also referred to as “stereoscopic images”.

In the 3DCG space, for example, objects of various configurations, such as desks, chairs, whiteboards, walls, floors, windows, doors, and ceilings, which constitute the conference hall of the remote conference, are arranged. Similarly, avatars for multiple conference participants participating in the remote conference are arranged in the 3DCG space. Avatars are objects that can be manipulated by conference participants. An object placed in the 3DCG space is, for example, a collection of polygons, which are rendering elements. Each polygon forming an object has a common coordinate axis. The shape, position, orientation, etc. of an object in the world coordinate system within the 3DCG space are represented by three-dimensional coordinates (X, Y, Z) on the coordinate axes.

A conference participant is equipped with, for example, a device that can visually recognize a stereoscopic image such as an HMD (hereinafter also referred to as a 3D image device), a sensor device that detects the movement and position of the body, and an audio device such as a microphone or headphones. I will participate in. The state of the remote conference in progress is provided to the conference participants in real time as a stereoscopic image through each avatar's viewpoint. Conference participants view 3D images of the process accompanying the progress of the conference from the avatar's point of view, operate the avatars while watching the 3D images, move around the conference venue, and interact with and encourage other conference participants (behavior). I do. Speech in a remote conference is made through an audio device such as a microphone or headphones, for example. Events (for example, actions such as conversations, proposals, leaving seats, etc.) corresponding to the actions of each conference participant occurring in the process accompanying the progress of the conference are reflected in the stereoscopic video being viewed.

The information processing system 10 according to the present embodiment has a function of relaying a 3DCG space in which a remote conference proceeds as a stereoscopic image viewed from an arbitrary viewpoint position. A viewpoint position in the 3DCG space where the remote conference proceeds is designated by, for example, a virtual camera. The virtual camera can be moved to any position within the 3DCG space where the remote conference is proceeding. The virtual camera relays objects and avatars placed in the conference hall, and phenomena such as events occurring in the process accompanying the progress of the conference as stereoscopic images viewed from arbitrary viewpoint positions. If there is a relay relaying the 3DCG space in which the remote conference is proceeding, the viewpoint position of the virtual camera is set according to the relay operator's operation.

The information processing system 10 according to this embodiment has a function of converting a stereoscopic video relayed by a virtual camera into a two-dimensional video. The converted two-dimensional video is distributed to viewers watching the remote conference in real time or after the fact, for example. A viewer views a delivered video of a remote conference via a device capable of viewing two-dimensional video (hereinafter also referred to as a 2D video device) such as a smartphone, a mobile phone, a tablet PC (Personal Computer), for example.

The information processing system 10 according to the present embodiment has a function of receiving messages (reaction signals such as text, voice, and actions) from viewers who view two-dimensional images. The information processing system 10, for example, moves the viewpoint position of the virtual camera in the 3DCG space where the remote conference progresses, based on the received message. For example, the information processing system 10 notifies the relayer of the received message, and moves the viewpoint position of the virtual camera according to the operation of the notified relayer. When the remote conference is in progress, the stereoscopic video relayed by the virtual camera is updated according to the movement of the viewpoint position, and the viewpoint position received from the viewer is reflected in the two-dimensional video being viewed.

The information processing system 10 according to this embodiment has a function of recording events in the 3DCG space in which the remote conference proceeds, and virtual camera information (viewpoint position, focal length, rotation, zoom parameters, etc.). The event in the 3DCG space includes the coordinates of each object that moves within the 3DCG space according to the event that occurs in the process of progressing the conference and that accompanies the actions of each conference participant. Events in the 3DCG space include voice data of conference participants. For example, minutes or the like of the remote conference are created based on the recorded voice data or the like. The information processing system 10 generates a 3D image based on the viewpoint position of the virtual camera based on the recorded information, and distributes a 2D image based on the generated 3D image to viewers after the event. Messages from viewers are also received in subsequent viewings, and viewpoint positions of the viewers after viewing are reflected in the two-dimensional video being viewed.

Next, the configuration of the information processing system 10 shown in FIG. 1 will be described.
The information processing system 10 shown in FIG. 1 mainly includes a scene manager SM, a virtual camera VC, a camera processor C, a renderer R, and a distribution server CH. Each of the above configurations is connected via a network such as a LAN (Local Area Network). Multiple virtual cameras VC, camera processors C, and renderers R can be connected to the network. FIG. 1 illustrates virtual cameras VC1, VC2, VC3, camera processors C1, C2, C3, and renderers R1, R2, R3. In the following description, a plurality of virtual cameras VC1, VC2, and VC3 are collectively referred to as virtual camera VC. The camera processor C and renderer R are the same.

Also, the scene manager SM and distribution server CH are connected to an external network. External networks include public networks such as the Internet, mobile phone networks, and wired and wireless LANs (Local Area Networks). A plurality of 2D video devices of viewers who view 2D video are connected to the external network. The information processing system 10 may include a plurality of scene managers SM and distribution servers CH. Also, the information processing system 10 may have the functions of the virtual camera VC, the camera processor C, and the renderer R integrated.

In FIG. 1, A1 represents the stereoscopic video being viewed by the distributor U1 who operates the viewpoint position of the virtual camera VC1. The stereoscopic video A1 being viewed by the distributor U1 includes the avatars of the conference participants (P1, P2, P3) of the remote conference viewed from the viewpoint position of the virtual camera VC1. Note that the distributor U1 who operates the viewpoint position of the virtual camera VC1 is not drawn in the stereoscopic video of the avatar's viewpoint position.

The virtual cameras VC2 and VC3 in the stereoscopic video A1 are virtual cameras VC arranged as objects in the 3DCG space. Virtual camera VC2 follows the movement of conference participant P2, for example. Virtual camera VC3 follows the movement of conference participant P3, for example.

A distributor U1 wears an HMD1 for viewing stereoscopic video from a virtual camera VC1 operated by himself. The HMD 1 worn by the distributor U1 includes audio devices such as headphones and an intercom. Also, the distributor U1 has a controller CN for designating the viewpoint position of the virtual camera VC1 in the 3DCG space, for example. The controller CN is a user input device that operates two viewpoint positions for stereoscopically viewing a 3DCG space. The controller CN includes a sensor (acceleration sensor, gyro sensor, geomagnetic sensor, image sensor, etc.) that detects the rotation angle of the virtual camera VC1. Note that the virtual cameras VC1, VC2, and VC3 may have a function of switching the stereoscopic video being relayed. Also, there may be a plurality of distributors who operate the viewpoint position of the virtual camera VC1.

The scene manager SM is a platform on which an application program that creates stereoscopic video in a 3DCG space such as a remote conference operates. The platform is, for example, a computer system configured by an information processing device such as a server and WS (WorkStation).

The scene manager SM creates a 3DCG space image by executing an application program, and provides remote conference participants with a stereoscopic image in real time through the avatar's viewpoint. Similarly, the scene manager SM creates a stereoscopic image of the 3DCG space viewed from the viewpoint position of the virtual camera VC by executing the application program, and transmits it to the renderer R. FIG. The scene manager SM outputs stereoscopic video in the 3DCG space seen from the respective viewpoint positions of the virtual cameras VC1, VC2, and VC3 to the distributor U1 who operates the viewpoint position of the virtual camera VC1. The distributor U1 views stereoscopic video in the 3DCG space viewed from the viewpoint positions of the virtual cameras VC1, VC2, and VC3 via an HMD or the like. Note that the stereoscopic video displayed on the HMD or the like is switched via a controller CN that receives an operation input from the distributor U1.

The scene manager SM stores and manages the data of objects such as structures and avatars that make up the conference room in the 3DCD space in association with time codes. Here, the "time code" is a counter that expresses absolute time in the 3DCG space. All events related to events that occur in the process of progressing remote meetings are stored and managed by timecode.

The scene manager SM receives messages sent from 2D video devices such as smartphones connected to an external network. The message includes reaction signals such as characters, voices, and actions from viewers of the two-dimensional video distributed after the event or in real time. The scene manager SM moves the viewpoint position of the virtual camera VC within the 3DCG space in which the remote conference proceeds based on the received message. Alternatively, the scene manager SM notifies the distributor U1, who operates the viewpoint position of the virtual camera VC1 in the 3DCG space, of the viewpoint position based on the received message. The distributor receives instructions from the scene manager SM via, for example, notification of movement of the viewpoint position displayed on the HMD (characters, markers, etc.) or voice notification, etc., and moves the viewpoint position of the virtual camera VC1. . Note that the message received by the scene manager SM is stored in association with the time code.

The function provided by the scene manager SM by executing a computer program is an example of "moving image data generating means for generating multi-viewpoint three-dimensional moving image data obtained by photographing an object to be photographed from multiple viewpoints". Similarly, the function provided by the scene manager SM by executing a computer program is an example of "means for receiving reactions to moving images from viewer devices."
Similarly, the functions provided by the scene manager SM by executing a computer program are "means for recording set viewpoints, times in multi-viewpoint 3D video data, and reactions".
is an example.

The virtual camera VC is a virtual device that designates a viewpoint position for relaying stereoscopic video in a 3DCG space in which a remote conference is proceeding. The virtual camera VC has two viewpoint positions and a convergence point determined from the two viewpoint positions as camera information in order to relay the 3DCG space as a stereoscopic image. In the virtual camera VC that is not operated by the distributor U1, the arrangement position in the 3DCG space is determined by the scene manager SM. For example, the virtual camera VC may be rendered as an object visible to the conference participants in the 3DCG space, or may be invisible to the conference participants. Also, the objects of the virtual camera VC may be drawn only within the stereoscopic video viewed by the distributor U1, such as the virtual cameras VC2 and VC3 in the stereoscopic video A1, and may not be shown to the viewers. . Virtual camera information such as the focal length, zoom parameters, rotation, and viewpoint position of the virtual camera VC (hereinafter also simply referred to as "camera information") is controlled by the scene manager SM or the distributor U1 who operates the virtual camera VC. . The virtual camera VC is an example of "means for accepting setting of viewpoints for multi-viewpoint three-dimensional video data".

The camera processor C is a computer that records and saves the camera information of the virtual camera VC in association with the time code. Based on the camera information of the virtual camera VC, the camera processor C cooperates with the scene manager SM to generate stereoscopic video in the 3DCG space in which the remote conference is proceeding. The camera processor C outputs to the renderer R the camera information of the virtual camera VC and the stereoscopic video in the 3DCG space viewed from the viewpoint position of the virtual camera VC.

The renderer R is a computer that performs a so-called rendering process that converts a stereoscopic image in a 3DCG space, the viewpoint position of which is designated by the virtual camera VC, into a two-dimensional image (video). In rendering processing, coordinate transformation such as affine transformation of an object in 3DCG space to a two-dimensional plane, and drawing processing such as lighting processing are performed. converted to pixel-by-pixel data seen from . The renderer R creates a two-dimensional video from the frame-based two-dimensional image rendered based on the converted pixel-based data, and outputs the created two-dimensional video to the distribution server CH. The camera processor C and the renderer R are an example of ``means for generating a moving image when the subject is viewed from the received viewpoint''.

The distribution server CH is a computer that distributes a two-dimensional image (video) output from the renderer R to a 2D image device such as a smartphone connected to an external network. The distribution server CH performs distribution switching of the 2D video output from the plurality of renderers R1, R2, R3, etc. after the fact via the 2D video device or according to a request from the viewer who is watching in real time. The distribution server CH is an example of "means for providing the generated moving image to the viewer device".

[2. Device configuration〕
FIG. 2 is a configuration diagram showing an example of the hardware configuration of the computer. The scene manager SM, camera processor C, renderer R, and distribution server CH shown in FIG. 1 are implemented using the computer 100 shown in FIG. A computer 100 illustrated in FIG. 2 includes a CPU (Central Processing Unit) 101, a main storage device 102, an auxiliary storage device 103, a communication IF (Interface) 104, and an input/output IF 105, which are interconnected by a connection bus . CPU 101 is also called a processor. However, the CPU 101 is not limited to a single processor, and may have a multiprocessor configuration. Also, a single CPU 101 connected by a single socket may have a multi-core configuration. Moreover, each of the above components may be provided in plurality, or some components may not be provided.

A CPU 101 is a central processing unit that controls the entire computer 100 . The CPU 101 expands the programs stored in the auxiliary storage device 103 in a work area of the main storage device 102 in an executable manner, and controls peripheral devices through the execution of the programs to provide functions that meet predetermined purposes. The main storage device 102 is a storage medium in which the CPU 101 caches programs and data and develops work areas. The main storage device 102 includes, for example, flash memory, RAM (Random Access Memory), and ROM (Read Only Memory).

The auxiliary storage device 103 is a storage medium that stores an OS (Operating System), each program executed by the CPU 101, operation setting information, and the like. The auxiliary storage device 103 is, for example, a HDD (Hard-disk Drive), SSD (Solid State Drive), EPROM (Erasable Programmable ROM), flash memory, USB (Universal Serial Bus) memory, or the like. The auxiliary storage device 103 includes a CD (Compact Disc) drive device, a DVD (Digital Versatile Disc) drive device, a BD (Blu-ray (registered trademark) Disc) drive device, and the like. Examples of recording media include silicon disks including nonvolatile semiconductor memories (flash memories), hard disks, CDs, DVDs, BDs, USB (Universal Serial Bus) memories, SD (Secure Digital) memory cards, and the like.

The communication IF 104 is an interface with a network connected to the computer 100 and an external network. Communication IF 104 includes a wireless communication module that performs communication based on a predetermined standard. The input/output IF 105 is an interface for inputting/outputting data with other devices connected to the computer 100 .

Execution of programs by the CPU 101 provides processing of the scene manager SM, camera processor C, renderer R, and distribution server CH shown in FIG. However, at least part of each of the processes described above may be provided by a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a GPU (Graphics Processing Unit), or the like. Similarly, at least a part of each of the above processes is a dedicated LSI (large scale integration) such as an FPGA (Field-Programmable Gate Array), a numerical processor, a vector processor, an image processor, or other digital circuits. good too. At least part of the scene manager SM, camera processor C, renderer R, and distribution server CH may include analog circuits.

[3. Data structure]
FIG. 3 is a diagram showing an example of the data structure of objects in the 3DCG space recorded and saved in the scene manager SM. As shown in FIG. 3, data about objects in the 3DCG space recorded and saved in the scene manager SM are recorded as records for each time code. The above records are "TC", "Object", "Pos(x,y,z)", "Rot(p,y,r)", "FOV(f)", "Scale(s)", "Option {Trigger, Target, . . . }”.

The "TC" field stores the counter value of a counter that expresses absolute time in the 3DCG space. The counter value is measured with a period of 0.1 ms, for example. Note that the "TC" field may store time information calculated from the measured counter value. FIG. 3 exemplifies time information in the format of hourly, minutely, and secondly, such as "h:mm:ss".

The “Object” field stores identification information that uniquely identifies an object placed in the 3DCG space. An example of data stored in the "Object" field is identification information that uniquely identifies the virtual camera VC. In FIG. 3, operable avatars (P1, P2) of remote conference participants are illustrated. Similarly, a virtual camera VC1 (HID1, HID2) operated by the distributor U1 and a virtual camera VC2 (CAM1) following the avatar are exemplified. The virtual camera VC1 has two viewpoint positions (HID1, HID2) for stereoscopically viewing the 3DCG space. The viewpoint positions (HID1, HID2) of the virtual camera VC1 are operated via a user input device provided by the distributor U1.

The "Pos (x, y, z)" field stores the coordinates in the 3DCG space of the object stored in the "Object" field. When the object stored in the "Object" field is an avatar of a conference participant, barycentric coordinates of a plurality of polygons forming the object are stored. If the object stored in the "Object" field is the virtual camera VC, the coordinates of the viewpoint position are stored. In the case of the virtual camera VC1 operated by the broadcaster U1, coordinates of viewpoint positions are stored for each of HID1 and HID2.

The "Rot (p, y, r)" field stores parameter coordinates representing the rotation of the local coordinates accompanying movement of the object stored in the "Object" field. In the case of the virtual camera VC, parameter coordinates representing the rotation direction about the pitch axis, yaw axis, and roll axis of the viewpoint position are stored. In the case of the virtual camera VC1 operated by the distributor U1, the above coordinates for each of HID1 and HID2 are stored.

The “FOV(f)” field stores a parameter representing the angle of view as camera information of the virtual camera VC. The “Scale(s)” field stores parameters for enlargement/reduction associated with the movement of the object stored in the “Object” field in the 3DCG space. The “Option {Trigger, Target, . In the case of the virtual camera VC, for example, information such as a zoom target and start/stop of the virtual camera VC1 operated by the distributor U1 is stored.

[4. Processing flow]
The processing of the scene manager SM in the information processing system 10 will be mainly described below with reference to the flowcharts shown in FIGS. The scene manager SM, camera processor C, renderer R, and distribution server CH shown in FIG. , the processing shown in FIG. The same applies to the setting processing of the virtual camera VC1, which will be described later with reference to FIG.

FIG. 4 is a flow chart showing an event recording process relating to an event of the scene manager SM. The scene manager SM records exchanges between conference participants (avatars) occurring in the process accompanying the progress of the remote conference provided via the information processing system 10 and camera information of the virtual camera VC in association with the time code TC. . The scene manager SM records, in the auxiliary storage device 103, camera information of objects and virtual cameras VC placed in the 3DCG space, for example, using the data structure described with reference to FIG. Similarly, the scene manager SM records a message from the viewer of the 2D video distributed from the distribution server CH in association with the time code TC. In the scene manager SM, the 3DCG space image created by executing the application program is provided to the remote conference participants in real time as a stereoscopic image through the avatar's viewpoint.

In the flowchart of FIG. 4, the start of processing is exemplified by the execution of an application program that provides content such as a remote conference. The scene manager SM initializes (resets) the time code TC, which is a counter representing absolute time in the 3DCG space (S1). The scene manager SM arranges, for example, various objects that make up the conference hall on the world coordinate system in the 3DCG space. Then, the scene manager SM waits until the login of the conference participants is accepted (S2). However, if the number of conference participants participating in the remote conference, their positions, etc. are notified in advance, avatars assigned to the conference participants in advance may be arranged in the 3DCG space. Note that the above objects are registered in advance, for example, in the auxiliary storage device 103 referred to by the scene manager SM. In the standby state (S2) of FIG. 4, the scene manager SM is in a state of waiting for a plurality of events exemplified in the processes of S3 to S7.

The scene manager SM receives login and logout of conference participants (players) of the remote conference provided using the 3DCG space (S3). When the scene manager SM accepts the login and logout of the conference participants (S3, acceptance), the scene manager SM records the accepted login and logout of the conference participants as a key record associated with the time code TC (S8). . The key record includes, for example, identification information that uniquely identifies the conference participant, identification information of the avatar in 3D space associated with the conference participant, and the like.

The scene manager SM provides the participants who have logged in to the conference with stereoscopic images of the 3DCG space where the remote conference will be held as seen from the avatar's viewpoint. A logged-in conference participant, for example, operates an avatar while viewing a stereoscopic image via an HMD or the like, moves between conference venues, and interacts with or encourages other conference participants. Events caused by the actions of conference participants occurring in the 3DCG space are reflected in the stereoscopic video being viewed.

The scene manager SM receives logins and logouts of a plurality of distributors Un who relay the 3DCG space in which the remote conference proceeds (S4). When the scene manager SM accepts the login and logout of the distributor Un (S4, acceptance), the scene manager SM records the accepted login and logout of the distributor Un as a key record associated with the time code TC (S8). . The key record includes, for example, identification information that uniquely identifies the distributor Un and identification information of the virtual camera VCn operated by the distributor Un. Note that the initial placement position of the virtual camera VCn operated by the distributor Un in the 3DCG space is determined by the scene manager SM.

The distributor Un operates the viewpoint positions (HID1, HID2) of the virtual camera VCn to relay the 3DCG space in which the remote conference proceeds. The viewpoint positions (HID1, HID2) of the virtual camera VCn are set at arbitrary positions in the 3DCG space. The scene manager SM outputs a stereoscopic image of the 3DCG space seen from the viewpoint position of the virtual camera VCn to the distributor Un who has accepted the login. The distributor Un operates the viewpoint position of the virtual camera VCn while viewing the stereoscopic video displayed on the HMD or the like, and relays the 3DCG space in which the remote conference progresses. The stereoscopic video relayed through the operation of the virtual camera VCn reflects, for example, movement in the 3DCG space according to events accompanying the progress of the remote conference, zooming in on a specific avatar, and the like.

The scene manager SM receives an operation input to the avatar by the conference participant (S5). When the received operation input causes movement, rotation, or enlargement/reduction of an object placed in the 3DCG space in which the remote conference is progressing (S5, Yes), the scene manager SM associates it with the time code TC and performs the above Parameters representing object movement, rotation, enlargement/reduction are recorded as key records (S9). The scene manager SM uses, for example, the data configuration shown in FIG. 3 to record various parameters about the target object.

The scene manager SM receives an operation input to the virtual camera VCn by the distributor Un (S6). When the scene manager SM receives an operation input to the virtual camera VCn by the distributor Un (S6, Yes), the scene manager SM records the camera information of the virtual camera VCn as a key record in association with the time code TC (S10). . The scene manager SM uses, for example, the data configuration shown in FIG. 3 to record camera information about the virtual camera VCn to be operated.

The scene manager SM receives messages (viewer reactions) from viewers of the 2D video distributed from the distribution server CH (S7). Messages received from viewers include reaction signals such as text, voice, and actions. When the scene manager SM receives a message from the viewer of the 2D video distributed from the distribution server CH (S7, Yes), the received message is associated with the time code TC and recorded as a viewer reaction ( S11).

The content provider (service provider) reflects, for example, evaluations and requests regarding the distributed 2D video contained in the message recorded in the scene manager SM in the provided content and service details.

Next, setting processing of the virtual camera VC1 by the scene manager SM will be described with reference to the flowchart shown in FIG. FIG. 5 is a flow chart showing setting processing of the virtual camera VC1 of the scene manager SM. The start of the processing of the flowchart shown in FIG. 5 and the processing of S1 are the same as the processing shown in FIG.

The scene manager SM receives operation input of the virtual camera VCn operated by the distributor Un via the controller CN (S21). The operation input includes camera information of the virtual camera VCn, such as viewpoint position in 3DCG space (coordinates of HID1 and HID2), zoom-in/zoom-out (enlargement/reduction), camera orientation (rotation acquisition sensor), angle of view. , start of image acquisition (HID trigger ON/OFF), and the like. The scene manager SM acquires the received camera information of the virtual camera VCn and temporarily stores it in a predetermined area of the main storage device 102 . The process of the scene manager SM shifts to the process of S22.

If the operation input includes a parameter (HID trigger ON) indicating start of video acquisition (S22, Yes), the scene manager SM proceeds to the process of S23. If the operation input does not include a parameter (HID trigger ON) indicating start of video acquisition (S22, No), the scene manager SM returns to the process of S21.

The scene manager SM determines whether a virtual camera VCn relaying from an arbitrary viewpoint in the 3DCG space where the remote conference is provided has already been arranged (S23). If the virtual camera VCn is already arranged (S23, Yes), the scene manager SM proceeds to the process of S26. If the virtual camera VCn is not yet arranged (S23, No), the scene manager SM proceeds to the process of S24.

In the process of S24, the scene manager SM sets the viewpoint positions (HID1, HID2) of the virtual cameras VCn according to a camera direction setting algorithm which will be described later with reference to FIG. The scene manager SM arranges a virtual camera VCn that relays the 3DCG space from an arbitrary viewpoint at the viewpoint positions (HID1, HID2) set in the process of S24 (S25). The arrangement position of the virtual camera VCn in the 3DCG space is represented by the middle point between the two viewpoint positions (HID1, HID2). The process of the scene manager SM shifts to the process of S27.

In the process of S26, the scene manager SM updates the position, angle, etc. of the virtual camera VCn. The scene manager SM updates the position of the virtual camera VCn from the HID1 and HID2 coordinates received in the process of S21, and updates the rotation angle of the camera from the orientation of the camera. The process of the scene manager SM shifts to the process of S27.

In the process of S27, the scene manager SM records the camera information and the like of the virtual camera VCn set in the processes of S25 and S26 in the auxiliary storage device 103. FIG. The camera information is recorded, for example, in the data structure described with reference to FIG. The process of the scene manager SM shifts to the process of S28.

In the process of S28, the scene manager SM determines whether the count value of the time code TC has reached the recordable time. Here, the recordable time is a predetermined set time, and can be obtained from the memory capacity of the auxiliary storage device 103 or the like, for example. If the count value of the time code TC has reached the recordable time (S28, Yes), the scene manager SM ends the processing shown in FIG. If the count value of the time code TC has not reached the recordable time (S28, No), the scene manager SM shifts to the processing of S21 and continues the processing shown in FIG.

Next, referring to FIG. 6, the camera orientation algorithm will be described. FIG. 6 is an explanatory diagram for explaining the setting of the camera direction of the virtual camera VCn. FIG. 6 illustrates a top view in which the viewing direction of the virtual camera VCn is rotated to the right (clockwise). The state of the virtual camera VCn shown in FIG. 6 changes in the order of (1)→(2)→(3). Note that the state shown in (1) represents the initial state.

In FIG. 6, HID1 and HID2 represent the viewpoint positions of the virtual camera VCn in the 3DCG space, and the middle point P1 between HID1 and HID2 represents the arrangement position of the virtual camera VCn. The viewpoint position and arrangement position are expressed in the world coordinate system. The viewpoint positions of HID1 and HID2 are set by the distributor Un who operates the virtual camera VCn. Note that the distance between HID1 and HID2 is associated with the angle of view (FOV) of the virtual camera VCn or focus processing (f). The angle of view (FOV) of the virtual camera VCn is increased by widening the distance between HID1 and HID2, for example, by an operation input by the broadcaster Un or by the control of the scene manager SM. ) can be widened.

The gaze point in the 3DCG space of the virtual camera VCn placed at the intermediate point P1 is represented by "VD", and the arrow pointing from the intermediate point P1 to the gaze point VD represents the visual axis direction of the virtual camera VCn in the 3DCG space. . "VD1" represents the visual axis direction by HID1, and "VD2" represents the visual axis direction by HID2. Note that P2 and P3 shown in FIG. 6 represent left and right viewpoints of the distributor Un.

The camera direction setting algorithm shown in FIG. is used to set the gaze point VD of the virtual camera VCn whose camera direction moves.

As shown in FIG. 6(1), in the initial state, end point coordinates VP1 and VP2, gaze point VD
are on the same plane perpendicular to the visual axis of the virtual camera VCn. In the state (1) of FIG. 6, the sensor value of the rotation angle of the virtual camera VCn is input by the distributor Un's operation input. The scene manager SM, for example, rotates the visual axis of HID2 to the right along the input rotation angle while maintaining the direction of the visual axis of HID1 ((2) in FIG. 6).

As shown in FIG. 6(2), the scene manager SM sets the gazing point VD of the virtual camera VCn, for example, at the middle point between the end point coordinates VP1 of HID1 and the end point coordinates VP2 of the rotated HID2. The visual axis direction of the virtual camera VCn in the 3DCG space is a straight line direction connecting the middle point P1 between HID1 and HID2 and the point of gaze VD after rotational movement.

The scene manager SM rotates the visual axis of HID2 rightward along the input rotation angle from the state shown in FIG. 6(2), and rotates the visual axis of HID1 rightward ((3 )). At this time, the amount of rotational movement of the visual axis of HID1 in the right direction corresponds to the amount of rotational movement of the visual axis of HID2 shown in FIG. 6(2).

As shown in FIG. 6(3), the scene manager SM sets the gazing point VD of the virtual camera VCn at the intermediate point between the end point coordinates VP1 of HID1 after rotational movement and the end point coordinates VP2 of HID2. The visual axis direction of the virtual camera VCn in the 3DCG space is a straight line connecting the middle point P1 between HID1 and HID2 and the gaze point VD updated by the rotational movement.

The scene manager SM continuously executes the above-described processing until the input sensor value of the rotation angle is reflected, and rotates the gaze point VD and the visual axis direction of the virtual camera VCn. The scene manager SM executes the camera direction setting algorithm described with reference to FIG. 6 to reflect a more natural rotation operation of the virtual camera VCn in the stereoscopic video.

As described above, the information processing system 10 can set a virtual camera at an arbitrary position in the 3DCG space where the remote conference proceeds. The information processing system 10 can relay, via a virtual camera, events such as exchanges between conference participants that occur in the process of progressing a remote conference, as stereoscopic images viewed from arbitrary viewpoint positions. The information processing system 10 can convert the relayed stereoscopic video into a two-dimensional video. The information processing system 10 can deliver in real time to a 2D video device, such as a smart phone, owned by a viewer who watches the remote conference.

In the information processing system 10, a remote conference in progress in a 3D CG space can be viewed in real time without burdening the viewer with environment preparation such as a dedicated device for viewing 3D video or expanding the bandwidth of a communication line. .

The information processing system 10 can receive messages (reaction signals such as text, voice, and actions) from viewers who view two-dimensional video. The information processing system 10 can, for example, move the viewpoint position relayed by the virtual camera in the 3DCG space where the remote conference proceeds, based on the received message. The stereoscopic video relayed by the virtual camera whose viewpoint position has been updated is converted into a two-dimensional video and delivered to the viewer. The information processing system 10 can reflect the viewer's intention on the two-dimensional video being distributed. In the information processing system 10, two-dimensional video can be distributed in consideration of viewers' reactions and viewpoints. That is, for example, it is desirable to distribute video in consideration of the reactions and viewpoints of viewers who watch the exchanges between users participating as speakers in a conference after the event or in real time, and this can be realized by the information processing system 10. .

The information processing system 10 can record and save information indicating the state of the object in the 3DCG space where the remote conference is proceeding and the state of the virtual camera in association with the time code. The information processing system 10 can reproduce the stereoscopic video of the remote conference based on the information recorded and saved in association with the time code. The information processing system 10 can deliver the reproduced 2D video of the remote conference to the 2D video devices of the viewers who watch the event after the fact. The information processing system 10 can also receive messages from viewers regarding the two-dimensional video that is distributed after the fact. In the information processing system 10, it is possible to distribute two-dimensional video in consideration of viewers' reactions and viewpoints even after the fact.

According to the information processing system 10 according to the present embodiment, it is possible to provide a technology that enables real-time and asynchronous two-way participation of events in a three-dimensional space rendered by CG using a two-dimensional video device.

[Modification]
In the embodiment, the content provided by the information processing system 10 has been described as content using the 3DCG space. The information processing system 10, for example, photographs an actual conference hall with a camera equipped with an imaging device such as a CCD (Charge-Coupled Device) or a CMOS (Complementary Metal-Oxide Semiconductor), and displays an object such as an avatar in the photographed conference hall. may be arranged. By using the actual conference hall, it is expected to improve the sense of presence and immersion in remote meetings.

FIG. 7 is a configuration diagram showing an example of an information processing system provided with a camera RC that captures an image of a conference hall A2 in real space. The camera RC is connected to an external network and acquires an image of the real space used as the conference hall of the remote conference. There may be a plurality of cameras RC that capture images of the conference hall in the real space.

Assume that a camera RC that captures an image of the conference hall in real space is fixed with respect to the conference hall. The scene manager SM, for example, analyzes the moving image frames of the captured video using camera information such as the position, tilt, and angle of view of the camera RC, and estimates the visual axis of the camera that captures the conference hall. Then, the scene manager SM specifies a world coordinate system in which objects such as avatars rendered in 3DCG are arranged based on the fixed visual axis of the camera. The scene manager SM sets markers and the like that serve as references for arranging objects created by 3DCG in the world coordinate system. Then, the scene manager SM may arrange an object such as an avatar created by 3DCG at the marker position and provide the contents of the remote conference described in the embodiment. If the camera RC that captures the real space is not fixed, a marker (actual marker) that serves as a reference for estimating the capturing position may be placed in the real space.

Participants in a remote conference are provided with a three-dimensional space in which, for example, a conference hall being photographed in a real space and an avatar created by 3DCG are synthesized as a stereoscopic image viewed from the avatar's viewpoint. Even in a conference room photographed from the real space, conference participants can operate their own avatars and take actions such as remarks and dialogues in the conference, reaching out to others, and moving within the conference hall.
The distributor Un, who relays the state of the remote conference, can operate the virtual camera VCn in the same manner as in the embodiment to relay stereoscopic video seen from any viewpoint position in the synthesized three-dimensional space.

Note that the information processing system 10 may photograph a subject such as a conference participant or a distributor, and synthesize an image of the photographed subject in the real space into the 3DCG space. Chroma key synthesis is an example of synthesizing an image of a subject in real space and an image in 3DCG space. 8 and 9 are diagrams for explaining photographing of a subject in real space when using chromakey synthesis.

In FIGS. 8 and 9, the background 20 is a background of a predetermined color such as green background or blue background. In chromakey synthesis, a 3DCG image is synthesized in an area in which the background 20 of a predetermined color has been photographed. The image of the subject 21 is captured by the camera 22 with the real space of the predetermined color specified by the background 20 as the background. The camera 22 captures an image of the subject 21 in real space together with camera information such as the camera's position, tilt, and angle of view, for example. The position of subject 21 is specified by sensor 23 . Note that the camera 22 may be, for example, a depth camera capable of acquiring the depth of the space to be photographed.

As shown in FIG. 9, the subject 21 may be provided with an HMD 25 capable of detecting movement of its own head and a motion sensor 24 detecting movement of its arm. If the subject 21 is a distributor of stereoscopic video provided by the information processing system 10, as described in the embodiment, an operation device for the virtual camera VC that relays the synthesized 3DCG space from an arbitrary viewpoint position is provided. . Movements of the subject 21 detected via the HMD 25 and motion sensor 24 are reflected in motions and actions in the synthesized 3DCG space.

FIG. 10 is a diagram showing an example of a composite video. FIG. 10 illustrates a state in which an image 26 in the 3DCG space is combined with a partial area within the image against the background 20 of a predetermined color shown in FIG. In the information processing system 10, when the subject 21 is a conference participant, a stereoscopic image obtained by synthesizing the subject 21 and the image 26 created in the 3DCG space is provided to the conference participants in the remote conference.

FIG. 11 is a diagram showing an example of a composite image displayed on a 2D image device. In FIG. 11, in the screen displayed on the 2D image device 27, the subject 21 wearing the HMD or the like and the image 26 in the 3DCG space are displayed in a combined state. Note that the 2D video device 27 in FIG. 11 may be a device that allows the naked eye to view a stereoscopic video obtained by synthesizing the subject 21 wearing an HMD or the like with the video 26 in the 3DCG space. In the case where the subject 21 is a distributor of stereoscopic video provided by the information processing system 10, for example, while relaying an event in progress in the synthesized 3DCG space from an arbitrary viewpoint position, explanation of the event and leading to the event The background can be notified to the viewer by voice.

Next, a form in which a subject photographed by a camera participates in a remote conference will be described. FIG. 12 is an explanatory diagram of a form in which a subject wearing an HMD participates in a remote conference. In FIG. 12, "C11" represents a camera that captures the subject Ua described in FIGS. Also, "R11" represents an image displayed in real time on the HMD worn by the subject Ua, and "R12" represents an image distributed to viewers connected to the external network. "R13" represents the real-time video of the remote conference combined with the subject Ua captured by the camera C11. The subject Ua participates remotely while viewing the video displayed on the HMD. The HMD displays, for example, an image of a viewpoint position corresponding to the movement of the subject Ua's head detected via the HMD.

As described with reference to FIGS. 8 and 9, the subject Ua is provided with a background such as a green screen capable of chroma key synthesis, and the camera C11 capable of capturing an RGB image, or the background from the captured image area. The user himself/herself is photographed by a depth camera C11 capable of clipping. The camera C11 records camera information including at least the camera position (Pos (x, y, z)) and rotation (Rot (p, y, r)) in association with the time code TC. The video of Ua is output to the scene manager SM.

The scene manager SM synthesizes the video of the camera C11 and the 3DCG object by matching the time code TC. The video of the camera C11 synthesized by the scene manager SM and the object in the 3DCG space are combined into a real-time video including the subject Ua, which is subjected to chromakey synthesis, as shown in the video R13. In the process of generating the image R13, for example, the color space of the RGB image for each frame shot with a green screen or the like is rotated and converted into an HSB (Hue, Saturation, Brightness) space. In the generation process, the background removal process for the converted image and the synthesis process with the 3DCG image viewed from the visual axis direction of the camera C11 are performed in real time. The generated image R13 is distributed as a two-dimensional image to the viewer's 2D image device connected to the external network via the distribution server CH.

FIG. 13 is an explanatory diagram of a form in which a subject not wearing an HMD participates in a remote conference.
Even when the subject does not wear the HMD, the same processing as in FIG. 12 is performed. However, the subject Ua may participate while viewing the video displayed on a display device capable of viewing the 3D video of the remote conference progressing in real time with the naked eye, or the 2D video converted by the information processing system 10. The scene manager SM may save, for example, remote conference data obtained by synthesizing the video of the camera C11 and the object in the 3DCG space in the auxiliary storage device 103 as an offline moving image file (SV). It should be noted that the scene manager SM can store the data of the remote conference in the auxiliary storage device 103 in the same manner as in the form of FIG.

The form shown in FIG. 13 can also be applied, for example, to video of a remote conference played back after the event. FIG. 14 is an explanatory diagram of a form in which a subject not wearing an HMD participates in a remote conference that is reproduced after the fact. In FIG. 14, an image saved as an offline moving image file (SV) in the auxiliary storage device 103 or the like is reproduced as an image that can be viewed by the subject Ua. However, the post-event video to be reproduced may be a compatible moving image such as a moving image or a sequence in which other material images or scenes are recorded.

In FIG. 14, the scene manager SM of the information processing system 10 reads out and reproduces an offline moving image file (SV) stored in the auxiliary storage device 103 . The reproduced offline moving image file (SV) is displayed as a video R11 on a display device that can be viewed by the subject Ua. Here, the image R11 may be a stereoscopic image or a 2D image as long as the subject Ua can view it.

The subject Ua in FIG. 14 is set with a background such as a green screen capable of chromakey synthesis, and is captured by a camera C11 capable of capturing an RGB image or a depth camera C11 capable of extracting the background from the captured image area. to shoot. The camera C11 records camera information including at least the camera position (Pos (x, y, z)) and rotation (Rot (p, y, r)) in association with the time code TC. The video of Ua is output to the scene manager SM. The scene manager SM synthesizes the image of the camera C11 and the 3DCG object by matching the time code TC to generate an image R13 in which the subject Ua is chromakey-combined. Note that the processing for generating the image R13 has been described with reference to FIG.

The video R13 generated using the offline video file (SV) is distributed as a 2D video to the viewer's 2D video device connected to the external network via the distribution server CH. A two-dimensional image including the subject Ua participating in the remote conference as a participant after the event is delivered to the viewer in the stereoscopic image reproduced using the offline video file (SV). The scene manager SM can store in the auxiliary storage device 103 video data in which the subject Ua is combined with the reproduced offline moving image file (SV).

In the form shown in FIG. 14, for example, performances including actions such as gestures and hand gestures of the subject Ua are reflected in the played offline video file (SV). Depending on the performance of the subject Ua, for example, it is possible to synthesize a video as if the subject Ua were present in a remote conference held in the past, and record it in association with the time code TC.
As a result, in the information processing system 10 having the form shown in FIG. 14, it becomes possible for a person who could not participate in the conference in real time to participate in the conference without wearing an HMD and to record their remarks, thereby enabling sharing of data. .
Since the offline video files (SV) stored in the auxiliary storage device 103 can be used recursively, it is possible to have a "meeting beyond time" in which a plurality of meeting participants participate in sequence on the time code.

FIG. 15 is a flow chart showing event-related event recording processing when a camera for photographing a subject in real space is provided. When the information processing system 10 is provided with a camera for photographing a subject in real space, for example, the scene manager SM executes the processes of S31 to S33 shown in FIG. 15 before executing the process of S1 shown in FIG. do it.

In the processing of S31 in FIG. 15, the scene manager SM determines the presence of a camera (real camera) that captures a subject in real space. If there is a real camera (S31, Yes), the scene manager SM proceeds to the process of S33. On the other hand, if the actual camera does not exist (S31, no), the scene manager SM shifts to the process of S32. In the processing of S32, the scene manager SM generates content video in the 3DCG space described in the embodiment. Further, in the process of S33, the scene manager SM generates a content video (composite video) synthesized by matching the coordinate systems of the video of the subject captured by the real camera described in the modified example and the object in the 3DCG space. do. In the scene manager SM that executes the processing shown in FIG. 15, event recording processing related to the event shown in FIG. 4 is performed for the object of the video content generated in the processing of S32 or S33.

Note that the processing shown in S31 of FIG. 15 may be, for example, an operation input by a content provider or a distributor. For example, when the content generated by the process of S32 and the content generated by the process of S33 can be provided, the scene manager SM can select the content to be generated according to the operation input.

As another modification, the information processing system 10 may collect and analyze messages from viewers and evaluate the distributor's viewpoint. Since distributors with high evaluations tend to have a relatively large number of distribution applicants, for example, advertising effects, such as advertisements included in distributed video, can be expected. Further, in the information processing system 10, by collecting and analyzing messages from viewers, it becomes possible to grasp the tendency of viewers' interests. In addition, it is also possible to employ viewers as distributors as a form of post-video video distribution service provided by the information processing system 10 . It is possible to direct and edit the distribution video from the viewer's point of view.

In addition, the following additional remarks are disclosed regarding the above embodiment.

[Appendix 1]
a moving image data generating means for generating multi-view three-dimensional moving image data obtained by photographing an object to be photographed from multiple viewpoints;
means for receiving viewpoint setting for the multi-viewpoint 3D video data;
means for generating a moving image when the subject is viewed from the accepted viewpoint;
means for providing the generated animation to a viewer device;
means for receiving a reaction to the moving image from the viewer device;
means for recording the set viewpoint, the time in the multi-viewpoint 3D video data, and the reaction;
An information processing system comprising

[Appendix 2]
The means for receiving viewpoint setting for the multi-viewpoint 3D video data includes:
2. The information processing system according to claim 1, wherein a viewpoint for said multi-view three-dimensional video data is set based on a reaction received from said viewer.

[Appendix 3]
further comprising an imaging device,
3. The information processing system according to claim 1, wherein said moving image data generation means generates multi-viewpoint 3D moving image data including an object photographed by said imaging device.

[Appendix 4]
Computers that connect to each other
a step of generating multi-view three-dimensional video data obtained by photographing an object to be photographed from multiple viewpoints;
a step of generating a moving image when the shooting target is viewed from the received viewpoint with respect to the multi-viewpoint 3D moving image data;
providing the generated animation to a viewer device;
receiving a reaction to the moving image from the viewer device;
recording the set viewpoint, the time at which the viewpoint was set, and the reaction;
Information processing method that performs

[Appendix 5]
having the viewer view the generated video at the set viewpoint and the time at which the viewpoint was set;
2. A synthesizing process in which at least a coordinate system, a color tone, and a viewpoint direction are matched between the moving image generated by a photographing means for recording and storing a viewer and his/her reaction in a viewing state and another moving image. 5. The information processing method according to 4.

[Appendix 6]
means for storing the set viewpoint, the time at which the viewpoint was set, and the generated moving image;
At least the coordinate system, color tone, and viewpoint direction are established between the moving image generated by the photographing means for recording and storing the viewer and his/her reaction in the viewing state and the stored moving image or other compatible moving image. 5. The information processing method according to claim 4, wherein a combined synthesis process is performed.

10 information processing system 20 background 21 subject 22 camera 23 sensor 24 motion sensor 25 HMD
26 video 27 2D video device 100 computer 101 CPU
102 Main storage device 103 Auxiliary storage device 104 Input/output IF
105 communication interface
106 connection bus C, C1, C2, C3 camera processor C11 camera VC, VC1, VC2, VC3 virtual camera (virtual camera)
P1, P2 Avatar R, R1, R2, R3 Renderer RC Camera (actual camera)
SM Scene manager SV Offline movie file

Claims (11)

3D video data generating means for generating 3D video data including the first avatar;
2D video data generating means for generating 2D video data viewed from a viewpoint position of a first virtual camera that follows the first avatar, based on the 3D video data;
An information processing system comprising The three-dimensional video data includes a second avatar different from the first avatar,
2. The two-dimensional video data generating means according to claim 1, further generating two-dimensional video data viewed from a viewpoint position of a second virtual camera that follows the second avatar, based on the three-dimensional video data. Information processing system. According to a request from a viewer, the viewer device of the viewer is provided with the two-dimensional video data viewed from the viewpoint position of the first virtual camera, or the two-dimensional video data viewed from the viewpoint position of the second virtual camera is provided. 3. The information processing system according to claim 2, comprising providing means for switching between providing data. 2. The information processing system according to claim 1, further comprising minutes generating means for generating minutes data based on the audio data of said three-dimensional moving image data. 2. The 2D moving image data generating means according to claim 1, wherein said 2D moving image data generating means further generates 2D moving image data viewed from a viewpoint position of a third virtual camera operated by a distributor of the moving image, based on said 3D moving image data. Information processing system. 6. The information processing system according to claim 5, wherein the viewpoint position of said third virtual camera can be operated by a plurality of said distributors. The three-dimensional video data generating means generates three-dimensional video data viewed from the viewpoint position of the first virtual camera and three-dimensional video data viewed from the viewpoint position of a second virtual camera following a second avatar different from the first avatar. 6. The information processing system according to claim 5, wherein the 3D video data and the 3D video data viewed from the viewpoint position of the third virtual camera are transmitted to the stereoscopic video viewing device of the distributor. 6. The information processing system according to claim 5, wherein said three-dimensional moving image data generation means further generates two-dimensional moving image data viewed from a viewpoint position of a virtual camera in which said distributor's operation does not intervene. means for providing the two-dimensional video data to a viewer device;
means for receiving a reaction of a viewer from the viewer device;
further comprising
2. The information processing system according to claim 1, wherein said two-dimensional moving image data generating means moves a viewpoint position of said first virtual camera based on said reaction. 3D video data generating means generating 3D video data including the first avatar;
a step in which a two-dimensional video data generating means generates two-dimensional video data viewed from a viewpoint position of a first virtual camera that follows the first avatar, based on the three-dimensional video data;
Information processing method that performs generating three-dimensional video data including the first avatar;
Generating two-dimensional video data viewed from a viewpoint position of a first virtual camera that follows the first avatar, based on the three-dimensional video data.
An information processing program that causes a computer to execute processing.

Images