JP7065708B2 - Recording / playback device and program - Google Patents

Description

The present invention relates to a recording / playback device and a program for recording and playing back a television broadcast program and a viewing environment.

Conventionally, AR (Augmented Reality) and VR (Virtual Reality) are known as technologies for expanding the real environment perceived by humans by a computer, and the hardware market related to these is in the future. It is expected to expand toward.

For example, a 360-degree camera capable of omnidirectional shooting (see, for example, Patent Documents 1 and 2), a wearable 360-degree camera equipped to a user (see, for example, Non-Patent Document 1) and the like are known. It is possible to record 360-degree video captured by these cameras and provide an AR or VR experience service using all or part of the recorded content.

Japanese Unexamined Patent Publication No. 2011-182003 Japanese Unexamined Patent Publication No. 2011-160442

FITT360-The First 360 ° Neckband Wearable Camera, [online], Kickstarter, [Searched April 28, 2018], Internet <https://www.kickstarter.com/projects/467094941/fitt360-the-first-360 -neckband-wearable-camera?lang=ja ＞

A user who watches a television broadcast program (hereinafter referred to as "program") can easily access a past program at any time by recording the program on a recording / playback device.

Here, the user may want to recall the feelings at that time about the programs that have been watched in the past. The user can recall the emotion at that time by playing back the program that has been watched in the past by using the recording / playback device.

However, when not only the user but also family members or friends other than the user himself / herself are watching the program, it is not always possible to recall the same emotions just by playing the same program.

That is, it is not always possible to obtain the same viewing experience simply by playing back the same program using the recording / playback device. This is because the viewing experience is an important factor in the behavior, reaction, conversation, etc. of the family or friends who were watching together.

Therefore, the present invention has been made to solve the above-mentioned problems, and an object of the present invention is to provide a recording / playback device and a program capable of obtaining the same viewing experience as that of a program viewed in the past. It is in.

In order to solve the above problem, the recording / playback device according to claim 1 records a program video / audio signal including video and audio of a program played on a television in a recording unit, and a user who watches the program on a playback device. A recording / playback device that reads the program video / audio signal from the recording unit and transmits it to the playback device according to the operation, receives the program video / audio signal, and records the program video / audio signal as main content in the recording unit. 1 Receiving unit, an ambient video signal including an image of a person existing around the user and an image of the television on which the program is being played are received, and an ambient audio signal collected from the surroundings of the user. The person is extracted from the second receiving unit that receives the image and the surrounding image signal received by the second receiving unit, a person image signal for each person is generated, and the person image signal for each person is substituting. The position of the television is detected from the person extraction unit recorded in the recording unit as content and the ambient video signal received by the second receiving unit, and the program video signal in the program video / audio signal is generated by the playback device. A television position detection unit that generates television position data indicating a position to be reproduced and records the television position data as the sub-content in the recording unit, the program video / audio signal recorded in the recording unit, and the first. Based on the ambient audio signal received by the receiving unit of 2, the program corresponding to the program audio signal included in the ambient audio signal is specified, and the ambient audio signal and the program video audio signal of the program are used. A person voice extraction unit that obtains a difference between the program voice signal, generates a person voice signal of the person existing around the user, and records the person voice signal as the sub-content in the recording unit, and the person voice extraction unit. The main content and the sub-content are read from the recording unit, and the main content and the sub-content are transmitted to the playback device.

Further, the recording / playback device according to claim 2 is the recording / playback device according to claim 1, which includes a person 3D (three-dimensional) generation unit in place of the person extraction unit, and further includes a peripheral 3D generation unit. The person 3D generation unit extracts the person from the surrounding image signal received by the second receiving unit, generates a person image for each person, and estimates the posture of the person included in the surrounding image signal. , Bone data for each person showing the posture of the person is generated, and based on the person video signal and the bone data, person 3D data is generated as data of 3D CG animation of the person, and the person for each person. The 3D data is recorded in the recording unit as the sub-content, and the peripheral 3D generation unit extracts an object from the ambient video signal received by the second receiving unit, and obtains the 3D CG data of the object. The surrounding 3D data including the surrounding 3D data is generated, the surrounding 3D data is recorded as the sub-content in the recording unit, and the program video / audio signal is recorded in the recording unit as the main content, and the person 3D data for each person. The surrounding 3D data, the television position data, and the person voice signal are recorded as the sub-contents.

Further, the recording / reproducing device according to claim 3 is the recording / reproducing device according to claim 2, further comprising a third receiving unit, a line-of-sight detection unit, and a correction unit, and the third receiving unit is the second receiving unit. A video signal different from the ambient video signal received by the reception unit of the above, and a bird's-eye view video signal including the person existing around the user is received, and the person 3D generation unit receives the bird's-eye view video signal from the ambient video signal. The person is extracted, the person image is generated for each person, the posture of the person included in the surrounding image signal is estimated, and the bird's-eye view image signal included in the bird's-eye view image signal received by the third receiving unit is included. The posture of the person is estimated, and the bone data for each person indicating the posture of the person is generated based on the posture of the person estimated from the surrounding video signal and the bird's-eye view video signal, respectively, and the person video signal and the person are described. The person 3D data is generated based on the bone data, the line-of-sight detection unit detects the person's line of sight, the line-of-sight direction data for each person is generated, and the correction unit is generated by the person 3D generation unit. The person 3D data is corrected so that the line-of-sight direction of the person 3D data coincides with the line-of-sight direction indicated by the line-of-sight direction data generated by the line-of-sight detection unit, and the corrected person 3D for each person. The data is recorded in the recording unit as the sub-contents.

Further, in the recording / reproducing device according to the fourth aspect, in the recording / reproducing device according to the second aspect, the person 3D generation unit inputs 3D data of the person for each preset person and inputs the 3D data of the person to the ambient video signal. The posture of the person included is estimated, the bone data for each person showing the posture of the person is generated, and the prior person 3D is used as the data of the 3D CG animation of the person based on the 3D data and the bone data. Data is generated, the image of the person included in the surrounding image signal is projected and overwritten on the prior person 3D data, the person 3D data is generated, and the person 3D data for each person is used as the sub-content. It is characterized in that it records in the recording unit.

Further, in the recording / reproducing device according to claim 5, in the recording / reproducing device according to claim 3, the person 3D generation unit inputs 3D data of the person for each person preset and is used as the ambient video signal. The posture of the person included is estimated, the posture of the person included in the bird's-eye view video signal is estimated, and the posture of the person is estimated from the surrounding video signal and the bird's-eye view video signal, respectively. The bone data for each person showing the posture is generated, and the pre-person 3D data is generated as the data of the 3D CG animation of the person based on the 3D data and the bone data. It is characterized in that the image of the person included in the ambient video signal is projected and overwritten, the person 3D data is generated, and the person 3D data for each person is recorded in the recording unit as the sub-content.

Further, in the recording / playback device according to claim 6, in the recording / playback device according to any one of claims 2 to 5, the peripheral 3D generation unit obtains preset 3D data around the user. It is input as pre-peripheral 3D data, the object is extracted from the ambient video signal to generate an object image, and the corresponding object image is projected onto the object included in the pre-peripheral 3D data to overwrite the object. It is characterized in that the surrounding 3D data is generated and the surrounding 3D data is recorded in the recording unit as the sub-content.

The program according to claim 7 is characterized in that the computer functions as the recording / playback device according to any one of claims 1 to 6.

As described above, according to the present invention, it is possible to obtain the same viewing experience as that of a program that has been viewed in the past.

It is a block diagram which shows the structural example of the recording / reproduction apparatus of Example 1. FIG. It is a flowchart which shows the processing example of the recording-playback apparatus of Example 1. FIG. It is a block diagram which shows the structural example of the person voice extraction part. It is a block diagram which shows the structural example of the recording / reproduction apparatus of Example 2. FIG. It is a flowchart which shows the processing example of the recording / reproduction apparatus of Example 2. It is a block diagram which shows the structural example of the person 3D generation part in the 1st modification of Example 2. FIG. It is a block diagram which shows the structural example of the surrounding 3D generation part in the 2nd modification of Example 2. FIG. It is a block diagram which shows the structural example of the recording / reproduction apparatus of Example 3. FIG.

Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings. The present invention relates to an environment (viewing environment) in a room that includes the video and audio of a program viewed by the viewer himself (first party) or the like as a user, as well as the reaction of surrounding persons who view the program together. It is characterized by recording video and audio. Here, the first person is a viewer who wants to reproduce the viewing experience.

As a result, by reproducing the video / audio of the program and the video / audio of the viewing environment, the first party can obtain the same viewing experience as that of the program that has been viewed in the past. For example, it is possible to realize a service by AR or VR technology that can record and reproduce the viewing environment itself.

The first embodiment described below is an example of recording a 2D (Dimension) video and audio of a surrounding person as seen by a first party, in addition to the program video and audio. In the first embodiment, for example, by using AR glass, it is possible to superimpose and reproduce the program video sound and the video sound of a surrounding person by two-dimensional CG.

Further, the second embodiment is an example of recording a 3D (three-dimensional) video and audio of a surrounding person as seen by a first party, in addition to the program video and audio. In the second embodiment, for example, by using AR glass, the video / audio of the program and the video / audio of a surrounding person can be superposed and reproduced. Further, for example, by using a VR HMD (Head Mounted Display), it is possible to superimpose and reproduce surrounding people and the like on a three-dimensional CG.

Further, the third embodiment is an example of recording a highly accurate 3D image of a surrounding person in the second embodiment. In the third embodiment, it is possible to reproduce the surrounding people with a highly realistic three-dimensional CG.

[Example 1]
First, Example 1 will be described. As described above, the first embodiment is an example of recording 2D video and audio of a surrounding person as seen by a first party, in addition to the program video and audio. In the first embodiment, for example, by using AR glass, it is possible to superimpose and reproduce the program video sound and the video sound of a surrounding person by two-dimensional CG.

FIG. 1 is a block diagram showing a configuration example of the recording / playback device of the first embodiment, and FIG. 2 is a flowchart showing a processing example of the recording / playback device of the first embodiment. The recording / playback device 1-1 includes reception units 10 and 11, a person extraction unit 12, a television position detection unit 13, a person voice extraction unit 14, a recording unit 15, and a transmission unit 16.

It is assumed that the first person and the surrounding persons are watching the program played on the television. The receiving unit 10 receives the program video / audio signal for the program being viewed by the first party and surrounding persons (step S201). Then, the receiving unit 10 records the program video / audio signal as the main content in the recording unit 15 (step S202). The program video / audio signal is configured to include a program video signal and a program audio signal.

The receiving unit 11 receives a 360-degree video / audio signal from the wearable 360-degree camera 2 (step S203). The 360-degree video-audio signal is composed of a surrounding video signal centered on the first party and an audio signal picked up at the position of the first party. Hereinafter, for convenience of explanation, the former video signal is referred to as a 360-degree video signal, and the latter audio signal is referred to as a 360-degree audio signal.

The 360-degree video signal is a peripheral video signal of a lateral video including a video of a person around the first party, a video of a television on which a program is played, and the like, and a 360-degree audio signal is a video signal of the first party. It is an ambient audio signal picked up from the surroundings, including the audio of surrounding people.

Here, the wearable 360-degree camera 2 is worn by the first party, captures a 360-degree image around the first party, collects sound, and has a 360-degree video and a synchronized 360-degree video-audio signal. Is generated, and a 360-degree video / audio signal is transmitted to the recording / playback device 1-1.

The receiving unit 11 separates the 360-degree video-audio signal into a 360-degree video signal and a 360-degree audio signal (step S204). Then, the receiving unit 11 outputs the 360-degree video signal to the person extraction unit 12 and the television position detection unit 13, and outputs the 360-degree audio signal to the person voice extraction unit 14.

The receiving unit 11 may receive the synchronized 360-degree video signal and the 360-degree audio signal from the wearable 360-degree camera 2, respectively.

The person extraction unit 12 inputs a 360-degree video signal from the reception unit 11, analyzes the 360-degree video signal by a predetermined person extraction process, and extracts a person around the first person from the video of the 360-degree video signal. Extract the video. Then, the person extraction unit 12 records the video signal (personal video signal) for each person as sub-content in the recording unit 15 (step S205).

The person extraction process by the person extraction unit 12 is known, and for example, the following technique is used. This person extraction process is based on a segmentation technique using a CNN (Convolutional Neural Network), and is a process of separating a 360-degree video signal into a foreground with a person and a background without a person.
"Google (registered trademark) announces mobile-based real-time video segment technology that separates the foreground and background of people using CNN. Integrated into YouTube (registered trademark)", [online], [May 11, 2018 Day search], Internet <http://shiropen.com/2018/03/02/32805>

The television position detection unit 13 inputs a 360-degree video signal from the reception unit 11 and analyzes the 360-degree video signal by a predetermined object extraction process and a television position (including posture) detection process. Then, the television position detection unit 13 extracts the television image from the video of the 360-degree video signal, identifies the position (including the posture) of the television in the entire image, and generates the television position data indicating the position of the television. do. The television position detection unit 13 records the television position data as sub-content in the recording unit 15 (step S206).

The object extraction process and the television position (including posture) detection process by the television position detection unit 13 are known, and for example, the following techniques are used.
"Method of estimating the position and orientation of the TV screen by the camera of the mobile terminal", IPSJ Journal (CDS: Consumer Devices & Systems), vol.5, no.4, 2015, p.61-69
"A Study on Television Position Detection Method in Living Room", 2016 Proceedings of the Winter Games of the Institute of Image Information and Television Engineers, 22B-6, 2016

The television position data is data indicating a position where an AR glass or a VR HMD (Head Mounted Display), which is a reproduction device 3 described later, reproduces a program video signal. Further, the television position data can be used as reference data for specifying the reproduction position when the reproduction device 3 reproduces the person video signal for each person.

The person voice extraction unit 14 inputs a 360-degree audio signal from the reception unit 11, and reads a program video audio signal, which is the main content, from the recording unit 15. Then, the person audio extraction unit 14 separates the program audio signal from the program video audio signal, analyzes the 360-degree audio signal by a predetermined program audio specifying process, and determines whether or not the 360-degree audio signal includes the program audio signal. Is determined. The person voice extraction unit 14 identifies a program corresponding to the program voice signal included in the 360-degree voice signal.

The person voice extraction unit 14 obtains the difference between the 360-degree voice signal and the program voice signal of the specified program, and generates the voice signal (person voice signal) of the person around the first person, thereby 360. The person voice signal is extracted from the voice signal. Then, the person voice extraction unit 14 records the person voice signal as sub-content in the recording unit 15 (step S207). This person voice signal corresponds to the environmental sound of the viewing environment.

The program voice identification process by the person voice extraction unit 14 is known, and for example, the following technique is used. This program audio identification processing is based on ACR (Automatic Contents Recognition) technology, and matching that extracts feature points of a 360-degree audio signal and a program audio signal and compares the pattern of the feature points between the two signals. Is a process of specifying a program corresponding to the program audio signal from the 360-degree audio signal.
"Evixar ACR (Automatic Content Recognition), Sound Sensing", [online], [Search on May 11, 2018], Internet <https://www.evixar.com/evixaracr>

FIG. 3 is a block diagram showing a configuration example of the person voice extraction unit 14. The person voice extraction unit 14 includes a program identification unit 40 and a subtraction unit 41.

The program specifying unit 40 inputs a 360-degree audio signal from the receiving unit 11, reads out the program video audio signal which is the main content from the recording unit 15, and separates the program audio signal from the program video audio signal. Then, the program specifying unit 40 performs matching by comparing the pattern of the feature points between the 360-degree audio signal and the program audio signal by the above-mentioned ACR technique, and whether or not the 360-degree audio signal includes the program audio signal. Is determined.

When the program specifying unit 40 determines that the 360-degree audio signal includes the program audio signal, the program specifying unit 40 identifies the program of the program audio signal as a program of the program audio signal included in the 360-degree audio signal, and the 360-degree audio signal. Identify the location within. Then, the program specifying unit 40 outputs the program audio signal of the specified program to the subtracting unit 41.

On the other hand, when the program specifying unit 40 determines that the 360-degree audio signal does not include the program audio signal, that is, when the program corresponding to the program audio signal included in the 360-degree audio signal cannot be specified, the recording unit 15 starts from the recording unit 15. Read other program video and audio signals. Then, the program specifying unit 40 determines whether or not the 360-degree audio signal includes the program audio signal.

In this way, the program specifying unit 40 specifies the program included in the 360-degree audio signal and its position, and the program audio signal of the program is output to the subtracting unit 41.

The subtraction unit 41 inputs a 360-degree audio signal from the reception unit 11, and also inputs a program audio signal from the program identification unit 40. Then, the subtraction unit 41 obtains a difference signal by subtracting the program audio signal from the 360-degree audio signal with reference to the specified position in the 360-degree audio signal, and uses this as the person audio signal. The subtraction unit 41 records the person voice signal as sub-content in the recording unit 15.

Returning to FIGS. 1 and 2, a plurality of program video / audio signals are recorded in the recording unit 15 as the main content by steps S201 to S207. Further, as sub-contents, a person video signal, a person voice signal, and television position data relating to a person around the first person, corresponding to each of the plurality of program video / audio, are recorded.

Here, the first user, who is a user, wears AR glasses (reproduction device 3) and views the main content or the like recorded in the recording unit 15. In this case, the first party needs to perform viewing at the same position in the same room as when the main content or the like was recorded in the recording unit 15. The playback device 3 activates a predetermined application according to the operation of the first party, and selects the main content to be reproduced from a plurality of main contents according to the selection operation of the first party.

Then, the transmission unit 16 reads the program video / audio signal, which is the selected main content, from the recording unit 15, and the person video signal, the person audio signal, and the television, which are the sub-contents corresponding to the program video / audio signal. Read the position data.

The transmission unit 16 transmits the program video / audio signal, which is the main content, and the person video signal, the person voice signal, and the television position data for each person, which is the sub content, to the playback device 3 as data for reproducing the main content and the sub content. (Step S208).

The reproduction device 3 is, for example, a reproduction device for AR. The playback device 3 receives the program video / audio signal, which is the main content, and the person video signal, the person voice signal, and the television position data for each person, which is the sub-content, from the recording / playback device 1-1, and programs the program video / audio signal. Separate into video signals and program audio signals.

The playback device 3 reproduces the program video signal of the main content at the position indicated by the television position data. Further, the playback device 3 reproduces a person video signal for each person in the sub-content at a predetermined position around the first person (a predetermined position for each person based on the position indicated by the television position data), and further Plays program audio signals and person audio signals.

In this case, the predetermined position where the person video signal is reproduced may be preset by the first party as the position data for each person. The playback device 3 reproduces a person video signal for each person at each position according to preset position data for each person.

Further, the reproduction device 3 may receive from the recording / reproduction device 1-1 at a predetermined position where the person video signal is reproduced. Specifically, the person extraction unit 12 of the recording / playback device 1-1 uses the television position data detected by the television position detection unit 13 as a reference when extracting the person image from the video of the 360-degree video signal. Identify the position of the portrait image. Then, the person extraction unit 12 adds position data to the person video signal and records it in the recording unit 15. The transmission unit 16 transmits the person video signal and the position data to the reproduction device 3.

The playback device 3 receives position data as well as a person image signal for each person from the recording / playback device 1-1, and according to the position data for each person, the person image for each person is set at each position based on the position of the television. Play each signal.

As described above, according to the recording / playback device 1-1 of the first embodiment, the receiving unit 10 records the program video signal as the main content in the recording unit 15, and the receiving unit 11 records the 360-degree video-audio signal 360-degree. It is separated into a video signal and a 360-degree audio signal.

The person extraction unit 12 extracts a person image from the image of the 360-degree image signal, and records the person image signal for each person as sub-content in the recording unit 15. Further, the television position detection unit 13 detects the television position from the 360-degree video signal and records the television position data as sub-content in the recording unit 15.

The person voice extraction unit 14 separates the program audio signal from the program video audio signal read from the recording unit 15, determines whether or not the 360-degree audio signal includes the program audio signal, and identifies the program. Then, the person voice extraction unit 14 extracts the person voice signal from the 360 degree voice signal by obtaining the difference between the 360 degree voice signal and the program voice signal of the specified program. The person voice extraction unit 14 records the person voice signal as sub-content in the recording unit 15.

The transmission unit 16 follows the operation of the user who is the first person, and from the recording unit 15, the program video / audio signal which is the main content, the person video signal for each person which is the sub-content corresponding to the program video / audio signal, the person audio signal, and the person audio signal. Read the TV position data. Then, the transmission unit 16 transmits these data to the reproduction device 3.

As a result, for example, when the first party wears the AR glass which is the playback device 3, the main content is overlaid on the TV screen and the sound is reproduced on the AR glass, and at the same time, the sub content. The video and audio of surrounding people are played back. The first person will see the surrounding environment at the time of reproduction with AR glass for the surroundings other than the person.

In this way, by using the AR glass, the main content is reproduced and the reaction of the person around the first person is reproduced, so that the same viewing experience as at that time is obtained for the programs viewed in the past. be able to. In addition, a more reproducible viewing experience can be obtained as compared with the conventional reproduction of only the main content.

[Example 2]
Next, Example 2 will be described. As described above, the second embodiment is an example of recording a 3D image, an audio, and the like of a surrounding person as seen by a first party, in addition to the program video and audio. In the second embodiment, for example, by using AR glass, the video / audio of the program and the video / audio of a surrounding person can be superposed and reproduced. Further, for example, by using a VR HMD, it is possible to superimpose and reproduce surrounding people and the like by three-dimensional CG.

FIG. 4 is a block diagram showing a configuration example of the recording / playback device of the second embodiment, and FIG. 5 is a flowchart showing a processing example of the recording / playback device of the second embodiment. The recording / playback device 1-2 includes reception units 10 and 11, person 3D generation unit 20, surrounding 3D generation unit 21, television position detection unit 13, person voice extraction unit 14, recording unit 15, and transmission unit 16. ..

Similar to the first embodiment shown in FIG. 1, the receiving unit 10 receives the program video / audio signal (step S501) and records the program video / audio signal as the main content in the recording unit 15 (step S502).

Similar to the first embodiment shown in FIG. 1, the receiving unit 11 receives the 360-degree video / audio signal from the wearable 360-degree camera 2 (step S503), and the 360-degree video / audio signal is the 360-degree video signal and the 360-degree audio. Separate into signals (step S504). Then, the receiving unit 11 outputs the 360-degree video signal to the person 3D generation unit 20, the surrounding 3D generation unit 21, and the television position detection unit 13, and outputs the 360-degree audio signal to the person voice extraction unit 14.

The person 3D generation unit 20 inputs a 360-degree video signal from the reception unit 11, extracts a person image from the image of the 360-degree video signal, and estimates bone data indicating the posture of the person. Then, the person 3D generation unit 20 generates the person 3D data based on the person video signal and the bone data, and records the person 3D data for each person as sub-contents in the recording unit 15 (step S505). The person 3D data is the data of the person's 3D CG animation.

The person 3D generation unit 20 includes a person extraction unit 12, a posture estimation unit 22, and a person generation unit 23. Similar to the first embodiment shown in FIG. 1, the person extraction unit 12 inputs a 360-degree video signal from the receiving unit 11, analyzes the 360-degree video signal by a predetermined person extraction process, and uses the 360-degree video signal. Extract the texture information of the person video signal. Then, the person extraction unit 12 outputs the texture information of the person video signal for each person to the generation unit 23.

The posture estimation unit 22 inputs a 360-degree video signal from the reception unit 11, analyzes the 360-degree video signal by a predetermined posture estimation process, and analyzes the body, hand, face, etc. of the person image included in the 360-degree video signal. The part is detected and the posture of each person is estimated. The posture estimation unit 22 generates bone data indicating the posture of each person, and outputs the bone data for each person to the generation unit 23.

The posture estimation process by the posture estimation unit 22 is known, and for example, the following techniques are used. This posture estimation process is based on a skeleton detection algorithm, detects parts such as the body, hands, and face of a person image included in a 360-degree video signal, and determines a predetermined key point based on a key point such as a joint. It is a process of estimating the posture of each person by expressing it with a connected straight line.
"SMPLify: 3D Human Pose and Shape from a Single Image (ECCV 2016)", [online], [Search May 11, 2018], Internet <https://www.youtube.com/watch?v=eUnZ2rjxGaE ＞

The generation unit 23 inputs a person video signal for each person from the person extraction unit 12, and also inputs bone data for each person from the posture estimation unit 22. Then, the generation unit 23 analyzes the corresponding person video signal and bone data by a predetermined person 3D data generation process. The generation unit 23 projects and overwrites the bone data (three-dimensional model of the person obtained from the bone data) with the person video signal, and generates the person 3D data (three-dimensional CG animation of the person). The generation unit 23 records the person 3D data for each person as sub-contents in the recording unit 15.

The person 3D data generation process by the generation unit 23 is known, and for example, the following technique is used. This person 3D data generation process is based on a technique for synthesizing a person video signal and bone data, and is a process of pasting a person video signal to the bone data.
"SMPLify: 3D Human Pose and Shape from a Single Image (ECCV 2016)", [online], [Search May 11, 2018], Internet <https://www.youtube.com/watch?v=eUnZ2rjxGaE ＞
"Announcement of markerless performance capture technology that reconstructs from posture estimation of a person to clothes to be worn by inputting images taken with a monocular camera", [online], [Search on June 15, 2018], Internet <https //shiropen.com/seamless/vr-monoperfcap ＞

The surrounding 3D generation unit 21 inputs a 360-degree video signal from the receiving unit 11, analyzes the 360-degree video signal by a predetermined peripheral 3D data generation process, and based on the 360-degree video signal, surrounds the first person. Generates surrounding 3D data including 3D data of an object (people around the first person and objects around). The peripheral 3D generation unit 21 records the peripheral 3D data as sub-content in the recording unit 15 (step S506). The surrounding 3D data is data related to the surrounding image including the 3D CG of the object.

The peripheral 3D data generation process by the peripheral 3D generation unit 21 is known, and for example, the following techniques are used. This surrounding 3D data generation process is based on photogrammetry (photogrammetry), detects an object included in a 360-degree video signal, and generates 3D data of the object based on a plurality of video signals of the object. It is a process. Photogrammetry is a photogrammetry method for obtaining dimensions and shapes by analyzing parallax information from two-dimensional images obtained by photographing a 3D object from a plurality of observation points.
"Recreating reality in 3D by making full use of" photogrammetry "" Nagasaki Churches "VR content released", [online], [Search on May 11, 2018], Internet <http://www.moguravr .com / hacosco-nagasaki-vr />
"3DF Zephyr", [online], [Search on May 11, 2018], Internet <https://www.3dflow.net/3df-zephyr-pro-3d-models-from-photos/>

Similar to the first embodiment shown in FIG. 1, the television position detecting unit 13 inputs a 360-degree video signal from the receiving unit 11 and extracts a television image from the video of the 360-degree video signal. Then, the television position detection unit 13 identifies the position of the television in the entire video, generates television position data indicating the position of the television, and records the television position data as sub-content in the recording unit 15 (step S507). ..

Similar to the first embodiment shown in FIG. 1, the person voice extraction unit 14 inputs a 360-degree audio signal from the reception unit 11 and reads out a program video audio signal which is the main content from the recording unit 15. Then, the person audio extraction unit 14 separates the program audio signal from the program video audio signal and identifies the program. The person voice extraction unit 14 obtains the difference between the 360-degree voice signal and the program voice signal of the specified program, and generates the person voice signal to extract the person voice signal from the 360-degree voice signal. The person voice extraction unit 14 records the person voice signal as sub-content in the recording unit 15 (step S508).

As a result, a plurality of program video / audio are recorded in the recording unit 15 as the main content. Further, in the recording unit 15, as sub-contents, person 3D data for each person related to a person around the first person, 3D data around the first person, and a person corresponding to each of a plurality of program video / audio. Audio signals and television position data are recorded.

Here, it is assumed that the first user, who is the user, wears the AR glass or the HMD of the VR, which is the playback device 3, and watches the main content or the like recorded in the recording unit 15. The playback device 3 activates a predetermined application according to the operation of the first party, and selects the main content to be reproduced from a plurality of main contents according to the selection operation of the first party.

Then, the transmission unit 16 reads the program video / audio signal, which is the selected main content, from the recording unit 15, and also has the person 3D data, the surrounding 3D data, and the person, which are the sub-contents corresponding to the program video / audio signal. Read the audio signal and TV position data.

The transmission unit 16 reproduces the program video / audio signal as the main content, and the person 3D data, surrounding 3D data, person voice signal, and television position data for each person as the sub-content as data for reproducing the main content and the sub-content. It is transmitted to the device 3 (step S509).

The reproduction device 3 is, for example, a reproduction device for AR or a reproduction device for VR. The playback device 3 receives the program video / audio signal, which is the main content, and the person 3D data, surrounding 3D data, person voice signal, and television position data for each person, which is the sub-content, from the recording / playback device 1-2, and the program video. The audio signal is separated into a program video signal and a program audio signal.

When the playback device 3 is a playback device for AR, the playback device 3 reproduces the program video signal of the main content at the position indicated by the television position data, as in the first embodiment. Further, the reproduction device 3 reproduces the person 3D data of the sub-content at a predetermined position around the first person, and also reproduces the program audio signal and the person audio signal. The position data at the predetermined position where the person 3D data is reproduced may be preset by a first party as in the first embodiment, or may be received by the reproduction device 3 from the recording / reproduction device 1-2. You may do it.

When the playback device 3 is a playback device for VR, the playback device 3 reproduces the surrounding 3D data of the sub-content, detects the television from the surrounding 3D data, and reproduces the program video signal of the main content at that position. Further, the reproduction device 3 detects a person from the surrounding 3D data, reproduces the person 3D data of the sub-content at the position, and reproduces the program audio signal and the person audio signal.

As described above, according to the recording / playback device 1-2 of the second embodiment, the receiving unit 10 records the program video signal as the main content in the recording unit 15, and the receiving unit 11 records the 360-degree video-audio signal 360-degree. It is separated into a video signal and a 360-degree audio signal.

The person 3D generation unit 20 extracts a person image from the video of the 360-degree video signal by the person extraction unit 12, estimates the bone data of the person from the 360-degree video signal by the posture estimation unit 22, and the generation unit 23. Person 3D data is generated based on the person video signal and bone data. Then, the person 3D generation unit 20 records the person 3D data for each person as sub-contents in the recording unit 15.

The surrounding 3D generation unit 21 generates surrounding 3D data including 3D data of surrounding objects from a 360-degree video signal, and records the surrounding 3D data as sub-content in the recording unit 15. The television position detection unit 13 detects the television position from the 360-degree video signal and records the television position data as sub-content in the recording unit 15.

The person voice extraction unit 14 separates the program audio signal from the program video audio signal read from the recording unit 15, determines whether or not the 360-degree audio signal includes the program audio signal, and identifies the program. Then, the person voice extraction unit 14 extracts the person voice signal from the 360 degree voice signal by obtaining the difference between the 360 degree voice signal and the program voice signal of the specified program. The person voice extraction unit 14 records the person voice signal as sub-content in the recording unit 15.

According to the operation of the user who is the first person, the transmission unit 16 has the program video / audio signal which is the main content from the recording unit 15, the person 3D data for each person which is the sub-content corresponding to the program video / audio signal, and the surrounding 3D data. Read the person voice signal and the TV position data. Then, the transmission unit 16 transmits these data to the reproduction device 3.

As a result, for example, when the first party wears the AR glass which is the reproduction device 3, the main content is overlaid on the TV screen and the sound is reproduced on the AR glass. At the same time, the AR glass reproduces the image of the person 3D data, which is a sub-content, as a three-dimensional animation of the surrounding people, and reproduces the voice of the person. The first person will see the surrounding environment at the time of reproduction with AR glass for the surroundings other than the person.

Further, for example, when the first person wears the reproduction device 3 which is the HMD of VR, the object other than the surrounding person by the surrounding 3D data which is the sub-content is reproduced as the 3D CG in the HMD of VR. .. Further, on the VR HMD, the main content is overlaid on the television screen and the sound is reproduced. Further, in the VR HMD, along with the 3D CG other than the person, the image by the person 3D data which is the sub-content is reproduced as the 3D animation of the surrounding person, and the sound of the person is reproduced.

That is, on the VR HMD, the 3D CG animation of the person by the person 3D data which is the sub-content and the main content are superimposed and displayed against the background of the 3D CG other than the person by the surrounding 3D data which is the sub-content.

In this way, by using the AR glass or VR HMD, the main content is reproduced and the reaction of the person around the first person is reproduced, so the program that was watched in the past is the same as that time. A viewing experience and a new viewing experience can be obtained.

When the AR glass is used, it is possible to move to an arbitrary viewpoint and obtain a new viewing experience. Furthermore, during playback, the images around the first person other than the person are the surrounding conditions at the time of playback (current), so the difference between the past situation at the time of recording and the current situation at the time of playback was emphasized. You can get a viewing experience.

Further, even when the VR HMD is used, it is possible to move to an arbitrary viewpoint, and a new viewing experience can be obtained. Moreover, since the entire viewing environment can be reproduced, it can be reproduced at any place.

[First modification of Example 2]
Next, the first modification of Example 2 will be described. The first modification of the second embodiment is an example in which the 3D data of the person (preliminary person 3D data) obtained by the prior shooting is used when the person 3D data is generated in the second embodiment.

Comparing the recording / playback device 1-2 of the second embodiment with the recording / playback device 1-2 of the first modification of the second embodiment, the recording / playback device 1-2 of the first modification of the second embodiment is an embodiment. It differs from the person 3D generation unit 20 of 2 in that it has a person 3D generation unit 20'.

FIG. 6 is a block diagram showing a configuration example of the person 3D generation unit in the first modification of the second embodiment. The person 3D generation unit 20'includes a posture estimation unit 22, a pre-person 3D generation unit 24, and a generation unit 25.

The person 3D generation unit 20'inputs a 360-degree video signal from the reception unit 11, estimates bone data indicating the posture of the person from the 360-degree video signal, and presets preset human 3D data and estimated bone data. Pre-person 3D data for each person is generated based on. Then, the person 3D generation unit 20'generates the person 3D data by projecting and overwriting the signal of the person image included in the 360-degree video signal on the prior person 3D data, and the person 3D data for each person is generated. It is recorded in the recording unit 15 as sub-content.

The posture estimation unit 22 of the person 3D generation unit 20'inputs a 360-degree video signal from the reception unit 11 and estimates the posture of each person, as in the second embodiment shown in FIG. Then, the posture estimation unit 22 outputs bone data indicating the posture of each person to the prior person 3D generation unit 24.

The pre-person 3D generation unit 24 inputs bone data for each person from the posture estimation unit 22, and also inputs preset pre-person 3D data for each person. Then, the pre-person 3D generation unit 24 analyzes the pre-person 3D data and the bone data by a predetermined person 3D data generation process, and generates pre-person 3D data for each person showing a three-dimensional model of the basic person. ..

The person 3D data generation process by the prior person 3D generation unit 24 is known, and for example, the following technique is used. This person 3D data generation process is a rigging process that generates a mechanism (rig) for adding a person's movement after setting up bone data in advance person 3D data.
"Quick Rig Tool", [online], [Search June 15, 2018], Internet <https://knowledge.autodesk.com/ja/support/maya/learn-explore/caas/CloudHelp /cloudhelp/2018/JPN/Maya-CharacterAnimation/files/GUID-DC29C982-D04F-4C20-9DBA-4BBB33E027EF-htm.html>

The preset pre-person 3D data is 3D data constructed by 3D whole body scan processing or 3D modeling processing by a monocular camera for a person around the first person, and is data of 3D CG animation of the person. ..

3D whole body scan processing is known and, for example, the following techniques are used.
"SHUN'X Whole Body High Speed 3D Scanner System", [online], [Search on May 11, 2018], Internet <https: // http://www.vrcjp.com/home.jsf>

Further, 3D modeling processing by a monocular camera is known, and for example, the following techniques are used.
"Brownschweig Institute of Technology and others announced a method using machine learning to generate a human body 3D avatar with 4.5 mm accuracy from a single monocular camera", [online], [Search on May 11, 2018], Internet <https://shiropen.com/seamless/ai-3d-people-models>

The generation unit 25 inputs a 360-degree video signal from the reception unit 11 and inputs pre-person 3D data for each person from the pre-person 3D generation unit 24. Similar to the person extraction unit 12 shown in FIG. 4, 360 A person image is extracted from the image of the video signal. Then, the generation unit 25 associates the person video signal with the prior person 3D data by the matching process.

The generation unit 25 generates the person 3D data by projecting and overwriting the corresponding person video signal on the prior person 3D data by a predetermined projection process, and the person 3D data for each person is recorded as sub-contents. Record at 15.

As a result, it is possible to obtain person 3D data that reflects the current person image when the first person is watching the program, instead of the past person image when the person 3D data was obtained in advance by shooting in advance. ..

The projection process by the generation unit 25 is known, and for example, the following technique is used. This projection process is based on general projection mapping, and is a process of projecting a person image signal as a material onto the prior person 3D data which is an object.
"Projector", [online], [Search on May 11, 2018], Internet <https://docs.unity3d.com/ja/current/Manual/class-Projector.html>
"Easy Decal Decal can be applied even on large uneven surfaces! Terrain is okay! Dynamically changeable decal system", [online], [Search on May 11, 2018], Internet <http: // www. asset-sale.net/entry/Easy_Decal ＞

As described above, according to the recording / playback device 1-2 of the first modification of the second embodiment, the person 3D generation unit 20'estimates bone data indicating the posture of the person from the 360-degree image and sets it in advance. Pre-person 3D data is generated based on the pre-person 3D data and the estimated bone data. Then, the person 3D generation unit 20'generates the person 3D data by projecting and overwriting the person video signal included in the 360-degree video signal on the prior person 3D data, and sub-contents the person 3D data for each person. Is recorded in the recording unit 15.

As a result, since the person 3D data having higher accuracy than that of the second embodiment is recorded in the recording unit 15, the surrounding people based on the person 3D data can be reproduced as a three-dimensional animation in a state closer to reality at the time of reproduction. can.

[Second modification of Example 2]
Next, a second modification of the second embodiment will be described. The second modification of Example 2 is an example of using the 3D data around the first person (pre-surrounding 3D data) obtained by prior imaging when generating the surrounding 3D data in Example 2.

Comparing the recording / playback device 1-2 of the second embodiment with the recording / playback device 1-2 of the second modification of the second embodiment, the recording / playback device 1-2 of the second modification of the second embodiment is an embodiment. It differs from the peripheral 3D generation unit 21 of 2 in that it has a peripheral 3D generation unit 21'.

FIG. 7 is a block diagram showing a configuration example of the surrounding 3D generation unit in the second modification of the second embodiment. The peripheral 3D generation unit 21'includes a generation unit 26.

The generation unit 26 inputs a 360-degree video signal from the reception unit 11 and also inputs preset surrounding 3D data. Then, the generation unit 26 extracts an object from the 360-degree video signal and generates an object video signal.

The generation unit 26 extracts an object from the pre-surrounding 3D data, and associates the object extracted from the 360-degree video signal with the object extracted from the pre-surrounding 3D data by a matching process. Then, the generation unit 26 generates the surrounding 3D data by projecting the corresponding object video signal onto the object of the prior surrounding 3D data and overwriting it, and records the surrounding 3D data as sub-content in the recording unit 15.

As a result, it is possible to obtain surrounding 3D data that reflects the current object image when the first person is watching the program, instead of the past object image when the advance surrounding 3D data was obtained by the prior shooting. ..

The preset surrounding 3D data is shot while moving in the room where viewing is performed in advance using a wearable 360-degree camera 2, etc., and the above-mentioned surrounding 3D data generation processing is performed using the plurality of shot images. It is 3D data constructed by the photogrammetry of. This pre-peripheral 3D data includes 3D data in which the photographed object is represented by 3D CG.

As described above, according to the recording / playback device 1-2 of the second modification of the second embodiment, the surrounding 3D generation unit 21'projects the image of the object included in the 360-degree video signal onto the pre-peripheral 3D data. And overwrite it to generate surrounding 3D data. Then, the peripheral 3D generation unit 21'records the peripheral 3D data as sub-content in the recording unit 15.

As a result, the surrounding 3D data with higher accuracy than that of the second embodiment is recorded in the recording unit 15. Therefore, at the time of reproduction, the object other than the surrounding person by the surrounding 3D data is reproduced as a 3D CG in a state closer to reality. can do.

[Example 3]
Next, Example 3 will be described. As described above, Example 3 is an example of recording a highly accurate 3D image of a surrounding person in Example 2. In the third embodiment, it is possible to reproduce the surrounding people with a highly realistic three-dimensional CG.

FIG. 8 is a block diagram showing a configuration example of the recording / playback device of the third embodiment. The recording / playback device 1-3 includes receiving units 10, 11, 30, a person 3D generation unit 32, a surrounding 3D generation unit 21, a television position detection unit 13, a person voice extraction unit 14, a line-of-sight detection unit 31, and a correction unit 33. It includes a recording unit 15 and a transmitting unit 16.

Comparing the recording / playback device 1-2 of the second embodiment shown in FIG. 4 with the recording / playback device 1-3, both the recording / playback devices 1-2 and 1-3 generate the receiving units 10 and 11 and the surrounding 3D. It is the same in that it includes a unit 21, a television position detection unit 13, a person voice extraction unit 14, a recording unit 15, and a transmission unit 16. On the other hand, the recording / reproducing device 1-3 includes a person 3D generating unit 32 different from the person 3D generating unit 20 of the recording / reproducing device 1-2, and further includes a receiving unit 30, a line-of-sight detection unit 31, and a correction unit 33. It differs in that it is.

The recording / playback apparatus 1-3 performs the same processing as in steps S501 to S504 and S506 to 509 shown in FIG. In step S505, the recording / playback device 1-3 generates person 3D data based on the 360-degree video signal, the bird's-eye view video signal of the person, and the line-of-sight direction data, and the person 3D data for each person is used as sub-content in the recording unit 15. Record in.

Similar to the second embodiment shown in FIG. 4, the receiving unit 10 receives the program video / audio signal and records the program video / audio signal as the main content in the recording unit 15.

Similar to the second embodiment shown in FIG. 4, the receiving unit 11 receives the 360-degree video-audio signal from the wearable 360-degree camera 2 and separates the 360-degree video-audio signal into a 360-degree video signal and a 360-degree audio signal. .. Then, the receiving unit 11 outputs the 360-degree video signal to the person 3D generation unit 32, the surrounding 3D generation unit 21, and the television position detection unit 13, and outputs the 360-degree audio signal to the person voice extraction unit 14.

The receiving unit 30 receives the bird's-eye view video signal of the person from the bird's-eye view camera 4, and outputs the bird's-eye view video signal to the person 3D generation unit 32. The bird's-eye view camera 4 is installed in a room where viewing is performed, takes a picture of a person around the first person in a bird's-eye view, and transmits a bird's-eye view video signal of the person to the recording / playback device 1-3.

This bird's-eye view camera 4 is used by the person 3D generation unit 32 to generate high-precision bone data, and as a result, is used to generate high-precision person 3D data. The bird's-eye view video signal is a signal different from the 360-degree video signal output from the receiving unit 11, and is a signal in which a person is photographed from an angle different from the 360-degree video signal.

The person 3D generation unit 32 inputs a 360-degree video signal from the receiving unit 11, inputs a bird's-eye view video signal of the person from the receiving unit 30, extracts a person's image from the 360-degree video signal image, and extracts the person's image. To generate. The person 3D generation unit 32 estimates bone data for each person indicating the posture of the person from the person image signal and the bird's-eye view image signal of the person. Then, the person 3D generation unit 32 generates the person 3D data based on the person video signal and the bone data, and outputs the person 3D data for each person to the correction unit 33.

The person 3D generation unit 32 includes a person extraction unit 12, a posture estimation unit 27, and a person generation unit 23. Similar to the second embodiment shown in FIG. 4, the person extraction unit 12 inputs a 360-degree video signal from the receiving unit 11 and extracts texture information of the person video signal from the 360-degree video signal. Then, the person extraction unit 12 outputs the texture information of the person video signal for each person to the generation unit 23.

The posture estimation unit 27 inputs a 360-degree video signal from the receiving unit 11, and also inputs a bird's-eye view video signal of a person from the receiving unit 30. Then, the posture estimation unit 27 analyzes the 360-degree video signal by a predetermined posture estimation process, similarly to the posture estimation unit 22 shown in FIG. 4, and the body, hand, and the person image included in the 360-degree video signal. Detects parts such as the face and estimates the posture of each person. The posture estimation unit 27 analyzes the bird's-eye view video signal by a predetermined posture estimation process in the same manner as the 360-degree video signal, detects parts of the person image included in the bird's-eye view video signal, such as the body, hands, and face, and the person. Estimate each posture.

The posture estimation unit 27 uses the posture estimated from the 360-degree video signal and the posture estimated from the bird's-eye view video signal to optimize the naturalness of the posture without the joints coming off or having an unnatural angle. The three-dimensional position is specified in the manner of triangulation, and bone data showing the posture of each person is generated. Then, the posture estimation unit 27 outputs bone data for each person to the generation unit 23. As a result, the bone data becomes data that reflects the posture of the person with high accuracy, and the generation unit 23 in the subsequent stage can generate highly accurate person 3D data.

The posture estimation process by the posture estimation unit 27 is known, and the same “SMPLify” technique as the above-mentioned posture estimation unit 22 or the following “OpenPose” technique is used.
"OpenPose: Real-time multi-person keypoint detection library for body, face, and hands estimation", [online], [Search on May 11, 2018], Internet <https://github.com/CMU-Perceptual -Computing-Lab / openpose ＞

The generation unit 23 inputs the texture information of the person video signal for each person from the person extraction unit 12, and also inputs the bone data for each person from the posture estimation unit 27. Then, the generation unit 23 projects and overwrites the person video signal on the bone data in the same manner as in the second embodiment shown in FIG. 4, and generates the person 3D data. The generation unit 23 outputs the person 3D data for each person to the correction unit 33.

The line-of-sight detection unit 31 is based on an eyeball image detected by a camera of the line-of-sight detection glass while each person around the first person is wearing a wearable line-of-sight detection glass (not shown). Detects the line of sight of a person and generates line-of-sight direction data for each person. Then, the line-of-sight detection unit 31 outputs the line-of-sight direction data for each person to the correction unit 33.

Since the line-of-sight direction detection process is known, detailed description thereof will be omitted here. As the line-of-sight detection glass, for example, the glass introduced at the following site is used.
"Tobii Pro Glasses 2", [online], [Searched May 11, 2018], Internet <https://www.tobiipro.com/ja/product-listing/tobii-pro-glasses-2/>

The correction unit 33 inputs the person 3D data for each person from the person 3D generation unit 32, and inputs the line-of-sight direction data for each person from the line-of-sight detection unit 31. Then, the correction unit 33 obtains the line-of-sight direction indicating where the person was looking based on the line-of-sight direction data for each person. The correction unit 33 corrects the person 3D data by correcting the position of the black eye in the person 3D data so that the line-of-sight direction of the person 3D data matches the line-of-sight direction obtained from the line-of-sight direction data. The correction unit 33 records the person 3D data for each person whose black eye position has been corrected in the recording unit 15 as sub-contents.

As a result, the line-of-sight direction of the person based on the corrected person 3D data becomes an accurate direction, and high-precision person 3D data can be generated.

Since the television position detection unit 13, the person voice extraction unit 14, the recording unit 15, and the transmission unit 16 are the same as the components shown in FIG. 4, description thereof will be omitted here.

As a result, a plurality of program video / audio are recorded in the recording unit 15 as the main content. Further, in the recording unit 15, as sub-contents, person 3D data for each person related to the person around the first person and surrounding 3D related to the object around the first person corresponding to each of the plurality of program video / audio are provided. Data, person voice signals and television position data are recorded. The person 3D data is data with higher accuracy than that of the second embodiment.

As described above, according to the recording / playback device 1-3 of the third embodiment, the person 3D generation unit 32 extracts the person image from the image of the 360-degree video signal by the person extraction unit 12, and causes the posture estimation unit 27 to extract the person image. The bone data for each person is estimated from the 360-degree video signal and the bird's-eye view of the person. Then, the person 3D generation unit 32 generates the person 3D data for each person based on the person video signal and the bone data in the generation unit 23.

The line-of-sight detection unit 31 detects the line of sight of each person and generates line-of-sight direction data, and the correction unit 33 corrects the position of the black eye of the person 3D data based on the line-of-sight direction data, and the corrected person for each person. The 3D data is recorded in the recording unit 15 as sub-contents.

As a result, the person 3D data with higher accuracy than that of the second embodiment is recorded in the recording unit 15, so that the surrounding people can reproduce the person as a three-dimensional animation based on the person 3D data in a state closer to reality at the time of reproduction. can.

In this way, by using the AR glass or VR HMD, the main content is reproduced and the reaction of the person around the first person is reproduced, so the program that was watched in the past is the same as that time. A viewing experience and a new viewing experience can be obtained.

The recording / playback device 1-3 is a person 3D generation unit 32 provided with a person extraction unit 12, a posture estimation unit 27, and a generation unit 23, and is a person for each person based on a 360-degree video signal and a bird's-eye view video signal. I tried to generate 3D data. In this case, the recording / reproducing device 1-3 may include a person 3D generation unit 20'using the prior person 3D data shown in FIG. 6 instead of the person 3D generation unit 32.

The person 3D generation unit 20'of the recording / playback device 1-3 includes a posture estimation unit 22, a pre-person 3D generation unit 24, and a generation unit 25, as in the first modification of the second embodiment shown in FIG. There is. In this case, the posture estimation unit 22 inputs a bird's-eye view video signal from the reception unit 30 in addition to the 360-degree video signal, generates bone data, and generates a pre-person 3D, similarly to the posture estimation unit 27 shown in FIG. Output to unit 24.

Further, the recording / playback device 1-3 is configured to generate peripheral 3D data based on the 360-degree video signal by the peripheral 3D generation unit 21. In this case, instead of the peripheral 3D generation unit 21, the recording / playback device 1-3 uses the peripheral 3D generation unit 21'which uses the preliminary peripheral 3D data, as in the second modification of the second embodiment shown in FIG. You may be prepared. As described above, the peripheral 3D generation unit 21'generates peripheral 3D data based on the 360-degree video signal and the preliminary peripheral 3D data.

Although the present invention has been described above with reference to Examples 1, 2, and 3, the present invention is not limited to the above Examples 1, 2, and 3, and can be variously modified without departing from the technical idea. be. For example, the recording / playback device 1-3 of the third embodiment includes a person 3D generation unit 32 that uses a bird's-eye view video signal of a video of each person taken by the bird's-eye view camera 4, and a line-of-sight detection unit 31 that detects the line of sight of each person. I made it. On the other hand, the recording / reproducing device 1-3 may further include a facial expression detecting unit for detecting the facial expression of a person's face.

In this case, the facial expression detection unit generates facial expression data for the facial expression of the person detected by the facial expression detection device, and outputs this to the correction unit 33. The correction unit 33 inputs facial expression data for each person from the facial expression detection unit, and reflects the facial expression data on the face of the person 3D data for each person. Then, the correction unit 33 records new person 3D data for each person as sub-contents in the recording unit 15.

The facial expression detection process by the facial expression detection unit is known, and for example, the following techniques are used.
"Instantly reflect human eyes and facial expressions on avatars in VR space--360Channel develops" FACE "", [online], [Search on May 11, 2018], Internet <https://japan.cnet .com / article / 35101852 />

As the facial expression detection glass, for example, the glass introduced at the following site is used.
"AffectiveWear: Eyeglass-type device that recognizes the wearer's daily facial expressions", [online], [Search on May 11, 2018], Internet <https://www.jstage.jst.go.jp/article / tvrsj / 21/2 / 21_385 / _pdf ＞

As the hardware configuration according to the recording / playback devices 1-1, 1-2, 1-3 of the first, second, and third embodiments of the present invention, a normal computer can be used. The recording / playback device 1-1, 1-2, 1-3 is composed of a volatile storage medium such as a CPU and RAM, a non-volatile storage medium such as a ROM, and a computer provided with an interface and the like.

The functions of the receiving units 10 and 11, the person extracting unit 12, the television position detecting unit 13, the person voice extracting unit 14, the recording unit 15, and the transmitting unit 16 provided in the recording / reproducing device 1-1 describe these functions. Each is realized by having the CPU execute the program. In addition, each of the receiving units 10 and 11, the person 3D generation unit 20, the surrounding 3D generation unit 21, the television position detection unit 13, the person voice extraction unit 14, the recording unit 15, and the transmission unit 16 provided in the recording / playback device 1-2. The functions are also realized by causing the CPU to execute a program describing these functions. In addition, the receiving units 10, 11, 30, the person 3D generation unit 32, the surrounding 3D generation unit 21, the television position detection unit 13, the person voice extraction unit 14, the line-of-sight detection unit 31, and the correction unit provided in the recording / playback device 1-3. The functions of 33, the recording unit 15, and the transmitting unit 16 are also realized by causing the CPU to execute a program describing these functions.

These programs are stored in the storage medium, read by the CPU, and executed. In addition, these programs can be stored and distributed in storage media such as magnetic disks (floppy (registered trademark) disks, hard disks, etc.), optical disks (CD-ROM, DVD, etc.), semiconductor memories, etc., and can be distributed via a network. You can also send and receive.

1 Recording / playback device 2 Wearable 360-degree camera 3 Playback equipment 4 Bird's-eye view camera 10, 11, 30 Receiver 12 Person extraction unit 13 TV position detection unit 14 Person voice extraction unit 15 Recording unit 16 Transmission unit 20, 20', 32 Person 3D Generation unit 21,21'Around 3D generation unit 22,27 Posture estimation unit 23 Generation unit 24 Pre-person 3D generation unit 25,26 Generation unit 31 Line-of-sight detection unit 33 Correction unit 40 Program identification unit 41 Subtraction unit

Claims (7)

The program video / audio signal including the video and audio of the program played on the television is recorded in the recording unit, and the program video / audio signal is read from the recording unit and played back according to the operation of the user who views the program on the playback device. In the recording / playback device that sends to the device
A first receiving unit that receives the program video / audio signal and records the program video / audio signal as the main content in the recording unit.
A second second, which receives an ambient video signal including an image of a person existing around the user and an image of the television on which the program is being played, and also receives an ambient audio signal picked up from the surroundings of the user. Receiver and
Person extraction that extracts the person from the surrounding video signal received by the second receiving unit, generates a person video signal for each person, and records the person video signal for each person as sub-content in the recording unit. Department and
The position of the television is detected from the ambient video signal received by the second receiving unit, and television position data indicating the position where the program video signal in the program video audio signal is reproduced by the playback device is generated. A TV position detection unit that records the TV position data as the sub-content in the recording unit, and
Based on the program video audio signal recorded in the recording unit and the ambient audio signal received by the second receiving unit, the program corresponding to the program audio signal included in the ambient audio signal is specified. , The difference between the ambient audio signal and the program audio signal in the program video audio signal of the program is obtained, a person audio signal of the person existing around the user is generated, and the person audio signal is used. A person audio extraction unit that records in the recording unit as sub-content, and
A transmission unit that reads the main content and the sub-content from the recording unit and transmits the main content and the sub-content to the playback device.
A recording / playback device characterized by being equipped with. In the recording / playback device according to claim 1,
A person 3D (three-dimensional) generation unit is provided in place of the person extraction unit, and a surrounding 3D generation unit is further provided.
The person 3D generation unit
The person is extracted from the surrounding image signal received by the second receiving unit, a person image for each person is generated, the posture of the person included in the surrounding image signal is estimated, and the posture of the person is determined. Bone data for each person to be shown is generated, person 3D data is generated as data of the 3D CG animation of the person based on the person video signal and the bone data, and the person 3D data for each person is used as the sub-content. Recorded in the recording unit as
The surrounding 3D generation unit is
An object is extracted from the ambient video signal received by the second receiving unit, peripheral 3D data including 3D CG data of the object is generated, and the peripheral 3D data is used as the sub-content in the recording unit. Record and
In the recording unit
The recording / reproduction is characterized in that the program video / audio signal is recorded as the main content, and the person 3D data, the surrounding 3D data, the television position data, and the person audio signal for each person are recorded as the sub-content. Device. In the recording / playback device according to claim 2,
Further, a third receiving unit, a line-of-sight detection unit, and a correction unit are provided.
The third receiving unit is
A video signal different from the ambient video signal received by the second receiving unit, and a bird's-eye view video signal including the person existing around the user is received.
The person 3D generation unit
The person is extracted from the surrounding image signal, the person image is generated for each person, the posture of the person included in the surrounding image signal is estimated, and the bird's-eye view image received by the third receiving unit is used. The posture of the person included in the signal is estimated, and the bone data for each person indicating the posture of the person is generated based on the posture of the person estimated from the surrounding image signal and the bird's-eye view image signal, respectively. The person 3D data is generated based on the person video signal and the bone data.
The line-of-sight detection unit
Detects the line of sight of the person, generates line-of-sight direction data for each person, and
The correction unit
The person 3D data is corrected so that the line-of-sight direction of the person 3D data generated by the person 3D generation unit matches the line-of-sight direction indicated by the line-of-sight direction data generated by the line-of-sight detection unit, and for each person. A recording / playback device characterized in that the corrected person 3D data is recorded in the recording unit as the sub-content. In the recording / playback device according to claim 2,
The person 3D generation unit
The 3D data of the person for each person set in advance is input, the posture of the person included in the surrounding video signal is estimated, the bone data for each person showing the posture of the person is generated, and the 3D data is generated. And, based on the bone data, the pre-person 3D data is generated as the data of the 3D CG animation of the person, and the image of the person included in the surrounding video signal is projected onto the pre-person 3D data to overwrite the data. A recording / playback device for generating the person 3D data and recording the person 3D data for each person in the recording unit as the sub-contents. In the recording / playback device according to claim 3,
The person 3D generation unit
The 3D data of the person for each person set in advance is input, the posture of the person included in the surrounding image signal is estimated, and the posture of the person included in the bird's-eye view image signal is estimated, and the surrounding image is estimated. Based on the posture of the person estimated from the signal and the bird's-eye view video signal, the bone data for each person indicating the posture of the person is generated, and based on the 3D data and the bone data, the person is three-dimensional. Pre-person 3D data is generated as CG animation data, the image of the person included in the surrounding video signal is projected and overwritten on the pre-person 3D data, the person 3D data is generated, and the said for each person. A recording / playback device characterized in that person 3D data is recorded in the recording unit as the sub-contents. The recording / playback device according to any one of claims 2 to 5.
The surrounding 3D generation unit is
The preset 3D data around the user is input as the pre-peripheral 3D data, the object is extracted from the ambient video signal to generate an object image, and the object included in the pre-surrounding 3D data is combined with the object. A recording / playback device characterized in that the corresponding object image is projected and overwritten, the surrounding 3D data is generated, and the surrounding 3D data is recorded in the recording unit as the sub-content. A program for operating a computer as a recording / playback device according to any one of claims 1 to 6.

Images