JP2013046358A - Content reproduction apparatus and content reproduction program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a content which is provided with image information having individually suitable added value.SOLUTION: A content reproduction apparatus for reproducing a content using an image obtained by photographing a program displayed on a screen comprises: imaging means for acquiring an image including the screen on which the program is displayed; reproduction content setting means which sets a reproduction content with respect to the image obtained by the imaging means; and reproduction means which on the basis of setting information obtained by the reproduction content setting means, reproduces an image obtained by displaying a predetermined object at a preset position of the image obtained by the imaging means.

Description

  The present invention relates to a content reproduction apparatus and a content reproduction program, and more particularly, to a content reproduction apparatus and a content reproduction program for providing content to which video information with appropriate added value is individually added.

  Conventionally, various programs are produced on television stations and the like. The produced program is transmitted to each viewer's home or the like by a broadcast wave or the like, and displayed on the transmitted television receiver in the home or the like to be provided to each viewer.

  Conventionally, a mechanism for automatically creating a program using a computer is known (see, for example, Patent Document 1). In the technique shown in Patent Document 1, in the production of a television news program or the like, a program description language or TVML (TV program Making Language) is used, and program production is automatically performed by inputting production data and information data. Yes.

  TVML is an object-based description language for producing TV programs. Video and audio of TV programs can be described separately for material data and scripts (production contents, etc.). Can be read by software operating on a television receiver, PC (Personal Computer), etc., and the material data described in the script can be acquired and immediately reproduced and viewed as a TV program. .

JP 2001-24610 A

  Here, in the conventional program production as described above, for example, when setting of CG (Computer Graphics) characters appearing in a program, production data, and the like, a program displayed based on the setting is a television receiver. Displayed on the screen of a PC or the like. For example, in the technique as disclosed in Patent Document 1, effect data and information data are set in correspondence with preference information (for example, favorite genre) of a predetermined viewer (user) who performs the setting. Thus, a dedicated program that suits the viewer's preference can be displayed.

  However, for example, when a program displayed on the screen of one television receiver is viewed by a plurality of viewers (for example, family members or siblings), the above-described preference information is information of one individual who is a predetermined viewer. Therefore, only viewers corresponding to the set preference information can be satisfied, and it is not possible to give video information with individual added value to all of the plurality of viewers. Therefore, for example, when one of the viewers who watches at the same time is a hard of hearing, there has been no method for providing video information with appropriate added value such as displaying sign language only for the viewer.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to provide a content playback apparatus and a content playback program for providing content to which video information with appropriate added value is individually added. .

  In order to solve the above problems, the present invention employs means for solving the problems having the following characteristics.

  The present invention provides a content reproduction apparatus that reproduces content using an image obtained by photographing a program displayed on a screen, an imaging unit for obtaining an image including the screen on which the program is displayed, and an image obtained by the imaging unit. Based on the setting information obtained by the reproduction content setting means and the reproduction content setting means, a predetermined object is displayed at a preset position of the image obtained by the imaging means. And reproducing means for reproducing the recorded image.

  Further, the present invention is a content reproduction program for causing a computer to function as each unit included in the above-described content reproduction apparatus.

  ADVANTAGE OF THE INVENTION According to this invention, the content which provided the video information with the appropriate additional value individually can be provided.

It is a figure which shows the example of schematic structure of the content reproduction system in this embodiment. It is a figure which shows an example of the function structure in a content provision server. It is a figure which shows an example of the function structure in a portable terminal. It is a figure for demonstrating the specific example of the content provision in this embodiment. It is a figure which shows the example of a screen display at the time of the movement detection of the portable terminal in this embodiment. It is a figure for demonstrating the correspondence of viewing environment and virtual studio. It is a figure for demonstrating the concrete of the control parameter of a virtual camera. It is a figure which shows an example of CG character control corresponding to the motion of a portable terminal. It is a flowchart which shows an example of a content reproduction | regeneration processing procedure.

<About the present invention>
The present invention relates to a display device (for example, a television receiver, a PC display, a display unit of a portable terminal, etc.) that displays content such as a television program (hereinafter also referred to as “broadcast program” or “program”). The image (including the video) obtained by photographing the screen is provided with individual added value video information and the like to the viewer. Specifically, in the present invention, for example, when a broadcast TV program is displayed on the screen of a television receiver, the displayed screen is captured and the angle of view of the captured camera is supported. Processing such as displaying an object such as a CG character or a prop on the image space, or causing the displayed object to perform a predetermined effect (for example, sign language) according to the contents of the television program is performed.

  Furthermore, the present invention provides a portable terminal (for example, a tablet PC, a mobile phone, a digital camera, a game machine, a notebook type) that can be easily carried as an apparatus used when photographing the screen of the display device as described above. If the mobile terminal that captures the screen of the display device is moved, the orientation and operation of the object described above may be changed according to the positional relationship between the display device and the mobile terminal. it can. As a result, CG characters and the like can be displayed from various directions, and for example, video information with high added value such as changing the direction in which the sign language hand can be seen can be given.

  Hereinafter, a preferred embodiment of a content reproduction apparatus and a content reproduction program according to the present invention having the above-described features will be described in detail with reference to the drawings.

  In the present embodiment, a program is used as an example of content. In the present embodiment, TVML is used as an example of a script used for program playback or the like. However, the present invention is not limited to this.

<Example of schematic configuration of content playback system>
FIG. 1 is a diagram illustrating a schematic configuration example of a content reproduction system according to the present embodiment. A content playback system 10 shown in FIG. 1 includes a content providing server 11, a television receiver 12 as a display device, and one or a plurality of mobile terminals 13 as content playback devices (in the example of FIG. 1, mobile terminals 13-1). 13-2). The content providing server 11 and the mobile terminal 13 are connected in a state where data can be transmitted and received via a communication network 14 such as an Internet line or a wireless communication line.

  The content providing server 11 is a production-side device that produces television programs and the like to be provided to viewers based on various data such as images, videos, sounds, and text data stored in advance. The content providing server 11 may be a broadcasting station.

  The content distribution server 11 outputs the produced television program to the viewer-side display device 12 (in the example of FIG. 1, a television receiver) using, for example, the broadcast wave shown in FIG. Note that the transmission method of the television program is not limited to the broadcast wave. For example, when the television program is transmitted to the PC instead of the television receiver 12, it may be distributed from the content providing server 11 by a stream signal or the like.

  Further, the content providing server 11 outputs a TVML script corresponding to the television program transmitted to the viewer side by broadcast waves to the mobile terminals 13-1 and 13-2 via the communication network 14. Accordingly, in the present embodiment, CG data corresponding to a program displayed on the screen of the television receiver 12 on the mobile terminal 13 side (including various objects such as CG characters and props, images, videos, and the like). Etc. can be acquired individually. Therefore, it is possible to provide content to which video information having appropriate added value for each viewer is added.

  The television receiver 12 is a first display unit that displays a television program transmitted from the content providing server 11. The television receiver 12 may be a general display device such as a CRT (Cathode Ray Tube) display or a liquid crystal display. However, the present invention is not limited to this.

  The portable terminals 13-1 and 13-1 are owned by individual viewers 15-1 and 15-2, respectively, and can display imaging means such as a camera and captured images (including video). Display means (second display means) that can be used. The portable terminal 13 can display the program displayed on the screen of the television receiver 12 (first display means) on the second display means by photographing the program with the imaging means. The mobile terminal 13 may be a general terminal such as a mobile phone terminal with a camera, a tablet PC, or an HMD (Head Mounted Display), but is not limited to this in the present invention. Absent.

  At this time, the mobile terminal 13 displays the object 16 such as a CG character spatially superimposed on the video of the screen of the television receiver 12 by software (application) or the like in the present embodiment. The video information with individual added value is given to each viewer for each portable terminal 13. Thereby, even if the video displayed on the portable terminals 13-1 and 13-2 is a video of a program displayed on the television receiver 12, different video is additionally displayed. be able to. Note that the screen of the television receiver 12 that is a live-action image taken by the mobile terminal 13 is not necessarily an image that is actually displayed. For example, a television program received by a receiver attached to the mobile terminal 13 is overlaid. It is also possible to display images that are individually generated by the mobile terminal 13 and to display them.

  That is, in this embodiment, as shown in FIG. 1, the space between the television receiver 12 and the mobile terminal 13 is used as a three-dimensional virtual studio, and the object 16 exists in the space so that the object 16 exists. It is displayed on the screen. Therefore, the content displayed from the screen of the mobile terminal 13 is the content captured by the virtual camera (CG camera).

  If there is a viewer who does not need added-value video information, it is not necessary to view the program via the mobile terminal 13, and the program displayed on the screen of the television receiver 12 may be viewed as it is. It will be.

  In addition, a TVML script corresponding to a program played back on the television receiver 12 is transmitted from the content providing server 11 to the viewing terminal 13 via the communication network 14. Therefore, the viewing terminal 13 can generate the object 16 based on the content of the TVML script obtained from the content providing server 11, and can display and produce the object corresponding to the program display.

  Here, the number of the content providing server 11, the television receiver 12, and the portable terminal 13 constituting the content reproduction system 10 shown in FIG. 1 is not limited to the number shown in FIG. There can be one or more. That is, in the present embodiment, the relationship among the numbers of content providing server 11, television receiver 12, and portable terminal 13 can be L: M: N (L, M, and N are 1 or more). Next, a functional configuration example of the content providing server 11 and the mobile terminal 13 described above, which is the main configuration of the present embodiment, will be specifically described.

<Content providing server 11: functional configuration example>
FIG. 2 is a diagram illustrating an example of a functional configuration in the content providing server. The content providing server 11 shown in FIG. 2 includes an input unit 21, an output unit 22, a storage unit 23, a script generation unit 24, a program script generation unit 25, a program production unit 26, a distribution control unit 27, The reproduction unit 28, the transmission / reception unit 29, and the control unit 30 are configured.

  The input means 21 receives various inputs such as a script (production) generation instruction, a program script generation instruction, a program production instruction, a distribution control instruction, a reproduction instruction, and a transmission / reception instruction from an administrator or a program producer (user). Accept. Note that the input unit 21 includes, for example, a keyboard, a pointing device such as a mouse, a voice input device such as a microphone, and the like.

  The output means 22 displays the contents of instructions input by the input means 21, the data generated based on the contents of each instruction, the contents of produced programs, etc., and outputs sound. The output unit 22 includes a screen display device such as a display, an audio output device such as a speaker, and the like. Note that the input unit 21 and the output unit 22 in the present embodiment may have both input and output functions, such as a touch panel.

  The storage unit 23 stores, for example, a program script (hereinafter also simply referred to as “script data”), material data, a program production engine, a script, a program, and the like, and is read and written as necessary.

  Specifically, a program script in which all effects for the generated program are described, and a plurality of data such as images, video, audio, text data, etc. for generating the program described in the program script. Material data (characters, sets, sounds, actions, still images, moving images, etc.), program production contents and program production engine necessary for composing the program, TVML script corresponding to the produced program, program generated by TVML script Etc. are accumulated.

  Note that the program production engine is a CG character appearing in a program, or an event such as “title display”, “zoom-in”, or “CG character motion” defined in advance for each motion in the program. is there. In this embodiment, by using a program production engine, for example, when text data or the like from script data representing the contents of a program is input, a TVML script necessary for automatic program generation can be output based on the text data.

  Further, the storage means 23 stores other various setting information. The storage unit 23 can also acquire the various data described above from an external device connected to the communication network 14 or the like.

  The script generation means 24 generates script data including production information and the like for producing a program that is transmitted to the television receiver 12 and provided to the viewer. The script generation means 24 generates script data for providing content (program) to which an object is added to a program that is transmitted to the mobile terminal 13 and provided to the viewer.

  Therefore, when the program to be transmitted to the television receiver 12 has already been produced, the script generation unit 24 generates only the script data corresponding to the program. Therefore, for example, when a program such as a live action has already been produced, the script generation means 24 regards the program as one video data, and displays a CG character object or the like on the screen of the portable terminal 13 for the video data. Script data for generating the script is generated. The script generation unit 24 stores the generated script data in the storage unit 23.

  The program script generation means 25 is a program script corresponding to a program to be transmitted to the television receiver 12 using the script data generated by the script generation means 24, the material data stored in the storage means 23, the program production engine, or the like. (For example, a TVML script) is generated. This script includes effect information for the program, and additionally includes setting information such as CG characters displayed on the screen of the mobile terminal 13, various objects such as desks and chairs, and actions for each object. Further, the program script generation unit 25 stores the generated script in the storage unit 23.

  The program production means 26 produces a program that is actually displayed on the television receiver 12 in correspondence with the program script generated by the program script generation means 25. In this embodiment, as an example, the program generated by the program production means 26 is a program in a virtual space based on TVML. However, the present invention is not limited to this, and a program such as a live-action film is produced. May be. In this case, for example, editing content (for example, screen switching or telop such as wipe) that is directed by the program producer or the like using the input means 21 or the like with respect to a video shot by one or more cameras in a studio in a real space A predetermined program is produced based on the The program production means 26 accumulates the produced program in the accumulation means 23.

  The distribution control means 27 transmits a program, a program script, etc. to the viewer side. Specifically, the distribution control means 27 transmits the produced program to the television receiver 12 by broadcast waves or the like. The distribution control means 27 distributes a program script corresponding to the transmitted program to the viewer's mobile terminal 13. The distribution control means 27 may control the distribution of the corresponding program script when, for example, the portable terminal 13 receives an inquiry about the program script for the currently viewed program.

  The reproduction unit 28 reproduces the program script (TVML script) generated by the program script generation unit 25 and displays the video or audio as a viewer using the output unit 22. In the present embodiment, the function of the playback means 28 described above is referred to as a “TVML player”. Further, the reproducing means 28 accumulates the reproduced program in the accumulating means 23.

  The transmission / reception means 29 transmits various data such as control data, program, program script (TVML script), program production engine, material data, script data, etc. via the communication network 14 to the television receiver 12 constituting the content reproduction system 10 and the like. This is a communication interface for transmitting data to the mobile terminal 13 and receiving various data from the television receiver 12, the mobile terminal 13, and other terminals.

  Specifically, the transmission / reception means 29 transmits, for example, a program to the television receiver 12 and transmits a TVML script to the mobile terminal 13. The transmission / reception means 29 can receive a TVML script acquisition request (inquiry) from the mobile terminal 13, and the mobile phone that requested the TVML script corresponding to the request, the accompanying material data, the program production engine, and the like. It can be transmitted to the terminal 13.

  The control unit 30 controls the entire functional configuration in the content providing server 11. Specifically, the control unit 30 generates script data, generates a program or a program script, or creates a program based on input information from an administrator or a program producer input by the input unit 21. Various controls such as production, distribution control, and program playback are performed.

<Portable terminal 13: functional configuration example>
Next, a functional configuration example of the mobile terminal 13 in the content reproduction system 10 described above will be described with reference to the drawings. FIG. 3 is a diagram illustrating an example of a functional configuration in the mobile terminal. 3 includes an input unit 31, an output unit 32 as a second display unit, a storage unit 33, an imaging unit 34, a reproduction content setting unit 35, a reproduction unit 36, and movement detection. Means 37, transmission / reception means 38, and control means 39 are provided.

  The input unit 31 receives various inputs such as a shooting instruction from a viewer (user), a reproduction content setting instruction, a reproduction instruction, and a movement detection instruction. Note that the input unit 31 includes, for example, an operation button, a voice input device such as a microphone, and the like.

  The output means 32 displays the contents of instructions input by the input means 31, the contents of control data and programs generated based on the contents of each instruction, and outputs sound. The output unit 32 includes a screen display device such as a display, an audio output device such as a speaker, and the like. Note that the input unit 31 and the output unit 32 in the present embodiment may have both input and output functions, such as a touch panel.

  The storage unit 33 stores, for example, a program script (TVML script), a program production engine, material data, and the like, and is read and written as necessary. The storage means 33 is, for example, an image (including video) taken by the imaging means 34, setting information set by the reproduction content setting means 35, contents reproduced by the reproduction means 36, and detection results obtained by the movement detection means 37. Accumulate etc.

  Note that the data stored by the storage means 33 can be read and written as necessary. Further, the storage unit 33 stores other various setting information. Furthermore, the storage means 33 can also acquire the various data described above from an external device connected to a communication network or the like.

  The imaging unit 34 captures an image having a predetermined angle of view with a camera or the like. In the present embodiment, an image including the screen of the television receiver 12 is taken by the imaging unit 34.

  Note that the television receiver 12 may be photographed in all frames in a video when it is photographed as a video, or may be photographed in some frames. Therefore, in the present embodiment, it is preferable that the entire screen of the television receiver 12 is included in one or more frames of the video shot by the imaging unit 34. Further, in the present embodiment, it is only necessary that the screen of the television receiver 12 is included in a part of the video, and the size is not particularly limited, but a predetermined amount that can be visually recognized with respect to the entire screen. It is preferable that the screen has a ratio or more. This is because if it can be seen with the naked eye, the viewer can easily understand that video information with added value for the program is being provided.

  The reproduction content setting means 35 is, for example, the type of object (eg, CG character, chair, prop, table, graph, product, etc.) to be displayed as value-added video information during program reproduction, and the position (area) to be displayed (composited). Various reproduction contents such as a setting (for example, control parameters (camera parameters) of a virtual camera) of a size, a color, display contents, production contents, and an angle of view of an image displayed from the mobile terminal 13 are set. The reproduction content setting unit 35 sets the operation content of the object based on the movement content of the mobile terminal 13 itself detected by the movement detection unit 37 as the reproduction content.

  Here, the reproduction content may be set by the viewer inputting the setting for each condition using the input means 31 or the like, for example. The reproduction content setting unit 35 performs image analysis on the image obtained by the imaging unit 34, for example, and at least of the position, size, inclination, etc. of the screen frame of the television receiver 12 included in the image. One of them may be acquired, and based on the acquisition result, the playback content of the object described above may be set. Further, the reproduction content setting means 35 may analyze the program displayed on the screen of the television receiver 12 and set conditions for performing individual reproduction control based on information obtained from the result. it can.

  Thus, in the present embodiment, for example, an image on which a program is displayed is analyzed, and when the first or last of the program is displayed or a special display, telop, superimpose, or the like set in advance is displayed. It is possible to control the display or non-display of the object corresponding to, the operation content of the object, and the like. Furthermore, in the present embodiment, for example, it is possible to cause the CG character to perform sign language in accordance with the contents of the telop.

  As the image analysis method described above, the position and size of an object having a predetermined shape, displayed characters, and the like can be extracted based on luminance information for each pixel included in the image. When reading the position and size of an object by image analysis, for example, the viewer inputs an area for image analysis using the input unit 31 or the like, or inputs the screen size of the television receiver 12 or the like. Thus, it is possible to realize quicker and more accurate image analysis.

  Furthermore, in the present embodiment, the playback content of the object to be displayed can be changed by moving the mobile terminal 13 relative to the television receiver 12. In this case, the reproduction content setting means 35 takes the difference in the position information of the television receiver 12 between a plurality of frames included in the captured video, and the above-described reproduction content according to the movement status obtained from the difference information. Can also be set. When setting the playback content of the object for such movement of the mobile terminal 13, the playback content setting means 35 can also set the playback content using the detection result obtained from the movement detection means 37. . The reproduction content setting means 35 can also set the reproduction content based on the above-described difference information and detection result.

  Note that the playback content setting information described above may be acquired by being transmitted together with the TVML script from the content providing server 11 via the communication network 14. Further, the setting information of the reproduction content may be transmitted together with the program by a broadcast wave or a stream signal. In that case, the setting information of the reproduction content may be displayed together with the program on the television receiver 12, and the setting information may be acquired by photographing the image with the imaging unit 34 and analyzing the image.

  The reproduction unit 36 reproduces the TVML script generated by the content providing server 11 and causes the output unit 32 to display it as a viewer by video or audio. In the present embodiment, the function of the playback means 36 described above is referred to as a “TVML player”. Specifically, the playback unit 36 edits a video in which an object is combined with the video captured by the imaging unit 34 based on the playback content set by the playback content setting unit 35, for example, and the edited video content Play. Thereby, the content reproduced by the reproducing means 36 can be displayed on the display of the output means 32, for example.

  The movement detection unit 37 detects the movement (movement) of the mobile terminal 13 itself. Moreover, the movement detection means 37 acquires the positional information at the time of movement, etc. For example, a general acceleration sensor or a gyro sensor can be used as the movement detection unit 37, but the present invention is not limited to this. In the present embodiment, the reproduction content can be set by the reproduction content setting means 35 based on the movement detection content (position information and the like) obtained from the movement detection means 37, and the action of the object such as the corresponding CG character is set. Accordingly, individual content display is performed on each mobile terminal 13.

  The transmission / reception means 38 constitutes the content reproduction system 10 with various data such as control data, captured images (including video), set reproduction contents, reproduced programs, movement detection results, etc. via the communication network. It is a communication interface for transmitting to other terminals and receiving various data from other terminals.

  The control unit 39 controls the entire functional configuration of the mobile terminal 13. Specifically, the control means 39 shoots the television receiver 12 based on the input information from the viewer input by the input means 31, sets the playback content, and displays the program based on the playback content. And various controls such as movement detection.

<Specific examples of content provision>
Next, a specific example of content provision in the present embodiment will be described with reference to the drawings. FIG. 4 is a diagram for explaining a specific example of content provision in the present embodiment. In the example of FIG. 4, the specific data flow between the broadcasting station side (content providing server 11) and the viewer side (mobile terminal 13) is shown in a simplified manner.

  In this embodiment, as shown in FIG. 4, a TVML playback function (TVML player) as playback means 28 based on the above-described material data, program production engine, script data, etc. obtained from the storage means 23 of the content providing server 11. The broadcast program reproduced by is output to the television receiver 12 on the viewer side by the broadcast wave.

  In addition, the content providing server 11 transmits the material data and script data of the storage unit 23 corresponding to the program to the mobile terminal 13. In the example of FIG. 4, it is assumed that the program production engine is data that the mobile terminal 13 already has and does not transmit.

  The data transmission described above may be performed at the same time as the broadcast program transmission or at a different time. If the times are different, it is preferable that the mobile terminal 13 obtains the material data and the script data before displaying the program on the television receiver 12. Accordingly, it is possible to provide content that gives video information with individual added value for each viewer (mobile terminal) in accordance with the timing of program display by the television receiver 12.

  In the present embodiment, the broadcast station may generate material data and script data in response to a request from the viewer, and transmit the generated data. Thereby, it is not necessary to transmit extra data to a viewer who does not require it (mobile terminal 13), so that efficient data transmission can be performed.

  In the present embodiment, identification information is set for each television receiver 12 in advance, and information such as whether the power of the television receiver 12 is turned on or off on the broadcasting station side, Information about what program the television receiver 12 is watching is acquired together with the identification information described above, and the viewer is viewing the portable terminal 13 of the viewer set corresponding to the acquired identification information. Material data and script data corresponding to a program can also be transmitted. Accordingly, by transmitting the additional data only for the portable terminal 13 of a predetermined viewer corresponding to the television receiver 12, more efficient data transmission can be realized.

  In FIG. 4, on the viewer side (mobile terminal 13), information set by the processing such as TVML filter and command insertion / deletion corresponding to the playback content set by the playback content setting unit 35 is played back by the playback unit 36. To the TVML playback function (TVML player). Note that the TVML filter is an overlapping part of the contents set by the viewer in the description of the playback contents for the object and the control parameters of the virtual camera (video displayed on the mobile terminal 13) in the contents of the TVML script. In some cases, these descriptions are filtered (deleted).

  It should be noted that the deleted description content is temporarily stored in the storage means 33 or the like, and when the playback content set by the viewer is canceled, playback can be performed using the stored playback content. In addition, the viewer can insert a command into the deleted portion.

  Furthermore, in the present embodiment, as shown in FIG. 4, the reproduction control software as the reproduction content setting means 35 can analyze the image obtained from the television receiver and set the reproduction content. Also, the playback control software receives EIT (Event Information Table) of a TV program being broadcast, time information transmitted by an event message, etc. from the broadcasting station side, for example, via the TV receiver 12 or the Internet, and the received information The playback timing of the TVML script can be controlled with respect to the TVML playback function as the playback means 36.

  In addition, the mobile terminal 13 acquires the video from the television receiver 12 by the camera photographing function as the imaging unit 34, performs the above-described playback in the present embodiment by the playback unit 36, and outputs video, audio, and the like.

  The TVML playback function as the playback means 36 uses, for example, a live-action video by a camera shooting function as one of the materials, and displays it as a background video of an object (for example, a CG character) based on the playback content in a virtual studio, for example.

  Thus, when a TVML script representing sign language information corresponding to a program is distributed as additional information over the Internet for a program (for example, a live-action video) displayed on the television receiver 12, for example, on the mobile terminal 13. An object (for example, a CG character) of a sign language video can be generated, and the video of the television receiver 12 can be synthesized as a background and displayed on the screen of the mobile terminal 13. That is, the real space between the television receiver 12 and the portable terminal 13 can be used as a virtual studio. In addition, since the presence or absence of the display of an object can be individually set by each portable terminal 13, individual content display can be realized.

  Furthermore, in the present embodiment, by using an acceleration sensor, a gyro sensor, or the like as the movement detecting unit 37, for example, even when the screen frame of the television receiver 12 described above cannot be recognized, an image is displayed on the television receiver 12. The content of the object displayed on the screen of the mobile terminal 13 even when it is not displayed (for example, when the power is off) or when the entire frame of the television receiver 12 is not within the angle of view. Can be controlled.

<Example of screen display when detecting movement of portable terminal 13>
Next, a screen display example at the time of detecting the movement of the mobile terminal 13 in the present embodiment will be described with reference to the drawings. FIG. 5 is a diagram illustrating a screen display example at the time of detecting the movement of the mobile terminal in the present embodiment. FIG. 6 is a diagram for explaining the correspondence between the viewing environment and the virtual studio.

  Here, FIGS. 5A and 5B show a case where the mobile terminal 13 moves the shooting position of the television receiver 12, and FIG. 6A shows a sketch of the actual viewing environment. FIG. 6C shows a sketch in the virtual studio.

  For example, in the state of FIG. 5A, it is assumed that the television receiver 12 is included in the shooting range (view angle) shot by the camera lens provided in the imaging unit 34 included in the mobile terminal 13. At this time, in the present embodiment, a shooting space between the television receiver 12 and the portable terminal 13 is set as a virtual studio, and for example, a CG character as an object is set at a predetermined standing position (see FIG. 5 and FIG. 6). Area). The predetermined standing position (area) is stored in advance as reproduction content setting information.

  Here, when the portable terminal 13 is moved from the state of FIG. 5A to the state of FIG. 5B, the television receiver 12 is photographed by the camera photographing function (imaging means 34) of the portable terminal 13, As described above, it is possible to synthesize a CG character in front of the television receiver 12.

  In the present embodiment, as shown in FIG. 6A, the viewer moves or rotates the mobile terminal 13 being photographed to move the virtual camera of the TVML player as shown in FIG. 6B. The (CG camera) can be moved or rotated in real time. In this embodiment, as shown in FIG. 6B, a camera image display plate is set at the position of the television receiver 12, and the virtual camera is controlled according to the movement of the mobile terminal 13 in the viewing environment. it can.

  Here, a specific example of the control parameters of the virtual camera will be described with reference to the drawings. FIG. 7 is a diagram for explaining specific control parameters of the virtual camera. FIGS. 7A to 7C are diagrams illustrating the pan, the tilt, and the roll, respectively. Moreover, X, Y, and Z shown in FIG. 7 have shown the coordinate on the space which the portable terminal 13 image | photographs as a virtual camera.

  In this embodiment, when setting the reproduction content of an object with the movement of the portable terminal 13, for example, the control parameter of the virtual camera imaged by the portable terminal 13 is changed. Specifically, as shown in FIGS. 7A to 7C, movement amounts (Δx, Δy, Δz) obtained by moving the mobile terminal 13 being photographed within a predetermined time and By acquiring the rotation amount (Δpan, Δtilt, Δroll) or the like as the change amount of the control parameter, the reproduction content can be changed. Thereby, in this embodiment, for example, as shown in FIGS. 5A and 5B, the angle (orientation) at which the CG character is displayed can be freely changed.

  Further, in the present embodiment, for example, the CG character is controlled so as to face the mobile terminal 13 in the direction of the mobile terminal 13 according to the movement of the mobile terminal 13, or “Do not move” or “Do not shake up and down”. The message according to the movement of the mobile terminal 13 such as “Please do” can be controlled so as to be output as a voice by a utterance from the CG character.

  FIG. 8 is a diagram illustrating an example of CG character control corresponding to the movement of the mobile terminal. The example of FIG. 8 shows an example in which the direction of the CG character is directly facing the camera direction. For example, when the mobile terminal 13 that is shooting within a predetermined time moves as shown in FIG. 8, the rotation angle for confronting is calculated using the position coordinates of the virtual camera and the coordinate position of the CG character itself. . Thereby, in this embodiment, it can rotate so that a CG character may always face the front with respect to a virtual camera (mobile terminal 13). Furthermore, in this embodiment, it is also possible to perform control so that the CG character follows the movement of the virtual camera using the method described above.

  In addition, the process mentioned above can be performed for every portable terminal. For this reason, in this embodiment, even when a plurality of mobile terminals 13 are photographed from different positions, the CG characters displayed on the respective mobile terminals 13 are controlled so as to always face their own camera directions. Therefore, it is possible to provide content to which video information with appropriate added value is individually provided.

  Furthermore, in this embodiment, as an example of the object displayed on the image photographed with the virtual camera (mobile terminal 13), for example, a product related to the program, its price, a store, etc. are displayed, or the program is understood. It is possible to display a commentary prop for prompting and a CG character for presenting information in sign language.

  In this embodiment, an object such as a castle or a maze related to a program is displayed as an object, and the castle or maze is moved according to the movement or position of the virtual camera, a trap is set, or another CG is displayed. It is also possible to realize a so-called game function such as displaying a character. In the present embodiment, a switcher, an audio fader, or the like is displayed on the display screen of the mobile terminal 13, and the content itself displayed on the mobile terminal 13 is controlled by the viewer inputting instructions to the displayed content. For example, a program can be produced.

<Execution program>
Here, the above-described content playback device such as the mobile terminal 13 includes a CPU (Central Processing Unit), a volatile storage medium such as a RAM (Random Access Memory), a nonvolatile storage medium such as a ROM (Read Only Memory), It can be configured by a computer (hardware) including an input device such as a touch panel, operation buttons, and a pointing device, a display unit for displaying images and data, and an interface for communicating with the outside.

  Therefore, the functions of each means included in the content reproduction apparatus can be realized by causing the CPU to execute a program describing these functions. These programs can also be stored and distributed in a recording medium such as a magnetic disk (floppy disk, hard disk, etc.), optical disk (CD-ROM, DVD, etc.), semiconductor memory, or the like.

  That is, an execution program (content reproduction program) for causing a computer to execute the processing in each configuration described above is generated, and the program is installed in, for example, a general-purpose personal computer or server, thereby realizing content reproduction processing. be able to.

  Next, the processing procedure by the execution program according to the present invention will be described with reference to the flowchart of content reproduction processing in the content reproduction apparatus.

<Content playback processing procedure>
FIG. 7 is a flowchart illustrating an example of a content reproduction processing procedure. In the flowchart shown in FIG. 7, first, the mobile terminal 13 captures a live-action video from the camera (imaging means 34) (S01), and determines whether or not the television receiver 12 is shown (S02). The live-action video captured in the process of S01 means, for example, a program displayed on the screen of the television receiver 12, and may be all or part of the screen. Further, the determination in the process of S02 determines whether or not the entire screen frame of the television receiver 12 is included in the image.

  Here, when the television receiver 12 is reflected (YES in S02), image analysis processing of the television receiver 12 is performed (S03). Specifically, the contents of the image in the screen frame of the television receiver 12 are analyzed. In the process of S03, the position information of the screen frame (television receiver 12) included in the image may be acquired using image frames at predetermined intervals.

  If the television receiver 12 is not shown (NO in S02), data such as position information (for example, coordinate information) when the mobile terminal 13 is moved using movement detection means such as an acceleration sensor or a gyro sensor. Obtain (S04). In the process of S04, the position information of the mobile terminal 13 may be acquired every predetermined time.

  Next, the movement amount / rotation amount of the mobile terminal 13 is calculated (S05). Specifically, when the position information of the screen frame of the television receiver 12 can be grasped by the process of S03, the moving direction of the screen frame between image frames such as every predetermined interval included in the video before and after the movement, The inclination is acquired, and the movement amount and rotation amount of the mobile terminal 13 are calculated.

  Further, when the position of the mobile terminal 13 can be grasped by the process of S04, the movement amount and the rotation amount of the mobile terminal 13 are calculated from the difference in the position information every predetermined time. Further, the control parameter (camera parameter) of the virtual camera is acquired from the calculation result of the movement amount / rotation amount of the mobile terminal 13 obtained by the process of S05 (S06).

  Next, it is determined whether or not the script data is updated (S07). If there is an update (YES in S07), whether or not there is a description of the camera control system (for example, the control parameters described above). Judgment is made (S08). If there is a description of the camera control system (YES in S08), the camera control script data is filtered (S09).

  That is, in this embodiment, when the script data includes a description of the camera control system, the camera control is performed separately from the movement of the viewer's mobile terminal 13, so that the virtual control is performed as intended by the viewer. In order to prevent the camera from becoming inoperable, the script of the camera control system is deleted by a filter. If the script data includes a description of the camera control system, the description of the control system is stored in the storage means 33 or the like. The control system accompanying the movement of the terminal 13 can also be switched.

  If the script data is not updated in the process of S07 described above (NO in S07), it is determined whether or not the script data has been completed (S10). Here, if the script data has not been completed (NO in S10), if there is no description of the camera control system in the process of S08 (NO in S08), or after the process of S09 is completed, the camera described above The CG drawing (object) and the live-action video are combined in correspondence with the reproduction content set in advance based on the control system and the like, and after the processing is performed, the combined image (video) is displayed on the screen ( S11).

  In the above-described processing of S10, when the script data is completed (YES in S10), the content reproduction processing in the present embodiment is terminated.

  As described above, according to the present invention, when a broadcast program or the like is viewed, a screen on which the program is displayed is captured by a camera attached to a mobile terminal of each viewer, and the captured image (broadcast) By allowing viewers to view a video obtained by synthesizing an additional video such as a CG object with a video including a program video), it is possible to provide video information with appropriate added value individually.

  According to the present invention, for example, when there are a plurality of viewing terminals in the vicinity of one viewer, one content can be viewed from different viewpoints. In addition, according to the present invention, it is possible to generate an image in which a space surrounded by the viewing terminal is captured by a camera and a live action is combined with a CG character. In addition, according to the present invention, for example, a CG character can be displayed in front of a television receiver, and by causing the CG character to express a sign language accompanying the television program, the viewer can sign language video synchronized with the television program. Can be viewed on a mobile terminal. In addition, sign language changes the viewing angle and distance slightly, so that the readability of the fingers changes greatly. By changing the sign language expression of the CG character to a size and angle that is easy to see, a sign language image that is easier to understand can be obtained. It can be presented.

  Although the preferred embodiment of the present invention has been described in detail above, the present invention is not limited to the specific embodiment, and various modifications, within the scope of the gist of the present invention described in the claims, It can be changed.

DESCRIPTION OF SYMBOLS 10 Content reproduction system 11 Content provision server 12 Television receiver 13 Portable terminal (content reproduction apparatus)
14 communication network 15 viewer 16 object 21, 31 input means 22, 32 output means 23, 33 storage means 24 script generation means 25 program script generation means 26 program production means 27 distribution control means 28, 36 reproduction means 29, 38 transmission / reception means 30, 39 Control means 34 Imaging means 35 Reproduction content setting means 37 Movement detection means

Claims (7)

In a content playback apparatus that plays back content using an image of a program displayed on a screen,
Imaging means for obtaining an image including a screen on which the program is displayed;
Reproduction content setting means for setting the reproduction content for the image obtained by the imaging means;
Replaying means for replaying an image in which a predetermined object is displayed at a preset position of an image obtained by the imaging means based on setting information obtained by the replay content setting means. Content playback apparatus. The reproduction content setting means includes
2. The content reproduction apparatus according to claim 1, wherein the reproduction content of the object is set in correspondence with a position of a screen on which the program is displayed in an image obtained by the imaging unit. The reproduction content setting means includes
3. The screen content on which the program is displayed in the image obtained by the imaging means is subjected to image analysis, and the reproduction content of the object is set based on the result obtained by the image analysis. The content reproduction apparatus described in 1. Movement detection means for detecting the position of the content reproduction device;
4. The content reproduction apparatus according to claim 1, wherein the reproduction content setting unit sets the reproduction content of the object in correspondence with the position information obtained by the movement detection unit. . The reproduction content setting means includes
5. The content reproduction apparatus according to claim 1, wherein the content of reproduction of the object is set based on a movement amount or a rotation amount of the content reproduction apparatus during a predetermined time. The reproduction content setting means includes
6. The setting according to claim 1, wherein when the object is a CG character, the CG character is set to perform an action including sign language corresponding to the content of the program. Content playback device.   A content reproduction program for causing a computer to function as each unit included in the content reproduction apparatus according to any one of claims 1 to 6.