JP2012142860A - Arbitrary viewpoint video processing device, control method therefor, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To facilitate setting of a desired viewpoint when generating arbitrary viewpoint video at the desired viewpoint from video at plural viewpoints.SOLUTION: An arbitrary viewpoint video processing device includes a first change method for changing at least either a viewpoint position or a visual line direction at the viewpoint position when determining a viewpoint of arbitrary viewpoint video to be generated, and a second change method for selecting predetermined viewpoints according to a switching order. If the viewpoint of generated arbitrary viewpoint video is a viewpoint changed by the first change method, when an instruction to switch it to the previous or next viewpoint in the switching order by the second change method is input, processing is performed as follows. When an instruction to the previous viewpoint is input, the viewpoint is switched to a viewpoint selected by the second change method immediately before being changed by the first change method. Then, when an instruction to the next viewpoint is input, the viewpoint is switched to the next viewpoint of the viewpoint selected by the second change method immediately before being changed by the first change method in the viewpoint switching order.

Description

  The present invention relates to a technique for generating an arbitrary viewpoint video at an arbitrary viewpoint from videos at a plurality of viewpoints.

  With the start of digital broadcasting in recent years, various next-generation broadcasting systems have been researched and developed. In the next-generation broadcasting system, there is a technique for generating a video at an arbitrary viewpoint called an arbitrary viewpoint video or a free viewpoint video from videos at a plurality of viewpoints. With the technology for generating an arbitrary viewpoint video, for example, in a video content such as a soccer broadcast, it is possible to generate a video that is actually in a spectator seat or a field.

  In Patent Document 1, when providing video content that can generate an arbitrary viewpoint video as described above, information on a viewpoint that cannot generate an arbitrary viewpoint video is determined in advance so that the viewer can view the video content. A method for controlling an arbitrary viewpoint video is disclosed.

JP 2007-195091 A

  However, in the case of video content that can generate an arbitrary viewpoint video at the viewpoint set by the viewer as described above, the viewer needs to set the viewpoint of the arbitrary viewpoint video to be generated, for example, by operating a direction key of the remote controller. There is. Since the viewpoint information is defined by at least the viewpoint position and the gaze direction at the viewpoint position, in order to set such a desired viewpoint, the viewer needs to be forced to perform complicated operations to set each parameter. There is. Furthermore, until the desired viewpoint is set, the viewer may be confused because the viewpoint of the arbitrary viewpoint video being viewed cannot be grasped. In such a situation, the viewpoint should be set. I had to start over again.

  The present invention has been made in view of the above-described problems, and an object thereof is to facilitate setting of a desired viewpoint when generating an arbitrary viewpoint video at a desired viewpoint from a plurality of viewpoint videos. .

In order to achieve the above object, an arbitrary viewpoint video processing apparatus of the present invention comprises the following arrangement.
An arbitrary viewpoint video processing apparatus for generating an arbitrary viewpoint video at an arbitrary viewpoint from an image at a plurality of viewpoints, which is an image at a viewpoint defined by at least a viewpoint position and a gaze direction at the viewpoint position. Receiving means for receiving at least two or more predetermined viewpoint information, and setting means for setting a viewpoint switching order for at least two or more predetermined viewpoints received by the receiving means; The first change means for changing at least one of the viewpoint position and the line-of-sight direction of the viewpoint of the generated arbitrary viewpoint video and the instruction input to the next viewpoint or the instruction to the previous viewpoint are received and set Selecting one of at least two predetermined viewpoints in accordance with the viewpoint switching order set by the means; Accordingly, the second changing means for changing the viewpoint of the generated arbitrary viewpoint video and the video at the plurality of viewpoints received by the receiving means are changed by the first changing means or the second changing means. Generating means for generating an arbitrary viewpoint video at the selected viewpoint, wherein the second changing means is a previous one when the viewpoint of the generated arbitrary viewpoint video is a viewpoint changed by the first changing means. When the instruction input to the viewpoint is received, the first viewpoint changed by the second changing means is selected immediately before being changed by the first changing means, and when the instruction input to the next viewpoint is received, A second viewpoint set next to the first viewpoint in the switching order is selected.

  With this configuration, according to the present invention, it is possible to easily set a desired viewpoint when generating an arbitrary viewpoint video of a desired viewpoint from videos of a plurality of viewpoints.

The block diagram which showed the function structure of the television receiver and content delivery server which concern on embodiment of this invention. The figure for demonstrating the example of a structure of the recommendation viewpoint which concerns on embodiment Flow chart of viewpoint change processing according to the embodiment Flow chart of viewpoint selection processing when viewpoint information according to the embodiment is updated Flowchart of switching order determination processing according to modification The figure for demonstrating the switching order of the recommendation viewpoint which concerns on a modification

(Embodiment)
Hereinafter, a preferred embodiment of the present invention will be described in detail with reference to the drawings. In addition, one embodiment described below is an example in which the present invention is applied to a television receiver capable of generating an arbitrary viewpoint video at a desired viewpoint from videos at a plurality of viewpoints as an example of an arbitrary viewpoint video processing apparatus. explain. However, the present invention can be applied to any device capable of generating an arbitrary viewpoint video at a desired viewpoint from videos at a plurality of viewpoints. Further, in this specification, “viewpoint” is described as information defined by both a three-dimensional viewpoint position and a line-of-sight direction at the viewpoint position. However, the information is not limited to the above-described configuration as long as it is information related to a viewpoint necessary to generate a video at a set viewpoint. Note that the viewpoint may be composed of, for example, viewpoint position, gazing point position, and upward information at the viewpoint, and may further include information such as a field angle and an aspect ratio.

  FIG. 1 is a block diagram showing functional configurations of a television receiver 100 and a content distribution server 200 according to the embodiment of the present invention. In the present embodiment, the television receiver 100 is described as receiving video content generated by the content distribution server 200 as a broadcast signal, but the embodiment of the present invention is not limited thereto.

  The control unit 101 is a controller including peripheral circuits such as a microprocessor, a timer, an interrupt controller, and an input / output port, for example, and controls the operation of each block constituting the television receiver 100. The control unit 101 controls the operation of each block by reading out the operation program of each block constituting the television receiver 100 from the ROM 102 and executing the program on the RAM 103. The ROM 102 is a rewritable nonvolatile memory, and stores information such as setting values necessary for the operation of each block in addition to the operation program for each block provided in the television receiver 100. The RAM 103 is a rewritable volatile memory, and is used as an area for temporarily storing information and data output in the operation of each block included in the television receiver 100, for example.

  In the present embodiment, the operation of each block included in the television receiver 100 is described as being controlled by the operation program of the block. However, the present invention is not limited to this, and each block is similar to the operation program. You may comprise with the circuit which processes.

  The content receiving unit 104 generates a transport stream (TS) signal from the received broadcast signal. When receiving a broadcast signal using a broadcast wave, the content receiving unit 104 selects and demodulates the broadcast wave according to information on a broadcast station that is currently viewed and stored in the ROM 102, and outputs a TS signal. In addition, when the content receiving unit 104 receives a broadcast signal by IP broadcast, the content receiving unit 104 outputs a TS signal by decoding a packet of the received broadcast signal.

  Here, a method of generating a broadcast signal received in the present embodiment will be specifically described. The content distribution server 200 is a server that transmits a broadcast signal using a broadcast wave or an Internet communication network. In the present embodiment, the television receiver 100 receives video content generated by the content distribution server 200.

  The cameras 201-1 to 201-n are devices that shoot videos at a plurality of viewpoints included in the video content transmitted by the content distribution server 200. Video captured by each of the cameras 201-1 to 201-n is output to the content generation unit 202. In addition, information on the viewpoint position and line-of-sight direction (shooting viewpoint information) of each of the cameras 201-1 to 201-n is input to the viewpoint information generation unit 203.

  The content generation unit 202 outputs the video captured from a plurality of input viewpoints to the encoding unit 204 as one video content. The encoding unit 204 encodes the input video content into a transmission format and outputs the encoded video content to the multiplexing unit 205.

  The viewpoint information generation unit 203 determines at least two viewpoints as recommended viewpoints from viewpoints of an arbitrary viewpoint video that can be generated from videos at a plurality of viewpoints constituting the video content described above.

  For example, the recommended viewpoint may be information that can be set by the content creator, or a viewpoint that automatically captures a position where the motion vector in the time series of the subject in the time series is large in the angle of view is automatically selected. Information may be provided. For example, as shown in FIG. 2, the recommended viewpoint can be configured such that information on various viewpoint positions and line-of-sight directions can be identified by assigning, for example, names and identifiers (IDs). In this embodiment, it is assumed that the content distribution server 200 provides recommended viewpoint information including at least two recommended viewpoints together with video content.

  The viewpoint information generation unit 203 outputs the recommended viewpoint information generated in this way and the shooting viewpoint information of each of the cameras 201-1 to 201-n that has shot the video to the multiplexing unit 205.

  The multiplexing unit 205 multiplexes the encoded data of the video content including the videos at the plurality of viewpoints, the recommended viewpoint information, and the shooting viewpoint information, generates a TS signal for transfer, and sends it to the content transmission unit 206. Output. The content transmission unit 206 transmits the input TS signal as a broadcast wave, for example.

  The content reception unit 104 receives the TS signal generated in this way and outputs it to the separation unit 105. The separation unit 105 separates the input TS signal into encoded data of video content including videos at a plurality of viewpoints, recommended viewpoint information, and shooting viewpoint information. The decoding unit 106 decodes the encoded data of the video content separated by the separation unit 105, and outputs the obtained videos at a plurality of viewpoints to the video generation unit 107.

  Further, the recommended viewpoint information and the shooting viewpoint information separated by the separation unit 105 are respectively stored in the RAM 103 by the control unit 101, for example. At this time, the control unit 101 manages each recommended viewpoint included in the recommended viewpoint information as a viewpoint list as shown in Table 1.

  The viewpoint list is composed of “ID”, “name”, “viewpoint presence status”, and “viewpoint status” including specific parameters of the viewpoint, that is, viewpoint position, line-of-sight direction, and angle of view. In addition, when an arbitrary viewpoint video is generated for a certain viewpoint in the past, the specific parameters of the viewpoint when the arbitrary viewpoint video is generated are added to the list information as the item of the viewpoint as “past viewpoint state” Is done.

  In the viewpoint list, the plurality of recommended viewpoints included in the recommended viewpoint information are updated according to the ID and name information. In Table 1, the recommended viewpoint 2 has a difference between the current viewpoint state and the past viewpoint state, and the viewpoint has changed (updated) in the viewpoint state distributed in time series. The recommended viewpoint may increase or decrease in time series during the distribution of video content related to one program, for example. In the present embodiment, when a viewpoint provided as a recommended viewpoint is deleted, the item of the presence status of the viewpoint changes to “none”. When the number of recommended viewpoints increases, the viewpoint item is newly added to the viewpoint list.

  The video generation unit 107 is a block that generates an arbitrary viewpoint video at a designated viewpoint from videos input at a plurality of viewpoints. Specifically, the video generation unit 107 generates an arbitrary viewpoint video having the viewpoint position, line-of-sight direction, and angle of view included in the viewpoint state according to the viewpoint state instructed by the control unit 101.

  The drawing unit 108 receives an instruction from the control unit 101, generates GUI data to be displayed on the display unit 110 to be described later, and outputs the GUI data to the combining unit 109. When displaying the GUI data, the control unit 101 inputs the arbitrary viewpoint video generated by the video generation unit 107 to the synthesis unit 109, causes the synthesis unit 109 to combine the GUI data and the arbitrary viewpoint video, and outputs them to the display unit 110. Let

  The display unit 110 is a display device such as an LCD, for example, and displays a video signal of a video generated by the video generation unit 107 or a video synthesized with the GUI data generated by the synthesis unit 109.

  The operation input unit 111 is a user interface such as an infrared light receiving unit that receives an operation from a remote controller or a power button or a channel selection button provided in the television receiver 100, for example, and analyzes the operation content input by the viewer. The data is transmitted to the control unit 101.

(Viewpoint change processing)
A specific process of the viewpoint changing process of the television receiver 100 of the present embodiment having such a configuration will be described with reference to the flowchart of FIG. The processing corresponding to the flowchart can be realized by the control unit 101 reading, for example, a corresponding processing program stored in the ROM 102, developing the program in the RAM 103, and executing the program. The viewpoint changing process will be described as being started when the television receiver 100 is turned on by the viewer, receives the video content, and becomes ready to view the received video content. . In addition, since the recommended viewpoint information may be updated in time series, the viewpoint changing process may be repeatedly executed while the television receiver 100 is receiving video content, for example.

  In step S301, the control unit 101 registers the recommended viewpoints included in the recommended viewpoint information separated by the separation unit 105 in the viewpoint list in the order of ID. Specifically, the control unit 101 acquires recommended viewpoint information of the video content received by the content receiving unit 104, and configures the recommended viewpoint information according to the order of viewpoint IDs set in advance in the information. Register one viewpoint in the viewpoint list. At this time, if the viewpoint list is already stored in the RAM 103, the control unit 101 reads out the information and updates it with the newly received recommended viewpoint information. If the viewpoint list is not stored in the RAM 103, the control unit 101 starts a new viewpoint. A list is generated and stored in the RAM 103. In the present embodiment, the order of recommended viewpoints registered in the viewpoint list is the switching order when switching recommended viewpoints described later. In other words, in the present embodiment, the recommended viewpoints are switched in the order of predetermined IDs in the video content distributed from the content distribution server 200.

  In step S302, the control unit 101 causes the drawing unit 108 to generate GUI data that notifies the viewer that the viewpoint can be changed, and causes the synthesis unit 109 to superimpose the GUI data on the video in the viewpoint state being viewed. It is displayed on the display unit 110. It is assumed that the viewpoint state of the video being viewed is stored in the RAM 103, for example. Note that, after the television receiver 100 is activated, the viewpoint state of the video that first displays the received video content may be determined in advance as a viewpoint state of one of the recommended viewpoints as an initial viewpoint state, for example. The control unit 101 sets the initial viewpoint state as a viewing viewpoint state.

  In the present embodiment, the viewpoint state of the video to be displayed first is described as being set to the viewpoint state of one of the recommended viewpoints, but the present invention is not limited to this. That is, for the first video to be displayed, for example, the viewpoint state of the panoramic video of the video content is set, and the panoramic video may be displayed together with the GUI data notifying that the recommended viewpoint can be selected.

  In step S303, the control unit 101 determines whether an instruction input related to a viewpoint change has been made by the viewer. Specifically, the control unit 101 determines whether or not the viewpoint has been changed based on whether or not the operation input unit 111 has received that the instruction input relating to the viewpoint change has been made by the viewer.

In the present embodiment, the following three instruction inputs are described as being determined as viewpoint change instruction inputs.
・ Change instruction to the next recommended viewpoint (second change)
・ Change instruction to the recommended viewpoint displayed before (second change)
・ Instruction input to change the viewpoint state directly (first change)
Note that the instruction input for directly changing the viewpoint state is, for example, a method of increasing or decreasing the parameters of the viewpoint coordinates and the line-of-sight direction and the angle of view by a predetermined numerical value or directly changing to an arbitrary value. Assume that input is made. That is, in this step, the control unit 101 determines whether or not the above three instruction inputs have been made. The control unit 101 completes the process in step S304 when an instruction to change the viewpoint is input, or completes the process when no instruction is input or another instruction is input.

  In step S304, the control unit 101 determines which of the above-described three instruction inputs is the viewpoint change instruction input made by the viewer in step S303. The control unit 101 moves the process to S310 when the instruction change instruction input is an instruction to change to the next recommended viewpoint, and performs the process when it is an instruction to change to the recommended viewpoint displayed previously. Move to S316. If the instruction input for changing the viewpoint is an instruction input for directly changing the viewpoint state, the control unit 101 moves the process to S305.

  In step S <b> 305, the control unit 101 determines whether or not the viewpoint state being viewed stored in the RAM 103 is a viewpoint state set by the viewer directly changing the viewpoint state. If the viewpoint state being viewed is a viewpoint state that has been directly changed by the viewer, the control unit 101 moves the process to S307. If the viewpoint state is not directly changed, that is, one of the recommended viewpoints. If so, the process proceeds to S306.

  In step S306, the control unit 101 causes the RAM 103 to store the currently selected recommended viewpoint as the recommended viewpoint that was selected immediately before. As for the recommended viewpoint selected immediately before, the viewpoint ID is stored as information for identifying the viewpoint. At this time, the control unit 101 adds the viewpoint coordinates, line-of-sight direction, and angle-of-view information (viewpoint state) of the recommended viewpoint currently selected in the viewpoint list to the past viewpoint state items of the viewpoint. Remember. In other words, by storing the detailed information of the currently selected recommended viewpoint in the viewpoint list, the generated arbitrary viewpoint video can be generated again even when the viewpoint state of the viewpoint is changed. .

  In step S <b> 307, the control unit 101 updates the information in the RAM 103 with the viewpoint state changed by the instruction input as the viewpoint state being viewed. In step S308, the control unit 101 determines whether one of the recommended viewpoints is included in the angle of view of the video in the changed viewpoint state. The control unit 101 moves the process to S309 if any of the viewpoints of the recommended viewpoint information is included in the angle of view of the video in the viewpoint state being viewed, and completes the process if not included.

  In step S <b> 309, the control unit 101 causes the drawing unit 108 to generate GUI data for notifying that it is possible to change to the recommended viewpoint included in the angle of view in response to a change instruction to the next recommended viewpoint. Then, the combining unit 109 superimposes the GUI data on the video in the viewpoint state being viewed and displays it on the display unit 110 to complete the processing.

  In step S <b> 310, the control unit 101 determines whether or not the viewpoint state being viewed stored in the RAM 103 is a viewpoint state set by the viewer directly changing the viewpoint state. If the viewpoint state being viewed is a viewpoint state that has been directly changed by the viewer, the control unit 101 moves the process to S313. If the viewpoint state is not directly changed, that is, if any viewpoint of the recommended viewpoint information is selected. If it is in the viewpoint state, the process proceeds to S311.

  In step S <b> 311, the control unit 101 stores the currently selected recommended viewpoint ID in the RAM 103 as the recommended viewpoint selected immediately before. At this time, the control unit 101 adds the viewpoint coordinates, line-of-sight direction, and angle-of-view information (viewpoint state) of the recommended viewpoint currently selected in the viewpoint list to the past viewpoint state items of the viewpoint. Remember. In step S312, the control unit 101 updates the information in the RAM 103 with the viewpoint state of the viewpoint next to the currently selected viewpoint in the viewpoint list of the recommended viewpoint registered in step S301, and completes the process. To do.

  On the other hand, if the viewing state being viewed is a viewing state that has been directly changed by the viewer, in step S313, the control unit 101 includes any of the recommended viewpoints within the angle of view of the video in the viewing state being viewed. Judge whether or not. Specifically, the control unit 101 calculates, for each recommended viewpoint included in the viewpoint list, the positional relationship with the currently viewed viewpoint state and the direction from the current viewpoint, so that the recommended viewpoint is within the angle of view. Judge whether it is included. The control unit 101 moves the process to S314 if any of the recommended viewpoints is included in the angle of view of the video in the viewpoint state being viewed, and moves the process to S315 if not included.

  In step S <b> 314, the control unit 101 updates the information in the RAM 103 with the viewpoint state of the recommended viewpoint included in the angle of view of the video in the viewpoint state being viewed as the viewpoint state being viewed, and completes the process. In this way, the viewer can easily move to the viewpoint included in the angle of view of the video currently being viewed, and thus the viewer's confusion due to the viewpoint movement can be reduced.

  In S315, the control unit 101 sets the viewpoint state of the viewpoint (second viewpoint) next to the recommended viewpoint selected immediately before in the viewpoint list of the recommended viewpoint registered in S301 as the currently viewed viewpoint state in the RAM 103. Update the information to complete the process. Specifically, the control unit 101 reads the recommended viewpoint ID selected immediately before from the RAM 103, and sets the viewpoint state of the viewpoint having the ID next to the ID from the viewpoint list as the currently viewed viewpoint state.

  In step S <b> 316, the control unit 101 determines whether the recommended viewpoint that was selected immediately before is stored in the RAM 103. If the recommended viewpoint selected immediately before is stored in the RAM 103, the control unit 101 moves the process to S317, and if not, moves the process to S318.

  In step S <b> 317, the control unit 101 updates the information in the RAM 103 with the viewpoint state (first viewpoint) of the recommended viewpoint selected immediately before being stored in the RAM 103 as the currently viewed viewpoint state, and completes the process. In step S318, the control unit 101 causes the drawing unit 108 to generate GUI data notifying that the previously selected recommended viewpoint does not exist, and causes the synthesis unit 109 to display the GUI data in the viewpoint state being viewed. The process is superposed and displayed on the display unit 110 to complete the process.

  In this way, the viewer can easily view the video at the viewpoint provided as the recommended viewpoint without performing complicated operations. In addition, even if the viewer arbitrarily changes the viewpoint and falls into a situation where the viewpoint of the generated arbitrary viewpoint video cannot be grasped, the recommended viewpoint selected immediately before or the point selected immediately before is selected. You can switch to the next viewpoint after the recommended viewpoint.

  It has been described that the control unit 101 switches to the recommended viewpoint selected immediately before when it is determined that an instruction to change to the previously displayed recommended viewpoint has been issued. At this time, when the recommended viewpoint having the same ID as the viewpoint is updated, that is, when the same viewpoint provided from the time when the arbitrary viewpoint video is generated is updated, the control unit 101 illustrated in FIG. What is necessary is just to process like a flowchart.

  In step S <b> 401, the control unit 101 determines whether the viewpoint having the recommended viewpoint ID selected immediately before stored in the RAM 103 is present in the recommended viewpoint information output from the separation unit 105. Specifically, the control unit 101 determines the presence or absence by referring to the item of the presence state of the viewpoint having the recommended viewpoint ID selected immediately before in the viewpoint list stored in the RAM 103. The control unit 101 moves the process to S403 when the presence state of the recommended viewpoint selected immediately before is “present”, and moves the process to S402 when “no”.

  In step S <b> 402, the control unit 101 stores the viewpoint state of the viewpoint of the video that first displays the received video content as the currently viewed viewpoint state in the RAM 103 after the television receiver 100 is activated, and completes the process.

  In step S <b> 403, the control unit 101 determines whether or not information (viewpoint state) of the viewpoint coordinates, the line-of-sight direction, and the angle of view of the recommended viewpoint that was selected immediately before has been updated. Specifically, the control unit 101 determines whether or not the information of the current viewpoint state and the past viewpoint state is different in the recommended viewpoint item selected immediately before the viewpoint list. The control unit 101 moves the process to S <b> 404 when it is determined that the information on the viewpoint coordinates, the line-of-sight direction, and the angle of view of the recommended viewpoint that was selected immediately before has been updated, and the process when it is determined that they are the same. Move to S407.

  In step S <b> 404, the control unit 101 determines which of the current viewpoint information and the information when the arbitrary viewpoint video was generated in the past is used when generating the arbitrary viewpoint video using the recommended viewpoint selected immediately before. The drawing unit 108 is caused to generate GUI data for causing the image data to be generated. Then, the control unit 101 causes the combining unit 109 to display the generated GUI data on the display unit 110 by superimposing the generated GUI data on the video in the viewing state.

  In step S <b> 405, the control unit 101 waits for processing until the viewer inputs an instruction to select either the current viewpoint state or the viewpoint state when an arbitrary viewpoint video was generated in the past. The control unit 101 moves the process to S407 when the current viewpoint state is selected, and moves the process to S406 when the viewpoint state when the arbitrary viewpoint video is generated in the past is selected.

  In step S <b> 406, the control unit 101 reads from the viewpoint list stored in the RAM 103 the item of the previous viewpoint state of the viewpoint having the recommended viewpoint ID selected immediately before, and the viewpoint state in which the viewpoint state is being viewed. Is stored in the RAM 103. In step S <b> 407, the control unit 101 reads from the viewpoint list stored in the RAM 103 the item of the current viewpoint state of the viewpoint having the recommended viewpoint ID selected immediately before, and the viewpoint that is viewing the viewpoint state. The state is stored in the RAM 103.

  In this way, when the recommended viewpoint that has generated an arbitrary viewpoint video in the past is selected again, even if the recommended viewpoint has been updated, the viewer can select the arbitrary viewpoint for the viewpoint before or after the update. It is possible to select whether to generate a video. That is, when selecting the recommended viewpoint displayed before, even if the viewpoint is updated, the viewpoint before updating can be selected, and the updated viewpoint can be selected by recognizing that the viewpoint has been updated. Therefore, it is possible to reduce the occurrence of confusion for the viewer due to the change of the viewpoint.

  As described above, the arbitrary viewpoint video processing apparatus of the present invention can facilitate setting of a desired viewpoint when generating an arbitrary viewpoint video at a desired viewpoint from videos at a plurality of viewpoints. Specifically, the arbitrary viewpoint video processing device is predetermined with a first change method for changing at least one of the viewpoint position and the line-of-sight direction at the viewpoint position when determining the viewpoint of the arbitrary viewpoint video to be generated. A second changing method for selecting the selected viewpoint according to the switching order. When the generated arbitrary viewpoint video viewpoint is a viewpoint that has been changed by the first changing method, when an instruction input for switching to the previous viewpoint or the next viewpoint is received by the second changing method, the following is performed: To process. When an instruction input to the previous viewpoint is received, the viewpoint is switched to the viewpoint selected by the second change method immediately before the change is made by the first change method. When an instruction input to the next viewpoint is received, the viewpoint is switched to the next viewpoint of the viewpoint selected by the second change method immediately before the change by the first change method in the order of changing the viewpoint.

  In this way, the viewer can easily view the video at the viewpoint provided as the recommended viewpoint information without performing a complicated operation. In addition, even if the viewer arbitrarily changes the viewpoint and falls into a situation where the viewpoint of the generated arbitrary viewpoint video cannot be grasped, the recommended viewpoint selected immediately before or the point selected immediately before is selected. You can switch to the next viewpoint after the recommended viewpoint.

(Modification)
In the above-described embodiment, the switching order of the recommended viewpoints has been described as being the ID order of each viewpoint that is set in advance when the recommended viewpoint information is provided, but the implementation of the present invention is not limited thereto. In the present modification, a method for dynamically changing and setting the recommended viewpoint switching order will be described with reference to the flowchart of FIG.

  In this modification, it is assumed that the recommended viewpoint information provided to the television receiver 100 together with the video content further includes attribute information having at least one of the following three pieces of information. The attribute information includes, for example, at least one of a “theme” such as a sports broadcast competition name, a “subject” such as a person name or affiliation name of a main subject, and a “photographing place” such as a photographed place name. It is assumed that the viewpoint information generation unit 203 sets the information in advance.

  In addition, the viewpoint list generated in the television receiver 100 in the present modification includes information on whether or not the arbitrary viewpoint video generated for each recommended viewpoint has been viewed on the television receiver 100 in addition to the switching order information described above. In addition.

(Switching order determination process)
In step S501, the control unit 101 classifies the recommended viewpoints included in the recommended viewpoint information separated by the separation unit 105 into viewpoint groups having the same attribute information, and sets the viewpoint switching order for each group. Here, an example of classification of viewpoints having the same attribute information will be described in detail with reference to FIG.

  FIG. 6 two-dimensionally shows the viewpoint state of each recommended viewpoint when a plurality of competitions are shot using a plurality of cameras in one stadium, as in the case of Olympic athletics. FIG. In viewing such video content, the viewer can, for example, set whether to switch the recommended viewpoint in order to the viewpoint with the closest viewpoint position or to switch the viewpoint so that the viewpoints of the same competition are continuous.

  For example, in the example of FIG. 6, when the viewer is set to switch the recommended viewpoint in the order of the viewpoint position, refer to the shooting location item of the attribute information of the recommended viewpoint, And grouped into viewpoints having the attribute of “field”. Then, the switching order of the viewpoints having the attribute “outside track” is set in the order of the viewpoint coordinates in the order of 601 → 604 → 606 → 607 → 608. Further, the switching order of viewpoints having the attribute of “field” is set in the order of the viewpoint coordinates in the order of 603 → 602 → 605.

  Also, for example, in the example of FIG. 6, when the viewer is set to switch the recommended viewpoint for each competition, the item of the theme of the attribute information of the recommended viewpoint is referred to, and “short-distance running”, “cannon throw”, and “ The viewpoints having the “long jump” attribute may be grouped. Then, the switching order of viewpoints having the attribute of “short-distance running” is set in the order of the line of sight, such as 601 → 604 → 606 → 602 → 608. The switching order is set such that the viewpoint having the “cannonball throw” attribute is 603 → 605 in the line-of-sight direction, and the viewpoint having the “long jump” attribute is 607.

  In addition to this, for example, when the video content is a team competition such as a soccer broadcast or a baseball broadcast, it may be grouped for each team, or for a travel program, it may be grouped by region.

  In S502, the control unit 101 registers the recommended viewpoints included in the separated recommended viewpoint information in the viewpoint list stored in the RAM 103 together with the switching order information determined in S501.

  In step S <b> 503, the control unit 101 determines whether it is necessary to avoid selecting viewpoints having the same attribute information in switching the recommended viewpoints of the received video content. Specifically, the control unit 101 is set so as to avoid the selection of viewpoints having the same attribute information by the viewer in advance, or the content in which the received video content avoids the selection of viewpoints having the same attribute information. It is determined whether or not the genre is set. This is because, for example, when the video content is a travel program, it is assumed that it is desirable to present various shooting locations rather than presenting arbitrary viewpoint videos from the viewpoint of the same shooting location to the viewer. It is. The control unit 101 moves the process to S504 when it is necessary to avoid the selection of viewpoints having the same attribute information, and moves the process to S505 when it is not necessary to avoid the selection.

  In step S <b> 504, the control unit 101 deletes unnecessary viewpoints from the viewpoint list stored in the RAM 103 so that the viewpoints having the same attribute information have one viewpoint for each attribute information.

  In step S <b> 505, the control unit 101 refers to the viewpoint list stored in the RAM 103, and when there is a viewpoint that has already been used to generate an arbitrary viewpoint video, the control unit 101 sets the viewpoint switching order later. Sort viewpoints. Although the present embodiment has been described as sorting based on whether or not viewpoints were used for generation, the implementation of the present invention is such that recommended viewpoints are ordered in the order of recommended viewpoints that are selected less frequently. The switching order may be reset. The recommended viewpoints on the viewpoint list thus sorted are updated and stored in the switching order item again in S506, and the control unit 101 completes the processing.

  In this way, it is possible to dynamically set the switching order of the recommended viewpoints according to the attribute information of the recommended viewpoints, the setting by the viewer, and the content genre. That is, when the recommended viewpoint is switched and the arbitrary viewpoint video is presented, it is possible to set the related viewpoints to be continuous, so that it is possible to present the viewpoint switching that does not confuse the viewer.

  In the above-described embodiment, when the viewpoint state being viewed is a directly changed viewpoint state, and any of the recommended viewpoints is included in the angle of view of the video in the viewpoint state being viewed, In the above description, it is assumed that an instruction input to the viewpoint is switched to the viewpoint within the angle of view. At this time, since the viewpoint is switched to a viewpoint that is different from the set viewpoint switching order, the viewpoint list is not configured so that the viewpoint information already used for generating the arbitrary viewpoint video is later. Therefore, the control unit 101 may reset the viewpoint switching order in the viewpoint list by executing the switching order determination process described above.

  In the above-described embodiment and modification, it has been described that information on at least two or more predetermined viewpoints that are recommended viewpoints is received together with videos at a plurality of viewpoints. However, the present invention is not limited to this. Absent. That is, the recommended viewpoint information may be generated by the arbitrary viewpoint video processing apparatus based on the received videos at the plurality of viewpoints. For example, recommended viewpoints may be registered so that they are equally spaced in a space where arbitrary viewpoint video can be generated, or the image may be analyzed and the position of a specific pattern such as a corner of a road registered as a recommended viewpoint. Good.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (8)

An arbitrary viewpoint video processing device for generating an arbitrary viewpoint video at an arbitrary viewpoint from an image at a plurality of viewpoints, which is an image at a viewpoint defined by at least a viewpoint position and a gaze direction at the viewpoint position,
Receiving means for receiving the video at the plurality of viewpoints and information on at least two or more predetermined viewpoints;
Setting means for setting a viewpoint switching order for the at least two or more predetermined viewpoints received by the receiving means;
First changing means for changing at least one of a viewpoint position and a line-of-sight direction of the viewpoint of the generated arbitrary viewpoint video;
Upon receiving an instruction input to the next viewpoint or an instruction to the previous viewpoint, one viewpoint is selected from at least two predetermined viewpoints according to the viewpoint switching order set by the setting unit. Second changing means for changing the viewpoint of the generated arbitrary viewpoint video,
Generating means for generating an arbitrary viewpoint video at the viewpoint changed by the first changing means or the second changing means, using videos at a plurality of viewpoints received by the receiving means;
When the viewpoint of the generated arbitrary viewpoint video is the viewpoint changed by the first changing means, the second changing means is
Upon receiving an instruction input to the previous viewpoint, the first viewpoint changed by the second changing means immediately before being changed by the first changing means is selected,
When receiving an instruction input to the next viewpoint, the second viewpoint set next to the first viewpoint in the viewpoint switching order is selected.
An arbitrary viewpoint video processing device characterized by the above.   The said setting means sets the switching order of the said viewpoint so that the order of a viewpoint with few frequency | counts of the said selected at least 2 or more predetermined viewpoints may precede. Arbitrary viewpoint video processing device. The receiving means further receives, for each of the at least two or more predetermined viewpoints, attribute information including information on at least one of a theme, a subject, and a shooting location of an arbitrary viewpoint video at the viewpoint;
The said setting means sets the switching order of the said viewpoint so that the viewpoint with the same attribute information may continue the said at least 2 or more predetermined viewpoint. Arbitrary viewpoint video processing device.   The second changing means is the viewpoint in which the viewpoint of the generated arbitrary viewpoint video is changed by the first changing means, and is determined in advance within an angle of view of the arbitrary viewpoint video at the viewpoint. When any one of at least two viewpoints is included, the viewpoint included in the angle of view is selected when an instruction input to the next viewpoint is received. The arbitrary viewpoint video processing apparatus according to any one of claims 1 to 3.   5. The arbitrary setting according to claim 4, wherein when the viewpoint included in the angle of view is selected by the second changing unit, the setting unit sets the switching order of the viewpoint again. 6. Viewpoint video processing device. The second changing means includes
When any one of the viewpoint position and the line-of-sight direction of the first viewpoint is updated, when receiving an instruction input to the previous viewpoint position, the first viewpoint before the update and the first after the update An instruction means for receiving an instruction input for generating an arbitrary viewpoint video at any of the viewpoints of
The arbitrary viewpoint video processing apparatus according to claim 1, further comprising: A control method for an arbitrary viewpoint video processing apparatus that generates an arbitrary viewpoint video at an arbitrary viewpoint from an image at a plurality of viewpoints, which is an image at a viewpoint defined by at least a viewpoint position and a gaze direction at the viewpoint position,
A receiving step for receiving the video at the plurality of viewpoints and information on at least two or more predetermined viewpoints;
A setting step in which a setting unit sets a viewpoint switching order for at least two or more predetermined viewpoints received in the reception step;
A first changing step in which a first changing unit changes at least one of a viewpoint position and a line-of-sight direction of the viewpoint of the generated arbitrary viewpoint video;
The second changing means receives an instruction input to the next viewpoint or an instruction input to the previous viewpoint, and the at least two or more predetermined viewpoints are determined according to the viewpoint switching order set in the setting step. A second changing step of changing the viewpoint of the generated arbitrary viewpoint video by selecting one of the viewpoints;
A generating step for generating an arbitrary viewpoint video at the viewpoint changed in the first changing step or the second changing step using the video at the plurality of viewpoints received in the receiving step; Prepared,
In the second changing step, the second changing means, when the viewpoint of the generated arbitrary viewpoint video is the viewpoint changed in the first changing step,
Upon receiving an instruction input to the previous viewpoint, the first viewpoint changed in the second change step immediately before being changed in the first change step is selected,
When receiving an instruction input to the next viewpoint, the second viewpoint set next to the first viewpoint in the viewpoint switching order is selected.
A control method for an arbitrary viewpoint video processing device.   The program for functioning a computer as each means of the arbitrary viewpoint video processing apparatus of any one of Claims 1 thru | or 6.

Images