JP5495855B2 - Video processing apparatus and video processing method - Google Patents

Description

  The present invention relates to a video processing apparatus and a video processing method for processing multi-view video information.

  In the era of full-fledged digital broadcasting, various next-generation broadcasting systems are being researched and developed. One of them is a “multi-view video system” that distributes video data taken by arranging a number of cameras and allows viewers to freely select a desired video from a video source (camera video). . In the technique disclosed in Patent Document 1, a multi-view video is multiplexed with a broadcast stream and transmitted, and an arbitrary viewpoint video is taken out and displayed by a receiving device.

  In addition, it is assumed that high-angle and high-resolution video data is provided for video from each camera, and the range is freely cut out by the video processing apparatus and displayed on the video display unit. FIG. 10A shows an example in which a multi-viewpoint camera is arranged around the subjects A to C (see reference numerals 1304 to 1306) for photographing. Of the plurality of cameras, cameras A to C are denoted by reference numerals 1301 to 1303. FIG. 10B shows a shooting range 1402 corresponding to the field of view of the camera B 1302 and a shooting range 1403 corresponding to the field of view of the camera A 1301. Subjects A to C exist in the shooting range 1402, and subjects A and B exist in the shooting range 1403.

  The video processing apparatus cuts out the screen including the subject in the screen cutout range 1401 and displays the video in the range. For example, when the screen is cut out so as to include the subject B1305 in the video imaged by the camera B1302, the subject image is displayed in the center of the screen. On the other hand, when the image is taken by the camera A1301, the subjects A1304 and B1305 are within the photographing range 1403, so that each subject image can be displayed at the center of the screen by cutting out the screen. However, the subject C 1306 is not within the shooting range 1403. For this reason, when the image of the subject C1306 is displayed at the center of the screen, it is necessary to use the images provided from the cameras B and C. As described above, in a multi-view video system, it is necessary to select a video source to be used and a screen cutout range according to a range that the viewer desires to view.

Japanese Patent Laid-Open No. 5-130528

In the above-described video processing apparatus that processes video, as a video selection method, it is conceivable to select a video source and a display range according to an operation instruction from a viewer. For example, when a viewer views a program whose location changes frequently, such as a sports program, it is necessary to perform the operation frequently, and thus the operational burden is large.
Therefore, in order to automatically select the display range on the apparatus side, a video processing apparatus having a function of tracking a specific subject designated by the viewer and always displaying it in the center of the screen is conceivable. If the device detects a subject, selects a video source so that the subject image is captured at the center of the screen, and selects a display range, the viewer can view a desired video without any operation. In the future, multi-view video services will become more popular as next-generation digital broadcasting services, and the rise of devices with the above functions is expected.

When the video processing device reselects the video source, it is possible to continue displaying a desired subject image at the center of the screen. At that time, depending on the selection method of the video source, a situation may occur in which the video source is frequently switched in accordance with the movement of the subject. If the video source is frequently switched, a video that is very difficult for the viewer to view may be displayed.
Therefore, an object of the present invention is to improve user convenience by minimizing switching of video sources.

Image processing apparatus according to an embodiment of the present invention can process a plurality of image data having different shooting range, an image processing apparatus for tracking a predetermined object, a portion including the predetermined subject from the image data Output means for cutting out the range of the image and outputting it to the display unit, and time for detecting the movement of the predetermined subject to perform motion prediction processing, and the moving subject is included in each shooting range of the plurality of video data And control means for performing control for selecting one video data from the plurality of video data to display the subject on the screen based on the calculated time.

  According to the present invention, it is possible to provide an image that is easy for a viewer to view by reducing the frequency of screen switching in the tracking processing of a designated subject.

It is a block diagram which shows the structural example of an apparatus in order to demonstrate 1st Embodiment of this invention combined with FIG. It is the schematic which shows the example of camera arrangement | positioning regarding a multiview video. It is a flowchart which shows the process example. It is explanatory drawing of the imaging | photography range and screen cut-out range of the camera A. It is explanatory drawing of each imaging | photography range of Cameras A thru | or C, and a screen cutout range. It is explanatory drawing of the motion vector of a to-be-photographed object. It is explanatory drawing which shows the camera arrangement | positioning and subject movement based on 2nd Embodiment of this invention. FIG. 10 is a flowchart illustrating a processing example of a server in order to describe the third embodiment of the present invention in conjunction with FIG. 9. It is a flowchart which shows the process example of a video display apparatus. It is explanatory drawing of a multiview video system.

[First Embodiment]
FIG. 1 is a block diagram showing a configuration example of a system according to the first embodiment of the present invention. In this example, an IP (Internet Protocol) broadcast server (hereinafter simply referred to as a server) as a distribution device of a digital broadcasting station and a video display device as a video processing device are shown.
First, a block for acquiring multi-view video data from the server 200 will be described. The video display apparatus 100 is connected to the server 200 whose IP interface unit 101 provides multi-view video data of a digital broadcasting station via the Internet using the TCP / IP protocol. FIG. 1 representatively shows three viewpoint video data among the multi-view video data distributed by the server 200. The IP interface unit 101 receives data from the server 200, decodes it, and outputs it to the IP · TS (transport stream) processing unit 102. The IP interface unit 101 outputs a request to the server 200.

  The IP / TS processing unit 102 takes out the encrypted TS signal and outputs the decrypted signal to the IP / MPEG decoding unit 103. MPEG is an abbreviation for “Moving Picture Experts Group”. The IP / MPEG decoding unit 103 decompresses the compressed video data and outputs the decompressed video data to the subject extraction unit 109 as video data for subject selection. The IP / MPEG decoding unit 103 also outputs the decompressed video data to the output video generation unit 111.

  Next, a block for determining a screen cutout range for subject selection and tracking will be described. A remote controller (hereinafter referred to as a remote controller) 105 is used when a user transmits an operation instruction to the video display device 100. When the user uses the remote controller 105 to select a subject, the remote control signal receiver 106 receives a signal from the remote controller 105 and outputs an operation instruction signal to the controller 107. The control unit 107 determines a subject in response to a user operation instruction, and the subject selection unit 108 holds information on a subject selected from a plurality of subjects (subject information). The subject information is sent to a screen cutout range determination unit (hereinafter referred to as a range determination unit) 110, and the range determination unit 110 determines a screen cutout range based on the subject information. Information on the screen cutout range is transmitted to the output video generation unit 111. Here, a process of extracting a part of the video data transmitted from the IP / MPEG decoding unit 103 based on the cut-out range information is performed, and the processed video signal is transmitted to the output processing unit 112. The video signal output from the output processing unit 112 is sent to the display unit 113 to display the video.

  The control unit 107 controls the multi-view video selection instruction unit 104 based on the information from the range determination unit 110. That is, when the video source needs to be switched, the control unit 107 instructs the IP interface unit 101 to switch the video source via the multi-view video selection instruction unit 104. In accordance with this instruction, the IP interface unit 101 requests the server 200 to switch the multi-view video in order to change to the instructed video data. Thereby, change control to the data provided by the video source selected by the control unit 107 is performed. Each video source includes video data provided by shooting with a camera having a different shooting range.

  FIG. 2 shows an example of the arrangement of cameras that provide video sources. For example, it is assumed that a plurality of cameras having different shooting ranges are arranged around a stadium, and videos shot by each camera are supplied from a broadcasting station as multi-viewpoint videos. Cameras A to C (see reference numerals 201 to 203) are arranged adjacent to each other along one long side direction around a rectangular stadium in order to photograph a subject 204 indicated by a black dot. A camera D (see reference numeral 205) described in a second embodiment to be described later is disposed at a position facing the camera B on the side opposite to the cameras A to C across the stadium.

  When the video display apparatus 100 requests video data of the viewpoint video (A viewpoint video) of the camera A from the IP interface unit 101 to the server 200 (see FIG. 1), video data captured by the camera A 201 from the server 200 is displayed. Supplied. Similarly, when the video display device 100 requests the server 200 for the viewpoint video (B viewpoint video) of the camera B202 or the viewpoint video (C viewpoint video) of the camera C203 to the server 200, data of each viewpoint video is supplied. Is done. By displaying these videos on the display unit 113, a multi-viewpoint video system can be realized, and a desired viewpoint video can be provided to the viewer.

  Next, the operation of the video display apparatus 100 will be described using the flowchart of FIG. The following processing is realized by the control unit 107 interpreting and executing a program created according to the flowchart. First, the control unit 107 displays a selection screen on the display unit 113 in order to prompt the user to select a video source related to the display video (S301). In this case, options may be presented on the display screen to allow the user to select, or a predetermined video source may be displayed, or a video source to be displayed first on the broadcast station side may be designated. . Next, the control unit 107 executes processing for causing the user to select a desired subject (S302). As a method for the user to select a subject, for example, the control unit 107 performs control so that a part of the screen is designated by a user operation, identifies the subject by image recognition of the designated range, and registers data of the subject. To do. Alternatively, the control unit 107 may control the user to select a desired subject based on information (player information or the like) given by the metadata.

  After selecting a subject, the subject extraction unit 109 extracts image data of the selected subject. The range determination unit 110 cuts out a part of the video provided from the currently selected video source, sets the screen cut-out range so that the subject image is displayed in the center of the screen (S303), and the display unit 113. Display video on. In the next process, the control unit 107 determines whether or not the subject has moved (S304). As a result, if it is determined that the subject has moved, the process proceeds to S305, but if it is determined that the subject has no movement, the determination process in S304 is repeated.

  Next, the control unit 107 determines whether or not the subject image can be displayed at the center of the screen with the current video source (S305). When it is determined that the subject image can be displayed at the center of the screen (branch to YES), the process returns to S303, and the range determination unit 110 changes the screen cutout range. FIG. 4 shows a camera position, a screen cutout range 401, and a shooting range 402 when a video of a video source obtained by shooting the subject 204 using the camera A201 is used. Note that the shooting range 402 can be calculated and determined based on video data provided by the video source, or information indicating the range can be acquired from the server 200. The shooting range of each camera is determined by, for example, the coordinate position of the camera, the shooting direction of the camera, and the angle of view of the camera (the angle of the shooting range). A screen cutout range 401 corresponding to the display range of the display unit 113 is set so as to include the subject 204 in the center of the screen. When the subject 204 moves beyond the shooting range 402 of the camera A 201 due to the movement of the subject 204 and the subject image cannot be displayed at the center of the screen, it is necessary to switch the video source so that the subject image can be displayed at the center of the screen. . The video source selection process at this time will be described below.

  For example, if the subject image cannot be displayed at the center of the screen in the selected video source by the camera A 201 (branch to NO in S305), the control unit 107 performs processing for searching for another video source. That is, a video source capable of displaying the subject image at the center of the screen is searched. Search methods for video sources that can cut out the screen in the same range as before the change are the method of image recognition from video, or search for video sources that can cut out the screen in the same range based on the coordinate data at the time of screen cutting There is a way. Alternatively, the camera position information and the angle-of-view information of each video source may be acquired, and a range in which the screen can be cut out is calculated based on both information.

  FIG. 5A is a diagram for explaining a video source selection process to be switched next when the screen cutout range 501 including the subject 204 moves outside the shooting range 402 of the camera A. The screen cutout range 501 indicates a range in which the subject 204 can be displayed at the center of the screen, and the next video source capable of providing video in the range is a video source shot by the camera B202. That is, the screen cutout range 501 is within the shooting range 502 of the camera B202 and outside the shooting range 503 of the camera C203. In S307 of FIG. 3, the control unit 107 calculates the number of video sources that include a screen clipping range in which the subject image can be displayed at the center of the screen within the shooting range, and determines whether the number is 2 or more. To do. Immediately after the subject 204 moves out of the shooting range 402 of the camera A201 in FIG. 5A, the number of the video sources is 1, and the process proceeds in the direction of NO. Here, the video source, that is, the video source that supplies the video shot by the camera B 202 is selected (S310), and the process proceeds to S311.

  On the other hand, in S307, when the number of video sources including a screen cutout range in which the subject image can be displayed in the center of the screen is 2 or more, the process proceeds to S308. FIG. 5B shows an example in which the screen cutout range 601 including the subject 204 is within the shooting range 502 of the camera B202 and within the shooting range 503 of the camera C203. An image including the subject 204 in the center of the screen can be displayed with each image source by the cameras B and C. In this case, the control unit 107 selects the video source so that the number of switching of the video source is minimized. Specifically, in S308 of FIG. 3, the control unit 107 performs subject motion prediction processing to calculate a motion vector. Next, the control unit 107 selects a video source having the longest displayable time based on the calculated direction and magnitude of the motion vector (S309). For the motion vector calculation processing, the movement trajectory of the subject is stored in the memory and calculated based on this data, or information such as the movement direction and movement speed is given as metadata in addition to the coordinate information of the subject. In some cases, the data can be used for calculation. The “displayable time” can be calculated by predicting the time during which the screen cutout range can stay within the shooting range of each camera based on the motion vector.

  FIG. 6 is a diagram illustrating an example of a video source determination process based on a motion vector. In the example of FIG. 6A, the motion vector 701 is directed to the right as shown by the arrow, and the control unit 107 cannot perform the screen cutout process first when the subject 204 moves in that direction. Determine the video source. If the subject 204 continues to move in the direction of the motion vector 701, it is the video source from the camera B 202 that makes it impossible to perform the screen cutout process first. The subject image can be continuously displayed for a longer time by using the video source by the camera C203. Therefore, a video source that supplies video shot by the camera C203 that can reduce the frequency of switching the video source and can display a video for a longer time is selected.

FIG. 6B illustrates a case where the motion vector 801 is directed downward in the drawing. A video source capable of continuing to display the subject 204 at the center of the image for a long time is one that was captured by the camera B202 instead of the camera C203. Therefore, in this case, the video source that supplies the video shot by the camera B 202 is selected.
After S309 or S310 in FIG. 3, the process proceeds to S311. Here, the control unit 107 instructs the IP interface unit 101 to switch to the video source selected in step S309 or S310 via the multi-view video selection instruction unit 104. The IP interface unit 101 switches to the video of the video source.
According to the first embodiment, it is possible to track a desired subject while minimizing the number of times of switching video sources.

[Second Embodiment]
Next, a second embodiment of the present invention will be described. Since the configuration of the apparatus according to the second embodiment is the same as that of the first embodiment, the differences will be mainly described below.
FIG. 7 is a diagram for explaining video source selection processing when the camera D (see reference numeral 205) is arranged on the side opposite to the cameras A to C. A shooting range 901 in the figure indicates a range shot by the camera D205.

  FIG. 7A illustrates a case where the video supplied from the camera A 201 is used first. Here, it is assumed that the subject 204 has moved in the direction indicated by the motion vector 801 (downward in the figure), so that the video source from the camera A 201 cannot supply the video in the screen cutout range 601. In this case, switching of the video source is required as in the first embodiment, but as can be seen from the direction of the motion vector 801, the video source capable of displaying the video of the subject 204 for the longest time is the camera B202. This is due to the camera D205. However, when the image from the camera A201 is switched to the image from the camera D205, the direction of movement of the subject image is reversed compared to the previous image. If such video switching is automatically performed, it is difficult for the viewer to know the moving direction of the subject image, which may hinder viewing. Therefore, the control unit 107 lowers the priority at the time of selection for the video source whose direction of movement of the subject changes. In the example of FIG. 7A, when the priority order is represented by 1 to 3, priority 1 is an image captured by the camera B202, priority 2 is an image captured by the camera C203, and priority 3 is captured by the camera D205. It will be the image.

  FIG. 7B illustrates a case where the camera B202 does not exist. When the subject 204 moves in the direction of the arrow indicating the motion vector 801, the video source from the camera A201 is switched to the video source from the camera C203. That is, the video of the camera D205 that can display the image of the subject 204 in the center of the screen for a longer time is not selected because the priority is low, and the video shot by the camera C203 that does not change the direction of movement of the subject image on the screen. Is preferentially selected and displayed.

  According to the second embodiment, the video source selection control is performed so that the moving direction of the subject in the video source before the change and the moving direction of the subject in the video source after the change do not change or the amount of change becomes small. Is called. As a result, it is possible to realize switching of the video source with less discomfort for the viewer.

[Third Embodiment]
Next, a third embodiment of the present invention will be described. In the present embodiment, an example of processing for designating a video source to be selected from a transmission device as a video data supply source to a reception device will be described. Since the configuration of the apparatus according to the third embodiment is the same as that of the first embodiment, the differences will be mainly described below. In this embodiment, the video source selection criteria are the same as those in the above embodiment, except that the determination subject is not the control unit 107 of the video display device 100 but the control unit of the distribution device. In FIG. 1, the video display device 100 notifies the server 200 of information on the subject selected according to the user operation. The system control unit 210 of the server 200 has a transmission control function when transmitting video data of a video source to be used and screen cutout range instruction information corresponding to the display range to the video display device 100. The video display device 100 performs switching and display of video sources based on such information.

  A processing example of the server 200 will be described using the flowchart of FIG. Note that the following processing is realized according to a program interpreted and executed by the system control unit 210. The processes of S304, S305, and S307 to 310 are the same as those in the first embodiment shown in FIG. 3 except that the system control unit 210 performs them. Hereinafter, the differences S1101 to S1103 and S1110 will be described. .

  The system control unit 210 of the server 200 instructs a video source to the video display device 100 (S1101). The system control unit 210 receives subject information from the video display device 100 (S1102), and determines a screen cutout range so that the designated subject is displayed at the center of the screen. Then, the system control unit 210 controls to transmit instruction information indicating a screen cutout range including the subject image at the center of the screen to the video display device 100 (S1103). Thereafter, the process proceeds to the process of S304 (detection of subject movement). If the determination result in S305 is NO, the processes after S307 are executed.

After S309 or S310, the system control unit 210 performs transmission control to instruct the video display device 100 to switch to the selected video source (S1110).
Next, a processing example of the video display apparatus 100 will be described using the flowchart of FIG. The following processing is realized according to a program that is interpreted and executed by the control unit 107.
In accordance with the instruction to the video display apparatus 100 in S1101, the control unit 107 controls selection of the video source (S1201). Next, in response to a user operation instruction, subject selection processing is performed (S1202). The method for selecting a subject is the same as in the first embodiment. The control unit 107 performs control for notifying the server 200 of subject information indicating the selected subject (S1203). In accordance with the instruction in S1103, the range determination unit 110 determines the screen cutout range, and the display unit 113 displays the video in the range (S1204).

Thereafter, the system control unit 210 of the server 200 detects the movement (movement speed) of the subject (see S304 in FIG. 8) and determines whether or not the subject has moved. When the process reaches S1110 through S305 to 310 in FIG. 8, the system control unit 210 of the server 200 instructs the video display apparatus 100 on the selected video source. Upon receiving this instruction, the control unit 107 of the video display apparatus 100 determines whether to change the video source (S1205). As a result, when it is determined that the instruction is an instruction to change the video source (branch to YES), the control unit 107 sends an instruction to the IP interface unit 101 via the multi-view video selection instruction unit 104. As a result, the designated video source is changed (S1206), and the process proceeds to S1207. If it is determined in S1205 that the instruction is not to change the video source (branch to NO), the process proceeds to S1207. Here, the control unit 107 determines whether or not there is an instruction to change the screen cutout range. Here, the range determination unit 110 changes the screen cutout range, and the display unit 113 displays the video in the range. Then, the process returns to S1205. If it is determined in S1207 that there is no instruction to change the screen cutout range, the process returns to S1205.
In the third embodiment, a subject is tracked by a distribution device, and a video source selection instruction and a screen cutout range instruction are sent to the video display device. Thus, the subject image can always be displayed at the center of the screen while the frequency of screen switching is minimized in the video display device.

[Other Embodiments]
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

DESCRIPTION OF SYMBOLS 100: Image | video display apparatus 101: IP interface part 104: Multi viewpoint video selection instruction | indication part 107: Control part 108: Subject selection part 109: Subject extraction part 111: Output video generation part

Claims (14)

A video processing apparatus that processes a plurality of video data with different shooting ranges and tracks a predetermined subject,
An output means for cutting out a part of the range including the predetermined subject from the video data and outputting it to a display unit;
The movement of the predetermined subject is detected and motion prediction processing is performed, a time during which the moving subject is included in each of the shooting ranges of the plurality of video data is calculated, and the subject is displayed on the screen based on the calculated time. Control means for performing control for selecting one video data from the plurality of video data to be displayed on the video processing apparatus. The control means calculates a time during which the moving subject is included in each imaging range of the plurality of video data based on coordinate information and a motion vector of the moving subject. the video processing apparatus according to 1. Said control means, said plurality of video data, wherein the time included the subject of the mobile in the photographing range is higher selection priority of long video data, to claim 1 or claim 2 The video processing apparatus described. The control means increases the selection priority of video data with a small amount of change in the moving direction of the moving subject when the predetermined subject moves outside the imaging range of the video data. The video processing apparatus according to claim 1. 5. The video processing apparatus according to claim 1 , further comprising a receiving unit configured to receive a plurality of video data having different shooting ranges from an external apparatus. The video processing apparatus according to claim 1 , wherein the output unit determines a range to be cut out from the video data so that the predetermined subject is located at a center of a screen. An external device that is a supply source of a plurality of video data having different shooting ranges to the video processing device instructs the video processing device to select predetermined video data,
The output means selects the predetermined video data in accordance with an instruction from the external device, and cuts out a partial range including the predetermined subject from the selected predetermined video data . The video processing apparatus according to claim 1. A video processing method executed by a video processing apparatus that processes a plurality of video data with different shooting ranges and tracks a predetermined subject,
An output step in which the output means of the video processing device cuts out a part of the range including the predetermined subject from the video data and outputs it to the display unit;
A calculating step in which the control means of the video processing device detects movement of the predetermined subject to perform motion prediction processing, and calculates a time during which the moving subject is included in each shooting range of the plurality of video data; ,
And a selection step of selecting one video data from the plurality of video data so that the moving subject is displayed on a screen based on the time calculated in the calculation step by the control means of the video processing device. Video processing method. In the calculating step, the control means calculates a time during which the moving subject is included in each shooting range of the plurality of video data based on coordinate information and a motion vector of the moving subject. The video processing method according to claim 8. 9. The selection step according to claim 8, wherein, in the selection step, the control unit increases a selection priority of video data having a long time in which the moving subject is included in a shooting range among the plurality of video data. The video processing method according to claim 9. In the selection step, when the predetermined subject moves outside the imaging range of the video data, the control means increases the selection priority of video data with a small change amount in the moving direction of the moving subject. The video processing method according to any one of claims 8 to 10, wherein the video processing method is characterized. Characterized in that it has a receiving step of receiving means of the video processing apparatus receives a plurality of video data having the different image capturing range from the external device, image processing method according to any one of claims 8 to 11. Said output means in said output step, the predetermined subject is to be located in the center of the screen, and determines a range to be cut out from the image data, according to any one of claims 8 to 12 Video processing method. An external device that is a supply source of a plurality of video data having different shooting ranges to the video processing device instructs the video processing device to select predetermined video data,
In the output step, the output means selects the predetermined video data according to an instruction from the external device, and cuts out a partial range including the predetermined subject from the selected predetermined video data. wherein, the image processing method according to any one of claims 8 to 13.

Images