JP2009246917A - Video display device, and video processing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To enable a user to efficiently view video images by displaying a video image for which resolution improvement processing has been performed only upon a highlight region. <P>SOLUTION: A video display device includes: a video input unit to which a video image is inputted; a video dividing unit 206 for dividing the video image inputted to the video input unit into a plurality of regions; and a resolution improvement processing unit 211 for improving a resolution of the video image inputted to the video input unit. The resolution improvement processing unit improves a resolution for partial regions, divided by the video dividing unit, in the video image inputted to the video input unit. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to video processing for a highlight area in an input video.

  As background art of a video display device, Patent Document 1 records a low-resolution whole image and a high-resolution partial image under the use of a single surveillance video camera, and according to instructions from a surveillance operator, A technique for reproducing and displaying in high definition or enlarged is disclosed. In Patent Document 2, a hyperboloid mirror that inputs a wide-angle image centered on or about the meeting vertical direction, a plurality of microphones that input sound, and a sound source direction that detects a sound source direction based on the sound input to the microphone A video conference system having a detection unit and a recording unit that records a wide-angle image, audio, and a sound source direction so that an image of a predetermined area in a direction corresponding to the sound source direction among the recorded wide-angle images becomes a rectangular output image. The technique which deform | transforms into is disclosed.

JP 2006-33793 A JP 2003-18561 A

  In the technique disclosed in Patent Literature 1, there is a problem that the monitoring operator needs to manually set the region to which the user pays attention.

  In addition, Patent Document 2 has a problem that a voice is required to determine a direction in which a user focuses attention.

  The present invention has been made in view of the above problems, and an object thereof is to enable efficient video viewing by the user.

  The embodiment of the present invention may be configured as described in the claims, for example.

  According to the present invention, the user can efficiently view the video.

  Embodiments (examples) of an apparatus operating device according to the present invention will be described below with reference to the drawings. However, the present invention is not limited to the following examples.

  In each embodiment of the present application, the “highlight area” means a video area having a video feature that should be noted by a video viewer (user). In other words, it can be said that the video area has a high degree of attention and a high importance area. It can also be expressed as a rising area.

  In each embodiment of the present application, the “highlight score” is an index that should be noticed by the viewer (user) of the image for each region in the image, and can be said to be the degree of attention and importance. It can also be expressed as a degree of excitement.

  That is, it can be said that the region where the “highlight score” is higher is the region to be noted and the region having higher importance.

  In the first embodiment, a highlight area is detected within the same screen of the video, and the video that has been subjected to the high resolution processing is displayed only in the highlight area. This makes it possible to reduce the required computing resources and to allow users to efficiently view images.

  FIG. 1 is a diagram illustrating an example of a hardware configuration of the video display apparatus according to the first embodiment.

  The apparatus of this example includes a video input device 101, a central processing device 102, an instruction input device 103, a storage device 104, a secondary storage device 105, and a display device 106. Each device is connected by an internal bus 107, and data can be transmitted and received between the devices.

  The video input device 101 inputs video data. The video input device 101 is, for example, a video camera connected to the device, a device that reads video data stored in the storage device 104 or the secondary storage device 105 described later, or a television broadcast or the like. Can be a TV tuner.

  In addition, when a video such as a web video is input via a network, the video input device 101 can be a network card such as a LAN card.

  The central processing unit 102 is mainly composed of a microprocessor, and executes a program stored in the storage device 104 or the secondary storage device 105.

  The instruction input device 103 is a pointing device such as a remote controller, a keyboard, and a mouse. This accepts a user's image input, display instruction, and the like.

  The display device 106 is realized by, for example, a display adapter, a liquid crystal panel, a projector, and the like, and displays an image input by the instruction input device 103.

  The storage device 104 is realized by, for example, a random access memory (RAM), a read-only memory (ROM), and the like. The storage device 104 is a program executed by the central processing unit 102, data processed in the moving image reproducing device, or moving image data to be reproduced. And ranking data.

  The secondary storage device 105 is composed of, for example, a hard disk, a DVD or a CD and their drives, or a nonvolatile memory such as a flash memory. The secondary storage device 105 is a program executed by the central processing unit 102 or data processed in the video playback device. Alternatively, video data to be reproduced is stored.

  FIG. 2 is an example of a functional block diagram of the video display apparatus according to the first embodiment. Some or all of these functional blocks may be realized as hardware in addition to the hardware shown in FIG. 1, or may be realized as a software program executed by the central processing unit 102. In the following description, as an example, it is assumed that all of these functional blocks are software programs executed by the central processing unit 102.

  As shown in FIG. 2, the video input / playback apparatus includes a video input unit 201, an instruction input unit 202, a highlight detection unit 203, a highlight score table storage unit 204, a video generation unit 205, and a display unit 206. Have. The highlight detection unit 203 includes a video acquisition unit 207, a video division unit 208, a highlight score calculation unit 209, and a highlight score table generation unit 210. The video generation unit 205 includes a high resolution processing unit 211 and a display video generation unit 212.

  The video input unit 201 inputs a video for high resolution processing from the video input device 202. The video input unit 201 is executed by the central processing unit 102 when a video input is instructed by the user from the instruction input unit 202 or when instructed by a scheduler (not shown).

  The video acquisition unit 207 of the highlight detection unit 203 acquires, from the video input by the video input unit 201, a video of a frame at a time targeted for highlight detection processing described later. Hereinafter, a time stamp designating a time corresponding to a video is referred to as a PTS (Presentation Time Stamp). The video dividing unit 208 divides the video into a plurality of areas in order to calculate the highlight score for each area of the video and to perform the resolution enhancement processing only on the area with the highest highlight score. The highlight score calculation unit 209 calculates a highlight score to be described later for each divided region. The highlight score table generation unit records the calculated highlight score in the highlight score table in association with the corresponding PTS and each divided area. The highlight score table is held in the highlight score table storage unit 204. Note that the processing of each unit constituting the highlight detection unit 203 is executed by the central processing unit 102.

  The video generation unit 205 processes the input video with reference to the highlight score table to generate an output video. The high resolution processing unit 211 of the video generation unit 205 refers to the highlight score table stored in the highlight score table storage unit 204 to determine the divided region with the highest highlight score in the frame corresponding to each PTS. Then, the high resolution processing is performed on the corresponding image. The display video generation unit 212 synthesizes the divided area with high resolution and the divided area without high resolution, and generates a video to be displayed. Note that the processing of each unit constituting the video generation unit 205 is executed by the central processing unit 102.

  The display unit 206 displays the video generated by the video generation unit 205 on the display device 106. The display unit 206 displays the video generated by the display video generation unit on the display device 106 frame by frame. In this case, the processing of the display unit 206 is executed by the central processing unit 102 every time one frame worth of video is generated by the display video generation unit.

  FIG. 3 is a diagram for explaining an example of the resolution enhancement processing by the resolution enhancement processing unit 201.

  First, as shown in (a), a plurality (three in this case) of original images 301 to 303 adjacent to each other in time series, that is, continuous frames are prepared, and alignment is performed with decimal precision. (B) shows a state in which the other images 302 and 303 are aligned with the image 301 as the reference position. Positioning also takes into account movements such as rotation and enlargement / reduction. In this way, one composite image with the sampling position shifted from the integer pixel position is generated. Thereafter, as shown in (c), the high-resolution image 304 is generated by sampling again at the target resolution. In resampling, the pixel value is determined while convolving the influence of surrounding pixels by a method of interpolating and increasing the number of pixels (sampling points) using a low-pass filter or by applying an inverse function of a point spread function. . As a result, the high-frequency component of the original image is restored, and a high-resolution image having a resolution about twice that of the original image is generated. This high resolution processing is effective for continuous images in which a subject in the image is moving.

  Note that the resolution enhancement processing by the resolution enhancement processing unit 201 is not limited to the above method, and other conventional resolution enhancement processing may be used.

  FIG. 4 is an example of video division according to the first embodiment.

  The input video 401 input by the video input unit 201 and acquired by the video acquisition unit 207 has a plurality of two or more video dividing units 208 to calculate a highlight score for each area in one frame of video. Divide into images. For example, as shown in FIG. 4, the image is divided into four areas: a divided area A 402, a divided area B 203, a divided area C 404, and a divided area D 405. The input image 401 does not necessarily have to be a rectangle, and may have another shape, for example, a circle such as an all-around image or a fisheye image. Further, the shape of the divided area may be a rectangle or other shapes. Further, the size and shape of each divided region may be the same or different.

  A highlight score is a score indicating the importance of a video. As shown in FIG. 4, a certain frame of video is divided into a plurality of areas, and the highlight score of each divided area is calculated. As a result, the divided area having the highest highlight score can be regarded as the most important area in the frame, that is, the highlighted area.

  FIG. 5 is a diagram illustrating an example of a method for calculating a highlight score in the highlight score calculation unit.

  First, (a) is an N-1 frame divided area A501 obtained by dividing the video of the N-1 frame by the video dividing unit 208. Similarly, (b) is an N frame divided area A502. The image of the divided area A of two continuous frames is binarized, each pixel is compared, and an inter-frame difference of the image is detected. A binary image is generated with “1” for pixels having a difference and “0” for pixels having no difference, and this is defined as a difference image 503 in (c). The sum of the pixels having a difference indicated by “1” becomes the highlight score of the divided area A502 in N frames. Further, a specific moving object may be detected from the difference image 503 in both frames, and the magnitude of the moving distance may be reflected in the highlight score. As described above, since the highlight score indicates the area of the moving object and the moving distance, it is assumed that a divided region having a higher highlight score is a region that becomes a highlight with a higher degree of user attention. it can.

  In FIG. 5 showing an example of the method for calculating the highlight score, the method for calculating the highlight score from the area of the moving object and the movement amount using the inter-frame difference of the image has been described. However, the present embodiment is not limited to this method. For example, a method of giving a high highlight score to a divided region that matches an image assumed in advance by template matching may be used. Also, a method for calculating the highlight score may be defined according to the video or program genre. For example, in the case of a video of a baseball game, an image of the ball is stored in advance as a template, and a highlight score is calculated by a method using template matching. Thereby, the area including the ball can always be used as the highlight area and can be displayed as the area of the video image which the user pays attention. In addition, the highlight score may be calculated by a method using image processing such as giving a high highlight score to a divided region including a luminance or high-saturation region focused on by human eyes. Further, a region where the image is raised may be estimated from data other than the image accompanying the image, for example, audio data, and may be used as a highlight region.

  Next, the data structure of the highlight score table generated by the highlight score table generating unit 210 will be described.

  FIG. 6 is an example of the data structure of the highlight score table in the first embodiment. The highlight score table generation unit 210 associates the highlight score calculated by the highlight score calculation unit 209 with the corresponding PTS and each divided area, and the highlight score table stored in the highlight score table storage unit 204. To record.

  In the highlight score table 601 shown in FIG. 6, PTS 602 is the PTS of the frame for which the highlight score calculation unit 209 has calculated the highlight score. You may record the highlight score of each divided area of all the frames. Alternatively, the highlight score calculation unit 209 may calculate the highlight score every specific time (for example, 5 seconds) to reduce the processing amount, and the highlight score table generation unit 210 may record the highlight score. The segment score 603 is a highlight score of each segment corresponding to the PTS 602. The high resolution processing unit 211 refers to the highlight score table 601 and performs high resolution processing on the divided region having the highest highlight score in a frame corresponding to a certain PTS.

  FIG. 7 is a diagram illustrating an example of a processing flow for determining a highlight score of an input image according to the first embodiment. This processing is performed by the highlight score calculation unit 209, and the highlight score is calculated for a specific divided region of the video of the specific frame obtained by the video dividing unit 208. The higher the highlight score, the greater the degree of attention that the user pays attention to the divided area as the highlight area.

  The highlight score calculation unit 209 acquires a specific divided region (referred to as divided region A) of a certain frame (referred to as N-1 frame) (S701), and then similarly, adjacent frames (N A divided area A of a frame is acquired (S702). Subsequently, the divided areas of the N-1 frame and the N frame are compared (S703), and a difference image is generated according to the method shown in FIG. 5 (S704). The number of pixels that are the difference of the generated difference image is calculated (S705), and the calculated number of pixels is recorded in the highlight score table of the highlight score table storage unit 204 as an N frame highlight score (S706). ). If the sizes of the divided areas are not all the same, the highlight score may be normalized according to the number of pixels in each divided area itself. When there is a divided area having the same highlight score, a new highlight score may be calculated by adding the score at the time of calculating the highlight score in the frame corresponding to the previous PTS.

  Next, a processing method from video input to display in the first embodiment will be described.

  FIG. 8 is a diagram illustrating an example of a flow showing processing from video input to display in the first embodiment.

  As shown in FIG. 8, the video input unit 201 inputs a video to be subjected to a resolution enhancement process according to the first embodiment (S801). The video acquisition unit 207 acquires video for one frame corresponding to the input PTS (S802). The video dividing unit 208 divides the acquired video into a plurality of areas, and generates divided areas (S803). The highlight score calculation unit 209 calculates a highlight score for each of the divided areas using a highlight score calculation process described later (S804). The highlight score table generation unit 210 records the calculated highlight score of each divided area in the highlight score table 601 held in the highlight score table storage unit 204 (S805). The high resolution processing unit 211 refers to the highlight score table 601 and sets the divided region having the highest highlight score in the frame corresponding to a certain PTS as the highlight region (S806), and the high resolution described in FIG. (S807). The display video generation unit 212 combines the highlight region that has been subjected to the high resolution processing and the region that has not been subjected to the high resolution processing, and generates a video of a frame to be displayed (S808). The display unit 206 displays the generated video on the display device 106 (S809). If the user does not give an instruction to end the processing, or if the input video is not finished and the processing is not finished (S810), the video input is continued and the next frame processing is continued. If so, the process ends (S810).

  In the flowchart of processing from input to display shown in FIG. 8 above, highlight score calculation processing is performed for all frames of the input video. Here, in order to reduce the processing amount, for example, an image of a specific frame specified in advance is used as a highlight score calculation target, for example, a highlight score is calculated for an image every 15 frames, and a frame in the meantime is highlighted. As for the light score, the score of the frame calculated immediately before may be continued.

  When generating a frame image to be displayed by combining the highlight region that has been subjected to the high resolution processing and the region that has not been subjected to the high resolution processing (S808), the display video generation unit 212 sets the boundary between the combined divided regions. You may perform the process which changes a resolution in steps so that resolution may become smooth. Further, in order to prevent the change in resolution from being noticeable, a process of gradually lowering the resolution may be performed on the divided areas that do not require the high resolution process.

  FIG. 9 is a diagram illustrating an example of an image obtained by performing a process of changing the resolution stepwise so that the resolution becomes smooth at the boundary of the combined divided regions.

  In the video of FIG. 9A, the divided area A901 is an area that has undergone high resolution processing, the divided area B902 is an area that has not been subjected to high resolution processing, and the divided area A901 and the divided area B902 are: The images are combined by the display image generation unit 212. The resolution of the input video and the divided area B902 that has not been subjected to the high resolution processing is J, and the resolution of the divided area A901 that has been subjected to the high resolution processing is (2 × J). A resolution boundary line 903 is a boundary between the divided area A 901 and the divided area B 902. The display video generation unit 212 changes the resolution of the area on the divided area A 901 side from the resolution boundary line 903 in a stepwise manner, and generates a smooth video (b) in which the change in resolution is not noticeable. For example, from the resolution boundary line 903 to the divided area A 901 side, an area 904 in the H pixel range is set to a resolution ((5/3) × J), and an area 905 in the H or more (2 × H) pixel range is set to the resolution ( (4/3) × J).

  As a result of the above processing, the resolution of the boundary between the divided region A901 that has been subjected to the high resolution processing and the divided region B902 that has not been subjected to the high resolution processing generates a smooth image that is inconspicuous in the resolution change. Is possible.

  The video display device according to the first embodiment of the present invention described above displays a video that prompts the user to pay attention to the highlight position by detecting a highlight region of the input video and controlling a region where resolution is increased. It becomes possible. Further, it is possible to reduce the calculation resources used for the high resolution processing by performing the control so that the high resolution processing is performed only in a highlight area that requires high resolution display.

  In the second embodiment, the video display device downloads highlight metadata corresponding to a certain video from the server, and performs a resolution enhancement process.

  FIG. 10 is a diagram illustrating an example of a hardware configuration of the video display apparatus according to the second embodiment. In the configuration of FIG. 10, in addition to the hardware configuration of the first embodiment shown in FIG. 1, a network I / F 1001 for downloading highlight metadata described later from the server is provided.

  FIG. 11 is an example of a functional block diagram of the video display apparatus according to the second embodiment.

  FIG. 11 includes a communication device 1101 for downloading highlight metadata from a server in addition to the functional block diagram according to the first embodiment shown in FIG. Further, the highlight score calculation unit 209 and the highlight score table generation unit 210 according to the first embodiment are not provided, but a highlight / metadata analysis unit 1102 is provided. Further, the highlight score table storage unit 204 according to the first embodiment is not provided, but a highlight metadata storage unit 1103 is provided. Accordingly, the video display device itself does not perform highlight detection processing, and the highlight metadata analysis unit 1102 analyzes the highlight metadata that has been defined and downloaded in advance, thereby identifying the video highlight.

  In the video display device of FIG. 11, the functions and operations of the constituent elements having the same reference numerals as those of the constituent elements of FIG.

  Next, the data structure of the highlight metadata downloaded by the communication device 1101 will be described.

  FIG. 12 is an example of a data structure of highlight metadata in the second embodiment. The highlight metadata analysis unit 1102 analyzes the highlight metadata 1201 previously stored in the server and downloaded by the communication device 1101 and associates it with the corresponding video. The highlight metadata may be manually created by any creator and stored in the server, or may be created as a configuration in which the server has the highlight detection unit 203 shown in FIG. 2 of the first embodiment. Good.

  The PTS 1202 is a time stamp of a corresponding video frame. A highlight coordinate U1203 and a highlight coordinate V1204 are coordinates indicating a region to be a video highlight. A rectangle having the highlight coordinate U1203 and the highlight coordinate V1204 at both ends of the diagonal line is the highlight region of the image of the corresponding PTS 1201 frame. The highlight area does not necessarily have to be rectangular, and may be circular, for example. In this case, a circular center coordinate and a radius are designated as a highlight area. The PTS 1202, the highlight coordinate U1203, and the highlight coordinate V1204 may be set for all frames, or may be set every specific frame, for example, every 5 seconds. In this case, the highlight area of the frame in between may continue the highlight area set immediately before. Alternatively, the highlight area switching method 1205 may be specified, such as switching continuously (panning).

  By downloading the highlight metadata described above from the server via the network, the user can reduce the calculation resources for detecting the highlight area in the video display apparatus. In addition, it is possible to highlight an arbitrary area of an image at an arbitrary time by highlight metadata. For example, the advertiser can give an arbitrary area of the video to the user as an area to be highlighted. Moreover, a server may hold a plurality of different metadata for one video. Thereby, the user can view the video by using the highlight metadata corresponding to each user's subjectivity.

  FIG. 13 is an example of a video highlight area in a frame corresponding to a certain PTS according to the second embodiment.

  The resolution enhancement processing unit 211 of the video generation unit 205 and the highlight metadata analysis unit 1102 use the frame 1301 corresponding to the specific PTS of the video and the highlight coordinates U1302 indicated in the highlight metadata. A rectangle having the highlight coordinate V1303 at both ends of the diagonal line is determined to be the highlight area 1304 in the video of this frame. The highlight area does not necessarily have to be a rectangle, and may be another shape such as a circle.

  Next, a processing method from video input to display in the second embodiment will be described.

  FIG. 14 is an example of a flowchart illustrating processing from video input to display in the first embodiment.

  As illustrated in FIG. 14, the instruction input unit 202 inputs an instruction from the outside that designates an image to be subjected to the high resolution processing according to the second embodiment (S1401). Next, the communication device 1001 transmits a request for highlight metadata corresponding to the video to the server (S1402), and receives highlight metadata from the server (S1403). Next, the video input unit 201 starts input of video (S1404), and the video acquisition unit 207 acquires video (S1405). The highlight metadata analysis unit 1002 analyzes the received highlight metadata (S1406), and determines the highlight area of the frame corresponding to the PTS described in the highlight metadata. Thereafter, similar to the processing of the first embodiment described with reference to FIG. 8, the video is generated and displayed by the processing of S807 to S810.

  The video display apparatus according to the second embodiment of the present invention described above uses the highlight metadata downloaded from the server. As a result, it is not necessary to perform highlight detection processing in the video display device serving as a client, so that the calculation resources of the video display device can be reduced. In addition, arbitrarily created highlight metadata can be shared by a plurality of users through a network.

  For example, for a certain video, the advertiser creates highlight metadata indicating the coordinates of the PTS and the highlight position and causes the server to store the highlight metadata. Thereafter, the user who has downloaded the highlight metadata can be made to focus on a specific image area intended by the advertiser.

  For example, the server may be configured to hold a plurality of highlight metadata having different highlight determination criteria for the same video. In this case, the user can select highlight / metadata according to each individual's preference, increase the resolution of the video, and view the video.

  In the third embodiment, the video display device detects the highlight area of the input video and regenerates the video centered on the highlight area, thereby displaying the video centered on the highlight area focused on by the user. Is possible.

  The hardware configuration according to the third embodiment will be described as the same configuration as that of the first embodiment in the third embodiment. As in the first embodiment, the video input unit 201 is, for example, a video camera connected to a device, a device that reads video data stored in the storage device 104 or the secondary storage device 105, which will be described later, or a television broadcast or the like. When receiving, it may be a TV tuner. In the present embodiment, the video input unit 201 will be described as an all-around image sensor. The all-around image sensor is a camera that captures the entire periphery at the same time instead of a specific direction.

  FIG. 15 is a diagram illustrating an example of video shooting of the video input unit according to the third embodiment and an example of video shot and input to the video display device.

  As shown in FIG. 15A, an omnidirectional image sensor 1501, which is a video input unit 201, is installed in a space to be imaged. The video imaged by the omnidirectional image sensor is a circular video image represented by polar coordinates as shown in (b), and this image is an input video image 1502 of the video display device according to the third embodiment. The input video is indicated by polar coordinates (r, θ) 1503. The omnidirectional image sensor 1501 is excellent in that it can obtain images not only in one direction but also in all directions, and is frequently used in surveillance camera applications, etc., but it is generally difficult for the user to intuitively understand the images. It is.

  With the configuration of the present embodiment described below, the highlight area of the input video is detected, and the input video is subjected to coordinate conversion processing to generate a display video centered on the highlight area, so that the user can intuitively understand It is possible to provide an easy video.

  FIG. 16 is an example of a functional block diagram of the video display apparatus according to the third embodiment.

  As shown in FIG. 16, the video input / playback apparatus includes a video input unit 201, an instruction input unit 202, a highlight detection unit 203, a highlight score table storage unit 204, and a display unit 206, as in the first embodiment. The highlight detection unit 203 includes a video acquisition unit 207, a video division unit 208, a highlight score calculation unit 209, and a highlight score table generation unit 210.

  In this embodiment, the video generation unit 1601 includes a video coordinate conversion unit 1602 and a display video control unit 1603.

  In the video display device of FIG. 16, the functions and operations of the constituent elements having the same reference numerals as those of the constituent elements of FIG.

  FIG. 17 is an example of division of the input video 1502 according to the third embodiment.

  For the input video input by the video input unit 201 and acquired by the video acquisition unit 207, the video dividing unit 208 divides the input video into two or more regions. For example, the image is divided into four areas: a divided area A1701, a divided area B1702, a divided area C1703, and a divided area D1704.

  As in the first embodiment, the highlight detection unit 203 calculates the highlight score of each divided area according to the highlight score calculation method shown in FIG. 7, and records it in the highlight score table 601 shown in FIG.

  The video generation unit 1601 processes the input video with reference to the highlight score table, and generates a video to be displayed on the display unit 206. The video coordinate conversion unit 1602 of the video generation unit 1601 refers to the highlight score table stored in the storage unit and determines that the divided region with the highest highlight score in the frame corresponding to each PTS is the highlight region. To do. Next, coordinate conversion processing of the input video 1502 is performed so that the highlight area is centered. A display video control unit 1603 controls coordinate conversion when the highlight area changes. The image to be displayed may be switched at the time of the PTS at which the highlight area changes that can be determined from the highlight score table 601, and the highlight area is continuously moved so that the display image changes smoothly. The display image may be generated by performing control to continuously change the parameter of the coordinate conversion. That is, the video generation unit 1601 performs coordinate transformation on the video acquired from the video acquisition unit 207 and reconstructs it to generate a new video.

  In the configuration example of FIG. 17 described above, a highlight score calculation unit 209 and a highlight score table generation unit 210 are provided in the same manner as in the first embodiment to detect a highlight region, and the detected highlight region. Is stored in the highlight score table storage unit 204.

  However, the configuration of the video display apparatus according to the third embodiment is not limited to this. For example, the hardware configuration of the video display apparatus according to the third embodiment is the same as in the second embodiment illustrated in FIG. A hardware configuration having a network I / F 1001 for downloading may be used.

  In this case, the functional block of the video display apparatus according to the third embodiment illustrated in FIG. 16 includes a communication device 1101, a highlight / metadata analyzing unit 1102, and a highlight metadata storage unit 1103, as in the second embodiment. Alternatively, the video display device itself may be configured not to perform highlight detection processing. In this case, as in the second embodiment, the highlight metadata defined in advance is downloaded from the server and analyzed by the highlight metadata analysis unit 1102 to identify video highlights. Since the detailed operation in this case is as described in the second embodiment, the description is omitted.

  Further, by providing the display video control unit 1603 with the function of the high resolution processing unit 211 described in the first or second embodiment, in addition to performing coordinate conversion so that the highlight area is centered, It is also possible to increase the resolution of the highlight area. In this case, after reconstructing the area focused on by the user, the area can be provided to the user as a higher resolution video.

  FIG. 18 is a diagram illustrating an example of a processing flow from video input to display according to the third embodiment.

  As shown in FIG. 18, the processing from video input (S801) to highlight score calculation (S801) is the same as the flow of the first embodiment shown in FIG. Next, the video coordinate conversion unit 1602 refers to the highlight score table 601 and generates a display video by performing coordinate conversion on the video to be described later, centering on the divided region having the highest highlight score in a frame corresponding to a certain PTS. In accordance with this method, a display image is generated (S1801). The display unit 206 displays the generated video on the display device 106 (S1802). If the user does not give an instruction to end the process, or if the input video is not finished and the process is not finished (S1803), the video input is continued and the next frame is continued. If there is, the process is terminated.

  FIG. 19 is a diagram for explaining an example of a method for generating a display image by converting the coordinates of the image by the image generation unit 1601.

  The input video 1901 input by the video input unit 201 is acquired by the video acquisition unit 207 as shown in (a), and is recognized by polar coordinates (r, θ) 1902 as shown in (a). A region 1903 indicated by shading 0 <θ <2 / π is a highlight region. Polar coordinates are converted into orthogonal coordinates by equation (1).

The coordinates after conversion in the xy orthogonal coordinates are assumed to be (Xa, Ya). Next, as shown in (b), the highlight area is rotated by 4 / π so that the center of the highlight area overlaps the y-axis. If the coordinates of the image after rotation are (Xb, Yb1904), coordinate conversion by rotation is expressed by equation (2).

Next, as shown in (c), the z-axis is set, and the highlight region is projected onto a hemisphere represented by x ² + y ² + z ² = r ² , r> 0. The coordinates of the image projected on the spherical surface (Xc, Yc, Zc) 1905 are expressed by the equation (3).

The image (Xd, Zd) 1907 generated by orthogonally projecting the image projected on the spherical surface in (c) onto the xz plane 1906 as shown in (d) is used as a display image, and the display unit 206 provides the image to the user. .

  FIG. 20 is an example of a screen for displaying video to the user according to the third embodiment.

  Reference numeral 2001 denotes an example of a video display screen generated by the video generation unit 1601 using the method of generating a video by coordinate conversion described with reference to FIG. As in the screen example 2002, the orthographic projection may be a perspective projection, or as shown in the screen example 2003, a coordinate conversion process may be further performed on the rectangle. Furthermore, high resolution processing may be performed on the video generated centering on the “swelled area”. In addition, an image 2004 indicating a “swelling area” in the input image may be superimposed and displayed. Thereby, if necessary, the direction information in the all-around video is not lost, and it is possible to display the highlight area focused by the user in an easily understandable video with a stereoscopic display.

  According to the third embodiment, for example, a reconstructed image centered on a highlight region, that is, a video region having a video feature that a user should pay attention to, can be displayed with respect to an input image that is an all-around video of a surveillance camera or an in-vehicle camera. . The same effect can be obtained when the video of a TV program or home video is used as the input video.

  The video display device according to the third embodiment of the present invention described above detects a highlight area from an input video and reconstructs and displays a video centering on the highlight area, so that the user can display the video for each PTS. It is possible to provide the user with an image reconstructed around the area to be focused on.

1 is a diagram illustrating an example of a hardware configuration of a video display device according to Embodiment 1. FIG. FIG. 3 is a diagram illustrating an example of a software configuration of the video display apparatus according to the first embodiment. The figure explaining an example of the method of the high resolution process by the high resolution process part. FIG. 3 is a diagram illustrating an example of division of input video according to the first embodiment. FIG. 6 is a diagram illustrating an example of an inter-frame difference video according to the first embodiment. The figure which shows an example of the data structure of a highlight score table. The figure which shows an example of the flow of a calculation process of the highlight score of the division area of an input image | video. The figure which shows an example of the flow which detects a highlight area | region from an input image | video and performs high resolution processing. The figure which shows an example of the image | video which performed the process which changes the resolution in steps so that the resolution may become smooth in the boundary of the combined division area. FIG. 6 is a diagram illustrating an example of a hardware configuration of a video display device according to a second embodiment. FIG. 6 is a diagram illustrating an example of a software configuration of a video display device according to a second embodiment. The figure which shows an example of the data structure of highlight metadata. The figure which shows an example of the highlight area | region designated by highlight metadata. The figure which shows an example of the flow which carries out high resolution processing of the highlight area | region designated by highlight metadata. FIG. 10 is a diagram illustrating an example of an image input method and an input image according to the third embodiment. FIG. 10 is a diagram illustrating an example of a software configuration of a video display device according to a third embodiment. FIG. 10 is a diagram illustrating an example of division of an input video according to the third embodiment. The figure which shows an example of the flow of a process which produces | generates the image | video centering on a highlight area | region from an input image | video. The figure which shows an example of the method of the method of producing | generating the image | video centering on a highlight area | region. The figure which shows an example of the presentation screen of the produced | generated image | video which the display part 206 displays.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 ... Video input device 102 ... Central processing unit 103 ... Instruction input device 104 ... Storage device 105 ... Secondary storage device 106 ... Display device 201 ... Video input unit 202 ... Instruction input unit 203 ... Highlight detection unit 204 ... Highlight score Storage unit 205 ... Video generation unit 206 ... Display unit 207 ... Video acquisition unit 208 ... Video division unit 209 ... Highlight score calculation unit 210 ... Highlight score table generation unit 211 ... High resolution processing unit 212 ... Display video generation unit 901 ... Network I / F
DESCRIPTION OF SYMBOLS 1001 ... Communication apparatus 1002 ... Highlight / metadata analysis part 1003 ... Highlight / metadata storage part 1601 ... Video | video production | generation part 1602 ... Video coordinate conversion part 1603 ... Display video control part

Claims (10)

A video input unit to which video is input;
A video dividing unit that divides the video input to the video input unit into a plurality of regions;
A high resolution processing unit for increasing the resolution of the video input to the video input unit,
The video display device, wherein the high resolution processing unit performs high resolution on a part of the video divided by the video dividing unit in the video input to the video input unit. The video display device according to claim 1,
An importance calculation unit for calculating importance for a plurality of areas divided by the video dividing unit;
The video display device, wherein the high-resolution processing unit varies the resolution of the plurality of regions divided by the video dividing unit according to the importance calculated by the importance calculating unit. The video display device according to claim 2,
An image display device comprising: an image display unit that displays a display image including a plurality of areas having different resolutions by processing of the high resolution processing unit. The video display device according to claim 3,
The video display device is characterized in that the video display unit displays the resolution at a plurality of boundaries that are different in resolution from each other in stages. A video input section for input video;
An analysis unit that analyzes highlight metadata indicating a region of high importance of the input video;
Among the input video, a high-resolution processing unit that makes the video region specified by the highlight metadata analyzed by the analysis unit higher resolution than other video regions,
A video display device comprising: a video display unit that displays a display video including a video region that has been increased in resolution by the high resolution processing unit. The video display device according to claim 5,
The video display unit displays a display video in which the resolution at a boundary between a region whose resolution has been increased by the high resolution processing unit and a region whose resolution has not been increased by the high resolution processing unit is changed stepwise. A video display device characterized by displaying. A video input section for input video;
A storage unit for storing information on a highlight area in the input video;
A video reconstructing unit that coordinates-transforms a video in an area specified by information on a highlight area stored in the storage unit from the input video and reconstructs the video into a new video; Processing equipment. The video processing device according to claim 7,
A video dividing unit that divides the video input to the video input unit into a plurality of regions;
An importance calculation unit for calculating importance for a plurality of areas divided by the video dividing unit;
The video processing apparatus characterized in that the information on the highlight area stored in the storage unit in the storage unit is information on the importance in the plurality of divided areas calculated by the importance calculation unit. The video processing device according to claim 7,
An analysis unit for analyzing highlight metadata indicating a region of high importance of the input video;
The video processing apparatus according to claim 1, wherein the information related to the highlight area stored in the storage unit in the storage unit is information related to the highlight area specified by the highlight metadata analyzed by the analysis unit. The video processing device according to claim 7,
An image processing apparatus, comprising: a resolution enhancement processing unit configured to increase a resolution of an area corresponding to an area specified by information on the highlight area in the image reconstructed by the image reconstruction unit.

Images