JP2009296285A - Video reproducing device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide technique for maintaining reproduction (viewing) of video, etc., of high quality (high resolution) by a video reproducing device, even when a radio wave is weak, etc. <P>SOLUTION: The video reproducing device includes: an input process (S11) for high-resolution video, an input process (S21) for low-resolution video associated with the high-resolution video; a process (S22) for a super-resolution process, an enlarging process, etc., such that the low-resolution video is made to match the high-resolution video; a process (S12) for correcting some broken frame of frames of the high-resolution video, by using frames having been subjected to the processes, when the some frame is broken; and a process (S13) for outputting the normal frames of the high-resolution video and the corrected frame. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a technology of a video playback device having a function of receiving (inputting) or playing back (displaying) a video (moving image) such as a digital broadcast, and particularly relates to a technology for playing back a high resolution video.

  A mobile digital broadcast receiver or the like has a function of receiving and reproducing a high-quality (high resolution) video such as a so-called full segment and a function of receiving and reproducing a low-quality (low resolution) video such as a so-called one-segment.

  Examples of systems in which a user (viewer) views content such as video include wireless terrestrial video transmission / reception systems such as digital terrestrial broadcasting (simultaneous broadcasting), BS and CS broadcasting, mobile phone networks, and wireless LANs, the Internet, etc. Video transmission / reception system using the above information network.

  In a conventional video reproduction device, when a radio wave becomes weak while viewing (reproducing) a high-quality (high resolution) video such as a full segment, the video may be distorted or not displayed correctly. In a video format used in digital broadcasting or the like, for example, a format such as MPEG, for example, if an I frame is missing, a video may not be displayed for a certain period, or a hole may be opened in a macro block unit.

  For this reason, there is a conventional method of correcting using a past frame when a part or all of a video frame is missing. With this method, when there is a motion in the video or when the screen is switched, the video is distorted and becomes defective.

Further, as an image processing technique for obtaining a high-quality video, for example, there is a technique of super-resolution processing (also referred to as super-resolution display or simply super-resolution). Such a technique is described, for example, in JP-A-8-336046 (Patent Document 1).
JP-A-8-336046 JP 2003-274302 A

  As a conventional technique for solving the video disturbance in the video playback device, for example, when the radio wave is weakened, the video is switched to a low image quality (low resolution) video such as 1Seg and used by super-resolution processing. In such a case, the output can be continued (Japanese Patent Laid-Open No. 2003-274302 (Patent Document 2)).

  However, when a low-resolution video such as one segment is output after super-resolution processing, it is difficult to acquire a high-quality (high-resolution) video such as a full segment.

  The present invention has been made in view of the above problems, and its main purpose is to reproduce (view) high-quality (high-resolution) video and the like even when the radio wave is weak in the video reproduction device. It is to provide technology that can be maintained.

  Of the inventions disclosed in the present application, the outline of typical ones will be briefly described as follows. In order to achieve the above object, a typical embodiment of the present invention is a technology such as a video playback apparatus having a function of playing back video or the like of digital broadcasts, and has the following configuration. And

  In this embodiment, high resolution (relatively high quality) video (first data) such as the full segment is received (input) and reproduced (output), but at that time, radio reception radio waves are weakened. If some frames of the high-resolution video (first data) (the entire frame or a part of the frame) are damaged or missing due to temporarily disturbed video or missing packets in the communication network, etc. Deal as follows. In other words, in this embodiment, a part of the frame of the first data is converted using a low-resolution (relatively low quality) video (second data) such as the one-segment associated with the first data. Correct (complement) and output (play). For example, the second data is subjected to predetermined processing such as super-resolution processing or enlargement processing, and the frame of the damaged / missed first data (damaged / missed portion) using the processed second data. Are corrected and supplemented by overwriting (fitting). Further, the first data frame that is not damaged or missing is continuously output (reproduced). As a result, high-quality (high-resolution) video output (viewing by the user) can be maintained.

  In the present embodiment, for example, a first input unit that inputs a high-resolution video, a second input unit that inputs a low-resolution video associated with the high-resolution video, and a damaged or missing frame of the high-resolution video A determination unit that determines a range of the image, a first processing unit that performs a first process so that a frame of the low-resolution video is matched with a frame of the high-resolution video, and a part of the frame in the high-resolution video is damaged Or a correction unit that corrects or complements the frame using the frame on which the first processing has been performed when the frame is missing, and a normal frame and the corrected frame in the high-resolution video in the time direction. And an output unit for outputting in order.

  The present embodiment further compares the storage unit that stores the past frame of the high-resolution video, the past frame of the high-resolution video, and the frame after the first processing of the low-resolution video, A motion range detection unit that detects a motion range between frames, and the correction unit uses the past frame for a portion that does not move in the frame according to a result of the motion range detection unit. The portion with motion is corrected or complemented using the frame after the first processing.

  Among the inventions disclosed in the present application, effects obtained by typical ones will be briefly described as follows. According to a typical embodiment of the present invention, in a video playback device, playback (viewing) of a high quality (high resolution) video or the like can be maintained even when radio waves are weak.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that components having the same function are denoted by the same reference symbols throughout the drawings for describing the embodiment, and the repetitive description thereof will be omitted. In the following description, the case of video will be described, and processing such as audio will be omitted.

<Conventional technology>
The prior art as a premise will be briefly described with reference to FIG. Referring to FIG. 10, in the conventional technique, for example, in a high-resolution MPEG format video frame (picture) group (including I frame, P frame, and B frame), if a frame is damaged or missing, it cannot be displayed correctly. For example, in a damaged frame, a loss (indicated by a black rectangle) occurs in units of macroblocks in the frame. In particular, when an I frame is damaged or missing, the video remains broken for a certain period. Here, when a part of a past frame is used (fitted) with respect to a damaged / missed part in the conventional method, there is a problem when there is movement in the video or when the video is switched. . That is, the user cannot perform natural display / viewing.

  In the inter-frame prediction method for moving images, the I frame is a frame that is encoded without using inter-frame prediction, the P frame is a frame that is encoded using only forward prediction, and the B frame is forward prediction. It is a frame encoded by either backward prediction or bidirectional prediction.

  The damage / missing of a frame in a high-resolution video such as a full segment may be the entire frame or a part of the frame (such as a macroblock unit). Further, the frame may be normal before and after the damaged / missed frame. However, when switching from a high-resolution video to a low-resolution video by the conventional method, it is not used even if a normal frame is acquired.

<Feature configuration>
In FIG. 1, a characteristic configuration in the present embodiment (Embodiment 1 or the like) is simply shown. A high-resolution video (first data: D1) and a low-resolution video (second data: D2) that are associated with each other are prepared and input to the apparatus (S11, S21). When high-resolution video (D1) can be correctly displayed (decoded, etc.) (normal frame: F1), and when it cannot be correctly displayed (decoded, etc.) due to frame breakage / missing (macroblock missing, etc.) (damaged frame: F2) There is. If it can be displayed correctly, the normal frame (F1) can be output as it is (S13). On the other hand, if the display cannot be performed correctly, the low-resolution video (D2) associated with the high-resolution video (D1) is subjected to super-resolution processing or enlargement processing (S22) with respect to the damaged frame (F2). The correction / complement processing is used (S12).

  In super-resolution processing or enlargement processing (S22), for example, the corresponding normal frame (F3) in the low-resolution video (D2) is matched with the size of the corresponding damaged frame (F2) in the high-resolution video (D1). Thus, super-resolution processing or enlargement processing is performed. For this process, a predetermined process which is a known technique can be used. In the correction / complementation process (S12) using the post-processing frame (F4), for example, the corresponding part of the post-processing frame (F4) is overwritten on the missing part (macroblock or the like) of the damaged frame (F2) ( The modified frame (F5) is obtained by processing such as fitting. Then, using the corrected frame (F5) and the normal frame (F1), high-quality (high-resolution) video is played back (S13) and played back.

  Examples of systems to which the user can view content such as video include, for example, the above-mentioned digital terrestrial broadcasting (simultaneous broadcasting), a video transmission / reception system using radio, and a video transmission / reception system using an information network. .

(Embodiment 1)
The video playback apparatus according to the first embodiment of the present invention will be described with reference to FIG. In the first embodiment, super-resolution processing is performed as predetermined processing on the low-resolution video side, and a past frame is not used for correction, and correction is performed by specifying a correction range from the control unit.

  FIG. 2 shows a block configuration of the video reproduction apparatus according to the first embodiment. The video playback apparatus is a mobile device having a digital broadcast receiving function, for example.

  This video reproduction apparatus includes a control unit 10 (including a correction range determination unit 19), a high-resolution video input unit 11, a low-resolution video input unit 12, a high-resolution video decoding unit 13, a low-resolution video decoding unit 14, and a switch (switching). Part) 15, a super-resolution processing part 16, a correction part (damaged video correction / complementation part) 17, a video output part 18 and the like. In addition, the apparatus may include a built-in or connected IC such as a processor (CPU) and a memory, an input device, an output device (such as the display device 20), an external storage device, and a communication interface device.

  Each unit in the apparatus configuration is realized by, for example, a dedicated circuit unit or a software module. This configuration satisfies real-time characteristics (sufficiently high processing speed) for video output and the like. For example, in the case of a dedicated circuit configuration, the super-resolution processing unit 16 or the like is mounted with a dedicated circuit, and a memory or a buffer is incorporated in each circuit unit as necessary. For example, in the case of a program control configuration, the super-resolution processing unit 16 and the like are realized as processing for data on the memory by executing a control program on the memory by the processor.

  The control unit 10 performs control related to this feature, and the correction range determination unit 19 determines a video correction range and the like.

  The high resolution video input unit 11 inputs a high resolution video (first data: D1). The low resolution video input unit 12 inputs a low resolution video (second data: D2). For example, the high-resolution video (D1) and the low-resolution video (D2) are video data such as MPEG, the high-resolution video (D1) is a full-segment video program, and the low-resolution video (D2) is a corresponding one-segment video program. It is.

  The high resolution video decoding unit 13 decodes the input high resolution video (first data: D1). The low resolution video decoding unit 14 decodes the input low resolution video (second data: D2). The decoding process includes, for example, a process for generating the above-described MPEG format I frame, P frame, B frame, and the like. The data (d1) decoded by the high resolution video decoding unit 13 is input to the switch 15. The data (d2) decoded by the low resolution video decoding unit 14 is input to the super resolution processing unit 16.

  The high-resolution decoding unit 13 and the low-resolution decoding unit 14 can know the state of breakage / missing of the frame by looking at the decoding processing state of itself. Each of the high-resolution decoding unit 13 and the low-resolution decoding unit 14 notifies the correction range determination unit 19 of the control unit 10 when it detects damage / missing of a frame (c1).

  Since the high resolution decoding unit 13 sees the state of the decoding process by itself, the high resolution decoding unit 13 can know whether the frame is damaged or missing, so the high resolution decoding unit 13 sends the frame (d1) to the switch 15 or the normal state. In response to this, a switching instruction (c2) is input.

  From the switch 15, the normal frame data (d3), which is one of the outputs, is input to the video output unit 18 in accordance with the switching control (c2). Further, the data (d4) of the damaged / missing frame, which is the other output, is input to the correcting unit 17.

  On the other hand, the super-resolution processing unit 16 performs super-resolution processing on the data (d2) decoded by the low-resolution decoding unit 14. The super-resolved data (d5) is input to the correction unit 17. A known technique can be used for the super-resolution processing.

  The correction unit 17 performs correction (restoration) / complement processing on the damaged / missing frame data (d4) using the super-resolution data (d5). The modified frame data (d6) thus created is input to the video output unit 18. The corrected frame data (d6) is input to the high resolution decoding unit 13 as a past frame, and is used when the high resolution decoding unit 13 performs a decoding process using the past frame. Note that correction (restoration) / complementation is to obtain a frame of sufficiently high quality for viewing without having to acquire the same frame as the original high-resolution frame.

  Although not shown, the correction unit 17 or the like incorporates a memory (or buffer), temporarily stores frame data in the memory, and performs predetermined processing on the memory. Further, for example, a memory (or buffer) for accumulating data (d3) on the normal frame side and matching the timing with the low resolution side may be provided in the front stage of the video output unit 18 or the like.

  The video output unit 18 sequentially outputs a group of frames of the output video data to the display device 20 (for example, a liquid crystal display) in time series using the normal frame (d3) or the corrected frame (d6).

  The correction range determination unit 19 determines a video correction range based on the notification (c1) of the breakage / missing from the high resolution decoding unit 13 and the low resolution decoding unit 14, and instructs the correction unit to the correction range (c3). ). This correction range (c3) is specified, for example, in units of macroblocks in the frame.

  In accordance with the correction range instruction (c3), the correction unit 17 adds the corresponding range in the data (d5) of the super-resolved frame to the specified correction range in the damaged / missing frame data (d4). The process of combining the data at the same position (such as fitting by overwriting) is performed.

  The outline of basic control and the flow of processing procedures are as follows. First, the video (d2) output from the low resolution decoding unit 14 is super-resolved by the super-resolution processing unit 16. Since the low resolution video data can be processed faster than the high resolution video data, the time difference is utilized. Secondly, when the frame (d1) of the high-resolution video is damaged or missing, the correction range determination unit 19 instructs the correction unit 17 of a portion (correction range) that cannot be correctly displayed (c3). Then, the damaged frame (d4) is corrected and complemented by combining with the super-resolution processed data (d5), and the corrected frame (d6) is output. Third, a normal frame (d3) that is not damaged or missing in a high-resolution frame is output as it is. Thereby, high-quality reproduction is maintained.

  With the configuration of the first embodiment as described above, high-quality video playback (viewing) is maintained even in the case where deterioration of high-resolution video due to deterioration of radio wave conditions occurs in a video playback device such as a mobile digital broadcast receiver. can do.

  In the above description, the super-resolution processing unit 16 may be replaced with an enlargement processing unit. The switch 15 may be switched from the control unit 10. Moreover, it is good also as a form which controls to the high-resolution decoding part 13 center by providing the function of the control part 10 in the high-resolution decoding part 13. FIG. Alternatively, the controller 10 (correction range determination unit 19) may notify the correction unit 17 of the damage / missing range instead of the correction range, and the correction unit 17 may determine the correction range. For example, the degree of damage / missing of a video differs depending on whether the MPEG I, P, or B frame is damaged or missing, but the present invention is applicable to any case.

<Packet loss in communication network>
Next, FIG. 3 shows the relationship between the packet loss in the communication network and the configuration of the present invention (this embodiment).

  In FIG. 3A, a packet constituting a high-resolution video frame f1 from a packet output means (or transmission means) 201 to a packet input means (or reception means) 202 on a communication network 203 (path, link, etc.). It is assumed that a data group (packet p1 or the like) is flowed. The communication network 203 (route, link, etc.) has a relatively high bit rate and low reliability. A packet loss occurs due to disturbance 204 on the communication network 203 (for example, packet p2). In this case, on the packet input means 202 side (video reproduction side), the macroblock of the packet loss portion (p2) is missing in the received frame f1 '(corresponding to the above-described damaged frame (F2)).

  On the other hand, in FIG. 3B, on the communication network 207 having the same or similar characteristics, from the packet output unit 205 to the packet input unit 206, the low-resolution video related to the same video content as the high-resolution video in FIG. Assume that a packet data group constituting the frame f2 is flowed. The communication network 207 (route, link, etc.) has a relatively low bit rate and high reliability. With respect to the disturbance 208 (similar to the disturbance 204) on the communication network 207, packet loss or the like hardly occurs on this side due to high reliability. When no packet loss or the like occurs, no macroblock is lost in the received frame f2 'on the packet input means 206 side (corresponding to the normal frame (F3) on the low resolution side).

  In the present embodiment, the input units (11, 12) receive video data (D1, D2) from a communication network, for example. In the present embodiment, as described above, even if some data is damaged or missing on the high resolution side using the input of the two systems of high and low data, the above-described figure As shown in FIG. 1, the image data can be corrected and displayed using the video data on the low resolution side.

(Embodiment 2)
Next, the video playback apparatus according to the second embodiment of the present invention will be described with reference to FIGS. In the second embodiment, super-resolution processing is performed as predetermined processing on the low resolution side, and a past frame is selectively used for correction, and a correction range or past frame is specified from the control unit. Correct it. At the time of correction, a past frame is used for a portion where there is no movement between frames. Embodiment 2 shows an example of macroblock processing.

  FIG. 4 shows a basic configuration of the second embodiment. The high-resolution video input unit 301 inputs a frame of the high-resolution video (D1). In this frame (f11), for example, the data (macroblock) immediately after the slice data in the P frame or B frame is damaged. The frame of the high resolution video (D1) (when there is damage) (d11) is input to the correction unit 305.

  The low-resolution video input unit 302 inputs a frame of the low-resolution video (D2). In this frame (f21), for example, it is assumed that there is no damage. As described above, the frame (d12) of the low-resolution video (D2) is processed by the super-resolution processing unit 304, for example, and the processed frame (d15) is input to the motion range detection unit 306.

  The frame storage unit 303 stores at least one previous (previous) frame with respect to the current frame. For example, the frame storage unit 303 inputs and stores the frame (d13) from the high-resolution video input unit 301. In this frame (f12), for example, it is assumed that there is no breakage / missing. The past frame (d14) is input from the frame storage unit 303 to the motion range detection unit 306.

  The motion range detection unit 306 compares the super-resolution frame (d15) with the past frame (d14), and detects a motion range between the frames. This detection is based on a known motion detection process using a difference comparison between frames. As a result of the comparison and detection, if the video in the damaged range (location) in the frame is almost the same between frames (there is almost no motion), the correction unit 305 corrects / complements using the data of the past frame (d13). Process. For example, the past frame (d13) is input from the frame storage unit 303 (or the motion range detection unit 306) to the correction unit 305, and is processed according to the instruction (d16) from the motion range detection unit 306. In addition, when the image of the damaged range (location) in the frame has a motion (difference) between the frames by the comparison and detection, the correction unit 305 performs the super-resolution processing of the frame (d15). Perform correction / complementation processing using data. For example, a super-resolution processed frame (d15) from the super-resolution processing unit 304 (or the motion range detection unit 306) is input to the correction unit 305 and processed according to an instruction (d16) from the motion range detection unit 306. The video output unit 307 performs output processing using the corrected high-resolution video frame (f13) in the data (d17) from the correction unit 305. The corrected high-resolution video frame (f13) is corrected at least than the high-resolution video frame (f11) at the time of input, and the quality is improved.

  FIG. 5 shows a more detailed configuration of the configuration of the second embodiment shown in FIG. As a part different from the configuration of the first embodiment in FIG. 2, the past frame memory 21 corresponding to the frame storage unit 303 and the motion detection unit 22 corresponding to the motion range detection unit 306 are provided.

  The past frame memory 21 stores the frame data (past frame) from the high resolution video decoding unit 13 and outputs the frame data to the correction unit 17 and the motion detection unit 22. The motion detection unit 22 receives the data (d6) from the super-resolution processing unit 16 and the data (d7) from the past frame memory 21, detects the motion range between the frames, and according to the result The control instruction (c4) described above is input to the correction unit 17. In accordance with the control instruction (c4), the correction unit 17 uses the data (d7) of the past frame (previous frame) when there is no motion in the part of the frame, and performs super-resolution processing when there is motion. Correction processing is performed using the frame (d5).

  According to the second embodiment, the processing can be made efficient by using the past frame.

  In the macroblock processing example of the second embodiment, a mode in which a frame of a high-resolution video (D1) is reduced, the frame and a frame of a low-resolution video are compared, and motion detection / determination is performed. Is possible.

(Embodiment 3)
Next, the video playback apparatus (and system) according to Embodiment 3 of the present invention will be described with reference to FIGS. In the third embodiment, the reception state or the radio wave state in the wireless reception unit as input means is further determined and controlled accordingly.

  6 and 7 show the apparatus and system configuration of the third embodiment. FIG. 6 shows a system on the video distribution (transmission) side, and FIG. 7 shows a system on the video reception side (this video reproduction apparatus). Specifically, as an example, the distribution (transmission) side system is a wireless monitoring camera device, and the reception side system is the monitor device, and the video captured by the wireless monitoring camera device is displayed on the monitoring device.

  First, in the video distribution (transmission) system shown in FIG. 6, high-resolution video and low-resolution video are distributed (transmitted) through a wireless communication network. The video input unit 400 inputs the original video or video data (D0). The high resolution encoding unit 401 encodes the input video (D0) at a high resolution to obtain high resolution video data (corresponding to D1). Similarly, the low resolution encoding unit 403 encodes the input video (D0) with low resolution to obtain low resolution video data (corresponding to D2). The high-speed wireless transmission unit 402 performs high-speed wireless transmission processing on high-resolution video data (corresponding to D1). This transmission output corresponds to an output on the communication network 203 (high bit rate, low reliability). The low-speed wireless transmission unit 404 performs low-speed wireless transmission processing on the low-resolution video data (corresponding to D2). This transmission output corresponds to an output on the communication network 207 (low bit rate, high reliability).

  On the other hand, in the video receiving system (this video playback apparatus) shown in FIG. 7, the high resolution video (D1) and the low resolution video (D2) are received from the distribution system via the wireless communication network. The apparatus includes, for example, a wireless IC unit 30, and the wireless IC unit 30 includes a high-speed wireless receiving unit 31 and a low-speed wireless receiving unit 32. The wireless IC unit 30 receives, for example, a digital data broadcast stream. In addition, the control unit 10 of the present apparatus includes a wireless stability determination unit 33. The high-speed wireless reception unit 31 receives a high-resolution video (D1) such as a full-segment video by wireless. The low-speed wireless reception unit 32 receives a low-resolution video (D2) such as a one-segment video by wireless. Each received data (d31, d32) is input to the above-described decoding unit (13, 14).

  The wireless stability determination unit 33 determines the wireless reception state and stability in the high-speed wireless reception unit 31 based on the radio wave or signal strength (c5). The wireless stability determination unit 33 may be provided in the wireless IC unit 30, the high-speed wireless reception unit 31, and the like.

  In this control, first, if it is determined by the determination by the wireless stability determination unit 33 that the signal received by the high-speed wireless reception unit 31 is strong and stable wireless reception is possible, the wireless stability determination is performed. This control (function) is turned off from the unit 33 to each part related to processing using the low-resolution video (D2), that is, the low-resolution video decoding unit 14, the super-resolution processing unit 16, the correction unit 17, and the like. The instruction (c6) of “pause” is input. Accordingly, each part (14, 16, 17) is turned off. Thereby, the effect of a power saving is acquired.

  Second, when it is determined by the determination by the wireless stability determination unit 33 that the strength of the signal received by the high-speed wireless reception unit 31 is weak and stable wireless reception has become difficult (for example, the room door is closed). In some cases, such as rain or fog, the video data may be damaged. Therefore, the wireless stability determination unit 33 instructs each part (14, 16, 17) to turn on (restart) control (function). Enter (c6). Accordingly, each part (14, 16, 17) is turned on.

(Embodiment 4)
Next, a video playback device (and system) according to Embodiment 4 of the present invention will be described with reference to FIGS. In the fourth embodiment, the data exchange and buffer status in the network interface unit, which is an input means, are monitored and judged, and the reception status of high and low resolution video data is controlled and adjusted accordingly. is there. In the fourth embodiment, high and low resolution videos (D1, D2) are received simultaneously on the network.

  8 and 9, the apparatus and system configuration of the fourth embodiment are shown. FIG. 8 shows a system on the video content distribution (transmission) side, and FIG. 9 shows a system including the video content reception side and the reception side. Specifically, as an example, a video content distribution / reception system on the Internet, the distribution (transmission) side is a server device 81 connected to the Internet 503, and the reception side is a user device 82 (PC, network TV, mobile device). Etc.).

  The distribution-side server device 81 includes a high-resolution content distribution unit 501 and a low-resolution content distribution unit 502, and distributes (transmits) the corresponding video content (D1, D2) through the Internet 503 or the like. The high-resolution content distribution unit 501 and the low-resolution content distribution unit 502 may distribute from individual server devices.

  The content receiving unit 504 of the user device 82 accesses the video content of the server device 81 through the Internet 503 or the like, and acquires a high resolution video (D1) and a low resolution video (D2). The content receiving unit 504 is, for example, a network interface (I / F) unit and a content receiving program. The content receiving unit 504 receives video content (D1, D2) having high and low resolutions, for example, simultaneously in parallel. In addition, when an access request is made from the user device 82 to the high-resolution content distribution unit 501 of the server device 81, the high-resolution video (D1) is not only distributed (transmitted) from the high-resolution content distribution unit 501, but is associated with it. The low resolution content distribution unit 502 may automatically distribute (transmit) the low resolution video (D2).

  On the Internet 503, for example, a packet constituting a high resolution video (D1) is transmitted using a UDP protocol (relatively high speed and low reliability). Further, a packet constituting the low resolution video (D2) is transmitted using the TCP protocol (relatively low speed and high reliability). As a result, as shown in FIG. 3 described above, the difference in characteristics in the input (reception) of data of each resolution becomes large.

  On the other hand, in the video content receiving system shown in FIG. 9 (this video playback apparatus as the user device 82), the high resolution video (D1) data and the low resolution video (D2) are transmitted from the distribution system via the Internet 503. ) Data. The apparatus includes, for example, a network I / F unit 40, and the network I / F unit 40 includes a high-speed network reception unit 41, a low-speed network reception unit 42, and a network transmission unit 47. For example, the network I / F unit 40 transmits and receives a packet group constituting the content data using a predetermined protocol. In addition, the apparatus includes buffers such as a synchronization input buffer 43, an input buffer 44, and an output buffer 45. In addition, the control unit 10 of the present apparatus includes a buffer capacity monitoring unit 46, a content confirmation unit 48, an effect setting unit 49, and the like. Each buffer absorbs and adjusts the difference in processing timing in high and low resolution data.

  The high-speed network receiver 41 receives high-resolution video (D1) data through the high-speed network I / F. The low-speed network receiving unit 42 receives low-resolution video (D2) data through the low-speed network I / F. The received data (d41, d42) are input to the decoding unit (13, 14) via the buffer (43, 44).

  The received high resolution video (D1) data (d41) is temporarily stored in the synchronization input buffer 43. Data (d43) in the synchronization input buffer 43 is input to the high-resolution video decoding unit 13. The received data (d42) of the low-resolution video (D2) is temporarily stored in the input buffer 44. The data (d44) in the input buffer 44 is input to the low resolution video decoding unit 14. The data (d2) from the low resolution video decoding unit 14 is temporarily stored in the output buffer 45. Data (d45) in the output buffer 45 is input to the super-resolution processing unit 15.

  In this control, first, the low resolution video decoding unit 14 sends information (c13) of the frame being processed (current frame) to the synchronization input buffer 34, thereby synchronizing video at high and low resolutions. Take.

  Secondly, the buffer capacity monitoring unit 46 confirms the capacity (frame data storage amount) of the buffers (43, 44) on the input side at high and low levels. Using the transmitting unit 47, the server device 81 on the content distribution side is notified. This adjusts the amount of data received at high and low, thereby adjusting or synchronizing the timing of processing at high and low.

  Third, the low resolution video decoding unit 14 decodes the low resolution video (D2, d44) and stores the data (d2) in the output buffer 45. When the high-resolution video (D1) is damaged or missing, the data in the output buffer 45 is sent to the correction unit 17 via the super-resolution processing unit 16. The output buffer 45 can also be provided at the subsequent stage of the super-resolution processing unit 16.

  Fourth, the low resolution video decoding unit 14 performs the decoding process a little earlier than the high resolution video decoding unit 13. Therefore, the content of the video content can be confirmed by referring to the low resolution video (c15) by the content confirmation unit 48 before the high resolution video (D1) is output using the time margin. Furthermore, based on the information (c16) from the content confirmation unit 48, the effect setting unit 49 sets the video effect (acoustic effect, etc.) to the video output unit 18 earlier, thereby providing suitable video / audio and the like. Output.

  In the fourth process, in particular, on the low resolution side, the low resolution video (D2) data is received and accumulated in the input buffer 44 ahead of the high resolution side in time (accordingly, the time margin described above). Is secured). Even in the case where the preceding reception is not performed, since it is low resolution video (D2) data, the data amount and the processing amount are smaller than those on the high resolution side, and network reception tends to be faster. There is room for decoding processing a little earlier, and the same processing can be performed thereafter.

  From the effect setting unit 49, for example, instruction control (c17) such as data processing for effect setting is performed on the video data of the video output unit 18.

  In the content confirmation unit 48, as content confirmation, for example, luminance is determined, and video processing control (c17) such as contrast adjustment is executed for the video effect set from the effect setting unit 49 according to the luminance state.

  In the third and fourth embodiments, the reception of the high- and low-resolution images (D1, D2) with respect to the wireless IC unit 30, the communication network I / F unit 40, and the like is input to two systems (multiple systems). Not only the form performed individually by the unit (receiver), but also a form collectively performed by one input unit (receiver) is possible. A form in which the wireless IC unit 30 and the network I / F unit 40 are individualized according to high / low resolution processing is also possible. Also, on the time axis, a form of simultaneous reception of high and low resolution images (D1, D2), a form of receiving a low side video a little earlier than the high side, for example, low side, high It is also possible to adopt a form in which reception is continued in the order of the side, or a form in which the low side is received in advance and used for reception on the high side.

<Super-resolution processing>
In FIG. 11, as a supplement, the concept of super-resolution processing (known technology) by the super-resolution processing unit 16 is simply shown. Super-resolution processing uses a motion between a plurality of frames (images) of a video to generate a frame (image) having a resolution higher than the original resolution image (for example, a resolution in which the length and width are doubled). It is a process to generate.

  The input image (video) 900 has a plurality of frames (images) continuous in the time (t) direction. The input image 900 has a subject (moving part). A plurality of frames to be processed are extracted from the input image 900 (A). For example, a first frame 901 and a second frame 902 having a first resolution are included. A square or triangular point in the frame indicates a pixel. With respect to the moving part (subject) between these frames, alignment processing (B) is performed (frame 903 after processing). Here, motion vector information between frames is used. Then, a high-definition process (C) such as pixel complementation is performed (processed frame 904). Here, high definition is realized by utilizing the positional deviation of the pixels. If there is no movement, there is no effect. The frame 904 after this processing has a higher resolution (second resolution) than the original first resolution frame (such as 901). By arranging such frames 904 in the time (t) direction (D), a super-resolution image (video) 905 having a higher resolution than the original input image 900 is obtained.

  The enlargement process (known technique) uses a nearest neighbor method, a linear interpolation method, or the like.

  As mentioned above, the invention made by the present inventor has been specifically described based on the embodiment. However, the present invention is not limited to the embodiment, and various modifications can be made without departing from the scope of the invention. Needless to say.

  The present invention can be used for various devices having a video reproduction function, such as a display device, a receiver, and an information processing device, and a system for broadcasting and communication.

It is a figure which shows the characteristic structure in the video reproduction apparatus in one embodiment of this invention, a typical process, an example of a video frame. It is a figure which shows the block configuration of the principal part in the video reproduction apparatus in Embodiment 1 of this invention. (A). (B) is a diagram showing packet loss in a communication network in relation to the configuration of the present embodiment, (a) is a packet input on the high resolution video side, and (b) is a packet on the low resolution video side. Indicates packet input. It is a figure which shows the basic composition in the video reproduction apparatus in Embodiment 2 of this invention. It is a figure which shows the block structure of the principal part in the video reproduction apparatus in Embodiment 2 of this invention. It is a figure which shows the structure of the system by the side of the delivery in the video reproduction apparatus and system in Embodiment 3 of this invention. It is a figure which shows the block structure of the principal part of the system of the receiving side in the video reproduction apparatus and system in Embodiment 3 of this invention, and a video reproduction apparatus. It is a figure which shows the structure of the system of the delivery side in the video reproduction apparatus and system in Embodiment 4 of this invention, and a receiving side. It is a figure which shows the block structure of the principal part of the system of the receiving side in the video reproduction apparatus and system in Embodiment 4 of this invention, and a video reproduction apparatus. It is a figure shown about the case of reproduction | regeneration of the high-resolution image | video of an MPEG format, and frame corruption as a prior art used as a premise with respect to this Embodiment. It is a figure shown about the outline | summary of a super-resolution process as a prior art used as a premise with respect to this Embodiment.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 10 ... Control part, 11, 301 ... High resolution video input part, 12, 302 ... Low resolution video input part, 13 ... High resolution video decoding part, 14 ... Low resolution video decoding part, 15 ... Switch, 16, 304 ... Super Resolution processing unit, 17, 305 ... correction unit (damaged video correction / complementation unit), 18, 307 ... video output unit, 19 ... correction range determination unit, 20 ... display device, 21 ... past frame memory, 22 ... motion detection , 30... Wireless IC unit, 31... High speed wireless reception unit, 32... Low speed wireless reception unit, 33 .. wireless stability determination unit, 40... Network I / F unit, 41. 43 ... Synchronization input buffer, 44 ... Input buffer, 45 ... Output buffer, 46 ... Buffer capacity monitoring unit, 47 ... Network transmission unit, 48 ... Content confirmation unit, 49 ... Effect setting 81, server device, 82, user device, 201, 205 ... packet output means, 202, 206 ... packet input means, 203, 207 ... communication network, 204, 208 ... disturbance, 303 ... frame storage unit, 306 ... motion Range detection unit, 400 ... Video input unit, 401 ... High resolution encoding unit, 402 ... High speed wireless transmission unit, 403 ... Low resolution encoding unit, 404 ... Low speed wireless transmission unit, 501 ... High resolution content distribution unit, 502 ... Low resolution Content distribution unit, 503... Internet, 504. Content reception unit, 900... Input image (video), 901 to 904... Frame, 905 ... super-resolution image (video), D1.

Claims (11)

A first input unit that inputs a high-resolution video; a second input unit that inputs a low-resolution video associated with the high-resolution video;
A determination unit for determining a range of damage or missing of the frame of the high-resolution video;
A first processing unit that performs a first process so as to match the frame of the low-resolution video with the frame of the high-resolution video;
A correction unit that corrects or complements the frame using the frame subjected to the first processing when a part of the frame in the high-resolution video is damaged or missing;
An image reproduction apparatus comprising: an output unit that sequentially outputs a normal frame and the corrected frame in the high-resolution image in a time direction. The video playback device according to claim 1,
A storage unit for storing past frames of the high-resolution video;
A motion range detecting unit that compares a past frame of the high-resolution video with a frame after the first processing of the low-resolution video and detects a motion range between the frames;
In the correction unit, according to the result in the motion range detection unit, the past frame is used for a portion where there is no motion in the frame, and the frame after the first processing is used for a portion where there is motion, A video reproducing apparatus characterized by correcting or complementing. The video reproduction apparatus according to claim 1 or 2,
The video reproduction apparatus characterized in that the first process is a super-resolution process of the frame of the low resolution video. The video reproduction apparatus according to claim 1 or 2,
The video playback apparatus according to claim 1, wherein the first process is an enlargement process of the frame of the low resolution video. In the video reproduction device according to any one of claims 1 to 4,
A first decoding unit that decodes the input high-resolution video; and a second decoding unit that decodes the input low-resolution video;
The determination unit determines a range of damage or missing of the decoded high-resolution video frame and indicates a correction range,
The first processing unit performs a first process so as to match the frame of the decoded low-resolution video with the high-resolution video,
In a case where a part of a frame in the decoded high-resolution video is damaged or missing, the correction unit follows the instruction of the correction range and applies the first or second part to the damaged or missing part of the frame. A video reproduction device characterized in that a corresponding portion of a frame subjected to processing is synthesized to correct or complement the frame. In the video reproduction device according to any one of claims 1 to 5,
The determination unit determines a range of damage or loss in units of blocks in the frame,
The video reproducing apparatus, wherein the correction unit corrects or supplements a damaged or missing block in the frame using a block in the frame on which the first processing has been performed. The video playback device according to claim 1,
The first input unit includes a first wireless reception unit that wirelessly receives the high-resolution video,
The second input unit includes a second wireless reception unit that wirelessly receives the low-resolution video,
A wireless determination unit for determining a state or stability of wireless reception of the high-resolution video by the first wireless reception unit;
The operation of the second input unit, the first processing unit, and the correction unit is switched on and off according to the wireless reception state or the stability determination result in the wireless determination unit. Video playback device. The video playback device according to claim 1,
The first input unit includes a first network receiving unit that receives and processes the high-resolution video data through a communication network;
The second input unit includes a second network receiving unit that receives and processes the low-resolution video data through a communication network;
A network transmission unit for transmitting information to a device on the delivery side of the data of the high resolution video or the low resolution video through a communication network;
A first buffer for storing data received by the first network receiver;
A second buffer for storing data received by the second network receiver;
A monitoring unit for monitoring the data capacity of the first buffer and the second buffer;
In accordance with the monitoring result of the data capacity in the monitoring unit, the network transmission unit is used to adjust the reception timing of the high-resolution video data and the low-resolution video data to the distribution-side device. For reproducing the information for adjusting the timing of processing of the high-resolution video and the low-resolution video. The video reproduction device according to claim 7 or 8,
The video reproducing apparatus, wherein the first and second input units input the high resolution video and the low resolution video simultaneously in parallel. The video reproduction apparatus according to claim 8, wherein
The first and second input units receive the low-resolution video data before the high-resolution video data, and perform the first processing on the low-resolution video data early. A video reproducing apparatus characterized by comprising: The video reproduction apparatus according to claim 8, wherein
A video playback apparatus characterized in that the content is confirmed by referring to the data of the low-resolution video, and accordingly, processing for setting a predetermined video effect is performed on the video output from the output unit.

Images