WO2007023440A2 - Video processing apparatus - Google Patents

Video processing apparatus Download PDF

Info

Publication number
WO2007023440A2
WO2007023440A2 PCT/IB2006/052876 IB2006052876W WO2007023440A2 WO 2007023440 A2 WO2007023440 A2 WO 2007023440A2 IB 2006052876 W IB2006052876 W IB 2006052876W WO 2007023440 A2 WO2007023440 A2 WO 2007023440A2
Authority
WO
WIPO (PCT)
Prior art keywords
image sequence
video signal
processing
temporal image
video
Prior art date
Application number
PCT/IB2006/052876
Other languages
French (fr)
Other versions
WO2007023440A3 (en
Inventor
Stephan O. Mietens
Original Assignee
Koninklijke Philips Electronics N.V.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Koninklijke Philips Electronics N.V. filed Critical Koninklijke Philips Electronics N.V.
Publication of WO2007023440A2 publication Critical patent/WO2007023440A2/en
Publication of WO2007023440A3 publication Critical patent/WO2007023440A3/en

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/01—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level
    • H04N7/0125—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level one of the standards being a high definition standard
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/01—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level

Definitions

  • the invention relates to a video processing apparatus for processing a video signal comprising at least one temporal image sequence.
  • the invention further relates to a method of processing a video signal comprising at least one temporal image sequence and a computer program product for executing such a method.
  • a video processing apparatus is arranged to receive a video signal transmitted through a wired or wireless medium, store, process and deliver the processed video signal to another apparatus for example, to a display or a storage or a further transmission means.
  • a video signal comprises a plurality of temporal sequences of image frames. Processing functions such as noise removal, stability improvement, de- interlacing, spatial up-conversion, temporal up-conversion, subjective quality improvement and various other enhancements and modifications can be applied on the video signal.
  • Such a video processing apparatus can be implemented on hardware and/or a software platform at a consumer location or at a video re-distribution location. Processing resources available on such platforms are generally subject to cost and size considerations.
  • some of the enhancements of the video signal can be preferably carried out off-line in order to achieve high-quality video. Otherwise, real-time algorithms for video enhancement can be applied, wherein the obtainable video quality is substantially lower than the video quality that can be achieved with off-line processing.
  • time taken for the entire video stream to be enhanced off-line may be longer than the duration of the video signal itself. In such cases, a viewer has to wait for a substantially long time after receiving the video stream to cater for the entire video stream to be enhanced by an off-line processing.
  • a third electronic means for storing the processed signal is provided for outputting the enhanced signal and/or the modified enhanced signal in real time.
  • This is a real time, on-line system, carrying out a limited off-line enhancement with the processing resources that are available after the real-time processing. In such a system, off-line enhancement is severely restricted. Moreover, this is not an efficient way of handling the on-line and off-line processing of the video signal.
  • the video processing apparatus comprises : a first storage means arranged to receive and store the video signal, an image sequence analyzer arranged to analyze the at least one temporal image sequence of the stored video signal and arranged to assign a priority value to the image sequence to indicate a preference for a first complexity processing of the at least one temporal image sequence, a first processor arranged to process the at least one temporal image sequence according to the priority value and arranged to store the processed at least one temporal image sequence in a second storage means, a second processor arranged to process the stored video signal according to a second complexity processing; and a combiner arranged to combine the processed at least one temporal image sequence and the processed video signal (VP) into a combined video signal.
  • a first storage means arranged to receive and store the video signal
  • an image sequence analyzer arranged to analyze the at least one temporal image sequence of the stored video signal and arranged to assign a priority value to the image sequence to indicate a preference for a first complexity processing of the at least one temporal image sequence
  • a first processor arranged
  • the first complexity processing may refer to a higher complexity
  • off-line processing and the second complexity processing may refer to a low-complexity, on- line processing.
  • the first complexity processing for e.g., may refer to a time and resources intensive image enhancement algorithm.
  • the inventor has recognized that the first complexity processing of the entire video signal for image quality enhancement can be carried out to temporal parts of the stored video signal according to an assigned priority value.
  • a priority value can be assigned to each temporal image sequence part to indicate a preference for the first complexity processing of the temporal part.
  • a relatively higher priority value is assigned to a temporal part of the video signal which, when presented with an enhanced quality, increases the viewing pleasure of a viewer.
  • a combination of the first complexity processed temporal parts and the second complexity processed video signal is delivered.
  • the video processing apparatus comprises a first storage means, an image sequence analyzer, a first processor, a second processor, a second storage means and a combiner.
  • the first storage means is arranged to receive a stream of video signal and store it for further processing.
  • the image sequence analyzer is arranged to analyze a temporal image sequence part and assign a priority value to each part. The prioritization is useful in deciding the order in which the temporal image sequences will be subjected to first complexity processing in the first processor.
  • the first processor is arranged to process algorithms of relatively higher complexity for image quality enhancement.
  • the processed output of the first processor can be stored in the first storage means.
  • the second processor is arranged to process algorithms of relatively lower complexity, preferably online, in real-time.
  • the combiner is arranged to deliver a combined output of first complexity processed temporal image sequences and second complexity processed video signal.
  • the first complexity processing such as relatively more complex algorithms for image quality enhancements can be carried out in the first processor. Due to the time and processor resource constraints such processing may be carried out offline.
  • the second processor is arranged to handle the second complexity processing for example, lower complexity processing of the video signal on-line, in real-time.
  • the combiner is arranged to combine the first complexity processed and the second complexity processed temporal image sequences.
  • the first storage means is arranged to comprise the second storage means.
  • the first storage means when arranged to comprise the second storage means, reduces the complexity of the video processing system. This arrangement facilitates a central storage and control resulting in a simpler and compact architecture compared to having an additional second storage means.
  • the combiner is arranged to comprise the second storage means.
  • the capacity of the first storage means can be more advantageously utilized to store a larger part of the video stream or multiple video streams.
  • the received video signal is conforming to a standard definition television (SDTV) video signal and the combined video signal is conforming to a high definition television (HDTV) video signal.
  • SDTV standard definition television
  • HDTV high definition television
  • Conversion of SDTV video signal to HDTV video signal according to this invention involves off-line, high complexity processing and on-line, low complexity processing of video signals.
  • the apparatus according to the invention can be advantageously employed to convert the SDTV video signal to HDTV video signal by employing a combination of off-line and on-line processing. A time-shifted recording and delivery can simultaneously be arranged.
  • An SDTV video signal can be converted in real-time to a HDTV video signal. The conversion involves some of the well-known processing for example de- interlacing, spatial up-conversion, frame rate conversion, and aspect ratio conversion.
  • the first processor is arranged to process the temporal image sequences for spatial up-conversion.
  • the first processor is arranged to carry out more complex processing that involves substantially increased processing resources and time.
  • One such processing is spatial up-conversion of the video signal with high complexity algorithms.
  • the first processor is arranged to process the temporal image sequences for image quality enhancement and/or noise removal.
  • the first processor is arranged to carry out more complex processing that involves substantially increased processing resources and time such as quality enhancement and/or noise removal.
  • a processing image sequences sharpness increase can be carried out using a linear or non-linear transform function.
  • Non-linear sharpness enhancement is resources intensive and can be carried out as first complexity processing.
  • These types of quality improvement processing can be applied to selected scenes in the order of priority in the first processor.
  • the first processor is arranged to process the image sequence for three-dimensional depth calculation.
  • the first processor is arranged to carry out more complex processing that involves substantially increased processing resources and time.
  • Three dimensional depth calculation of video signals is a resource intensive processing and such processing can be advantageously carried out as first complexity processing by the first processor in the proposed apparatus according to the invention.
  • the combiner is arranged to combine the processed at least one temporal image sequence and the processed video signal by means of a weighted combination.
  • the combined output signal comprises the first complexity processed temporal image sequences and the second complexity processed video signal.
  • the first complexity processed image sequence comprises a quality enhanced image sequence compared to the second complexity processed video signal.
  • the second complexity processed video signal is available at the combiner as a continuous stream whereas the first complexity processed temporal image sequences according to a priority value are generally discontinuous.
  • One form of combining can be by substituting the first complexity processed image sequences at their respective locations in the continuous stream of second complexity processed video signal. Such substitutions result in abrupt changes in the quality of output video from a relatively low quality to high quality and vice versa.
  • a few of the image frames on the temporal domain on either side of the change over point can be combined by a predetermined weighted combination of high and low quality processed image frames.
  • the weighted combination can result in a gradual increase or decrease of quality from low to high and vice versa. It is advantageous to process images of a temporal sequence, sometimes called as a shot by a particular type of complexity for example first complexity. Combining two shots, processed by different types of processing is easier in such cases than a single shot processed by two different types of processing.
  • the video processing apparatus is arranged to receive a viewer input to assign a priority value (PR) to the at least one temporal image sequence to indicate a preference for a first complexity processing of the at least one temporal image sequence
  • PR priority value
  • a priority value can be assigned to each temporal sequence by means of a viewer preference for high complexity processing.
  • the input can be in the form of voice, handwriting, key-stroke or touch.
  • the video processing system need not necessarily be provided with an image sequence analyzer, thereby reducing the complexity and cost comparable to a system with the image sequence analyzer.
  • the high quality image output based on viewer's choice is likely to enhance the viewing pleasure of the viewer.
  • the video processing method comprises : receiving and storing the video signal, analyzing the at least one temporal image sequence (TS) of the stored video signal and arranged to assign a priority value (PR) to the at least one temporal image sequence to indicate a preference for a first complexity processing of the at least one temporal image sequence, processing the at least one temporal image sequence (TS) according to the priority value (PR) and arranged to store the processed at least one temporal image sequence (TP) in a second storage means (110), processing the stored video signal (VS) according to a second complexity processing; and combining the processed at least one temporal image sequence (TP) and the processed video signal (VP) into a combined video signal (CP).
  • a priority value PR
  • the video processing method for processing a video signal is arranged to handle a first complexity processing and a second complexity processing of the video signal in an efficient manner.
  • a first step the image sequences are analyzed and each sequence, also called as a scene is assigned with a priority value indicating the order of preference of high complexity processing involving higher processor resources and time. Scenes that benefit most from the off-line, high complexity processing are first selected to undergo the first complexity processing.
  • a second complexity processing for example with a lesser complexity and lesser processor resources may be carried out on the entire length of the video signal. The second complexity processing may start as soon as the apparatus receives the demand for delivery of video signal.
  • a combining step is provided to combine the first complexity processed and second complexity processed image sequences and deliver a mixed video signal at the output.
  • the computer program product to be loaded by a computer arrangement, comprising instructions for processing video signal comprising at least one temporal image sequence, the computer arrangement comprising processing unit and a memory, the computer program product, after being loaded, providing said processing unit with the capability to carry out the following tasks: receiving and storing the video signal, analyzing the at least one temporal image sequence (TS) of the stored video signal and arranged to assign a priority value (PR) to the at least one temporal image sequence to indicate a preference for a first complexity processing of the at least one temporal image sequence, processing the at least one temporal image sequence (TS) according to the priority value (PR) and arranged to store the processed at least one temporal image sequence (TP) in a second storage means (110), processing the stored video signal (VS) according to a second complexity processing; and combining the processed at least one temporal image sequence (TP) and the processed video signal (VP) into a combined video signal (CP).
  • a priority value PR
  • the computer program product is provided to carry out the method of first complexity processing and second complexity processing of a video signal efficiently with reference to the method as mentioned above.
  • the computer program product is useful compared to the computer program product that carry out the entire processing on-line or offline or off-line processing with the available resources not used for on-line processing.
  • Fig.l is a schematic illustration of an embodiment of a video processing apparatus according to the invention
  • Fig. 2 is an illustration of a flow diagram of an embodiment of a video processing method according to the invention
  • Fig. 3 is an illustration of an embodiment of combining a first complexity and second complexity processed video signals
  • Figs. 4a-c are an illustration of another embodiment of combining the first complexity and second complexity processed video signals
  • Fig. 5 is a schematic illustration of a block diagram of a computer program product according to the invention.
  • Video streams for example movies or sport events conforming to various compression standards, resolutions and formats are available to a viewer through a number of means such as wired/wireless broadcast, Internet or optical storage discs.
  • Storage and processing of such video streams in a video processing apparatus for enhancing the image quality is a topic of current interest.
  • Image quality enhancement algorithms aimed at increasing the viewing pleasure of a viewer generally demand intensive processor and storage resources.
  • the received vide streams present a challenge of processing large amounts of data within the processor resources and available time. Due to the time and processor constraints, for example at a consumer location or a redistribution center, some of the image quality enhancements are required to be processed off-line. In such cases, a viewer may have to wait for durations comparatively longer than the total duration of a video stream. While delivering the video stream to a receiving apparatus for example a display device, some amount of real time, on-line processing is also required.
  • Fig.l is a schematic illustration of an embodiment of a video processing apparatus according to the invention.
  • the video processing apparatus 100 comprises a first storage means 110 arranged to receive a video signal stream VS.
  • the video signal stream can be in the form of an analog video stream or a digital video stream.
  • the video signal is encoded in the video processing apparatus by means of an encoder to convert the analog signal to a digital video stream before storage in the first storage means.
  • An image sequence analyzer 120, a first processor 130 and a second processor 140 are coupled to the storage means 110 and are arranged to receive the stored video signal.
  • the first storage means 110 may be in the form of a hard disc drive (HDD) or a personal video recorder (PVR) or an optical disc such as Blu-ray disc.
  • the video processing apparatus 100 may be a personal video recorder (PVR)
  • the image sequence analyzer 120 receives parts of video signal in the form of image sequences TS comprising a temporal sequence of image frames and analyses the contents with an object of assigning a priority value PR to each one of the image sequences.
  • the image sequence analyzer 120 need not wait for the entire video signal to be received and stored in the storage means 110. It starts analyzing one of the image sequence TS after it has been received at the storage means 110.
  • the priority value PR is an indication of the order of preference in which a high complexity enhancement operation can be applied to the temporal image sequence TS.
  • the image analyzing means analyses each frame of the temporal sequence of image frames, looking for and accumulating evidences that suggest the benefit expected by off-line enhancement of the particular temporal image sequence.
  • an image sequence with faster moving objects is given a higher priority value compared to an image sequence with comparatively slower movements because the image sequence with faster moving objects is considered to be a better candidate for first complexity processing involving image enhancement algorithms of higher complexity.
  • Motion vectors are useful evidences that indicate the degree of movement of objects in video streams conforming to certain well-known image coding standards such as moving picture experts group (MPEG) standards.
  • MPEG moving picture experts group
  • a sum of absolute motion vector lengths can be computed on a frame-by- frame basis and a higher priority value assigned to temporal image sequences that have higher sums.
  • longer image sequences are assigned a higher priority compared to shorter image sequences.
  • a priority number can be stored in a separate database and recalled. Alternately, the priority number can be attached to the temporal image sequence in the compressed bit stream.
  • a viewer can allot a priority number.
  • a preview of the image sequence can be arranged to be shown to the viewer and an input is requested from the viewer.
  • the input can be obtained through different types of user interlaces, for example, touch, speech, keyboard or any other interactive inputting device.
  • the input also may come from a sub-system that learns with such user preference indications and pre-assigns priority values according to the expected user preference.
  • some of the processing methods such as de-interlacing and temporal up-conversion may introduce visible artefacts. Relatively more complex algorithms requiring higher processor resources are needed in such cases. Processing using more complex algorithms need to be taken up as first complexity processing due to time constraints. Consequently, higher quality of processed image sequence with lesser artefacts can be expected at the output.
  • a number of enhancements for example, de-interlacing and temporal up-conversion can be used to increase the viewing experience of the user.
  • Some examples of first complexity processing for image improvements are motion-compensated de-interlacing, spatial up-conversion, motion compensated temporal up-conversion, linear/non- linear sharpness enhancement, noise reduction or picture stability improvement especially for fast moving scenes.
  • Parts of the video signal that benefit most from the high-complexity processing can be identified and an order of priority assigned for each part.
  • a high complexity processing can be carried out in the order of priority. If the time allowance between the storage and delivery of the video signal is sufficient, the entire length of the video stream can be subjected to high complexity processing. On the other hand, if the delivery has to start before the complete video stream is processed, a combined video signal comprising a low complexity processed video signal and parts of high complexity processed video signal are delivered. Thus the processor resources of the apparatus are efficiently utilized to enhance the viewing experience.
  • the first processor 130 processes the image sequences in the order of assigned priority.
  • the first processor 130 is arranged to process image enhancement algorithms of higher complexity.
  • the processed temporal image sequence TP is stored in a second storage means 160.
  • the second storage means 160 and the output of the second processor 140 are coupled to a combiner 150.
  • Examples of high complexity processing are de-interlacing, frame rate up- conversion, spatial up-conversion, noise removal restoration of scenes when the scenes have high motion content.
  • high complexity processing is de-interlacing, frame rate up- conversion, spatial up-conversion, noise removal restoration of scenes when the scenes have high motion content.
  • Such down-conversions using complex algorithms can also be called high-complexity processing. In such cases, a higher compression ratio can be achieved for an increased waiting time.
  • Examples of low complexity processing are de-interlacing of static scenes, frame rate up conversion for static scenes, spatial up-conversion, noise removal, enhancement of scenes, filtering, format conversion when the scenes are of substantially static type.
  • the second processor 140 carries out real-time, on-line processing of the video signal.
  • the output of the second processor is available as a continuous stream of processed video signal VP at the input of the combiner 150.
  • the combiner 150 replaces parts of high complexity processed temporal image sequences TP in the processed video stream VP and delivers a combined output video signal CP to the subsequent stages.
  • the abrupt changes may be annoying to some viewers.
  • a few image frames on either side of the change over point can be combined at the combiner 150 with a weighted combination of the image frames and delivered at the output.
  • the invention is advantageous when the video stream has to be viewed or transmitted further when the image quality enhancement of the full stream has not been completed. In other words, viewing or transmitting the enhanced video stream is possible while the recording and enhancement processing of a time-shifted part of the video stream is still in progress.
  • the invention provides a means of enhancing the viewing experience while reducing the waiting time for a viewer. Depending upon the time available for off-line enhancement, either the entire length of the video stream or preferred temporal parts may be made available after the first complexity processing.
  • the video processing system can deliver the processed video signal after a variable duration as desired by a viewer. Thus a flexibility of operation of the video processing apparatus is introduced compared to the prior art systems.
  • the flexibility of operation can be illustrated with the following example. Assume that a single video stream is recorded in the evening of a day without the intention of watching the recording on the same day.
  • the recording may be processed off-line during the night in order to be fully enhanced for watching on the next day.
  • this is an ideal situation for off-line processing and does not consider a situation wherein a viewer does not want to wait for a video processing task to be finished until he/she can again use the system or watch the recoded video stream.
  • the invention provides a facility by which the viewer can view the video stream before the enhancement of the complete video stream is carried out. Thus, an optimal viewing experience with partially processed video recordings can be achieved in a lesser waiting time.
  • a plurality of video streams may be queued up for off-line video enhancement.
  • the video processing system is arranged to convert the SDTV video signal into HDTV video signal.
  • a priority number is assigned after an analysis.
  • each temporal image sequence of SDTV format is processed to convert it to HDTV video format in the first processor.
  • Prioritization of video parts continue as the video is being received and recorded in the first storage means.
  • the second processor and the combiner start processing the SDTV video signal to HDTV video signal.
  • the second processor sequentially carries out an on-line real time conversion of the video stream.
  • the combiner combines the first complexity processed HDTV video signal and the second complexity processed HDTV video signal and delivers a combined output HDTV video signal.
  • the conversion of SDTV video signal to HDTV video signal continues in the order of priority. New priority values continue to be allotted to the parts of recently received video signal streams.
  • the first complexity processing can switch according to the recently assigned priority numbers. It is possible to receive multiple video streams simultaneously and record them in the first recording means. In such cases, prioritization of parts of video streams from different video streams can be processed according to the assigned priority numbers.
  • the first storage means 110 can be arranged to comprise the second storage means 160. Both the storage means can be arranged in a hard disc drive (HDD) or an optical storage such as Blu-ray disk. In such an arrangement, the processed temporal image sequence TP is coupled back to the first storage means 110. The first storage means 110 directly supplies the processed temporal image sequence TP to the combiner 150.
  • the first storage means can be a HDD or Blu-ray disc.
  • the combiner 150 can be arranged to comprise the second storage means 160.
  • a combination of various types of storage means for example magnetic or optical storage means can be provided.
  • the first processor 130 can be arranged to comprise the second processor 160.
  • a single processor or a plurality of processors can be provided, each processor being assigned with a single or plurality of processing functions.
  • the processing functions include, but are not restricted to first complexity processing, second complexity processing, image sequence analysis, combining processed video signals and overall control, coordination and operation of the video processing apparatus.
  • FIG. 2 is an illustration of a flow diagram 200 of an embodiment of a video processing method according to the invention.
  • An analog or digital video stream VS is received and stored in the first step
  • An analog video signal after digitization can be converted to a digital video stream according to one of the well-known image compression standards before storage.
  • a part of video stream TS in the form of a temporal image sequence is subjected to an image sequence analysis in the next step 220 and a priority value PR is assigned to the temporal image sequence TS.
  • the priority value PR is an indication of the order of preference in which the temporal image sequence TS can be subjected to a first complexity processing in a first processing step 230.
  • the processed temporal image sequence is stored in the next step 250.
  • a second complexity processing is carried out in step 240 on the entire video stream VS to generate a processed video stream VP.
  • the second complexity processing in step 240 may be a real time, on-line processing carried out on the entire video stream VS.
  • the processed temporal image sequences TP and the processed video stream VP are combined in the combining step 260 to deliver a combined video signal CP.
  • Fig. 3 is an illustration of an embodiment of combining a first complexity and second complexity processed video signals.
  • a first complexity processed output 310 and a second complexity processed output 320 available at the combiner are shown in Fig. 3.
  • the first complexity processed output is available for some discontinuous parts (323, 324 and 327) of the video stream.
  • the first complexity processing has not been done on some parts (321, 322, 325 and 326) of the video stream and these parts are not available at the combiner.
  • the first complexity processed parts of the video stream are synchronized with the second complexity processed video signal by using their respective timing information.
  • the combined output in one of the embodiments, comprises the processed video stream 330, replaced by the processed temporal image sequences (333, 334 and 337) wherever available.
  • Fig. 4 is an illustration of another embodiment of combining the first complexity and second complexity processed video signals.
  • first complexity and second complexity processed video signals is illustrated with reference to a sequence of image frames as they appear in a video compression scheme for example, MPEG scheme as mentioned above.
  • the I, P and B frames shown in Fig. 4 refer to the non-predictive, predictive and bi-directionally predictive image frames, respectively according to the MPEG scheme.
  • a temporal image sequence in such a series may be defined as a part of the video stream that starts and ends with an I-frame in the display order. Therefore, consecutive image sequences share a common I-frame according to this definition.
  • FIG. 4 An example of a Group of Pictures (GOPs) comprising 12 image frames with 3 independent image sequences (410, 420 and 430), visualized in the display order as shown in Fig. 4 (a), (b) and (c).
  • SD video 450 indicates that this stream is subjected to the second complexity processed SD to HD conversion.
  • a part 420 of the video stream is subjected to first complexity processing involving a higher complexity.
  • a partial decoding of frames in the SD part may be required.
  • the I-frames (411, 431) at transitions have to be decoded in order to decode the subsequent B and P-frames.
  • the second complexity processing can do this decoding.
  • Fig. 4(b) an overlapping sequence of four frames IBBP at the transition is shown.
  • the video stream in this case changes from a low complexity processed, low quality output to a high complexity processed, higher quality output.
  • the combination can be made such that the higher quality of the video is perceived gradually.
  • the weights for IBBP frames in such a combination can be identified, for example as (0.2, 0.8), (0.4, 0.6), (0.6, 0.4), (0.8, 0.2) whereas for a set of weights (x,y) , x denotes the weight of a frame from the SD stream and y denotes the weight of a corresponding frame from the HD stream.
  • Fig 4 (c) shows a transition from a HD stream to a SD stream with an overlapping sequence of four frames PBBI.
  • the effect of HD is made to gradually reduce and the effect of SD is made to gradually increase.
  • the same sets of weights, for example, as mentioned above can be used. This weighted combination will enable a gradual increase or decrease of output quality and it may result in a less annoying visual experience to the viewer compared to abrupt changes in visual quality.
  • Fig. 5 is a schematic illustration of a block diagram of a computer program product according to the invention.
  • the computer program product (300) can be loaded into a computing machine and enable of operating the machine comprising a processing unit and a memory, the computer program product, after being loaded, providing said processing unit with the capability to carry out the processing of a video signal comprising temporal sequences of images.
  • the computer program product provides the processing unit with the capability to analyze temporal image sequences and assign a priority value for each sequence indicating the preference of a first complexity processing of the temporal image sequences.
  • the computer program product further provides the processing unit with the capability to carry out a first complexity processing and/or a second complexity processing of temporal image sequences.
  • the computer-processing product to achieve the results according to the invention can control a single processing unit or a plurality of processing units.
  • the computer program product can be handled in a standard comprised or detachable storage, for example a flash memory or a compact disk or a hard disk.
  • the computer program product can be embedded in a computing machine as embedded software or kept pre-loaded or loaded from one of the standard memory devices.
  • the computer program product can be designed in any of the known codes such as machine language code or assembly language code and made to operate on any of the available platforms such as personal computers or servers.
  • the computer program product can be stored inside the apparatus or carried in a detachable storage medium such as an optical storage disc and transferred to the apparatus.
  • the computer program product can be used in various forms of consumer apparatus for example, set top box, video receiver, video recorder/player, hand-held mobile communication or entertainment devices providing video displays.
  • the computer program product can be implemented on various platforms such as personal computers or signal processors.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Television Systems (AREA)
  • Studio Circuits (AREA)
  • Television Signal Processing For Recording (AREA)

Abstract

A video processing apparatus (100) for processing a video signal comprising at least one temporal image sequence is provided. The apparatus processing the video comprises a first storage means (110) arranged to receive and store the video signal, an image sequence analyzer (120) arranged to analyze the at least one temporal image sequence (TS) of the stored video signal and arranged to assign a priority value (PR) to the at least one temporal image sequence to indicate a preference for a first complexity processing of the at least one temporal image sequence, a first processor (130) arranged to process the at least one temporal image sequence (TS) according to the priority value (PR) and arranged to store the processed at least one temporal image sequence (TP) in a second storage means (160) , a second processor (140) arranged to process the stored video signal (VS) according to a second complexity processing and a combiner (150) arranged to combine the processed at least one temporal image sequence (TP) and the processed video signal (VP) into a combined video signal (CP) .

Description

Video processing apparatus
The invention relates to a video processing apparatus for processing a video signal comprising at least one temporal image sequence.
The invention further relates to a method of processing a video signal comprising at least one temporal image sequence and a computer program product for executing such a method.
Generally, a video processing apparatus is arranged to receive a video signal transmitted through a wired or wireless medium, store, process and deliver the processed video signal to another apparatus for example, to a display or a storage or a further transmission means. In general, a video signal comprises a plurality of temporal sequences of image frames. Processing functions such as noise removal, stability improvement, de- interlacing, spatial up-conversion, temporal up-conversion, subjective quality improvement and various other enhancements and modifications can be applied on the video signal. Such a video processing apparatus can be implemented on hardware and/or a software platform at a consumer location or at a video re-distribution location. Processing resources available on such platforms are generally subject to cost and size considerations. Due to constraints of time and processing resources, some of the enhancements of the video signal can be preferably carried out off-line in order to achieve high-quality video. Otherwise, real-time algorithms for video enhancement can be applied, wherein the obtainable video quality is substantially lower than the video quality that can be achieved with off-line processing. In many cases, time taken for the entire video stream to be enhanced off-line may be longer than the duration of the video signal itself. In such cases, a viewer has to wait for a substantially long time after receiving the video stream to cater for the entire video stream to be enhanced by an off-line processing.
Several schemes have been proposed in the prior art for handling an on-line and an off-line processing of video signals simultaneously within the available resources of a video processing systems. An embodiment of a signal processing system for an off-line enhancement of a signal is known from prior art for example, refer PCT application IB2005/000497, filed by the applicant on 24 February 2004 (Attorney docket number PHFR040025). In this document, a signal processing system comprising a first electronic means for storing an input signal and a second processing means for a real-time processing and also for an off-line signal enhancement are described. The off-line signal enhancement is applied to the input signal using the available processing resources in the second processing means, not still used for real-time or on-line use. In one of the embodiments, a third electronic means for storing the processed signal is provided for outputting the enhanced signal and/or the modified enhanced signal in real time. This is a real time, on-line system, carrying out a limited off-line enhancement with the processing resources that are available after the real-time processing. In such a system, off-line enhancement is severely restricted. Moreover, this is not an efficient way of handling the on-line and off-line processing of the video signal.
It is desirable to provide a video processing apparatus that is capable of handling an off-line enhancement and an on-line processing of a video signal in a more efficient manner.
This is achieved in that the video processing apparatus according to the opening paragraph comprises : a first storage means arranged to receive and store the video signal, an image sequence analyzer arranged to analyze the at least one temporal image sequence of the stored video signal and arranged to assign a priority value to the image sequence to indicate a preference for a first complexity processing of the at least one temporal image sequence, a first processor arranged to process the at least one temporal image sequence according to the priority value and arranged to store the processed at least one temporal image sequence in a second storage means, a second processor arranged to process the stored video signal according to a second complexity processing; and a combiner arranged to combine the processed at least one temporal image sequence and the processed video signal (VP) into a combined video signal.
The first complexity processing, for e.g., may refer to a higher complexity, off-line processing and the second complexity processing may refer to a low-complexity, on- line processing. The first complexity processing, for e.g., may refer to a time and resources intensive image enhancement algorithm. The inventor has recognized that the first complexity processing of the entire video signal for image quality enhancement can be carried out to temporal parts of the stored video signal according to an assigned priority value. A priority value can be assigned to each temporal image sequence part to indicate a preference for the first complexity processing of the temporal part. A relatively higher priority value is assigned to a temporal part of the video signal which, when presented with an enhanced quality, increases the viewing pleasure of a viewer. At the output of the video processing apparatus, a combination of the first complexity processed temporal parts and the second complexity processed video signal is delivered.
According to the invention, the video processing apparatus comprises a first storage means, an image sequence analyzer, a first processor, a second processor, a second storage means and a combiner. The first storage means is arranged to receive a stream of video signal and store it for further processing. The image sequence analyzer is arranged to analyze a temporal image sequence part and assign a priority value to each part. The prioritization is useful in deciding the order in which the temporal image sequences will be subjected to first complexity processing in the first processor. The first processor is arranged to process algorithms of relatively higher complexity for image quality enhancement. The processed output of the first processor can be stored in the first storage means. The second processor is arranged to process algorithms of relatively lower complexity, preferably online, in real-time. The combiner is arranged to deliver a combined output of first complexity processed temporal image sequences and second complexity processed video signal.
Due to prioritization of temporal image sequences, image sequences that benefit most from the image quality enhancement, as viewed by the viewer are subjected to high complexity processing. Thus, the first complexity processing such as relatively more complex algorithms for image quality enhancements can be carried out in the first processor. Due to the time and processor resource constraints such processing may be carried out offline. The second processor is arranged to handle the second complexity processing for example, lower complexity processing of the video signal on-line, in real-time. Further, the combiner is arranged to combine the first complexity processed and the second complexity processed temporal image sequences. Thus the invention provides an efficient means for handling the off-line enhancement and the on-line processing of video signals to enhance the viewing pleasure of a viewer.
In an embodiment of the video processing apparatus according to the invention, the first storage means is arranged to comprise the second storage means.
The first storage means, when arranged to comprise the second storage means, reduces the complexity of the video processing system. This arrangement facilitates a central storage and control resulting in a simpler and compact architecture compared to having an additional second storage means. In an embodiment of the video processing apparatus according to the invention, the combiner is arranged to comprise the second storage means.
By having a separate storage means for storing the processed temporal image sequence, the capacity of the first storage means can be more advantageously utilized to store a larger part of the video stream or multiple video streams.
In an embodiment of the video processing apparatus according to the invention, the received video signal is conforming to a standard definition television (SDTV) video signal and the combined video signal is conforming to a high definition television (HDTV) video signal. Conversion of SDTV video signal to HDTV video signal according to this invention involves off-line, high complexity processing and on-line, low complexity processing of video signals. The apparatus according to the invention can be advantageously employed to convert the SDTV video signal to HDTV video signal by employing a combination of off-line and on-line processing. A time-shifted recording and delivery can simultaneously be arranged. An SDTV video signal can be converted in real-time to a HDTV video signal. The conversion involves some of the well-known processing for example de- interlacing, spatial up-conversion, frame rate conversion, and aspect ratio conversion.
In an embodiment of the video processing apparatus according to the invention, the first processor is arranged to process the temporal image sequences for spatial up-conversion.
The first processor is arranged to carry out more complex processing that involves substantially increased processing resources and time. One such processing is spatial up-conversion of the video signal with high complexity algorithms.
In an embodiment of the video processing apparatus according to the invention, the first processor is arranged to process the temporal image sequences for image quality enhancement and/or noise removal.
The first processor is arranged to carry out more complex processing that involves substantially increased processing resources and time such as quality enhancement and/or noise removal. For example, a processing image sequences sharpness increase can be carried out using a linear or non-linear transform function. Non-linear sharpness enhancement is resources intensive and can be carried out as first complexity processing. These types of quality improvement processing can be applied to selected scenes in the order of priority in the first processor. In an embodiment of the video processing apparatus according to the invention, the first processor is arranged to process the image sequence for three-dimensional depth calculation.
The first processor is arranged to carry out more complex processing that involves substantially increased processing resources and time. Three dimensional depth calculation of video signals is a resource intensive processing and such processing can be advantageously carried out as first complexity processing by the first processor in the proposed apparatus according to the invention.
In an embodiment of the video processing apparatus according to the invention, the combiner is arranged to combine the processed at least one temporal image sequence and the processed video signal by means of a weighted combination.
The combined output signal comprises the first complexity processed temporal image sequences and the second complexity processed video signal. Generally, the first complexity processed image sequence comprises a quality enhanced image sequence compared to the second complexity processed video signal. The second complexity processed video signal is available at the combiner as a continuous stream whereas the first complexity processed temporal image sequences according to a priority value are generally discontinuous. One form of combining can be by substituting the first complexity processed image sequences at their respective locations in the continuous stream of second complexity processed video signal. Such substitutions result in abrupt changes in the quality of output video from a relatively low quality to high quality and vice versa. In order to overcome this problem, a few of the image frames on the temporal domain on either side of the change over point can be combined by a predetermined weighted combination of high and low quality processed image frames. Thus the weighted combination can result in a gradual increase or decrease of quality from low to high and vice versa. It is advantageous to process images of a temporal sequence, sometimes called as a shot by a particular type of complexity for example first complexity. Combining two shots, processed by different types of processing is easier in such cases than a single shot processed by two different types of processing.
In an embodiment of the video processing apparatus according to the invention, the video processing apparatus is arranged to receive a viewer input to assign a priority value (PR) to the at least one temporal image sequence to indicate a preference for a first complexity processing of the at least one temporal image sequence
A priority value can be assigned to each temporal sequence by means of a viewer preference for high complexity processing. The input can be in the form of voice, handwriting, key-stroke or touch. In such cases, the video processing system need not necessarily be provided with an image sequence analyzer, thereby reducing the complexity and cost comparable to a system with the image sequence analyzer. Moreover the high quality image output based on viewer's choice is likely to enhance the viewing pleasure of the viewer.
It is desirable to provide a video processing method that is capable of handling an off-line enhancement and an on-line processing of a video signal simultaneously in a more efficient manner.
This is achieved in that the video processing method according to the opening paragraph comprises : receiving and storing the video signal, analyzing the at least one temporal image sequence (TS) of the stored video signal and arranged to assign a priority value (PR) to the at least one temporal image sequence to indicate a preference for a first complexity processing of the at least one temporal image sequence, processing the at least one temporal image sequence (TS) according to the priority value (PR) and arranged to store the processed at least one temporal image sequence (TP) in a second storage means (110), processing the stored video signal (VS) according to a second complexity processing; and combining the processed at least one temporal image sequence (TP) and the processed video signal (VP) into a combined video signal (CP).
The video processing method for processing a video signal is arranged to handle a first complexity processing and a second complexity processing of the video signal in an efficient manner. In this method, as a first step, the image sequences are analyzed and each sequence, also called as a scene is assigned with a priority value indicating the order of preference of high complexity processing involving higher processor resources and time. Scenes that benefit most from the off-line, high complexity processing are first selected to undergo the first complexity processing. A second complexity processing for example with a lesser complexity and lesser processor resources may be carried out on the entire length of the video signal. The second complexity processing may start as soon as the apparatus receives the demand for delivery of video signal. A combining step is provided to combine the first complexity processed and second complexity processed image sequences and deliver a mixed video signal at the output. Thus the constraints of time and processor resources are taken care of, in an efficient manner.
It is desirable to provide a computer program product to be loaded by a computer arrangement, comprising instructions for processing a video signal comprising at least one temporal image sequence, the computer program product capable of handling a first complexity processing and a second complexity processing of a video signal in a more efficient manner.
This is achieved in that the computer program product to be loaded by a computer arrangement, comprising instructions for processing video signal comprising at least one temporal image sequence, the computer arrangement comprising processing unit and a memory, the computer program product, after being loaded, providing said processing unit with the capability to carry out the following tasks: receiving and storing the video signal, analyzing the at least one temporal image sequence (TS) of the stored video signal and arranged to assign a priority value (PR) to the at least one temporal image sequence to indicate a preference for a first complexity processing of the at least one temporal image sequence, processing the at least one temporal image sequence (TS) according to the priority value (PR) and arranged to store the processed at least one temporal image sequence (TP) in a second storage means (110), processing the stored video signal (VS) according to a second complexity processing; and combining the processed at least one temporal image sequence (TP) and the processed video signal (VP) into a combined video signal (CP). The computer program product is provided to carry out the method of first complexity processing and second complexity processing of a video signal efficiently with reference to the method as mentioned above. The computer program product is useful compared to the computer program product that carry out the entire processing on-line or offline or off-line processing with the available resources not used for on-line processing.
The above object and desirable features of the present invention will be more apparent from the following description of the preferred embodiments with reference to the drawings, wherein: Fig.l is a schematic illustration of an embodiment of a video processing apparatus according to the invention,
Fig. 2 is an illustration of a flow diagram of an embodiment of a video processing method according to the invention, Fig. 3 is an illustration of an embodiment of combining a first complexity and second complexity processed video signals,
Figs. 4a-c are an illustration of another embodiment of combining the first complexity and second complexity processed video signals,
Fig. 5 is a schematic illustration of a block diagram of a computer program product according to the invention.
Video streams for example movies or sport events conforming to various compression standards, resolutions and formats are available to a viewer through a number of means such as wired/wireless broadcast, Internet or optical storage discs. Storage and processing of such video streams in a video processing apparatus for enhancing the image quality is a topic of current interest. Image quality enhancement algorithms aimed at increasing the viewing pleasure of a viewer generally demand intensive processor and storage resources. Thus the received vide streams present a challenge of processing large amounts of data within the processor resources and available time. Due to the time and processor constraints, for example at a consumer location or a redistribution center, some of the image quality enhancements are required to be processed off-line. In such cases, a viewer may have to wait for durations comparatively longer than the total duration of a video stream. While delivering the video stream to a receiving apparatus for example a display device, some amount of real time, on-line processing is also required.
Fig.l is a schematic illustration of an embodiment of a video processing apparatus according to the invention.
The video processing apparatus 100 comprises a first storage means 110 arranged to receive a video signal stream VS. The video signal stream can be in the form of an analog video stream or a digital video stream. In the case of analog video stream, the video signal is encoded in the video processing apparatus by means of an encoder to convert the analog signal to a digital video stream before storage in the first storage means. An image sequence analyzer 120, a first processor 130 and a second processor 140 are coupled to the storage means 110 and are arranged to receive the stored video signal. The first storage means 110 may be in the form of a hard disc drive (HDD) or a personal video recorder (PVR) or an optical disc such as Blu-ray disc. The video processing apparatus 100 may be a personal video recorder (PVR)
The image sequence analyzer 120 receives parts of video signal in the form of image sequences TS comprising a temporal sequence of image frames and analyses the contents with an object of assigning a priority value PR to each one of the image sequences. The image sequence analyzer 120 need not wait for the entire video signal to be received and stored in the storage means 110. It starts analyzing one of the image sequence TS after it has been received at the storage means 110. The priority value PR is an indication of the order of preference in which a high complexity enhancement operation can be applied to the temporal image sequence TS. The image analyzing means analyses each frame of the temporal sequence of image frames, looking for and accumulating evidences that suggest the benefit expected by off-line enhancement of the particular temporal image sequence. For example, an image sequence with faster moving objects is given a higher priority value compared to an image sequence with comparatively slower movements because the image sequence with faster moving objects is considered to be a better candidate for first complexity processing involving image enhancement algorithms of higher complexity. Motion vectors are useful evidences that indicate the degree of movement of objects in video streams conforming to certain well-known image coding standards such as moving picture experts group (MPEG) standards. As an example, a sum of absolute motion vector lengths can be computed on a frame-by- frame basis and a higher priority value assigned to temporal image sequences that have higher sums. In another example, longer image sequences are assigned a higher priority compared to shorter image sequences. A priority number can be stored in a separate database and recalled. Alternately, the priority number can be attached to the temporal image sequence in the compressed bit stream.
In another embodiment, a viewer can allot a priority number. In such a scheme, a preview of the image sequence can be arranged to be shown to the viewer and an input is requested from the viewer. The input can be obtained through different types of user interlaces, for example, touch, speech, keyboard or any other interactive inputting device. The input also may come from a sub-system that learns with such user preference indications and pre-assigns priority values according to the expected user preference.
In some scenes, for example scenes with fast motion, some of the processing methods such as de-interlacing and temporal up-conversion may introduce visible artefacts. Relatively more complex algorithms requiring higher processor resources are needed in such cases. Processing using more complex algorithms need to be taken up as first complexity processing due to time constraints. Consequently, higher quality of processed image sequence with lesser artefacts can be expected at the output. A number of enhancements for example, de-interlacing and temporal up-conversion can be used to increase the viewing experience of the user. Some examples of first complexity processing for image improvements are motion-compensated de-interlacing, spatial up-conversion, motion compensated temporal up-conversion, linear/non- linear sharpness enhancement, noise reduction or picture stability improvement especially for fast moving scenes.
Parts of the video signal that benefit most from the high-complexity processing can be identified and an order of priority assigned for each part. A high complexity processing can be carried out in the order of priority. If the time allowance between the storage and delivery of the video signal is sufficient, the entire length of the video stream can be subjected to high complexity processing. On the other hand, if the delivery has to start before the complete video stream is processed, a combined video signal comprising a low complexity processed video signal and parts of high complexity processed video signal are delivered. Thus the processor resources of the apparatus are efficiently utilized to enhance the viewing experience.
The first processor 130 processes the image sequences in the order of assigned priority. The first processor 130 is arranged to process image enhancement algorithms of higher complexity. The processed temporal image sequence TP is stored in a second storage means 160. The second storage means 160 and the output of the second processor 140 are coupled to a combiner 150.
Examples of high complexity processing are de-interlacing, frame rate up- conversion, spatial up-conversion, noise removal restoration of scenes when the scenes have high motion content. In some scenarios for example, when HDTV video signal is received and it has to be down-converted to SDTV or further down-converted to make it recordable in mobile hand-held devices such as mobile phones. Such down-conversions using complex algorithms can also be called high-complexity processing. In such cases, a higher compression ratio can be achieved for an increased waiting time. Examples of low complexity processing are de-interlacing of static scenes, frame rate up conversion for static scenes, spatial up-conversion, noise removal, enhancement of scenes, filtering, format conversion when the scenes are of substantially static type. The second processor 140 carries out real-time, on-line processing of the video signal. The output of the second processor is available as a continuous stream of processed video signal VP at the input of the combiner 150. The combiner 150 replaces parts of high complexity processed temporal image sequences TP in the processed video stream VP and delivers a combined output video signal CP to the subsequent stages. In such cases there is a sudden or abrupt change in the output quality of video signal observed by a viewer as the high complexity processed image sequences are expected to be of higher quality than the real time, on-line processed video signal. The abrupt changes may be annoying to some viewers. In order to minimize the effect of abrupt changes, a few image frames on either side of the change over point can be combined at the combiner 150 with a weighted combination of the image frames and delivered at the output.
The invention is advantageous when the video stream has to be viewed or transmitted further when the image quality enhancement of the full stream has not been completed. In other words, viewing or transmitting the enhanced video stream is possible while the recording and enhancement processing of a time-shifted part of the video stream is still in progress. The invention provides a means of enhancing the viewing experience while reducing the waiting time for a viewer. Depending upon the time available for off-line enhancement, either the entire length of the video stream or preferred temporal parts may be made available after the first complexity processing. The video processing system can deliver the processed video signal after a variable duration as desired by a viewer. Thus a flexibility of operation of the video processing apparatus is introduced compared to the prior art systems.
The flexibility of operation can be illustrated with the following example. Assume that a single video stream is recorded in the evening of a day without the intention of watching the recording on the same day. The recording may be processed off-line during the night in order to be fully enhanced for watching on the next day. However, this is an ideal situation for off-line processing and does not consider a situation wherein a viewer does not want to wait for a video processing task to be finished until he/she can again use the system or watch the recoded video stream. The invention provides a facility by which the viewer can view the video stream before the enhancement of the complete video stream is carried out. Thus, an optimal viewing experience with partially processed video recordings can be achieved in a lesser waiting time. In a realistic scenario, a plurality of video streams may be queued up for off-line video enhancement. In an embodiment of the video processing system according to the invention, the video processing system is arranged to convert the SDTV video signal into HDTV video signal. For each temporal image sequence, a priority number is assigned after an analysis. In the order of priority, each temporal image sequence of SDTV format is processed to convert it to HDTV video format in the first processor. Prioritization of video parts continue as the video is being received and recorded in the first storage means. When the HD-TV video signal output is required to be delivered to an output device such as a display, the second processor and the combiner start processing the SDTV video signal to HDTV video signal. The second processor sequentially carries out an on-line real time conversion of the video stream. The combiner combines the first complexity processed HDTV video signal and the second complexity processed HDTV video signal and delivers a combined output HDTV video signal. The conversion of SDTV video signal to HDTV video signal continues in the order of priority. New priority values continue to be allotted to the parts of recently received video signal streams. The first complexity processing can switch according to the recently assigned priority numbers. It is possible to receive multiple video streams simultaneously and record them in the first recording means. In such cases, prioritization of parts of video streams from different video streams can be processed according to the assigned priority numbers.
In another embodiment of the video processing system 100, the first storage means 110 can be arranged to comprise the second storage means 160. Both the storage means can be arranged in a hard disc drive (HDD) or an optical storage such as Blu-ray disk. In such an arrangement, the processed temporal image sequence TP is coupled back to the first storage means 110. The first storage means 110 directly supplies the processed temporal image sequence TP to the combiner 150. The first storage means can be a HDD or Blu-ray disc.
In another embodiment of the video processing system 100, the combiner 150 can be arranged to comprise the second storage means 160. A combination of various types of storage means for example magnetic or optical storage means can be provided.
In another embodiment of the video processing system 100, the first processor 130 can be arranged to comprise the second processor 160. A single processor or a plurality of processors can be provided, each processor being assigned with a single or plurality of processing functions. The processing functions include, but are not restricted to first complexity processing, second complexity processing, image sequence analysis, combining processed video signals and overall control, coordination and operation of the video processing apparatus.
Fig. 2 is an illustration of a flow diagram 200 of an embodiment of a video processing method according to the invention. An analog or digital video stream VS is received and stored in the first step
210. An analog video signal after digitization can be converted to a digital video stream according to one of the well-known image compression standards before storage. A part of video stream TS in the form of a temporal image sequence is subjected to an image sequence analysis in the next step 220 and a priority value PR is assigned to the temporal image sequence TS. The priority value PR is an indication of the order of preference in which the temporal image sequence TS can be subjected to a first complexity processing in a first processing step 230. The processed temporal image sequence is stored in the next step 250. A second complexity processing is carried out in step 240 on the entire video stream VS to generate a processed video stream VP. The second complexity processing in step 240, for example may be a real time, on-line processing carried out on the entire video stream VS. The processed temporal image sequences TP and the processed video stream VP are combined in the combining step 260 to deliver a combined video signal CP.
Fig. 3 is an illustration of an embodiment of combining a first complexity and second complexity processed video signals. A first complexity processed output 310 and a second complexity processed output 320 available at the combiner are shown in Fig. 3. The first complexity processed output is available for some discontinuous parts (323, 324 and 327) of the video stream. The first complexity processing has not been done on some parts (321, 322, 325 and 326) of the video stream and these parts are not available at the combiner. The first complexity processed parts of the video stream are synchronized with the second complexity processed video signal by using their respective timing information. The combined output, in one of the embodiments, comprises the processed video stream 330, replaced by the processed temporal image sequences (333, 334 and 337) wherever available. In some cases, it is possible to complete the first complexity processing on the entire video stream and the output may be available as a continuous stream. The second complexity processing is not necessary in such cases. In one of the embodiments, the second complexity processing need not necessarily be carried out fully or partially carried out depending upon the availability of the first complexity processed parts, resulting in saving of second processor resources. Fig. 4 is an illustration of another embodiment of combining the first complexity and second complexity processed video signals.
The combining of first complexity and second complexity processed video signals is illustrated with reference to a sequence of image frames as they appear in a video compression scheme for example, MPEG scheme as mentioned above. The I, P and B frames shown in Fig. 4 refer to the non-predictive, predictive and bi-directionally predictive image frames, respectively according to the MPEG scheme. A temporal image sequence in such a series may be defined as a part of the video stream that starts and ends with an I-frame in the display order. Therefore, consecutive image sequences share a common I-frame according to this definition.
An example of a Group of Pictures (GOPs) comprising 12 image frames with 3 independent image sequences (410, 420 and 430), visualized in the display order as shown in Fig. 4 (a), (b) and (c). In these figures, an example of converting standard definition (SD) video stream into a high definition (HD) video stream is illustrated. In the illustration, SD video 450 indicates that this stream is subjected to the second complexity processed SD to HD conversion. In a stream shown as HD video, a part 420 of the video stream is subjected to first complexity processing involving a higher complexity. At transitions, while combining the first complexity processed part 420 with the second complexity processed video stream (410, 430) a partial decoding of frames in the SD part may be required. The I-frames (411, 431) at transitions have to be decoded in order to decode the subsequent B and P-frames. The second complexity processing can do this decoding.
Abrupt variation in quality can be avoided or made smoother by a weighted combination. In Fig. 4(b) an overlapping sequence of four frames IBBP at the transition is shown. The video stream in this case changes from a low complexity processed, low quality output to a high complexity processed, higher quality output. The combination can be made such that the higher quality of the video is perceived gradually. The weights for IBBP frames in such a combination can be identified, for example as (0.2, 0.8), (0.4, 0.6), (0.6, 0.4), (0.8, 0.2) whereas for a set of weights (x,y) , x denotes the weight of a frame from the SD stream and y denotes the weight of a corresponding frame from the HD stream. There are various combination of weights that can be selected to satisfy the smoothing criterion. Similarly, the illustration in Fig 4 (c) shows a transition from a HD stream to a SD stream with an overlapping sequence of four frames PBBI. In this case the effect of HD is made to gradually reduce and the effect of SD is made to gradually increase. The same sets of weights, for example, as mentioned above can be used. This weighted combination will enable a gradual increase or decrease of output quality and it may result in a less annoying visual experience to the viewer compared to abrupt changes in visual quality.
Fig. 5 is a schematic illustration of a block diagram of a computer program product according to the invention. The computer program product (300) can be loaded into a computing machine and enable of operating the machine comprising a processing unit and a memory, the computer program product, after being loaded, providing said processing unit with the capability to carry out the processing of a video signal comprising temporal sequences of images. The computer program product provides the processing unit with the capability to analyze temporal image sequences and assign a priority value for each sequence indicating the preference of a first complexity processing of the temporal image sequences. The computer program product further provides the processing unit with the capability to carry out a first complexity processing and/or a second complexity processing of temporal image sequences. The computer-processing product to achieve the results according to the invention can control a single processing unit or a plurality of processing units.
The computer program product can be handled in a standard comprised or detachable storage, for example a flash memory or a compact disk or a hard disk. The computer program product can be embedded in a computing machine as embedded software or kept pre-loaded or loaded from one of the standard memory devices. The computer program product can be designed in any of the known codes such as machine language code or assembly language code and made to operate on any of the available platforms such as personal computers or servers.
The computer program product can be stored inside the apparatus or carried in a detachable storage medium such as an optical storage disc and transferred to the apparatus. The computer program product can be used in various forms of consumer apparatus for example, set top box, video receiver, video recorder/player, hand-held mobile communication or entertainment devices providing video displays. The computer program product can be implemented on various platforms such as personal computers or signal processors.
It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps other than those listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the system claims enumerating several means, several of these means can be embodied by one and the same item of computer readable software or hardware. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Claims

CLAIMS:
1. A video processing apparatus (100) for processing a video signal comprising at least one temporal image sequence, the apparatus comprising: a first storage means (110) arranged to receive and store the video signal, an image sequence analyzer (120) arranged to analyze the at least one temporal image sequence (TS) of the stored video signal and arranged to assign a priority value (PR) to the at least one temporal image sequence to indicate a preference for a first complexity processing of the at least one temporal image sequence , a first processor (130) arranged to process the at least one temporal image sequence (TS) according to the priority value (PR) and arranged to store the processed at least one temporal image sequence (TP) in a second storage means (110), a second processor (140) arranged to process the stored video signal (VS) according to a second complexity processing; and a combiner (150) arranged to combine the processed at least one temporal image sequence (TP) and the processed video signal (VP) into a combined video signal (CP).
2. A video processing apparatus (200) according to claim 1 wherein the first storage means (110) is arranged to comprise the second storage means (160).
3. A video processing apparatus (300) according to claim 1 wherein the combiner (150) is arranged to comprise the second storage means (160).
4. A video processing apparatus according to claim 1 wherein the received video signal is conforming to a standard definition television (SDTV) video signal and the combined video signal is conforming to a high definition television (HDTV) video signal.
5. A video processing apparatus according to claim 1 where the first processor is arranged to process the at least one temporal image sequence according to the priority value for spatial up-conversion.
6. A video processing apparatus according to claim 1 wherein the first processor is arranged to process the at least one temporal image sequence according to the priority value for image quality enhancement and/or noise removal.
7. A video processing apparatus according to claim 1 wherein the first processor is arranged to process the at least one temporal image sequence according to the priority value for three-dimensional depth calculation.
8. A video processing apparatus according to claim 1 wherein the combiner is arranged to combine the processed at least one temporal image sequence and the processed video signal by means of a weighted combination into the combined video signal.
9. A video processing apparatus according to claim 1 wherein the video processing apparatus further comprises input means arranged to receive an input and arranged to assign the priority value (PR) based on the input to the at least one temporal image sequence to indicate the preference for the first complexity processing of the at least one temporal image sequence.
10. A method of processing a video signal comprising at least one temporal image sequence, the method comprising: receiving and storing the video signal, analyzing the at least one temporal image sequence (TS) of the stored video signal and assigning a priority value (PR) to the at least one temporal image sequence to indicate a preference for a first complexity processing of the at least one temporal image sequence, processing the at least one temporal image sequence (TS) according to the priority value (PR) and storing the processed at least one temporal image sequence (TP), processing the stored video signal (VS) according to a second complexity processing; and - combining the processed at least one temporal image sequence (TP) and the processed video signal (VP) into a combined video signal (CP).
11. A computer program product to be loaded by a computer arrangement, comprising instructions for processing a video signal comprising at least one temporal image sequence, the computer arrangement comprising a processing unit and a memory, the computer program product, after being loaded, providing said processing unit with the capability to carry out the following tasks: receiving and storing the video signal, - analyzing the at least one temporal image sequence (TS) of the stored video signal and assigning a priority value (PR) to the at least one temporal image sequence to indicate a preference for a first complexity processing of the at least one temporal image sequence, processing the at least one temporal image sequence (TS) according to the priority value (PR) and storing the processed at least one temporal image sequence (TP), processing the stored video signal (VS) according to a second complexity processing ; and combining the processed at least one temporal image sequence (TP) and the processed video signal (VP) into a combined video signal (CP).
PCT/IB2006/052876 2005-08-22 2006-08-21 Video processing apparatus WO2007023440A2 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
EP05107667.7 2005-08-22
EP05107667 2005-08-22

Publications (2)

Publication Number Publication Date
WO2007023440A2 true WO2007023440A2 (en) 2007-03-01
WO2007023440A3 WO2007023440A3 (en) 2007-05-31

Family

ID=37671368

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IB2006/052876 WO2007023440A2 (en) 2005-08-22 2006-08-21 Video processing apparatus

Country Status (1)

Country Link
WO (1) WO2007023440A2 (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111258738A (en) * 2020-01-20 2020-06-09 深圳市商汤科技有限公司 View information processing method and device, electronic equipment and storage medium
CN112689200A (en) * 2020-12-15 2021-04-20 万兴科技集团股份有限公司 Video editing method, electronic device and storage medium

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020071663A1 (en) * 2000-12-07 2002-06-13 O'donnel John Setel Digital video recording system having multi-pass video processing
WO2005094060A1 (en) * 2004-03-02 2005-10-06 Koninklijke Philips Electronics N.V. Signal processing system
WO2005107264A1 (en) * 2004-04-30 2005-11-10 British Broadcasting Corporation Media content and enhancement data delivery

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020071663A1 (en) * 2000-12-07 2002-06-13 O'donnel John Setel Digital video recording system having multi-pass video processing
WO2005094060A1 (en) * 2004-03-02 2005-10-06 Koninklijke Philips Electronics N.V. Signal processing system
WO2005107264A1 (en) * 2004-04-30 2005-11-10 British Broadcasting Corporation Media content and enhancement data delivery

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111258738A (en) * 2020-01-20 2020-06-09 深圳市商汤科技有限公司 View information processing method and device, electronic equipment and storage medium
CN112689200A (en) * 2020-12-15 2021-04-20 万兴科技集团股份有限公司 Video editing method, electronic device and storage medium

Also Published As

Publication number Publication date
WO2007023440A3 (en) 2007-05-31

Similar Documents

Publication Publication Date Title
US20180192063A1 (en) Method and System for Virtual Reality (VR) Video Transcode By Extracting Residual From Different Resolutions
CN112073737B (en) Recoding predicted image frames in live video streaming applications
US6532593B1 (en) Transcoding for consumer set-top storage application
US7295757B2 (en) Advancing playback of video data based on parameter values of video data
US6507615B1 (en) Video signal processing apparatus and video signal processing method
US8275247B2 (en) Method and apparatus for normal reverse playback
US9058840B2 (en) Method of displaying moving image and image playback apparatus to display the same
EP1871098A2 (en) Processing of removable media that stores full frame video & sub-frame metadata
US20080152004A1 (en) Video coding apparatus
US20070217505A1 (en) Adaptive Decoding Of Video Data
EP2757793A1 (en) Video processor with frame buffer compression and methods for use therewith
US20100166081A1 (en) Video stream processing apparatus and control method, program and recording medium for the same
Lippman Feature sets for interactive images
US6891547B2 (en) Multimedia data decoding apparatus and method capable of varying capacity of buffers therein
US7343084B2 (en) Image processing apparatus and method, computer program, and computer readable storage medium
KR20040094441A (en) Editing of encoded a/v sequences
JP2011151721A (en) Image processing apparatus, image processing method, and program
WO2007023440A2 (en) Video processing apparatus
JP4902854B2 (en) Moving picture decoding apparatus, moving picture decoding method, moving picture decoding program, moving picture encoding apparatus, moving picture encoding method, moving picture encoding program, and moving picture encoding / decoding apparatus
US8363714B2 (en) Video stream modifier
JP4462559B2 (en) Trick playback content creation method and device, trick playback compressed video data transmission method and device, and trick playback content creation program
CN113228674B (en) Video encoding and video decoding
US7558466B2 (en) Moving image playback apparatus and its control method, computer program, and computer-readable storage medium
US20100232768A1 (en) Recording device, reproducing device, and method
US20090304089A1 (en) Reproduction processing apparatus, reproduction processing method, and computer program

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application
NENP Non-entry into the national phase in:

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 06795709

Country of ref document: EP

Kind code of ref document: A2