CN101076952A

CN101076952A - Compression rate control system and method with variable subband processing

Info

Publication number: CN101076952A
Application number: CNA2005800426450A
Authority: CN
Inventors: 约翰·D·罗尔斯顿; 克拉西米尔·D·克拉罗夫; 史蒂文·E·桑德斯
Original assignee: Droplet Technology Inc
Current assignee: Droplet Technology Inc
Priority date: 2004-10-12
Filing date: 2005-10-12
Publication date: 2007-11-21
Anticipated expiration: 2025-10-12
Also published as: JP2008516565A; WO2006042330A9; KR20070085316A; EP1800415A4; WO2006042330A3; WO2006042330A2; CN101076952B; AU2005295132A1; EP1800415A2; CA2583603A1

Abstract

Systems and methods are provided for compressing and decompressing still image and video image data in mobile devices. Corresponding mobile device architectures, and service platform architectures for transmitting, storing, editing and transcoding still images and video images over wireless and wired networks and viewing them on display-enabled devices are also provided.

Description

Mobile imaging application, apparatus structure and service platform structure

Related application

The application requires the priority of following provisional application: that submitted on October 12nd, 2004 is entitled as MOBILE IMAGING APPLICATION, DEVICEARCHITECTURE, the provisional application that No. the 60/618th, 558, the U.S. Patent application of AND SERVICE PLATFORM ARCHITECTURE; That submitted on October 13rd, 2004 is entitled as VIDEO MONITORING APPLICATION, DEVICEARCHITECTURES, the provisional application that No. the 60/618th, 938, the U.S. Patent application of AND SYSTEM ARCHITECTURE; That submitted on February 16th, 2005 is entitled as MOBILE IMAGING APPLICATION, DEVICE ARCHITECTURE, the provisional application that No. the 60/654th, 058, the U.S. Patent application of AND SERVICE PLATFORM ARCHITECTURE AND SERVICES; Its full content is incorporated into this with for referencial use.

The application is the U.S. Patent application of submitting on September 16th, 2004 the 10/944th that is entitled as MULTIPLECODEC-IMAGER SYSTEM AND METHOD, the continuity of No. 437 part, its be now on May 19th, 2005 the disclosed U.S. disclose US2005/0104752 number; That submitted on April 17th, 2003 is entitled as SYSTEM, the U.S. Patent application the 10/418th of METHOD AND COMPUTER PROGRAM PRODUCT FOR IMAGEAND VIDEO TRANSCODING, the continuity of No. 649 part, its be now on November 6th, 2003 the disclosed U.S. disclose US2003/0206597 number; That submitted on April 17th, 2003 is entitled as WAVELETTRANSFORM SYSTEM, the U.S. Patent application the 10/418th of METHOD AND COMPUTER PROGRAMPRODUCT, the continuity of No. 363 part, its be now on October 23rd, 2003 the disclosed U.S. disclose US2003/0198395 number; The U.S. Patent application the 10/447th that is entitled as PILE-PROCESSING SYSTEM AND METHODFOR PARALLEL PROCESSORS that on May 28th, 2003 submitted to, the continuity of No. 455 part, its be now on December 11st, 2003 the disclosed U.S. disclose US2003/0229773 number; The U.S. Patent application the 10/447th that is entitled as CHROMATEMPORAL RATE REDUCTION AND HIGH-QUALITY PAUSESYSTEM AND METHOD that on May 28th, 2003 submitted to, the continuity of No. 514 part, its be now on December 25th, 2003 the disclosed U.S. disclose US2003/0235340 number; The U.S. Patent application the 10/955th that is entitled as SYSTEM AND METHOD FORTEMPORAL OUT-OF-ORDER COMPRESSION AND MULTI-SOURCE COMPRESSION RATE CONTROL that on September 29th, 2004 submitted to, the continuity of No. 240 part, its be now on May 19th, 2005 the disclosed U.S. disclose US2005/0105609 number; The U. S. application of submitting on September 20th, 2005 that is entitled as COMPRESSION RATE CONTROL SYSTEM AND METHOD WITHVARIABLE SUBBAND PROCESSING (agent's document 74189-200301/US number) the _ _ _ _ number the part continuity, it requires the priority of No. the 60/612nd, 311, the provisional application submitted on September 21st, 2004; The U. S. application of submitting on September 21st, 2005 that is entitled as MULTIPLE TECHNIQUE ENTROPYCODING SYSTEM AND METHOD (agent's document 74189-200401/US number) the _ _ _ _ number CIP (part continuity), it requires the priority of No. the 60/612nd, 652, the provisional application submitted on September 22nd, 2004; The U. S. application of submitting on September 21st, 2005 that is entitled as PERMUTATIONPROCRASTINATION (agent's document 74189-200501/US number) the _ _ _ _ number CIP (part continuity), it requires the priority of No. the 60/612nd, 651, the provisional application submitted on September 22nd, 2004; Its full content is incorporated into this with for referencial use.The United States Patent (USP) the 6th that is entitled as MULTIPLECODEC-IMAGER SYSTEM AND METHOD that the application also publishes on November 30th, 2004,825, the United States Patent (USP) the 6th that is entitled as SYSTEM ANDMETHOD FOR A DYADIC-MONOTONIC (DM) CODEC No. 780 and that on January 25th, 2005 published, 847, No. 317 full content is incorporated into this with for referencial use.

Technical field

The present invention relates to data compression, more particularly, the still image that relates in mobile device writes down and the video image record, relates to being used for still image and video image being transmitted, store, editing and become the corresponding mobile device structure of sign indicating number and that on visual device they are shown and across a network and device distribution and upgrading coding decoder and serve platform structure on wireless and cable network.

Background technology

The still image of Direct Digitalization and video need many " positions ".Therefore, generally in order to store, to transmit and other application and compressed image and video.Known several basic compression methods, and the many specific modification in these methods.Usual way is feature with the tertiary treatment: conversion, quantification and entropy coding.Many images and video compressor are shared this and are had the basic structure of various variations.

The information that the purpose of conversion stage is harvest energy (energy) or source picture in the video compressor is so that local similar and pattern are converted into a kind of form of compression as far as possible in this picture or the sequence by utilizing.Compressor reducer is designed to work well in " typical case " input, fails to compress " at random " or " irrational " input and may neglect it.Use discrete cosine transform (DCT) as conversion stage such as many image compression such as MPEG-2 and MPEG-4 and video-frequency compression method.Compression waits some newer image compression and video-frequency compression method to use various wavelet transformations as conversion stage such as the MPEG-4 static structure.

Quantize behind this conversion stage, just to abandon usually information.Thereby the decompressing image of process reconstruct is not the complete regeneration of original image just.

The entropy coding process is generally harmless step: this step obtain the remaining information after the quantification and as a rule it is encoded so that its in decoder by complete regeneration.Therefore be not subjected to the influence of following entropy coding level usually with the relevant design decision which kind of information will abandon in the transform and quantization level.

Video compression/decompression (coding decoder) technology limitation based on DCT is exactly, originally be developed and be used for video broadcasting and stream is used the coding that these technology depend on the video content of studio environment, wherein high complexity encoder may operate on the computer workstation.The encoder of this calculation of complex allows to calculate simple and relatively inexpensive decoder (player) and is installed in user's playing device.Yet the matching of this asymmetric coding/decoding technology and mobile multimedia device (wherein may catch (and coding) and displaying video message in mobile phone self inside in real time) is very poor.Because relatively little computing capability and the power supply of mobile device, the result, (in other consumer products) video image in the mobile device is limited to very little picture size and very low frame rate usually.

Summary of the invention

The shortcoming that the present invention is directed to existing compress technique some solutions have been proposed and provide can be used as full software (or mixing) application program and on cell phone, carry out, high complexity but the image compression (coding decoder) of efficient calculation, thereby reduced the complexity of handset structure and the complexity of mobile imaging service platform structure.The many aspects of full software of the present invention or hybrid video coding decoder solution have reduced or eliminated the baseband processor in the multimedia handset and the cost and the needs of video accelerator basically.Combine with download the function that the coding decoder post-production is installed by OTA, the present invention in full software or the hybrid solution has reduced complexity, risk and the cost of mobile phone development and video messaging service structure and deployment basically.And according to many aspects of the present invention, the software video code converter starts the MMS that has disposed the control automatic inter-network of (MMSC) foundation structure (OTN) upgrading that cell phone is carried out and coding decoder is disposed or upgrading.Small echo code converter of the present invention provides complete interoperability between small echo video format and other measured and proprietary video format for operator.This full software or mixed video platform allow not have prior art can with, the new MMS service of Balance Treatment speed and video product accuracy disposes fast.This wavelet coding decoder they efficient processing still image and the ability of video aspect also be unique, and therefore can replace independent MPEG and JPEG coding decoder with the single low cost that can support dynamic picture mail and video messaging service and other service simultaneously and low-power consumption solution.

Description of drawings

Fig. 1 shows the actual displayed size between the ordinary video display format and the difference of resolution.

Fig. 2 schematically shows the system that is used for combined signal source-chnnel coding.

Fig. 3 shows the mobile imaging handset structure.

Fig. 4 shows mobile imaging service platform structure.

Fig. 5 has schematically compared the difference of the processing resource between DCT encoder and the improvement wavelet coders of the present invention.

Fig. 6 schematically shows the improvement system that is used for combined signal source-chnnel coding.

Fig. 7 shows improved mobile imaging handset structure.

Fig. 8 shows improved mobile imaging service platform structure.

Fig. 9 shows the block diagram of the aerial upgrade that is used to carry out video gateway.

Figure 10 shows the enforcement option that is used for the software imaging applications.

Figure 11 shows the enforcement option that is used for hardware-acceleration imaging applications.

Figure 12 shows the enforcement option that is used for mixed hardware acceleration and software imaging applications.

Embodiment

Image processing based on small echo

Wavelet transformation comprise with one dimension or more than the form repeated application wavelet filter of one dimension to data set.For the still image compression, can use 2-D wavelet transformation (level is with vertical).Video coder-decoder can use 3-D wavelet transformation (level, vertical and time).Improved, symmetrical 3-D video compression/decompression (coding decoder) device based on small echo can be reduced to the computation complexity of mobile device and power consumption based on needed those computation complexities of the coding decoder of DCT with below the power consumption satisfactorily, and can support to handle still image and video image in single coding decoder simultaneously.This method of supporting still image and video image in single coding decoder simultaneously can be eliminated the needs to independent MPEG (video) and JPEG (still image) coding decoder, perhaps greatly improve compression performance and therefore come with the relevant storage efficiency of mobile JPEG coding decoder.

Mobile picture message transmission

According to many aspects of the present invention, promoted the more abundant content of cell phone and service field, it utilizes more bandwidth and brings higher income per capita (ARPU) for the mobile service provider.The multimedia that mobile multimedia service (MMS) is based on the Short Message Service (SMS) of text develops.Many aspects of the present invention have made things convenient for new MMS to use.This new application is exactly the video messaging transmission.According to the present invention, video messaging is transmitted as the demand of response target audience personal information communication interchange and highly improved system is provided.This mobile picture message transmission need be added digital camera functionality (still image) and/or field camera function (video image) to cell phone, they wanted the video messaging that sends so that the user both can catch (coding), and can play (decoding) their video messaging of receiving again.

Even exist some to move picture message transmission service and application at present, they be limited in catching with other multimedia device (see figure 1)s such as transfer ratio such as TV, personal computer and digital camera on catch is much smaller with the video image displayed size and frame rate is much lower usually video image.As shown in Figure 1, minimum current format, SubQCIF110 (SubQ-CLV Common Intermediate Format) be 128 pixels (picture element) wide * 96 pixel height, QQVGA 120 (QQ-vector graphics array) is 160 * 120 pixels, QCIF 130 is 176 * 144 pixels, QVGA 140 is 320 * 240 pixels, CIF 150 is 352 * 288 pixels, and VGA 160 is 640 * 480 pixels, and the form of current maximum, D1/HDTV (HDTV (High-Definition Television)) is 720 * 480 pixels.Can support that frame rate is that mobile picture message transmission service and the application of 30fps or higher (as being provided by many aspects of the present invention and starting) VGA (or bigger) is long-expected.

Adaptability combined signal source-chnnel coding

With such as text, audio frequency and still image etc. other the transmission of data/medium type compare owing to need higher data rate usually, the video transmission on the mobile network is challenging in itself.In addition, limited and the channel width that changes are brought further restriction and difficulty with mobile network's vibration noise and error characteristics to video transmission.According to many aspects of the present invention, can use various combined signal source-channel coding technologies to make video bit stream be adapted to different channel condition (see figure 2)s.And combined signal source-channel coding method of the present invention can be measured (scalable), so that adapt to the channel width and the error characteristics of variation.Further, its supports the scalability of multicast scene, wherein, has different restrictions at the different device of the receiving terminal of video flowing on decoding computing capability and Presentation Function.

As shown in Figure 2, and according to many aspects of the present invention, at first by 220 pairs of source video sequence 210 source codes of source encoder (that is, compression), then by error correcting code (ECC) chnnel coding 230 to its coding.In the prior art mobile network, source code usually use such as H.263, MPEG-4 or motion JPEG etc. are based on the compress technique of DCT.This coding techniques can not can carry out real-time adjustment to the hierarchy compression of carrying out in the source encoder as in the present invention.Particularly when real-time video or be hunted down near real-time, encode and when transmitting by communication network (embodiment that is hunted down, encodes and store to be used for transmission subsequently with video compares), the present invention just provides significant advantage on the one hand at this.Exemplary channel coding method has Reed-Solomon compiling method, Bose-Chaudhuri-Hocquenghem Code method, FEC compiling method and Turbo compiling method.The joint source-channel coding video bit stream is complementary with regard to the demand of transmitting through-rate controller (Rate control) 240 and channel width subsequently, and reaches the best video quality through reconstruct simultaneously.Rate controller 240 sends video bit stream 250 with the discrete velocity distortion computation of transmission by the relevant compressed video bitstream of execution before the channel 260 at it.Because the finiteness of computing capability aspect in the mobile device, common rate controller is only considered available channel width, and does not consider the error characteristics of transmission channel clearly.According to many aspects of the present invention, source encoder has the function of the compression adjusted, so that obtain the various values of little to 1% to 5% and 1% to 10% compression ratio.When different compressibility factors be applied to separately but the data of representing one or more vedio datas together when giving frequency band just have this function especially.

Shown in Fig. 2 b, in decode procedure, receive joint source-channel coding bit stream 250 and in step 270, it is carried out the ECC channel-decoding, and in step 280, it is carried out the source decoding to present reconstructing video 290 by channel 260.

The invention provides based on having the more improvement adaptability joint source-channel coding of the algorithm of Computationally efficient, therefore, instant and channel width prediction and error condition can be used among source encoder 220, channel encoder 230 and rate controller 240 all these threes, realize the maximization of the instant and average quality (relation between video rate and the distortion) of the vision signal of reconstruct is controlled.

Also allow cellular carrier and MMS service provider to provide wider service quality (QoS) performance and price level by improved adaptability joint Source-Channel Coding Technology provided by the present invention, therefore by using their maximum revenue that wireless network infrastructure produced for its consumption user and corporate client.

The multicast scene needs the single adaptive video bit stream that can be decoded by many users.This is even more important in extensive heterogeneous network of modern times, and wherein the restriction of the network bandwidth makes that to be transmitted as the special tuning a plurality of radio hookup vision signals of each user unrealistic.The multicast of single adaptive video bit stream has greatly reduced bandwidth demand, is used for a plurality of users video bit stream of (comprise the high-end user with broadband wireless or wired connection and have the wireless telephone user that finite bandwidth is connected with fallibility) but need generation to decode.Because the limitation of the computing capability of mobile device, the granularity of adaptability rate controller is very coarse usually, for example only generates 2 layers of bit stream that comprise a basal layer and a reinforced layer.

By being that it is at the heterogeneous network that can support higher level aspect channel type (wireless and wired), channel width, interchannel noise/error characteristics, user's set and the user's service based on another advantage that more the present invention of the algorithm of Computationally efficient improves the adaptability joint source-channel coding and provided is provided.

The mobile imaging handset structure

Referring now to Fig. 3,, append to that portable digital cameras function on the cell phone relates to following hardware, software or as the function of the combined aspects of its hardware and software:

. have the corresponding preamplifier and the imager array 310 (being generally CMOS or CCD pel array) of analog to digital (A/D) signaling conversion circuit

. such as the image processing function 312 of pre-treatment, coding/decoding (coding decoder), reprocessing

. be used for by the non real-time transmission of wireless or cable network or the buffering area 314 of the real-time processing image that flows

. such as one or more image display panels of touch-screen 316 and/or color monitor 318

. the local video memory on internal memory 320 or the changeable storage 322

The coding decoder of use such as MPEG-4, commercial cell phone available, can imaging based on dct transform be limited to can only catch with usually on such as other multimedia device such as TV, personal computer and portable digital video camera catch compare with video image displayed size will be little and frame rate will be low video image.These devices of back are usually with the above form of VGA (640 * 480 pixel), catch/display video image with 30 frame per seconds (fps) or higher demonstration speed, and to recently seeing, commercial cell phone available, can imaging is limited to can only be with QCIF (176 * 144 pixel) or littler form, come capture video images with 15fps or lower demonstration speed.The Video Capture ability that this is lowered is owing to needed excessive power consumption of processing unit of quantity, type and sequence and the buffering memory finished relevant for the calculation procedure of the video compression/decompression of using dct transform.Even, also need custom-designed integrated circuit (IC) chip is implanted in the mobile phone hardware so that finish compression and decompression for this Video Capture function through the commercial available cell phone that reduced.

Use commercial available video coder-decoder and microprocessor technology can cause, power consumption very complicated so that 30fps or higher frame rate are caught the mobile imaging mobile phone of VGA (or bigger) video and expensive structure, and it have very long design and make delivery cycle (lead times) for attempting.This handset structure need utilize software program and the coding decoder of the further combination of hardware accelerator and the bigger buffer storage piece 314 (typical memory capacity is more than the 1MB) in the combinations such as operating in reduced instruction set computer (RISC) processor 324, digital signal processor (DSP) 326, application-specific integrated circuit (ASIC) (ASIC) 328 and reconfigurable processing unit (RPD) 330.These codec functions can use such as the integrated circuit (IC) of the such separation of risc processor 324, DSP 326, ASIC 328 and RPD 330 and implement, or being integrated in packaging assembly system (system-in-a-package together, SIP) or system-on-a-chip (system-on-a-chip, SoC) Nei one or more risc processors 324, DSP326, ASIC 328 and RPD 330 combine.

The risc processor 324 or the codec function on the DSP 326 that operate in above combination of hardware can be software routines, and they have for error correction and function upgrading purpose and this advantage that can be modified at any time.The shortcoming of carrying out the codec function of some complexity, repetition as software is that whole processor resource that is produced and power consumption needs surpass the available quantity in the mobile communications device usually.The codec function that operates on the ASIC 328 is generally mounting hardware enforcement complicated, repeated calculation procedure, and it has such advantage: the hardware-accelerated overall power consumption that can reduce coding decoder greatly of custom-made.The shortcoming of carrying out some codec function in mounting hardware comprises: the longer and more expensive design cycle, the risk of recall expense costliness and will being added in function newly developed can not be finished under the situation of imaging application the fixing function of silicon is upgraded under the situation of discovery mistake in the execution of fixing silicon.The codec function that operates on the RPD 330 had normally both needed hardware-accelerated also requirement can add or revise a plurality of routines of terminal mobile imaging mobile phone products function.The shortcoming of carrying out some codec function on RPD 330 is: implement to compare with fixing ASIC 328, need more substantial Si-gate and Geng Gao power consumption with support hardware reconstruct.

The imaging applications of being constructed has reduced or eliminated codec function complicated, repeatability so that make the mobile imaging mobile phone catch VGA 160 (or bigger) video with the frame rate of 30fps with full software configuration according to certain aspects of the invention.This arrangement has been simplified above structure and has been made mobile phone cost and business promotion in enormous quantities adapt.

New multimedia handset may also need not only to support picture and video messaging function, and support multiple additional multimedia function (voice, music, figure) and radio access pattern (2.5G and the access of 3G honeycomb, WLAN, bluetooth, GPS etc.).For the product recall of promoting new extra earning service and application more efficiently and avoiding both expensive, relate to exploitation, promote and support the complexity of this product and risk to make aerial (OTA) distributions of many function and applications and the management urgent need that becomes.Make that by full software imaging applications that many aspects of the present invention provided the OTA that carries out imaging applications by mobile operator distributes and management becomes possibility.

Mobile Java uses

The Java technology again to mobile device, focuses under a certain language and a kind of technology from the server to the desktop computer together installing on a large scale.Although be used for the application difference of the device of this scope, the function served as bridge that the Java technology plays in these difference of its statistics, thereby the developer who allows at some psychomotor domains can crossover device and range of application bring into play the technical ability of his door.

At first the J2ME (Java2, miniature version) that is incorporated into the Java field in June, 1999 by the Sun micro-system is the part among the extensive motivation that satisfies the different demands of Java developer better.Use Java 2 platforms, Sun has redefined Java technical pattern, and it is classified as three versions.Standard Edition (J2SE) is developed for desktop and commercial application of low side provides actual solution.Enterprise's version (J2EE) is at being absorbed in the developer that corporate environment is used.Miniature version (J2ME) is to introduce for using such as PDA, cell phone, beep-pager, TV set-top box, remote measurement parts and many other consumption user electronics and developers with device work of limited hardware resource of embedded equipment or the like.

The target of J2ME be to have little to 128KB RAM and have than the device of the much smaller processor of the function that is applied in the processor on typical desktop computer and the server unit.In fact J2ME is made up of a configuration set file (profile).Every kind of configuration file all is defined for the device (cell phone, PDA etc.) of particular type, and all comprises: the minimum class libraries collection that the particular type device is required and for supporting the standard of the Java Virtual Machine that this device is required.The virtual machine of appointment needn't be the same with employed virtual machine in Java2 Standard Edition (J2SE) and the Java2 enterprise version (J2EE) in any J2ME configuration file.

For all devices listed above all define single J2ME technology optimum or approaching optimum is very difficult.Difference aspect processor ability, memory, permanent memory and user interface is too big really.For addressing this problem, Sun divides the device definition that is suitable for J2ME and is subdivided into plurality of sections then again.Use first to divide (slice), the handle assembly that should be used for that Sun does not look ahead based on disposal ability, memory and memory capacity is divided into two wide in range kinds.The said firm defines the Java language compact version in the constraint that will be operated in those devices in each kind then again, and MIN at least Java language function still is provided simultaneously.

Secondly, use similar role identification in Sun each in these two device kinds---therefore, for example, no matter manufacturer is for Whom, all cell phones all drop in the class.Partner in its Java standardization body (JCP) helps down, and just be directed to each vertical division specially defines additional function to Sun then.

First divides (division) has created two J2ME configurations: jockey configuration (CDC) and limited jockey configuration (CLDC).Configuration is the selected group of Java Virtual Machine (JVM) that runtime environment is provided for device, and minimum class libraries collection and minimum API collection.The least common denominator subclass of Java language has been specified in configuration, that is, a kind of by the suitable language that is used for the resource constraint that its device family that once was developed brings.Even owing to also in user interface, function and application facet big like this variation is arranged in a kind of the configuration, Typical Disposition can not define such part and parcel as definition of user's interface tool set and permanent storage API.On the contrary, the definition of this function belongs to a kind of thing that is called configuration file.

The J2ME configuration file is that it is meant addressing such as beep-pager and the so concrete device class of cell phone by the Java API collection of the leading group of industry appointment.Every kind of configuration file all is provided in by the top of the Java language least common denominator subclass that is provided by its configuration, and is meant replenishing this configuration.The two kind configuration files important to mobile handsets are: replenish the configurations file of CDC and the mobile information apparatus configuration file (MIDP) of additional CLDC.Work contains more configuration file, should occur soon in standard and the reference implementation.

Be used for the Java technology (JTWI) of wireless industrial standard, JSR 185, defined the follow-on industrial standard platform of the mobile phone that is used to use the Java technology.JTWI is to be by the definition of the expert group of main mobile device manufacturer, cellular carrier and software vendor by Java standardization body (JCP).JTWI has specified and must be included in all devices that meet JTWI (CLDC 1.0 (JSR 30), MIDP 2.0 (JSR 118) and WMA1.1 (JSR 120), and, if applicable, CLDC 1.1 (JRS 139) and MMAPI (JSR 135)) within technology.Two additional JTWI standards that defined the technology that is used for the mobile multimedia device and interface are JSR-135 (" mobile media API ") and JSR-234 (" high level multimedia replenishes ").

The JTWI standard has increased the function bar that is used for high capacity machine, minimizes the API fragment simultaneously and expanded to be developed the basic foundation that is used for mobile telephone application.The advantage of JTWI comprises:

. interoperability: the target of this work is for the Application developer provides predictable environment, and provides referable function collection for equipment manufacturers.The benefit that both adopt this JTWI standard all is huge: manufacturer has therefrom obtained wide in range compatible application, and the software developer has therefrom obtained the equipment of wide in range its application program of support.

. security specification: JSR 185 standards for be defined in MIDP 2.0 standards in " security strategy of advising at the device of deferring to GSM/UMTS " relevant unreliable application introduced many explanations.It has expanded the basic MIDlet suite security framework that is defined in the MIDP 2.0.

. figure shows the way: the importance of JTWI standard is the figure that shows the way, the general picture of the common function that it can be expected in deferring to the device of JTWI for the software developer.See its first in February, 2003 in a series of figure of showing the way of expectation with six to nine months interval appearance, it has been described and the corresponding to additional function of the development of mobile phone.This figure that shows the way makes each side have more confidence to the plan in future: operator can plan their application deployment strategy better, equipment manufacturers can determine their product plan better, and the content development personnel can see the path more clearly that is used for its application and development work.In the future, operator depend on especially Java VM come from current erosion public internet such as extraction/protection low layer radio frequency/network function the security breaches of virus, worm and other " attack ".

According to many aspects of the present invention, aforementioned imaging applications is based on Java's, to consider aspect all mobile phones, the Java VM safety that can resist virus, worm and other mobile network's safety " attacks " and the powerful property of mobile phone/network and the download portability of " once write, move everywhere " on the process of the OTA coding decoder of simplifying with Java function.According to aspect further, follow JTWI standard JSR-135 (" mobile media API ") and JSR-234 (" high level multimedia replenishes ") based on the imaging applications of Java.

Mobile imaging service platform structure

The assembly (see figure 4) of mobile imaging service platform structure may comprise:

. cell phone 410

. mobile base station (BTS) 412

. base station controller/radio frequency network controller (BSC/RNC) 414

. mobile switching center (MSC) 416

. gateway service node (GSN) 418

. mobile multimedia service controller (MMSC) 420

The exemplary functions (see figure 4) that is included in the MMSC comprises:

. video gateway 422

. telecommunications company (Telco) server 424

.MMS application server 426

. storage server 428

Video gateway 422 in the MMSC 420 provides between the different video form that the imaging service platform is supported and becomes the sign indicating number service.Become sign indicating number also is used for supporting to be applied to mobile telephone network by cellular carrier different speech codec, and its corresponding voice code converter is integrated among the RNC 414.Process with this mobile imaging service platform of the structure updating shown in Fig. 4 generally includes: dispose new cell-phone 410, and manually add new hardware in MMSC420 video gateway 422.

The automatic OTA upgrading of the mobile phone that the full software mobile imaging application service platform support of constructing according to many aspects of the present invention is disposed, and the automatic OTN upgrading of the MMSC 420 that is disposed.The Java of above-mentioned cell phone imaging applications has implemented to provide that antagonism is antiviral, worm and other " attacks ", the powerful property of the improved mobile phone/network of institute, thereby the permission Mobile Network Operator provides desired service quality of national legislation person and reliability.

The mobile video messenger service maps out the plan and exposes the basic limitation relevant with current video compression technology.On the one hand, this mobile video service will be put to and be equal to now and have home theater quality broadcasting the video of (such as the full size image form with the VGA 160 of 30 frame per seconds).On the other hand, use original development to be used to broadcast and the processing procedure of this mass data that flows the existing video technique of application has substantially exceeded the computational resource and the power of battery that the real-time videos that can be used in the cell phone 410 are caught (coding).The coding of broadcasting and the video content of stream application-dependent in the environment of studio, the encoder of its medium or high complexity may operate on the computer workstation.Owing to must catch video messaging in real time in mobile phone self, it just is confined to very little size and very low frame rate.

The result, expect for a long time to compare from the video imaging service that the digital portable video camera that is equipped with the function video receiver that is used for answer (replicate) obtains with the user, present mobile video imaging service is original: picture little (QCIF 130) and discontinuous (10fps).Offer now the mobile subscriber the raw video image quality also lack very much the clear and definite high definition video that the industry is propagated in the life style advertisement.The mobile subscriber extensively adopt be field camera phone and relevant mobile video messenger service payment services expense before, VGA 160,30fps performance (that is, as their hand-held camera) that needs are complete.Because their 2.5G and 3G commercial modules seriously are subjected to risk, cellular carrier is urgently being sought the solution feasible to above problem.

Even after extremely expensive and development sequence consuming time, competitive video coder-decoder provider still can only be for VGA 130,30fps performance provide complicated hybrid software coding decoder and hardware accelerator solution, and its whole cost and power consumption are considerably beyond business needs and technical capability.Therefore mobile phone is limited to little discontinuous image, or expensive energy-dissipation structure.Big expensive service arrangement and too low service quality makes it can not penetration and promotion in the general marketplace scope.

New if desired hardware, upgrading MMSC foundation structure 420 also is expensive.In order to realize preferentially to choose full software ASP platform to the automatic OTA upgrading of mobile phone with to the OTN upgrading of MMSC 420 video gateway 422.

Improved image processing based on small echo

According to an aspect of the present invention, the 3-D wavelet transformation can by utilization design aspect the computation complexity far below video compression/decompression (coding decoder) device 410 based on coding decoder 420 (see figure 5)s of DCT.Can utilize 3-D wavelet coding decoder according to certain aspects of the invention to reduce significantly such as employed processing resource in color recovery and demodulation 430, image transform 440, memory 450, motion estimation 460/ time change 470 and quantification, rate controlled and entropy coding 480 these processing procedures.This application of wavelet conversion level also makes the design of quantification and entropy coding level have the computation complexity that reduces greatly.Further advantage according to the 3-D wavelet coding decoder 410 that is developed the particular aspects of the present invention that is used for mobile imaging application, device and service comprises:

. symmetry, low complex degree video coding and decoding

. not only be used for software but also be used for the lower part reason device ability need that the hardware encoding decoder is implemented

. have full software coding and decoding with the compatible mutually processor demand of existing commercial cell phone, use as local code and Java with VGA 160 (or bigger) video of the frame rate of 30fps

. be used for the integrated lower door number ASIC kernel of SoC

. low-buffer storage requirement more

. support still image (～JPEG) and video (～MPEG) single coding decoder

. owing to the simplification video editing (shearing, insertion, text cover) of shorter set of pictures (GOP)

. owing to shorter GOP synchronous with simplification speech codec

. owing to the low delay that is used to strengthen video flowing of shorter GOP

. be used for the fine granular yardstick of adaptability rate controlled, multicast and joint source-channel coding

. according to the low complex degree performance of emerging HDTV video format

According to many aspects of the present invention, above advantage is by unique combination obtains to following technology.

Use the wavelet transformation that promotes the short binary integer filter factor in the structure: for example, can use Haar, 2-6, and 5-3 small echo and their mutation.These uses add, subtract and little fixed point displacement---do not need multiplication or floating-point operation.

Lifting scheme calculates: can use the lifting scheme that allows original place (in place) to calculate to calculate above filter easily.Can be at Sweldens, Wim, The LiftingScheme:A custom-design construction of biorthogonal wavelets.Appl.Comput.Harmon.Anal.3 (2): 186-200, find the complete description to lifting scheme in 1996, its full content is incorporated into this with for referencial use.In this application, implement this lifting scheme and make the use of register and interim ram location minimize, and keep the localization of quoting efficient use high-speed buffer.

Wavelet transformation with pyramid form of customization pyramid structure: can on the basis of a half data that generates by last small echo level, advantageously calculate each level of wavelet transformation sequence, so overall calculation almost is independent of the quantity of level.This pyramid can be customized the advantage of balanced above lifting scheme and further save register and used and the cache memory bandwidth.

Block structure: compare with most of wavelet compression embodiments, picture can be divided into rectangular blocks easily, each piece all with the processing of coming of other piece branch.This will keep localization and can use the data that are retained in the processor high speed buffer storage to finish whole conversion pyramid with regard to allowing memory reference, thereby the mass data of having saved in most of processors moves.Block structure is particular importance in hardware embodiment, and this is because it has avoided the interior demand to big intermediate storage capacity of signal flow.

Block boundary filter: can all use improved filter to calculate easily on the border of each piece to avoid sharp-pointed pseudomorphism, be entitled as WAVELET TRANSFORM SYSTEM as what submit to applicant April in 2003 17, the U. S. application the 10/418th of METHOD AND COMPUTERPROGRAM PRODUCT, described in No. 363, its publication number is 2003/0198395, and its full content is incorporated into this with for referencial use.

The colourity time removes: in certain embodiments, can avoid handling the color difference signal in each territory, use single colourity territory for GOP on the contrary.This is described in the U. S. application the 10/447th that is entitled as CHROMA TEMPORAL RATE REDUCTION ANDHIGH-QUALITY PAUSE SYSTEM AND METHOD that the applicant submitted to May 28 in 2003, in No. 514, its publication number is 2003/0235340, and its full content is incorporated into this with for referencial use.

Use the time compression of 3D small echo: in certain embodiments, do not use such as the calculating aspect of conventional video compression methods such as MPEG very expensive motion search and operation of motion compensation.On the contrary, can computational fields to the time wavelet transformation in territory.This is a kind of very cheap calculating.The preferred short integer filter that has this paper lifting scheme that uses.

Binary quantizes: in certain embodiments, the quantization step of compression process is operated by unified use binary system displacement on the coefficient positions scope and is finished.This has just been avoided desired every sample multiplication of conventional quantization or division.

Pile up: in certain embodiments, at first reduce the data volume for the treatment of that entropy coder is handled by carrying out zero stream (run-of-zeros) conversion.Preferably, use is with the method for parallel processing structure to zero flow accounting, the U. S. application the 10/447th that is entitled as PILEPROCESSING SYSTEM AND METHOD FOR PARALLELPROCESSORS as applicant's 28 submissions May in 2003, described in No. 455, its publication number is 2003/0229773, and its full content is incorporated into this with for referencial use.Attention: most of modern processing platforms have some parallel function that can use by this way.

Circulation practical entropy coding: in certain embodiments, use the tradition table is searched the entropy coding step that the technology that combines with direct calculating on the incoming symbol is finished compression process.Symbol distribution characterization in source static image or the video has been caused using such as these simple entropy coders such as Rice-Golomb, exp-Golomb or binary dullnesses.The selection of entropy coder details is often according to the processor platform changes of function.The details of Rice-Golomb and exp-Golomb encoder is described in the following document: Golomb, S.W. (1966), " Run-length encodings ", IEEE Transaction onInformation Theory, IT-12 (3): 399-401; R.F.Rice, " Some PracticalUniversal Noiseless Coding Techniques ", Jet Propulsion Laboratory, Pasadena, California, JPL publishes 79-22, in March, 1979; And J.Teuhola, " A Compression Method for Clustered Bit-Vectors ", InformationProcessing Letters, the 7th volume, 308-311 page or leaf, in October, 1978 (having introduced term " exp-Golomb ").The details of the dull encoder of binary is described in No. the 6th, 847,317, the United States Patent (USP) that is entitled as SYSTEM AND METHOD FOR ADYADIC-MONOTONIC (DM) CODEC that the applicant announced on January 25th, 2005.More than the full content of each list of references all be incorporated into this with for referencial use.

Rate control

The method of a kind of adjustment decrement (speed of the carry-out bit that is generated) is exactly to change the amount of information that is abandoned in the quantized level of this calculating.Quantize usually by with each coefficient all divided by previously selected number (i.e. " quantization parameter "), and the remainder that abandons this division is finished.Therefore, the scope of coefficient value can be represented by identical single value (that is the merchant of this division).

When compressed image or GOP were decompressed, this inverse quantization treatment step multiply by the merchant (known) quantization parameter.This just returns to these coefficients the scope of its original size, to be used for further calculating.

Yet in many execution modes, division (or multiplication of equivalence) aspect power and the time loss and on hardware cost all is being the high operation of cost.Notice that this quantization operation is applied to each coefficient, and, the quantity of coefficient usually with the quantity of input pixel as many.

In other method, except division (or multiplication), quantize also to be restricted to the approximate number of 2 power.This can be by the advantage that the displacement operation of binary number is implemented with regard to having.Displacement is the very low operation of a kind of cost in many enforcements.Example is the enforcement of integrated circuit (FPGA or ASIC); Multiplier circuit is very big, but shifter circuit is just little a lot.In addition, compare with shifting function, on many computers, multiplication also needs the longer time to finish, and parallel work-flow still less perhaps in commission is provided.

Though quantize for calculating very effectively by the displacement process, have defective in some respects: it can only adjust compression ratio (carry-out bit speed) cursorily.According to many aspects of the present invention, can observe in practice: but nearly 2 times variation of the bit rate that generates by minimum energy (+1 or-1) change that quantizes shift parameters has been caused.Concerning some compression applications, this is an acceptable.Use for other, then need meticulousr ratio control.

Do not abandon the efficient of displacement quantization for the roughening problem that overcomes above-mentioned prior art, this quantizes just by unitized.Replace as in the past each coefficient all being used single public displacement parameter, we also provide different displacement parameters, so that it is applied to each zero independent stream compressed storage area or accumulation.The parameter value of each this district or accumulation all is recorded in the compression output file.Piling up is data store organisation, and wherein, data are represented with zero (or other common value) sequence of compression.Should be noted that sub-band can comprise several independent accumulations or memory block.In addition, accumulation or memory block can comprise several independent sub-bands.

Now, this solution makes the scope of significance bit speed be applied between resulting immediate two speed of quantization parameter of all coefficients by unified.For example, consider that all sub-bands all use identical quantization parameter Q except a sub-frequency bands (sub-band x), and that sub-band (sub-band x) uses the situation of Q+1.The whole bit rate that from quantization step, generates with in quantification, all use Q and compare and reduced for all sub-bands, but do not drop to the degree of all using Q+1 for all sub-bands.This just provides the interposition speed between unified Q of application or the resulting bit rate of the unified Q+1 of application, thereby has provided better, meticulousr compression control.

Notice that this computational efficiency almost completely is equal to the computational efficiency of pure displacement quantization, this is still to be shifting function usually because be applied to the operation of each coefficient.Can use any amount of sub-band.Be generally four to 100 sub-frequency bands.Be typically most 32 sub-frequency bands.Ask in the U.S. that is entitled as COMPRESSION RATE CONTROL SYSTEM AND METHOD WITHVARIABLE SUBBAND PROCESSING (agent's document 74189-200301/US number) that the further information of relevant rate controlled was submitted on September 20th, 2005 by the applicant _ _ _ _ number in provide, its full content is incorporated into this with for referencial use.

Improved adaptability joint source-channel coding

Referring now to Fig. 6,, the fine granular yardstick of above-mentioned improved coding decoder based on small echo has brought the control of improved adaptability ratio, multicast and joint source-channel coding.The computation complexity that is reduced of improved wavelet algorithm and the computational efficiency of Geng Gao make the channel width of instant and prediction and error condition information can be used among all these three of source encoder 620, channel encoder 630 and rate controller 640, so that influence is maximized control through the instant of vision signal 690 (see figure 6) quality (video rate and distortion correlation) of reconstruct with average compression ratio.For example, the available transmission bandwidth between mobile device 410 and cellular transmission tower 412 (as shown in Figure 4) may change based on visit the number of users of tower 412 in the specific time.Similarly, the transmission quality between mobile phone 410 and the tower 412 (being error ratio) may be based on the distance between phone 410 and the tower 412 and barrier and is changed.Can receive with phone 410 and correspondingly adjust compression ratio about the information of current available bandwidth and error ratio and with it.For example, when bandwidth reduces and/or the error ratio rising, can reduce compression ratio (and relevant reproduced picture quality of bringing thus) so that still can the complete compressed signal of real-time Transmission.On the contrary, when available bandwidth increase and/or error ratio reduction, just compression ratio can be reduced so that transmit higher-quality picture.Based on this feedback, then can by real-time processing in source encoder 620, channel encoder 630 or rate controller 640 variation or utilize the variation of these element combinations to adjust compression ratio.

Example rate of change increment can from 1% to 5%, from 1% to 10%, from 1% to 15%, from 1% to 25%, and changes from 1% to 40%.

Improved adaptability joint Source-Channel Coding Technology allows cellular carrier and MMS service provider to provide wider quality services (QoS) performance and price level for its consumption user and corporate client.Utilization is based on having the heterogeneous network of higher level of can making an uproar at channel type (wireless and wired), channel width, channel/support aspect error characteristics, user's set and the user's service of the improved adaptability joint source-channel coding of Computationally efficient more.

Improved mobile imaging handset structure

Fig. 7 shows improved mobile imaging handset structure.As shown in the figure, imaging applications can be used as with local code or Java and uses the full software application enforcement that operates on the risc processor.The Java encoding operation quickens can be at risc processor self internal implementation, or uses independent Java accelerator IC.This Java accelerator can be used as independent IC and implements, or this IC can also be integrated with other function in SIP or the SoC.

Improved mobile imaging handset structure shown in Fig. 7 has been eliminated being used for the independent DSP 326 that mobile imaging uses or the demand of ASIC 328 processing blocks (as shown in Figure 3), and has greatly reduced image processing in the cell phone 715 to the demand of buffer storage 714.

Improved mobile imaging service platform structure

Referring now to Fig. 8,, the primary clustering of improved mobile imaging service platform structure comprises:

. cell phone 810

. mobile base station (BTS) 812

. base station controller/radio frequency network controller (BSC/RNC) 814

. mobile switching center (MSC) 816

. gateway service node (GSN) 818

. mobile multimedia service controller (MMSC) 820

. imaging service Download Server 821

The exemplary functions (see figure 8) that is contained within the MMSC may comprise:

. video gateway 822

. telecommunications company's server 824

.MMS application server 826

. storage server 828

Relating to the step of disposing improved imaging service platform comprises:

Step 1.

Transmit a signal to the video gateway code converter and use 830 networks that are used in through upgrading on the video gateway of disposing 822.In other words, but when new 830 times spent of decoder software, Download Server 821 transmits a signal to the video gateway 822 on this available network.

Step 2.

Install and construct video gateway code converter software application 830 by automatic OTN 832 deployment or by manual program process (also seeing Fig. 9).

Step 3.

Transmit a signal to mobile video imaging applications 834 (for example, the video coder-decoder of upgrading) and can be used for the user mobile phone downloading and install.

Step 4.

If the user accepts, and completes successfully the affairs arrangement, then download and on cell phone 810, install mobile video imaging applications 834 by OTA 836 program processs.

Step 5.

Transmit a signal to the network that the mobile phone upgrading is finished.Activate service and relevant application.Upgrading user moon bill record is used for the new expense of mobile video imaging applications with reflection.

Performance

This improved mobile imaging application based on small echo, joint source-channel coding, handset structure and service platform structure have reached higher moving video image quality, lower mobile phone cost and complexity and through the target of the service arrangement cost that reduces.

Strengthen

Referring now to Figure 10,,, can consider to be used for the several enforcement options of full software in certain embodiments based on the imaging applications 1012 of small echo as the reinforcement of mobile imaging mobile phone 1010 structures.Can download 1014 by OTA imaging applications 1012 is installed to base band multimedia processing section, removable storage device 1016, image-forming module 1018 or other position of mobile phone 1010.If want, can also be in manufacture process or imaging applications 1012 is installed to base band multimedia processing section, removable storage device 1016, image-forming module 1018 or other position of mobile phone 1010 in the point of sale.Along with the development of mobile device structure also has other enforcement option.

In order to utilize the advanced technology of sustainable development in mobile device computing hardware (ASIC, DSP, RPD) and the integrated technology (SoC, SIP), can rely on some computing elements of hardware based processing resource further to improve the performance of mobile imaging mobile phone by acceleration, and further reduce cost and power consumption.Can consider that several devices at full hardware options are to be used for that these hardware based processing resources (the base band multimedia processing section, removable storage device 1116 or the image-forming module 1118 that comprise mobile phone 1110) are integrated into mobile phone 1110 (seeing Figure 11).

As shown in figure 12, the mixed structure that is used for imaging applications can calculate that strengthen, that repeat, fixing function and implement those in software and may need or require to carry out improved function after manufacturing multinomial reinforcement is provided by implement some at hardware.

Advantage

Full software imaging solution embodiment described herein has greatly reduced the baseband processor in the multimedia handset and the cost and the demand of video accelerator.Combine with download the function that the coding decoder post-production is installed by OTA, this full software solution just can greatly reduce the two cost of complexity, risk and mobile phone development and video messaging service arrangement.

Be further noted that: when some video coder-decoder of using according to many aspects of the present invention, can arrive MMSC to the transfer of data of representing the specific compression video by long-distance communication network, and this data possibility connecting band is useful on the decoder of compressed video.In mode according to many aspects of the present invention, can all or part of cancellation be necessary video gateway to the code converter video data that is input to MMSC aspect other.This just partly obtains conveniently, even this is because each compressed video section all is connected with the decoder of oneself, does not also need for MMSC video format to be become sign indicating number to the particular video formats by the wireless device appointment that is receiving.On the contrary, the wireless device as example 810 these receptions can use the decoder of connection to receive compressed video and this video of broadcast on receiving system 810 platforms.This is just at the cost that significant efficient and saving are provided aspect MMSC structure and its operation.

Additional aspect of the present invention is: small echo is handled on the video that can be designed in just handling and is finished additional video processing function.For example, small echo is handled and can be designed to finish color space conversion, black/white balanced, image stabilization, digital zooming, brilliance control and size adjustment and other function.

Another specific advantages of many aspects of the present invention is to have realized significantly improved voice synchronous.In an embodiment of the present invention, voice synchronous is in the interval of video frame.By relatively, MPEG4 only with voice synchronous in each the 15th frame.This just makes voice and video seriously asynchronous, and is especially such when faulty video transmission occurring when what occur usually on the moving network of image drift.In addition, when this video is implemented in MMSC, make voice synchronous in the interval of video frame just for such as automatically or finish in the program of video editing that can remote activation that video among the MMSC of video editing provides efficiently and video editing is fast got ready.In addition, many aspects of the present invention appear in the present encoding technology of all metadata in the video that just is generated and compresses of allow implanting more or easier implantation as much as possible.Except other, this metadata also comprises the once captive place of time, video (identifying as the position system from cell phone) and makes user of this film or the like.And, with reference frame in per 15 frames of video in the MPEG-4 compressed video relatively, because have the reference frame in the intervals of video frame in certain embodiments of the present invention, embodiments of the invention just provide the effective search of video and the editor of video, and better audio sync performance is provided.

Conclusion

Provide improved mobile imaging application, handset structure and service platform structure by various aspects of the present invention, each side of the present invention combines and has just greatly reduced and provide high-quality static technical sophistication degree and the cost relevant with the video imaging service for the cellphone subscriber.Improved adaptability joint Source-Channel Coding Technology makes cellular carrier and service provider should be able to use the maximum revenue that its wireless network infrastructure generated thereby make for its consumption user and corporate client provide wider service quality (QoS) performance and price level mutually.Can support higher levels of heterogeneous network based on having the more improved adaptability joint source-channel coding of Computationally efficient algorithm aspect channel type (wireless and wired), channel width, interchannel noise/error characteristics, user's set and the user's service.

Though more than be the complete description of the preferred embodiment of the present invention, can use different replacements, modification and equivalent.Therefore, more than describe the restriction that should not be counted as scope of the present invention, scope of the present invention is defined by the following claims.

Claims

1. a joint source-channel coding improves one's methods, wherein, described joint source-channel coding is handled source video to be compressed to generate the bit stream through joint source-channel coding in order in source encoder level, channel encoder level and rate controller level, and described improvement comprises:

Determine at least one the variation among the two of transfer bandwidth parameter and transmission error ratio parameter;

Change among described source encoder level, described channel encoder level and described this three of rate controller level the processing procedure of at least one in response to described at least one determined variation.

2. method according to claim 1, wherein, at least one is instant parameter in the described parameter.

3. method according to claim 1, wherein, at least one is a Prediction Parameters in the described parameter.

4. method according to claim 1, wherein, at least one is a mean parameter in the described parameter.

5. method according to claim 1, wherein, described improvement also comprises: provide can measure with the source encoder level of utilizing small echo.

6. method according to claim 1, wherein, at least one receives from the cellular phone signal tower in the described parameter.

7. method according to claim 1 wherein, changes the rate of change increment that the processing procedure of at least one in the described level causes about 1% to 40% scope.

8. method according to claim 1 wherein, changes the rate of change increment that the processing procedure of at least one in the described level causes about 1% to 5% scope.