WO2002080563A2 - Scalable expandable system and method for optimizing a random system of algorithms for image quality - Google Patents
Scalable expandable system and method for optimizing a random system of algorithms for image quality Download PDFInfo
- Publication number
- WO2002080563A2 WO2002080563A2 PCT/IB2002/001040 IB0201040W WO02080563A2 WO 2002080563 A2 WO2002080563 A2 WO 2002080563A2 IB 0201040 W IB0201040 W IB 0201040W WO 02080563 A2 WO02080563 A2 WO 02080563A2
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- module
- video
- video processing
- oiq
- image quality
- Prior art date
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T9/00—Image coding
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N17/00—Diagnosis, testing or measuring for television systems or their details
Definitions
- the present invention relates to methods and systems for optimizing video quality. More particularly, the present invention relates to an expandable scheme of video algorithms used to improve image quality.
- the video signal gets processed by a number of video functions (e.g. for sharpness enhancement, noise reduction, color correction, etc.)
- a number of video functions e.g. for sharpness enhancement, noise reduction, color correction, etc.
- Each of these functions may need a (small or large) number of control parameters.
- the order of applying the various functions could be a parameter (before building the hardware, which carries out the video processing functions as modeled by software, or if we have a highly flexible reconfigurable hardware), or the video system is static and cannot be altered.
- An objective image quality (OIQ) evaluator unit may vary in complexity from a simple measure of simple signals (like the rise time of the luminance signal) to a fairly complicated system that simulates the psychophysics of the human vision system (HVS).
- the optimization process may vary in complexity from a greedy exhaustive search engine (which requires huge computational resource, almost impossible to have in most practical situations) to a smart heuristic search methodology with less computational requirements.
- a system for optimizing video quality includes a scalable optimization paradigm for providing the best attainable objective image quality for the available computational resources.
- An optimizing video processing system comprises:
- the video processing module for processing an input of a video stream, the video processing module comprising architectural parameters for identifying an order of cascaded video functions and determining a bit precision between any consecutive cascaded functions according to an associated complexity level which correlates with a value of available computational resources;
- an optimizer module for optimizing processing of the video stream, the optimizer module being in communication with the video processing module, the optimizer module comprising a plurality of optimization engines each having an associated complexity level, the optimizer module includes means for selecting an optimization engine according to a complexity level which correlates with the value of available computational resources;
- an Object Image Quality (OIQ) evaluator module for evaluating an image quality of an output of the video stream from the video processing module, the OIQ evaluator comprising a plurality of objective image quality metrics having an associated complexity level, and the OIQ evaluator module includes means for selecting a metric according to a correlation factor ri and a complexity level for the value of available computation resources.
- the means for selecting the metric by the OIQ evaluator module may include determining a correlation factor R determined according to the following equation:
- F is a final metric (of the quality of the video as judged by the system), F being determined by finding a set of weights Wj_ which when multiplied by each individual metric f, (which ranges from 1 to n) of the plurality of objective metrics maximizes the correlation factor R with a predetermined subjective evaluation.
- the system may also have a computational resource analyzer for selecting the associated complexity level of at least one of the video processing module, the optimizer module, and the OIQ evaluator module.
- the optimizer module may include both deterministic and non-deterministic optimization engines.
- the optimizer module may include heuristic search engines comprising at least one of genetic algorithms (GA), simulated annealing (S A), tabu search (TS), simulated evolution (SE), and stochastic evolution.
- GA genetic algorithms
- S A simulated annealing
- TS tabu search
- SE simulated evolution
- stochastic evolution stochastic evolution.
- At least one of the video processing module, optimizer module and OIQ evaluator module can be scalable.
- the computational resource analyzer module may select the level of complexity for at least one of the video processing module, the optimizer module, and the
- OIQ evaluator module by detecting available computational resources for one of the modules.
- a method for optimizing video algorithms for available computation resources comprises:
- the evaluating of the objective image quality in step (c) may include determining a correlation factor R determined according to the following equation:
- F is a final metric (the quality of the video as judged by the system), F being determined by finding a set of weights Wj , which when multiplied by each individual metric fj (which ranges from 1 to n) of the plurality of metrics maximizes the correlation factor R with a predetermined subjective evaluation.
- the method may further comprise:
- step (d) selecting the associated complexity level of at least one of step (a), (b) and (c) by a computational resource analyzer.
- the plurality of optimization methods selected in step (b) may include both deterministic and non-deterministic optimization methods.
- the plurality of optimization methods include heuristic search engines comprising at least one of genetic algorithms (GA), simulated annealing (S A), tabu search (TS), simulated evolution (SE), and stochastic evolution.
- GA genetic algorithms
- S A simulated annealing
- TS tabu search
- SE simulated evolution
- stochastic evolution stochastic evolution
- the associated complexity level selected in step (d) may include detecting computational resources available for at least one of steps (a), (b) and (c).
- Step (b) may include providing a scalable optimizer for selecting the optimization method.
- Step (c) may include providing a scalable objective image quality evaluator for evaluating the objective image quality.
- the system may also comprise a video-processing module, an optimizer module, a scalable Objective Image Quality (OIQ) evaluator module, and a computational resource analyzer.
- OIQ scalable Objective Image Quality
- the video processing module comprises a plurality of video processing functions Fi, F 2j F n .
- Each function has a set of parameter P réelle 1 ⁇ i ⁇ n, which is sorted ascendingly in terms of their effect on the resulting image quality.
- the video processing module has its own set of architectural parameters, which describe the cascaded video processing functions' order as well as the bit precision of the data bus between any two consecutive functions.
- the optimizer module is a scalable optimizer with a plurality of possible optimization mechanisms.
- the optimizer module may comprise a number of optimization search engines varying in complexity and the corresponding required resources.
- the search engines may be exhaustive or heuristic.
- the scalable OIQ-evaluator module comprises a plurality of OIQ metrics having different levels of complexity.
- a table of complexity levels is kept by the OIQ- evaluator module which contains all the constituent metric methods and the presumed complexity for each metric.
- the computational resource analyzer module is an arbitrator, which based on the available computational resources will decide on which level of complexity for all other modules should be invoked.
- Fig. 1 is an overview of the scalable optimization system according to the present invention.
- Fig. 2 is a detailed diagram of the optimizer module shown in Fig. 1.
- Fig. 3 A is a detailed diagram of the objective image quality evaluator shown in Fig. 1.
- Fig. 3B is an illustration of the flow of a scalable dynamic objective metric.
- Fig. 4 is a flowchart of a method of the present invention.
- Fig. 5 is a continuation of the flowchart shown in Fig. 4.
- Fig. 1 illustrates an overview of a scalable optimizing system according to the present invention. According to Fig. 1, there is a video processing module 100, a system optimizer module 200, an objective image quality evaluator module 300 and an optional computational resource analyzer module 400.
- the video processing module 100 comprises architectural parameters for identifying an order of cascaded video functions, and for determining a bit precision between data of any consecutive cascaded functions. As shown in Fig. 1, there are a number of video processing functions 102
- each function having a set of architectural parameters Pi 105, ranging from Pi to P Organic.
- the set of parameters Pi (where 1 ⁇ i ⁇ n) and which are sorted ascendingly in terms of their effect on resulting image quality.
- Fig. 2 shows a detailed example of the optimizer module 110 shown in Fig. 1.
- This module comprises of a number of optimization engines (m search engines) which can be referred to as optimization methods 220, which vary in complexity, representation, and required computational resources.
- the optimization methods 220 may comprise a simple exhaustive search methodology (which will perturb all the pre-defined parameters over their range of values), as well as a number of heuristic search engines.
- the optimizer module also keeps a record of each method's presumed complexity level in table 230. The optimizer module is expandable since any sought engine could be appended to it, as long as its relative complexity level with other complexity levels of other methods is defined.
- the parameter and control signals dispatcher 235 in the optimizer module invokes the suitable optimization engine.
- the dispatcher contains control signals for invoking the suitable method (i.e. engine) and architectural parameters.
- a recommended complexity level is selected and/or supplied by the computational resource analyzer 130 (shown in Fig. 1) but the computational resource analyzer is an optional feature.
- the recommended complexity level may be selected by the optimizer module, for example.
- some of the methods in the optimizer module can be heuristic methods that may vary from a greedy method, wherein a good solution is constructed in stages, to more local heuristic search methods, e.g., genetic algorithms (GA's), simulated annealing (SA), tabu search (TS), simulated evolution (SE), stochastic evolution (SE) any hybrid of any number of these methods.
- GA's genetic algorithms
- SA simulated annealing
- TS tabu search
- SE simulated evolution
- SE stochastic evolution
- Video processing algorithms when used with heuristic methods may use, for example, genetic algorithms (GA).
- GA genetic algorithms
- the GA method will evolve toward a system configuration permitting the best image quality.
- GA's are iterative procedures that maintain a group of potential “candidate” solutions, which are evaluated and assigned a fitness value. GA's are known procedures to solve complex problems, and the section entitled book “Genetic Algorithms in Optimization and Adaptation” of a book entitled Advances in Parallel Algorithms, by Kronsjo and Shumshesuddin, pages 227-276 (1990) are hereby incorporated by reference as background material. GA's are iterative procedures that maintain a population of candidate solutions encoded in the form of chromosome strings. Each chromosome defines a certain way in which different video processing modules are connected, and thus, the way the sequences are processed.
- each chromosome comprises a number of genes, which in the case of video optimization process are the video processing functions as well ass their order.
- Simulated annealing is a methodology, not a fixed algorithm, in which a global minimum is calculated for a solution in regard to the complexity level that will be used by the optimizer module.
- TABU search is an adaptive procedure used for solving combinatorial optimization problems, which may direct a heuristic to continue exploration of a descending hill without falling back into a previous optimum from which it previously emerged.
- Simulated Evolution is a method by which a series of equations are used for determining the fitness for a complexity level over a series of generations.
- Fig. 3 A is a detailed illustration of the OIQ evaluator module.
- the OIQ evaluator module 300 consists of a number of objective image quality metrics (K metrics 320) that vary in complexity.
- K metrics 320 objective image quality metrics
- the OIQ module keeps a record of its constituent metrics methods as well as each method's presumed complexity level in table 330.
- the OIQ module is extendable, since any proposed metric could be appended to it, as long as its relative complexity level is priory defined. Based on the appropriate complexity level, which could be afforded by the available resources, the video stream dispatcher 310 in the OIQ module invokes the suitable OIQ metric.
- each objective metric 320 has a rating according to the desired level of performance and the allowable complexity, referred to as a figure of merit.
- the figure of merit represents the quality of the video signal based on that individual metric.
- a correlation factor with the human perception of video quality permits a scalable model, and new objective metrics can be added to or removed from the system so long as its correlation with human perception is defined.
- Fig. 3B is an illustration of a scalable objective metrics 320 shown in Fig 3 A with more detail of the table of complexity levels 330.
- Each of the metrics has a correlation factor (R, 1 ⁇ i ⁇ n) with the "1" from first metric fi. and the "n" from the last metric fate . Based on each single correlation factor, the evaluator gives a weight Wj for each figure of merit, while trying to maximize the overall correlation factor R of the final composite metric F with the predetermined subjective result, according to the equation:
- the computational resource analyzer module 400 may provide for the detection of the available computational resources, and decide on the appropriate analyzer complexity level as well as the suitable complexity level for the OIQ module.
- a value of the computing resources availability can be provided to the OIQ evaluator module to remove certain metrics from selection because the resources would exceed the available capacity. This value could also be received by the system optimizer module 200 whereby the optimization method 220 selected would have to fit within the given available resources.
- the algorithms chosen are optimized according to the available resources available to achieve the best objective image quality.
- This objective image quality correlates to the subject image quality of the human vision system.
- different algorithms and/or different metrics might be selected for a given image.
- This flexible approach maximizes image quality because with a static system, there would need to be a conservative threshold in terms of selecting an algorithm or metric so as not to overrun the availability of resources. If the resources are overrun by the requirement of the algorithm or metric, there could be a system interruption, and at the very least, a perceivable lapse by human vision while a substitute algorithm is chosen to fit within a resource capacity at a given moment in time.
- FIGs. 4 and 5 are flowcharts illustrating a basic overview of the method according to the present invention.
- step (a) there is an identifying of an order of cascaded video functions.
- Step (b) recites that there is a selecting of an optimization method for optimizing the processing of the video stream.
- Step (c) recites that there is an evaluating of an objective image quality of the video stream after the video stream is output from the video processing module by selecting a metric according to a correlation factor and an associated complexity level for the value of computational resources.
- Fig. 5 recites the evaluating of the objective image quality in step (c) by determining a correlation according to the previously recited equation.
- the computational resource module could be bypassed, if there is a desire to dictate a certain level of complexity on either/both of the optimizer module and/or the OIQ module.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- General Health & Medical Sciences (AREA)
- Signal Processing (AREA)
- Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
- Image Processing (AREA)
- Picture Signal Circuits (AREA)
Abstract
Description
Claims
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2002577435A JP2004527172A (en) | 2001-03-29 | 2002-03-28 | Scalable system and method for optimizing a random system of algorithms for image quality |
EP02713136A EP1433134A2 (en) | 2001-03-29 | 2002-03-28 | Scalable expandable system and method for optimizing a random system of algorithms for image quality |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US27981501P | 2001-03-29 | 2001-03-29 | |
US60/279,815 | 2001-03-29 | ||
US09/912,468 US6813390B2 (en) | 2001-07-25 | 2001-07-25 | Scalable expandable system and method for optimizing a random system of algorithms for image quality |
US09/912,468 | 2001-07-25 |
Publications (2)
Publication Number | Publication Date |
---|---|
WO2002080563A2 true WO2002080563A2 (en) | 2002-10-10 |
WO2002080563A3 WO2002080563A3 (en) | 2004-03-11 |
Family
ID=26959901
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/IB2002/001040 WO2002080563A2 (en) | 2001-03-29 | 2002-03-28 | Scalable expandable system and method for optimizing a random system of algorithms for image quality |
Country Status (5)
Country | Link |
---|---|
EP (1) | EP1433134A2 (en) |
JP (1) | JP2004527172A (en) |
KR (1) | KR20030005409A (en) |
CN (1) | CN1511303A (en) |
WO (1) | WO2002080563A2 (en) |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102170581A (en) * | 2011-05-05 | 2011-08-31 | 天津大学 | Human-visual-system (HVS)-based structural similarity (SSIM) and characteristic matching three-dimensional image quality evaluation method |
US8422795B2 (en) | 2009-02-12 | 2013-04-16 | Dolby Laboratories Licensing Corporation | Quality evaluation of sequences of images |
US9934043B2 (en) | 2013-08-08 | 2018-04-03 | Linear Algebra Technologies Limited | Apparatus, systems, and methods for providing computational imaging pipeline |
US11768689B2 (en) | 2013-08-08 | 2023-09-26 | Movidius Limited | Apparatus, systems, and methods for low power computational imaging |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CA2646808C (en) * | 2003-08-22 | 2013-01-22 | Nippon Telegraph And Telephone Corporation | Video aligning apparatus, video aligning method, and video quality assessing apparatus |
CN101345875B (en) * | 2008-09-03 | 2013-08-07 | 北京中星微电子有限公司 | Video algorithm development platform and its development method |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO1997039417A2 (en) * | 1996-03-29 | 1997-10-23 | Sarnoff Corporation | Method and apparatus for training a neural network to use a fidelity metric |
US5835627A (en) * | 1995-05-15 | 1998-11-10 | Higgins; Eric W. | System and method for automatically optimizing image quality and processing time |
-
2002
- 2002-03-28 WO PCT/IB2002/001040 patent/WO2002080563A2/en not_active Application Discontinuation
- 2002-03-28 EP EP02713136A patent/EP1433134A2/en not_active Withdrawn
- 2002-03-28 CN CNA028009525A patent/CN1511303A/en active Pending
- 2002-03-28 JP JP2002577435A patent/JP2004527172A/en active Pending
- 2002-03-28 KR KR1020027016075A patent/KR20030005409A/en not_active Application Discontinuation
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5835627A (en) * | 1995-05-15 | 1998-11-10 | Higgins; Eric W. | System and method for automatically optimizing image quality and processing time |
WO1997039417A2 (en) * | 1996-03-29 | 1997-10-23 | Sarnoff Corporation | Method and apparatus for training a neural network to use a fidelity metric |
Non-Patent Citations (1)
Title |
---|
SHYU M-S ET AL: "A genetic algorithm approach to color image enhancement" PATTERN RECOGNITION, PERGAMON PRESS INC. ELMSFORD, N.Y, US, vol. 31, no. 7, 31 July 1998 (1998-07-31), pages 871-880, XP004130994 ISSN: 0031-3203 * |
Cited By (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8422795B2 (en) | 2009-02-12 | 2013-04-16 | Dolby Laboratories Licensing Corporation | Quality evaluation of sequences of images |
CN102170581A (en) * | 2011-05-05 | 2011-08-31 | 天津大学 | Human-visual-system (HVS)-based structural similarity (SSIM) and characteristic matching three-dimensional image quality evaluation method |
US9934043B2 (en) | 2013-08-08 | 2018-04-03 | Linear Algebra Technologies Limited | Apparatus, systems, and methods for providing computational imaging pipeline |
US10360040B2 (en) | 2013-08-08 | 2019-07-23 | Movidius, LTD. | Apparatus, systems, and methods for providing computational imaging pipeline |
US11042382B2 (en) | 2013-08-08 | 2021-06-22 | Movidius Limited | Apparatus, systems, and methods for providing computational imaging pipeline |
US11567780B2 (en) | 2013-08-08 | 2023-01-31 | Movidius Limited | Apparatus, systems, and methods for providing computational imaging pipeline |
US11768689B2 (en) | 2013-08-08 | 2023-09-26 | Movidius Limited | Apparatus, systems, and methods for low power computational imaging |
Also Published As
Publication number | Publication date |
---|---|
JP2004527172A (en) | 2004-09-02 |
WO2002080563A3 (en) | 2004-03-11 |
EP1433134A2 (en) | 2004-06-30 |
KR20030005409A (en) | 2003-01-17 |
CN1511303A (en) | 2004-07-07 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
KR20200022739A (en) | Method and device to recognize image and method and device to train recognition model based on data augmentation | |
CN112116001B (en) | Image recognition method, image recognition device and computer-readable storage medium | |
CN110956202B (en) | Image training method, system, medium and intelligent device based on distributed learning | |
CN111008631B (en) | Image association method and device, storage medium and electronic device | |
CN109344314A (en) | A kind of data processing method, device and server | |
CN110557633B (en) | Compression transmission method, system and computer readable storage medium for image data | |
CN113778691B (en) | Task migration decision method, device and system | |
US6813390B2 (en) | Scalable expandable system and method for optimizing a random system of algorithms for image quality | |
CN111783935A (en) | Convolutional neural network construction method, device, equipment and medium | |
CN111160458A (en) | Image processing system and convolution neural network thereof | |
CN112862023B (en) | Object density determination method and device, computer equipment and storage medium | |
EP1433134A2 (en) | Scalable expandable system and method for optimizing a random system of algorithms for image quality | |
Verma et al. | A" Network Pruning Network''Approach to Deep Model Compression | |
CN111783936A (en) | Convolutional neural network construction method, device, equipment and medium | |
WO2020039790A1 (en) | Information processing device, information processing method, and program | |
CN114830137A (en) | Method and system for generating a predictive model | |
JP7073171B2 (en) | Learning equipment, learning methods and programs | |
JP7009971B2 (en) | Process scheduling device and process scheduling method | |
CN111506753B (en) | Recommendation method, recommendation device, electronic equipment and readable storage medium | |
KR102442891B1 (en) | System and method for updating weight of artificial neural networks | |
CN114327925A (en) | Power data real-time calculation scheduling optimization method and system | |
CN112418114A (en) | Time sequence behavior detection and response method and device, equipment and medium | |
CN109308565B (en) | Crowd performance grade identification method and device, storage medium and computer equipment | |
CN113762403B (en) | Image processing model quantization method, device, electronic equipment and storage medium | |
CN113079389B (en) | Resource self-adaptive adjusting method in edge computing environment |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AK | Designated states |
Kind code of ref document: A2 Designated state(s): CN JP KR |
|
AL | Designated countries for regional patents |
Kind code of ref document: A2 Designated state(s): AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE TR |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2002713136 Country of ref document: EP |
|
WWE | Wipo information: entry into national phase |
Ref document number: 1020027016075 Country of ref document: KR |
|
WWE | Wipo information: entry into national phase |
Ref document number: 028009525 Country of ref document: CN |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application | ||
WWP | Wipo information: published in national office |
Ref document number: 1020027016075 Country of ref document: KR |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2002577435 Country of ref document: JP |
|
WWP | Wipo information: published in national office |
Ref document number: 2002713136 Country of ref document: EP |
|
WWW | Wipo information: withdrawn in national office |
Ref document number: 2002713136 Country of ref document: EP |