CN109343692B - Mobile device display power saving method based on image segmentation - Google Patents

Mobile device display power saving method based on image segmentation Download PDF

Info

Publication number
CN109343692B
CN109343692B CN201811089845.7A CN201811089845A CN109343692B CN 109343692 B CN109343692 B CN 109343692B CN 201811089845 A CN201811089845 A CN 201811089845A CN 109343692 B CN109343692 B CN 109343692B
Authority
CN
China
Prior art keywords
image
layer
pixel
convolution
following
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201811089845.7A
Other languages
Chinese (zh)
Other versions
CN109343692A (en
Inventor
渠慎明
苏靖
程普
张东生
刘珊
渠梦瑶
王青博
张济仕
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Henan University
Original Assignee
Henan University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Henan University filed Critical Henan University
Priority to CN201811089845.7A priority Critical patent/CN109343692B/en
Publication of CN109343692A publication Critical patent/CN109343692A/en
Application granted granted Critical
Publication of CN109343692B publication Critical patent/CN109343692B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F1/00—Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
    • G06F1/26—Power supply means, e.g. regulation thereof
    • G06F1/32—Means for saving power
    • G06F1/3203—Power management, i.e. event-based initiation of a power-saving mode
    • G06F1/3234—Power saving characterised by the action undertaken
    • G06F1/325—Power saving in peripheral device
    • G06F1/3265—Power saving in display device
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20048—Transform domain processing
    • G06T2207/20056—Discrete and fast Fourier transform, [DFT, FFT]

Abstract

The invention provides a power saving method for a display of mobile equipment based on image segmentation, which solves the problem of inaccurate segmentation of image areas and non-image areas in the existing similar algorithm through convolution operation, context regularization operation and circular iteration operation, particularly solves the problems of large area error of predicted pictures and unclear edge segmentation, improves the accuracy degree of image segmentation, further reduces the brightness of the non-image areas in the images, and enables the power consumption degree of the display to be adjusted along with the use preference of users while effectively maintaining the image quality.

Description

Mobile device display power saving method based on image segmentation
Technical Field
The invention relates to the field of power saving of OLED self-luminous displays, in particular to a power saving method of a display of a mobile device based on image segmentation.
Background
In the current society, smart mobile devices such as mobile phones and tablet computers are widely used. The display is mainly classified into a non-self-luminous display and a self-luminous display as an indispensable interface for human-computer interaction in the smart mobile device. Organic Light-Emitting diodes (OLEDs), as a modern emerging self-emissive display technology, are different from conventional non-self-emissive displays in that each pixel provides a Light source, and the brightness of each pixel can be individually adjusted according to the content of the displayed image, thereby facilitating effective control of battery consumption. Therefore, the power constraint image enhancement algorithm applying the OLED self-luminous display is a research hotspot.
There are many power constrained image enhancement algorithms for existing self-emitting displays.
Lee et al propose a histogram-based modifiable correction technique that improves the rejection rate of the Jan et al approach. Although these methods can maintain image quality, their power saving rate is low. Thus, Lee et al propose a histogram-based modifiable correction technique that improves the rejection rate of the Jan et al approach. Although these methods can maintain image quality, their power saving rate is low. Therefore, based on OLED displays, it is currently recognized that the technique of reducing power consumption is based on changing the brightness value of the pixels to increase the battery life, and this method has received much attention in recent years. Another energy-saving method for OLED displays, which is specifically applied to video players, is a dynamic tone mapping method based on scene classification. While this approach optimizes the energy consumption of the display, the video classifier fails in certain specific scenarios. Park et al propose an alpha blending method to reduce human perception in images to reduce the power consumption of the display. This method, although achieving a low computational complexity of the video processing, requires initialization of certain parameters in the processing of the different images, which is difficult to achieve in practice. Nam et al propose a power-limited contrast enhancement (SDMSR) based multi-order retinal cortical method in which the input picture is divided into different sub-blocks and appropriate gains for each local adjusted pixel intensity are calculated until the target consumption of energy is reached. However, using these defined assumptions or a priori contrast enhancement to excessively calibrate the intensity of each pixel, the enhanced image may appear as a less than desirable artifact or image degradation. To reduce the degradation of image pixel intensity, calibration-based artificial image appearance, such as Chang et al, proposes a dimmable perceptual pixel, using a subtraction factor to reduce the pixel brightness value, which allows to directly judge the pixel intensity of the display content without prior knowledge of any device display, through an evaluation of Structural Similarity (SSIM). The energy saving method proposed by Chondro et al, applied to AMOLED displays and aimed at perceiving hue, suppresses the local brightness of the input picture by the color of the neighboring areas. The energy consumption is further reduced by increasing the structure of each pixel according to the tone conversion, and although the method has a certain balance between energy saving and image quality, the calculation complexity is too high, and the use of the video player is influenced. The method of Chang et al is a further enhancement to Chondro et al by suppressing the over-exposed regions and blue spectrum. The idea of the PQPR algorithm is to convert the input picture linearly and perform image enhancement based on the obtained gray-scale image curve. An image quality energy control (IQPC) based algorithm in OLED displays was subsequently proposed, which is an improvement over the Kang et al method. The IQPC method proposes a curved objective function in order to reduce errors to avoid image quality loss. The raw brightness diode energy control (oledppc) algorithm, proposed later, is a direct extension to IQPC. Chondro et al propose a histogram equalization based display power constrained contrast enhancement algorithm (HDPCCE). The algorithm minimizes the objective function using convex optimization theory while achieving contrast enhancement and power saving. The OLEDPC method designs another objective function to determine the gray-scale graph curve and globally calibrate the pixel intensity of the input picture based on image quality. Recently, Chondro et al proposed a hue saturation value color map (HSV) using a color preserving pixel dimming (HPPD) approach, with the aim of suppressing the power consumption of AMOLED displays.
However, the existing power constrained image enhancement algorithms have 2 significant disadvantages. First, visual perception is affected. The existing method directly adjusts the whole picture, and the operation can lose the detail information of the picture and influence the visual perception. Second, the degree of power saving is small. When the existing method is actually applied to the intelligent mobile equipment, a user cannot flexibly adjust the required power saving degree according to the self requirement, and the experience feeling is poor. Therefore, the power constraint image enhancement algorithm applying the OLED self-luminous display is a research hotspot.
Disclosure of Invention
The invention aims to provide a power saving method for a display of a mobile device based on image segmentation, which can reduce the brightness of a non-image area in an image according to a segmented image of a context regularized cycle depth learning frame, and can adjust the power consumption degree of the display according to the use preference of a user while effectively maintaining the image quality.
In order to achieve the purpose, the invention adopts the following technical scheme:
the power saving method for the display of the mobile equipment based on image segmentation is characterized by comprising the following steps:
step 1: for input picture
Figure GDA0003080130920000021
Performing convolution operation in a VGG19-FCN network, wherein the VGG19-FCN network is composed of 18 convolution layers, 5 pooling layers and 3 deconvolution layers; the method specifically comprises the following steps:
step 1.1: the convolutional layer operation was carried out using the following method:
suppose that
Figure GDA0003080130920000022
Is the i-th layer feature map of the i-th layer convolution layer, defines the feature map
Figure GDA0003080130920000023
In order to input the quantity of the input,
Figure GDA0003080130920000024
for the output quantity, i.e., the binary mask map, the convolution operation is shown in equation (1-1):
Figure GDA0003080130920000031
where N is the total number of feature maps,
Figure GDA0003080130920000032
is a convolution kernel of the i-th convolutional layer of the first convolutional layer,
Figure GDA0003080130920000033
is a deviation parameter of the ith convolution layer of the first convolution layer; n is
Figure GDA0003080130920000034
The number of feature maps in (a) is denoted as convolution operation, f (-) is the activation function;
step 1.2: the maximum pooling operation of the pooling layer is expressed by the following formula (1-3):
Figure GDA0003080130920000035
wherein Ω (m, n) represents a feature vector
Figure GDA0003080130920000036
Is the position (m, n) of the space vector of (a), and Δ represents
Figure GDA0003080130920000037
In layer 7 of the framework of the algorithm;
step 1.3: and (3) performing deconvolution layer operation by adopting the following method:
deconvoluting the output of the 5 th layer of the first layer of convolutional layer to the size of original figure, and then deconvoluting the output of the 4 th layer of the first layer of convolutional layer and the output of the 3 rd layer in turn to obtain
Figure GDA0003080130920000038
The deconvolution operation is expressed by the following equation (1-4):
Figure GDA0003080130920000039
wherein
Figure GDA00030801309200000310
Is a convolution kernel of the i-th convolutional layer of the first convolutional layer,
Figure GDA00030801309200000317
denoted as a deconvolution operation;
step 1.4: the learning rate is adjusted by the following method:
the adjustment of the learning rate is shown in the formula (1-5):
Figure GDA00030801309200000311
wherein R istFor the learning rate, ε is the threshold for trigger attenuation, t is the change count, t is 0, 1, 2, 3; alpha is a decay index; ρ is 0.90;
step 2: binary mask map output to convolutional layer
Figure GDA00030801309200000312
The context regularization operation specifically includes the following steps:
step 2.1: definition of
Figure GDA00030801309200000313
Is composed of
Figure GDA00030801309200000314
Pixel at position (m, n), XgIs a gray scale image, then XgThe pixel I (m, n) at position (m, n) is:
Figure GDA00030801309200000315
where η is the most primitive error, η is:
Figure GDA00030801309200000316
wherein the content of the first and second substances,
Figure GDA0003080130920000041
in order to minimize the error parameter η,
Figure GDA0003080130920000042
represents Frobenius norm operation;
step 2.2: defining a constraint function:
Figure GDA0003080130920000043
wherein the content of the first and second substances,
Figure GDA0003080130920000044
is a pixel
Figure GDA0003080130920000045
Pixels of the surrounding 8 directions;
step 2.3: defining a weight function W (m, n):
Figure GDA0003080130920000046
when W (m, n) is 0, the corresponding context constraint between m and n will be cancelled;
step 2.4: constructing a weight function W (m, n) based on a method of squared difference between vectors of two adjacent pixels:
Figure GDA0003080130920000047
where σ is a predetermined parameter, σ is 0.5, and I (m + Δ m, n + Δ n) is Xg8 directional pixels around the input pixel;
step 2.5: adding a weighted context constraint in the image domain, rewriting equations (1-9) as:
Figure GDA0003080130920000048
wherein, ω represents different directions of the pixels at 8 positions;
step 2.6: defining a higher order filter DΔm,ΔnLet D beΔm,ΔnThe value at each position (Δ m, Δ n) satisfies:
Figure GDA0003080130920000049
even if DΔm,ΔnThe value at each position (Δ m, Δ n) satisfies
Figure GDA00030801309200000410
Wherein, omega represents an index set,
Figure GDA00030801309200000414
is a multiplication operator for the pixel or pixels,
Figure GDA00030801309200000413
for convolution operators, DΔm,ΔnRepresenting a first order differential operator, WΔm,ΔnA weighting matrix representing pixels in (Δ m, Δ n) | | | · purple1Represents an estimate of manhattan distance;
step 2.7: the following objective function is defined and minimized, the objective function (1-14) being derived from equations (1-7) and equations (1-13):
Figure GDA00030801309200000411
where ξ is a regularization parameter that balances two conditions; for the
Figure GDA00030801309200000412
The following formula is satisfied:
Figure GDA0003080130920000051
step 2.8: defining auxiliary variables
Figure GDA0003080130920000052
Rewrite formula (1-15):
Figure GDA0003080130920000053
wherein, beta is a predefined scale factor,
Figure GDA0003080130920000054
initial value of beta0Is 1, maximum value betamaxIs 22 by a scale factor
Figure GDA0003080130920000055
Iteratively increasing β from a minimum value of 0 to a maximum value of 22;
step 2.9: first, fix
Figure GDA0003080130920000056
Optimization
Figure GDA0003080130920000057
Figure GDA0003080130920000058
Thus, optimization can be directly made in the location (m, n)
Figure GDA0003080130920000059
Figure GDA00030801309200000510
Wherein the content of the first and second substances,
Figure GDA00030801309200000511
a certain pixel of 8 pixels around the pixel representing the coordinates (m, n) in the image, sign (·)Is a function of the signal;
secondly, fix
Figure GDA00030801309200000512
Optimization
Figure GDA00030801309200000513
Figure GDA00030801309200000514
Since the formulae (1-19) are
Figure GDA00030801309200000515
The quadratic equation of (a), and thus the equations (1-19) can be rewritten:
Figure GDA00030801309200000516
step 2.10: optimization using two-dimensional Fourier transform and hypothetical cycle boundary conditions
Figure GDA00030801309200000517
Calculating an optimal solution Y:
Figure GDA0003080130920000061
where τ is the Fourier transform, τ (·)-1Is an inverse fourier transform;
and step 3: a loop iteration operation comprising the steps of:
step 3.1: after the context regularization calculation, the obtained optimal solution Y is compared with the input RGB picture
Figure GDA0003080130920000062
Multiplying pixel values to obtain a multiplied picture Y;
step 3.2: taking Y in the step 3.1 as input, repeatedly performing the step 1, the step 2 and the step 3.1, setting an entropy critical value of 6.92 based on the entropy of Y, and performing iteration from 0 to 6.92 until a binary mask image Y' closest to the true phase is predicted;
and 4, step 4: according to Y' in step 3.2, the image area of the picture displayed by the OLED self-luminous display is kept unchanged, and the non-image area is subjected to pixel brightness value reduction.
The image segmentation based mobile device display power saving method of claim 1, wherein: in step 1.1, the activation function uses a modified linear unit, as shown in formula (1-2):
f(x)＝max(0,x)；(1-2)
where x is the input value of the activation function.
In step 1.4, the learning rate RtInitial value R of0Set to 10-4.
In step 3.2, the method for judging whether the picture Y is the binary mask picture closest to the true phase comprises the following steps: the evaluation is judged by 4 evaluation indexes of precision, recall, F1_ measure and similarity.
The method for reducing the pixel brightness value of the non-image area in the step 4 comprises the following steps: defining the image after energy saving as Y', then:
Figure GDA0003080130920000063
where P is the desired power consumption level.
The invention has the beneficial effects that:
according to the image semantic segmentation method based on context regularization, the problem that the segmentation of image regions and non-image regions is inaccurate in the existing similar algorithm is solved through convolution operation, context regularization operation and circular iteration operation, the problems that a predicted image is large-area wrong and edge segmentation is not clear are particularly solved, the image segmentation accuracy is improved, the brightness of the non-image regions in the image is further reduced, and the power consumption degree of a display can be adjusted along with the use preference of a user while the image quality is effectively maintained.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
FIG. 1 is a flow chart of a method of the present invention;
FIG. 2 shows a high order filter D according to the present inventionΔm,ΔnSchematic structural diagram of (1).
Detailed Description
The technical solutions of the present invention will be described clearly and completely with reference to the accompanying drawings, and it should be understood that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
As shown in fig. 1: the invention relates to a power saving method for a display of mobile equipment based on image segmentation, which comprises the following steps:
step 1: performing convolution operation in a VGG19-FCN network, wherein the VGG19-FCN network is composed of 18 convolution layers, 5 pooling layers and 3 deconvolution layers; the method specifically comprises the following steps:
step 1.1: suppose that
Figure GDA0003080130920000071
Is the ith layer feature map of the ith convolutional layer
Figure GDA0003080130920000072
Is the amount of the input to be made,
Figure GDA0003080130920000073
is the output quantity, i.e., the binary mask map, as shown in equation (1-1):
Figure GDA0003080130920000074
where N is the total number of feature maps,
Figure GDA0003080130920000075
is a convolution kernel of the i-th convolutional layer of the first convolutional layer,
Figure GDA0003080130920000076
is a deviation parameter of the ith convolution layer of the first convolution layer; wherein n is
Figure GDA0003080130920000077
The number of feature maps in the graph is denoted as convolution operation, f (·) represents an activation function using a modified linear unit (ReLU), as shown in equation (1-2):
f(x)＝max(0,x)；(1-2)
where x is the input value of the activation function;
step 1.2: the convolution operation is followed by a pooling layer, and the pooling operation used in the present algorithm is maximal pooling, i.e. from the feature vectors
Figure GDA0003080130920000078
Taking the maximum value and keeping, and discarding the rest values in the space vector; the pooling operation can be expressed by the formula (1-3):
Figure GDA0003080130920000081
where Ω is represented in the feature vector
Figure GDA0003080130920000082
The (m, n) position of the space vector of (a), Δ represents the variable in layer 7 of the framework of the algorithm;
step 1.3: and (3) performing deconvolution layer operation by adopting the following method:
if the deconvolution operation of 32 times of amplification is directly carried out on the output of the layer 6 network, the obtained result is compared with that of the deconvolution operation
Figure GDA0003080130920000083
For the true phase binary diagram, the result is not accurate and has many errors; therefore, according to the sequence from back to front, after 16 times of the output of the 4 th layer is deconvoluted, 8 times of the output of the 3 rd layer is deconvoluted, and the obtained result is more accurate than the output result which is not subjected to the process; further, the deconvolution operation is expressed by the following equation (1-4):
Figure GDA0003080130920000084
wherein
Figure GDA0003080130920000085
Represents the value of the kernel of the i-th deconvolution layer of the first convolution layer,
Figure GDA00030801309200000814
denoted as a deconvolution operation; thus, after performing the deconvolution operation, a binary mask map is generated
Figure GDA0003080130920000086
(Binary mask) as a saliency-constrained map to separate image regions and non-image regions in the display image;
step 4, obtaining a predicted binary mask image after testing
Figure GDA0003080130920000087
Then, the invention makes a parameter adjustment to obtain the best effect; the adjustment of the learning rate is shown in the formula (1-5):
Figure GDA0003080130920000088
wherein R istFor the learning rate, ε is the threshold for trigger decay, t represents the change count, t is 0, 1, 2, 3; r0Is an initial value of the learning rate, which is set to 10-4(ii) a Alpha is a decay index; the invention carries out convolution operation of 15 epochs in total, wherein every 3 epochs are reduced to alpha times of the previous epochs and are changed for 4 times in total; the present invention sets the value of the attenuation index α to 0.90;
step 2: the context regularization operation specifically includes the following steps:
step 2.1: definition of
Figure GDA0003080130920000089
Is that
Figure GDA00030801309200000810
Value at position (m, n), XgIs a gray scale map because of XgIs the graph closest to the true phase, and I (m, n) is the grayscale graph XgThe pixel at position (m, n) can be expressed by the formula (1-6):
Figure GDA00030801309200000811
where η is the most primitive error and can be expressed by equations (1-7):
Figure GDA00030801309200000812
the error parameter eta is minimized and the error parameter eta,
Figure GDA00030801309200000813
represents Frobenius norm operation;
step 2.2: defining a constraint function:
Figure GDA0003080130920000091
wherein the content of the first and second substances,
Figure GDA0003080130920000092
is a pixel
Figure GDA0003080130920000093
8 directions of pixels around;
step 2.3: setting a weight function to W (m, n):
Figure GDA0003080130920000094
the weighting function W (m, n) represents that at position (m, n) the weighting function acts as a "switch" for the constraint between m and n, when W (m, n) is 0, the corresponding context constraint between m and n will be cancelled; a very critical issue is how to choose a reasonable W (m, n);
step 2.4: constructing a weight function W (m, n) based on a method of squared difference between vectors of two adjacent pixels:
Figure GDA0003080130920000095
where σ is a defined parameter and has a value of 0.5, and I (m, n) and I (m + Δ m, n + Δ n) are each XgAnd 8 directional pixels around the input pixel;
step 2.5: adding a weighted context constraint in the image domain; for ease of calculation, equations (1-9) can be expressed as:
Figure GDA0003080130920000096
wherein, ω represents different directions of the pixels at 8 positions;
step 2.6: as shown in fig. 2: defining a higher order filter DΔm,Δn；
Set up DΔm,ΔnThe calculation of the value at each position (Δ m, Δ n) satisfies the formulas (1 to 12):
Figure GDA0003080130920000097
for convenience of calculation, equations (2-12) use more reasonable expression methods, such as equations (1-13):
Figure GDA0003080130920000098
wherein the content of the first and second substances,
Figure GDA0003080130920000099
a multiplier that represents a pixel of the image,
Figure GDA00030801309200000910
representing the convolution operator, WΔm,ΔnA weighting matrix representing pixels in (Δ m, Δ n) | | | · purple1Represents an estimate of manhattan distance;
the invention filters each pixel channel of the input picture according to a moving window of a minimum filter, and then takes the maximum value of each channel as XgAn estimate of the component of (a);
step 2.7: the following objective function is defined and minimized to find an optimal function, which is given by equations (1-7) and (1-13):
Figure GDA0003080130920000101
where ξ is a regularization parameter that balances two conditions. For the
Figure GDA0003080130920000102
The following formula is satisfied:
Figure GDA0003080130920000103
step 2.8: for the convenience of calculation, the invention adopts an optimization method based on separation variables, and the basic idea of the methodIntroducing a plurality of auxiliary variables, constructing a series of simple subproblems, and converging the final solution to the optimal solution of the original problem; defining auxiliary variables
Figure GDA0003080130920000104
Rewrite formula (1-15):
Figure GDA0003080130920000105
where β is a predefined scale factor set to
Figure GDA0003080130920000106
Further, the initial value beta0Is 1, maximum value betamaxIs 22(ii) a By a scale factor
Figure GDA0003080130920000107
Repeatedly increasing beta to make the cycle from minimum value 0 to maximum value 22；
Step 2.9: first, fix
Figure GDA0003080130920000108
Simplification of
Figure GDA0003080130920000109
Then fixed
Figure GDA00030801309200001010
Simplification of
Figure GDA00030801309200001011
The process is repeated until convergence, so that the problem can be effectively solved; the method comprises the following steps:
first, fix
Figure GDA00030801309200001012
Simplification of
Figure GDA00030801309200001013
Figure GDA00030801309200001014
Thus, optimization can be directly made in the location (m, n)
Figure GDA00030801309200001015
Figure GDA00030801309200001016
Wherein the content of the first and second substances,
Figure GDA00030801309200001017
a certain pixel of 8 pixels around a pixel representing coordinates (m, n) in the image, sign (·) being a signal function;
secondly, fix
Figure GDA00030801309200001018
Optimization
Figure GDA00030801309200001019
Figure GDA00030801309200001020
Since the equations (2-19) are
Figure GDA00030801309200001021
The quadratic equation of (a), and thus the equations (1-19) can be rewritten:
Figure GDA00030801309200001022
step 2.10: optimization using two-dimensional Fourier transform (2D FFT) and hypothetical cycle boundary conditions
Figure GDA0003080130920000111
Can directly calculate
Figure GDA0003080130920000112
Optimal solution Y of (a):
Figure GDA0003080130920000113
where τ is the Fourier transform, τ (·)-1Is the inverse fourier transform of the signal to be processed,
Figure GDA0003080130920000115
representing pixel multiplication, in equations (1-21), the division is also calculated in pixel fashion; in an iterative process, by a scaling factor
Figure GDA0003080130920000114
Repeatedly increasing beta from a minimum value of 0 to a maximum value of 22。
And step 3: a loop iteration operation comprising the steps of:
step 3.1: after the context regularization calculation, the obtained optimal solution Y is compared with the input RGB picture Xi lMultiplying pixel values to obtain a multiplied picture Y; the value of each pixel point in the image Y area is 0, and the non-image area is kept unchanged;
step 3.2: taking Y in the step 3.1 as an input, repeating the step 1, the step 2 and the step 3.1, and setting an entropy critical value 6.9 based on the entropy of Y2Make iteration from 0 to 6.92And the process is carried out until the binary mask image Y' closest to the true phase is predicted.
The method judges whether the final result is the binary mask image closest to the true phase by calculating Precision, Recall, F1-Measure and similarity.
Wherein, the True Positive (TP) indicates that the prediction result is 0 when the answer is 0; a True Negative (TN) example shows that when the answer is 1, the prediction result is 1; the False Positive example (FP) shows that when the answer is 0, the prediction result is 1; false Negative (FN) indicates that the answer is 1, the prediction result is 0.
Precision (Precision) is the ratio of the true correct number to the whole result; that is, under the criterion that the correct answer should be 0, the number of correct predictions is proportional to the number of 0 values of all the predictions.
The Recall rate (Recall, also called Recall rate) is the proportion of the true correct number in the whole data set; i.e. the ratio of the number of predicted correct answers to the number of correct answers of 0 in the whole data set, under the criterion that the correct answer should be 0.
F1-Measure is the Precision rate Precision and Recall weighted harmonic mean.
The similarity (similarity) is the sum of the number of true correct numbers accounting for prediction errors and the false alarm rate and the missing alarm rate; that is, under the criterion that the correct answer should be 0, the number of correct answers is predicted to be a ratio of the number of correct answers of 0 to the sum of the number of correct answers of 1 and the number of predicted answers of 0 in the entire data set.
The values of the 4 evaluation indexes of precision, recall, F1_ measure and similarity are between 0 and 1, wherein the closer the value is to 1, the closer the final result is to the true phase.
And 4, step 4: and 4, step 4: according to the Y' in the step 3.2, keeping the image area of the picture displayed by the OLED self-luminous display unchanged, and reducing the pixel brightness value in the non-image area; the specific method comprises the following steps:
after a predicted binary mask image Y' is obtained by a context regularization-based cyclic depth learning framework, linear transformation is performed to reduce the power consumption of a non-image area in a displayed image, and an image after energy saving is defined as Y ″, then:
Figure GDA0003080130920000121
where P is the desired power consumption level; after this operation, only the non-image area pixel luminance values are reduced, but the image area is unchanged.
The objective evaluation indexes of 680 pictures with 80% of power consumption, 680 pictures with 70% of power consumption and 680 pictures with 60% of power consumption are compared respectively: the power consumption of 80% means that the power consumption after the power saving operation is performed is 80% of the power consumption without the power saving operation, and the power consumption is described as 80% hereinafter; the power consumption amount of 70% means that the power consumption amount after the power saving operation is performed is 70% of the power consumption amount without the power saving operation, and is hereinafter described as 70% of the power consumption amount; the power consumption amount of 60% means that the power consumption amount after the power saving operation is performed is 60% of the power consumption amount without the power saving operation, and hereinafter, the power consumption amount is described as 60%.
From table 1 to table 3, 5 objective evaluation indexes, namely Peak Signal to Noise Ratio (PSNR), Structural Similarity (SSIM), Visual Information Fidelity (VIF), Measure of Enhancement (EME) and Color Quality Enhancement (CQE), are compared for all comparison methods; wherein, the larger the PSNR value is, the smaller the representative image distortion is; the closer the value of SSIM is to 1, the higher the similarity with the original image is; the quality of the image is measured by the VIF through calculating the mutual information with the original image, and the larger the value is, the better the visual effect is; a larger EME value represents a higher image quality; comparing the CQE calculation with the original image in terms of color, wherein the larger the value is, the better the image color is; all objective index values used in the invention are higher in value, and represent better visual effect of the image.
TABLE 1 Objective evaluation index comparison of Power saving Effect at 80% Power consumption
Figure GDA0003080130920000122
TABLE 2 Objective evaluation index comparison of Power saving Effect at 70% Power consumption
Figure GDA0003080130920000123
Figure GDA0003080130920000131
TABLE 3 Objective evaluation index comparison of Power saving Effect at 60% Power consumption
Figure GDA0003080130920000132
According to the data results in tables 1, 2 and 3, the method of the present invention has data values higher than those of the other methods in comparison with the 5 methods of SDMSR, PQPR, IQPC, HDPCCE and HPPD, in the case of different power consumption, and thus, the method of the present invention not only realizes the function of reducing the brightness of the non-image region while maintaining the image region, but also has better image visual retention than the other 5 methods, and has improved visual retention than the 5 methods of SDMSR, PQPR, IQPC, HDPCCE and HPPD in the 5 objective evaluation indexes of PSNR, SSIM, VIF, EME and CQE.
Compared with other 5 power saving methods, the power saving method has the advantages that the visual effect is better maintained under the condition that the power consumption is the same, and the experimental data of objective evaluation indexes are shown; therefore, the method provided by the invention better maintains the image quality, and has higher value and better visual perception.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.

Claims (5)

1. The power saving method for the display of the mobile equipment based on image segmentation is characterized by comprising the following steps:
step 1: for input picture
Figure FDA0003080130910000011
Performing convolution operation in a VGG19-FCN network, wherein the VGG19-FCN network is composed of 18 convolution layers, 5 pooling layers and 3 deconvolution layers; the method specifically comprises the following steps:
step 1.1: the convolutional layer operation was carried out using the following method:
suppose that
Figure FDA0003080130910000012
Is the i-th layer feature map of the i-th layer convolution layer, defines the feature map
Figure FDA0003080130910000013
In order to input the quantity of the input,
Figure FDA0003080130910000014
for the output quantity, i.e., the binary mask map, the convolution operation is shown in equation (1-1):
Figure FDA0003080130910000015
where N is the total number of feature maps,
Figure FDA0003080130910000016
is a convolution kernel of the i-th convolutional layer of the first convolutional layer,
Figure FDA0003080130910000017
is a deviation parameter of the ith convolution layer of the first convolution layer; n is
Figure FDA0003080130910000018
The number of feature maps in (a) is denoted as convolution operation, f (-) is the activation function;
step 1.2: the maximum pooling operation of the pooling layer is expressed by the following formula (1-3):
Figure FDA0003080130910000019
wherein Ω (m, n) represents a feature vector
Figure FDA00030801309100000110
Is the position (m, n) of the space vector of (a), and Δ represents
Figure FDA00030801309100000111
In layer 7 of the framework of the algorithm;
step 1.3: and (3) performing deconvolution layer operation by adopting the following method:
deconvoluting the output of the 5 th layer of the first layer of convolutional layer to the size of original figure, and then deconvoluting the output of the 4 th layer of the first layer of convolutional layer and the output of the 3 rd layer in turn to obtain
Figure FDA00030801309100000112
The deconvolution operation is expressed by the following equation (1-4):
Figure FDA00030801309100000113
wherein
Figure FDA00030801309100000114
Is a convolution kernel of the i-th convolutional layer of the first convolutional layer,
Figure FDA00030801309100000115
denoted as a deconvolution operation;
step 1.4: the learning rate is adjusted by the following method:
the adjustment of the learning rate is shown in the formula (1-5):
Figure FDA00030801309100000116
wherein R istFor the learning rate, ε is the threshold for trigger attenuation, t is the change count, t is 0, 1, 2, 3; alpha is a decay index; ρ is 0.90;
step 2: binary mask map output to convolutional layer
Figure FDA0003080130910000021
The context regularization operation specifically includes the following steps:
step 2.1: definition of
Figure FDA0003080130910000022
Is composed of
Figure FDA0003080130910000023
Pixel at position (m, n), XgIs a gray scale image, then XgThe pixel I (m, n) at position (m, n) is:
Figure FDA0003080130910000024
where η is the most primitive error, η is:
Figure FDA0003080130910000025
wherein the content of the first and second substances,
Figure FDA0003080130910000026
in order to minimize the error parameter η,
Figure FDA0003080130910000027
represents Frobenius norm operation;
step 2.2: defining a constraint function:
Figure FDA0003080130910000028
wherein the content of the first and second substances,
Figure FDA0003080130910000029
is a pixel
Figure FDA00030801309100000210
Pixels of the surrounding 8 directions;
step 2.3: defining a weight function W (m, n):
Figure FDA00030801309100000211
when W (m, n) is 0, the corresponding context constraint between m and n will be cancelled;
step 2.4: constructing a weight function W (m, n) based on a method of squared difference between vectors of two adjacent pixels:
Figure FDA00030801309100000212
where σ is a predetermined parameter, σ is 0.5, and I (m + Δ m, n + Δ n) is Xg8 directional pixels around the input pixel;
step 2.5: adding a weighted context constraint in the image domain, rewriting equations (1-9) as:
Figure FDA00030801309100000213
wherein, ω represents different directions of the pixels at 8 positions;
step 2.6: defining a higher order filter DΔm,ΔnLet D beΔm,ΔnThe value at each position (Δ m, Δ n) satisfies:
Figure FDA00030801309100000214
even if DΔm,ΔnAt each timeThe values of one position (Δ m, Δ n) all satisfy
Figure FDA0003080130910000031
Wherein, omega represents an index set,
Figure FDA00030801309100000319
is a multiplication operator for the pixel or pixels,
Figure FDA0003080130910000032
for convolution operators, DΔm,ΔnRepresenting a first order differential operator, WΔm,ΔnA weighting matrix representing pixels in (Δ m, Δ n) | | | · purple1Represents an estimate of manhattan distance;
step 2.7: the following objective function is defined and minimized, the objective function (1-14) being derived from equations (1-7) and equations (1-13):
Figure FDA0003080130910000033
where ξ is a regularization parameter that balances two conditions; for the
Figure FDA0003080130910000034
The following formula is satisfied:
Figure FDA0003080130910000035
step 2.8: defining auxiliary variables
Figure FDA0003080130910000036
Rewrite formula (1-15):
Figure FDA0003080130910000037
wherein, beta is a predefined scale factor,
Figure FDA0003080130910000038
initial value of beta0Is 1, maximum value betamaxIs 22By means of a scale factor
Figure FDA0003080130910000039
Repeatedly increasing beta from a minimum value of 0 to a maximum value of 22；
Step 2.9: first, fix
Figure FDA00030801309100000310
Optimization
Figure FDA00030801309100000311
Figure FDA00030801309100000312
Thus, optimization can be directly made in the location (m, n)
Figure FDA00030801309100000313
Figure FDA00030801309100000314
Wherein the content of the first and second substances,
Figure FDA00030801309100000315
a certain pixel of 8 pixels around a pixel representing coordinates (m, n) in the image, sign (·) being a signal function;
secondly, fix
Figure FDA00030801309100000316
Optimization
Figure FDA00030801309100000317
Figure FDA00030801309100000318
Since the formulae (1-19) are
Figure FDA0003080130910000041
The quadratic equation of (a), and thus the equations (1-19) can be rewritten:
Figure FDA0003080130910000042
step 2.10: optimization using two-dimensional Fourier transform and hypothetical cycle boundary conditions
Figure FDA0003080130910000043
Calculating an optimal solution Y:
Figure FDA0003080130910000044
where τ is the Fourier transform, τ (·)-1Is an inverse fourier transform;
and step 3: a loop iteration operation comprising the steps of:
step 3.1: after the context regularization calculation, the obtained optimal solution Y is compared with the input RGB picture Xi lMultiplying pixel values to obtain a multiplied picture Y;
step 3.2: taking Y in the step 3.1 as an input, repeating the step 1, the step 2 and the step 3.1, and setting an entropy critical value 6.9 based on the entropy of Y2Make iteration from 0 to 6.92Performing the operation until a binary mask image Y' closest to the true phase is predicted;
and 4, step 4: according to Y' in step 3.2, the image area of the picture displayed by the OLED self-luminous display is kept unchanged, and the non-image area is subjected to pixel brightness value reduction.
2. The image segmentation based mobile device display power saving method of claim 1, wherein: in step 1.1, the activation function uses a modified linear unit, as shown in formula (1-2):
f(x)＝max(0,x)；(1-2)
where x is the input value of the activation function.
3. The image segmentation based mobile device display power saving method of claim 1, wherein: in step 1.4, the learning rate RtInitial value R of0Is set to 10-4。
4. The image segmentation based mobile device display power saving method of claim 1, wherein: in step 3.2, the method for judging whether the picture Y is the binary mask picture closest to the true phase comprises the following steps: the evaluation is judged by 4 evaluation indexes of precision, recall, F1_ measure and similarity.
5. The image segmentation based mobile device display power saving method of claim 1, wherein: the method for reducing the pixel brightness value of the non-image area in the step 4 comprises the following steps: defining the image after energy saving as Y', then:
Figure FDA0003080130910000051
where P is the desired power consumption level.
CN201811089845.7A 2018-09-18 2018-09-18 Mobile device display power saving method based on image segmentation Active CN109343692B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201811089845.7A CN109343692B (en) 2018-09-18 2018-09-18 Mobile device display power saving method based on image segmentation

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201811089845.7A CN109343692B (en) 2018-09-18 2018-09-18 Mobile device display power saving method based on image segmentation

Publications (2)

Publication Number Publication Date
CN109343692A CN109343692A (en) 2019-02-15
CN109343692B true CN109343692B (en) 2021-07-23

Family

ID=65305612

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201811089845.7A Active CN109343692B (en) 2018-09-18 2018-09-18 Mobile device display power saving method based on image segmentation

Country Status (1)

Country Link
CN (1) CN109343692B (en)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112348766B (en) * 2020-11-06 2023-04-18 天津大学 Progressive feature stream depth fusion network for surveillance video enhancement
CN113608805B (en) * 2021-07-08 2024-04-12 阿里巴巴创新公司 Mask prediction method, image processing method, display method and device
CN114860059B (en) * 2022-05-12 2023-10-31 深圳市华云电源有限公司 LED display screen power supply control method and device, electronic equipment and medium
CN117198219B (en) * 2023-11-07 2024-02-27 惠科股份有限公司 Display method, device, storage medium and apparatus for display screen
CN117234320B (en) * 2023-11-15 2024-02-23 深圳市鸿茂元智光电有限公司 Energy-saving display method and system for LED display screen and display screen

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104460960A (en) * 2013-09-18 2015-03-25 联想(北京)有限公司 Data processing method and electronic device
CN107169974A (en) * 2017-05-26 2017-09-15 中国科学技术大学 It is a kind of based on the image partition method for supervising full convolutional neural networks more
CN107680037A (en) * 2017-09-12 2018-02-09 河南大学 The improved face super-resolution reconstruction method based on nearest feature line manifold learning
CN108062756A (en) * 2018-01-29 2018-05-22 重庆理工大学 Image, semantic dividing method based on the full convolutional network of depth and condition random field
CN108062531A (en) * 2017-12-25 2018-05-22 南京信息工程大学 A kind of video object detection method that convolutional neural networks are returned based on cascade

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10417788B2 (en) * 2016-09-21 2019-09-17 Realize, Inc. Anomaly detection in volumetric medical images using sequential convolutional and recurrent neural networks

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104460960A (en) * 2013-09-18 2015-03-25 联想(北京)有限公司 Data processing method and electronic device
CN107169974A (en) * 2017-05-26 2017-09-15 中国科学技术大学 It is a kind of based on the image partition method for supervising full convolutional neural networks more
CN107680037A (en) * 2017-09-12 2018-02-09 河南大学 The improved face super-resolution reconstruction method based on nearest feature line manifold learning
CN108062531A (en) * 2017-12-25 2018-05-22 南京信息工程大学 A kind of video object detection method that convolutional neural networks are returned based on cascade
CN108062756A (en) * 2018-01-29 2018-05-22 重庆理工大学 Image, semantic dividing method based on the full convolutional network of depth and condition random field

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs;Liang-Chieh Chen et.al;《IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE》;20180401;第40卷(第4期);834-845 *
基于深度学习的图像语义分割研究;肖旭;《中国优秀硕士学位论文全文数据库 信息科技辑》;20180115;I138-1011 *

Also Published As

Publication number Publication date
CN109343692A (en) 2019-02-15

Similar Documents

Publication Publication Date Title
CN109343692B (en) Mobile device display power saving method based on image segmentation
Wang et al. Simple low-light image enhancement based on Weber–Fechner law in logarithmic space
CN110232661B (en) Low-illumination color image enhancement method based on Retinex and convolutional neural network
CN103778900B (en) A kind of image processing method and system
KR20200014842A (en) Image illumination methods, devices, electronic devices and storage media
CN114723643B (en) Low-light image enhancement method based on reinforcement learning and aesthetic evaluation
CN115115554B (en) Image processing method and device based on enhanced image and computer equipment
CN111225162B (en) Image exposure control method, system, readable storage medium and camera equipment
Acharya et al. Directed searching optimized mean-exposure based sub-image histogram equalization for grayscale image enhancement
CN103295205B (en) A kind of low-light-level image quick enhancement method based on Retinex and device
Parihar et al. A comprehensive analysis of fusion-based image enhancement techniques
Liu et al. Progressive complex illumination image appearance transfer based on CNN
Zhang et al. Non-uniform illumination video enhancement based on zone system and fusion
CN110009574B (en) Method for reversely generating high dynamic range image from low dynamic range image
CN108550124B (en) Illumination compensation and image enhancement method based on bionic spiral
CN113034412B (en) Video processing method and device
CN116993616A (en) Single low-illumination scene image enhancement method and enhancement system
Miao et al. Novel tone mapping method via macro-micro modeling of human visual system
Li et al. A novel detail weighted histogram equalization method for brightness preserving image enhancement based on partial statistic and global mapping model
Han et al. High dynamic range image tone mapping: Literature review and performance benchmark
CN113222828A (en) Zero-reference-based image enhancement method for industrial Internet of things monitoring platform
CN114240767A (en) Image wide dynamic range processing method and device based on exposure fusion
Liu et al. An adaptive tone mapping algorithm based on gaussian filter
CN113191956A (en) Backlight image enhancement method based on depth cutout
Fang et al. Tone mapping based on fast image decomposition and multi‐layer fusion

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant