WO2022213261A1 - 图像处理方法及装置、电子设备及存储介质 - Google Patents

图像处理方法及装置、电子设备及存储介质 Download PDF

Info

Publication number
WO2022213261A1
WO2022213261A1 PCT/CN2021/085651 CN2021085651W WO2022213261A1 WO 2022213261 A1 WO2022213261 A1 WO 2022213261A1 CN 2021085651 W CN2021085651 W CN 2021085651W WO 2022213261 A1 WO2022213261 A1 WO 2022213261A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
loss function
preprocessed
blurred
label
Prior art date
Application number
PCT/CN2021/085651
Other languages
English (en)
French (fr)
Inventor
付艳艳
Original Assignee
深圳市欢太科技有限公司
Oppo广东移动通信有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳市欢太科技有限公司, Oppo广东移动通信有限公司 filed Critical 深圳市欢太科技有限公司
Priority to CN202180096597.2A priority Critical patent/CN117121048A/zh
Priority to PCT/CN2021/085651 priority patent/WO2022213261A1/zh
Publication of WO2022213261A1 publication Critical patent/WO2022213261A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/70—Denoising; Smoothing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/50—Maintenance of biometric data or enrolment thereof
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G06T2207/30201—Face

Definitions

  • the present application relates to the field of imaging technologies, and in particular, to a privacy-protected image processing method, an image processing apparatus, an electronic device, and a storage medium.
  • Embodiments of the present application provide an image processing method, an image processing apparatus, an electronic device, and a storage medium for privacy protection.
  • An image processing method for privacy protection includes: acquiring a preprocessed image to perform feature extraction on the preprocessed image to obtain a first feature vector; generating a noise image according to the preprocessed image to detect the noise Perform feature extraction on the image to obtain a second feature vector; according to the structural similarity between the preprocessed image and the noise image, the first feature vector and the second feature vector establish a loss function; The noise image is processed to obtain a blurred image; the preprocessed image is replaced with the blurred image.
  • a privacy-preserving image processing apparatus includes: an acquisition module for acquiring a preprocessed image to perform feature extraction on the preprocessed image to obtain a first feature vector; a generation module for generating a noise image to Perform feature extraction on the noise image to obtain a second feature vector, and the noise image corresponds to the preprocessed image; a processing module is configured to, according to the structural similarity between the preprocessed image and the noise image, the first feature vector.
  • a feature vector and the second feature vector establish the loss function; a blurring module is used to process the noise image by using the loss function to obtain a blurred image; a replacement module is used to replace the blurred image with the blurred image. preprocessed images.
  • the electronic device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the image processing method described above is implemented.
  • a non-volatile computer-readable storage medium of a computer program is characterized in that, when the computer program is executed by one or more processors, the image processing method described above is implemented.
  • FIG. 1 is a schematic flowchart of an image processing method according to an embodiment of the present application.
  • FIG. 2 is a schematic flowchart of an image processing method according to an embodiment of the present application.
  • FIG. 3 is a schematic flowchart of an image processing method according to an embodiment of the present application.
  • FIG. 4 is a schematic flowchart of an image processing method according to an embodiment of the present application.
  • FIG. 5 is a schematic flowchart of an image processing method according to an embodiment of the present application.
  • FIG. 6 is a schematic flowchart of an image processing method according to an embodiment of the present application.
  • FIG. 7 is a schematic flowchart of an image processing method according to an embodiment of the present application.
  • FIG. 8 is a schematic flowchart of an image processing method according to an embodiment of the present application.
  • FIG. 9 is a block diagram of an image processing apparatus according to an embodiment of the present application.
  • FIG. 10 is a block diagram of an image processing apparatus according to an embodiment of the present application.
  • FIG. 11 is a block diagram of an image processing apparatus according to an embodiment of the present application.
  • FIG. 12 is a block diagram of an image processing apparatus according to an embodiment of the present application.
  • the present application provides an image processing method for privacy protection, which is characterized in that it includes:
  • S20 Generate a noise image according to the preprocessed image to perform feature extraction on the noise image to obtain a second feature vector
  • the embodiments of the present application also provide an electronic device.
  • Electronic devices include memory and processors.
  • a computer program is stored in the memory, and the processor is used to obtain a preprocessed image to perform feature extraction on the preprocessed image to obtain a first feature vector, generate a noise image according to the preprocessed image to perform feature extraction on the noise image to obtain a second feature vector, and obtain a second feature vector according to the preprocessed image.
  • the structural similarity between the image and the noise image is processed, the first feature vector and the second feature vector establish a loss function, the noise image is processed by the loss function to obtain a blurred image, and the blurred image is used to replace the preprocessed image.
  • a preprocessed image is acquired.
  • the image is an image for which privacy protection is desired, and may exist in a photo, an image in a TV or a computer screen, or an object in a video that requires privacy protection.
  • the object may include a human face or other areas that require privacy protection, and the protection scope may be the entire image or a region of interest (Region Of Interest, ROI).
  • the acquired preprocessed image is an image that needs to be subjected to privacy protection after preprocessing.
  • the preprocessing includes processing methods such as image interception, feature extraction, and face detection in which the image is a face.
  • the image to be privacy protected is a human face, and the face region after face detection is acquired as a preprocessing image.
  • a fixed area in the image needs privacy protection, such as the watermark in the lower right corner, then the lower right corner area of a certain size of the image is intercepted as the preprocessed image.
  • feature extraction is performed on the preprocessed image to obtain a first feature vector.
  • different images have different feature vectors, and the feature vectors are the usual meanings in the field of image technology.
  • the feature vector of a face image is a vector of key points of the face, specifically including coordinate points with clear semantics on the face, such as the tip of the nose, the corner of the mouth, the corner of the eye, and the like.
  • the first is used to distinguish subsequent feature vectors.
  • a noise image is generated according to the preprocessed image to perform feature extraction on the noise image to obtain a second feature vector.
  • the noise image may be a noise image satisfying a preset mean value and a preset variance, or a randomly generated random noise image.
  • the feature extraction method for the noise image can be the same as the feature extraction method for the preprocessed image, and the feature vector can also be the same.
  • the first feature vector and the second feature vector establish a loss function, and use the loss function to process the noise image to obtain a blurred image.
  • the loss function can be established according to different parameters.
  • the loss function is established by extracting the structural similarity and feature vector between the preprocessed image and the noise image.
  • the structural similarity represents the similarity between the preprocessed image and the output image.
  • structural similarity By calculating the structural similarity of the input and output images, the structural similarity can be minimized, so that the difference between the generated blurred image and the preprocessed image can be maximized as much as possible.
  • structural dissimilarity may also be utilized.
  • the loss function includes a classification loss function such as a cross-entropy loss function to determine the closeness of the actual output to the expected output, and/or a quality loss function to determine the quality difference between the output image and the expected output, and/or a distance loss function to determine the distance of the feature vector.
  • a classification loss function such as a cross-entropy loss function to determine the closeness of the actual output to the expected output
  • a quality loss function to determine the quality difference between the output image and the expected output
  • a distance loss function to determine the distance of the feature vector.
  • one or more loss functions can be used for multi-objective optimization.
  • the established loss function is optimized such as minimizing the loss function so that the output image satisfies the maximum difference from the preprocessed image as much as possible, while the feature vectors are as close as possible.
  • a loss function such as Euclidean distance
  • the distance of the feature vectors can be minimized to make the feature vectors of the generated blurred image and the preprocessed image as close as possible.
  • the noise image is input into the loss function, and the desired blurred image is output by optimizing the loss function.
  • the established loss function is minimized and solved to optimize the intermediate parameters, so that the desired blurred image is output.
  • a cross-entropy loss function may be established to maximize blurring of the output blurred image from the preprocessed image.
  • a cross-entropy loss function may be established such that the output blurred image is the most blurred from the preprocessed image, and at the same time, the output blurred image is closest to the feature vector of the preprocessed image.
  • the cross-entropy loss function and the distance loss function can be established to maximize the blurring of the output blurred image and the preprocessed image, and at the same time, the output blurred image is closest to the feature vector of the preprocessed image.
  • step S50 the preprocessed image is replaced with the blurred image.
  • the blurred image is obtained, the blurred image is replaced with the pre-processed image, or the blurred image is restored to the pre-processed image.
  • the loss function is used to blur the face, and then the blurred image is restored to the real-time monitoring screen, so that only the blurred face can be displayed in the monitoring screen instead of the clear face. .
  • the blurred image can be resized and then restored to the real-time monitoring screen.
  • the image is face information, and subsequent applications such as face recognition may be further performed on the face.
  • a first feature vector is obtained by acquiring a preprocessed image to perform feature extraction on the preprocessed image
  • a second feature vector is obtained by generating a noise image according to the preprocessed image to perform feature extraction on the noise image.
  • the subsequent face recognition model can be more accurately restored according to the blurred image, and at the same time, the accuracy of face recognition and authentication can be effectively improved. Further, the application of subsequent services such as face recognition and authentication can be carried out through the face recognition model, so that services such as identification and authentication can be realized while effectively protecting the privacy of images.
  • obtaining a preprocessed image includes:
  • S11 Perform face detection on a scene image including face features to obtain a face region, and determine the face region as a preprocessed image.
  • the processor is configured to perform face detection on a scene image including face features to obtain a face region, and determine the face region as a preprocessed image.
  • a face detection model such as a multi-task convolutional neural network (MTCNN) model can be used to detect the scene image to obtain a face region, and then the face region can be used as a preprocessed face image for subsequent follow-up.
  • MTCNN multi-task convolutional neural network
  • the specific process is the same as that in the above-mentioned embodiment.
  • the face region can be extracted by the face detection model, so that the privacy protection of the face information can be fuzzed in the business application scenario related to the face information.
  • the image processing method further includes:
  • S60 Perform face recognition on the preprocessed face image to authenticate the face.
  • the processor is configured to perform face recognition on the preprocessed face image to authenticate the face.
  • face recognition is performed on the preprocessed face image to authenticate the face.
  • face recognition models can be used to recognize faces, for example, Euclidean distance or cosine distance can be used in feature extraction.
  • the Euclidean distance measures the absolute distance between each point in the two images.
  • the feature extraction is performed on the face through the Euclidean distance, and then the face is identified by comparing the Euclidean distance with the threshold.
  • the specific Euclidean distance calculation formula is omitted here, and the present application can use the Euclidean distance to perform face recognition on the preprocessed face image.
  • face recognition can be performed on the scene image by using the cosine distance.
  • the cosine distance uses the cosine value of the angle between the two vectors in the vector space as a measure of the difference between the two images, and the face is characterized by the cosine distance. Extraction, and then compare the cosine distance with the threshold to identify the face.
  • the specific cosine distance calculation formula is omitted here, and the present application can use the cosine distance to perform face recognition on the preprocessed face image.
  • the human face can be authenticated on the basis of fuzzing the human face to protect privacy.
  • the face authentication-related business can effectively protect the privacy of the face information and improve the security of the face authentication.
  • step S11 further includes:
  • S111 Perform size transformation on the face region to transform the original first size into the second size to obtain a preprocessed image.
  • step S50 includes:
  • the processor is configured to perform size transformation of the face region to transform the original first size into the second size to obtain the preprocessed image, and to transform the size of the blurred image into the first size to replace the preprocessed image .
  • size transformation such as linear difference transformation is performed on the extracted face region to transform the original first size into the second size to obtain the preprocessed image.
  • size transformation such as linear difference transformation
  • the noise image is a random noise image
  • the size of the noise image is the second size
  • step S30 includes:
  • S31 Establish a first loss function Determine the loss function, and minimize the first loss function to obtain the blurred image, where SSIM is the structural similarity, T2 is the intermediate iterative image, T0 is the preprocessing image, E2 is the second feature vector, and E1 is the first feature vector. , ⁇ is the penalty term.
  • the processor is configured to establish a first loss function to obtain a loss function, and to minimize the first loss function to obtain a blurred image.
  • the input of the first loss function is the noise image T1 and the preprocessing image T0, and the output is the optimization result T generated by the continuous iteration of the intermediate result T2, that is, the blurred image.
  • SSIM(T2, T0) is the structural similarity between T2 and T0.
  • the calculation formula of structural similarity is omitted here. The larger the value of SSIM(T2, T0), the more similar T2 and T0 are, and the difference between the generated blurred image and the preprocessed image is maximized by minimizing the function. At the same time, the feature vectors of the two images are as close as possible.
  • ⁇ is the penalty term of the loss function, which is a penalty factor used to control the approximate degree of the feature vector in the optimization process.
  • the loss function is determined by the above-mentioned first loss function, and the blurred image can be obtained by minimizing the first loss function.
  • the image processing method further includes:
  • y i is the function label label
  • y predict is the probability of prediction
  • n is the number of samples
  • S80 Minimize a first comprehensive loss function to obtain a blurred image, where the first comprehensive loss function includes a second loss function and a first loss function.
  • the processor is configured to establish a second loss function and minimize the first comprehensive loss function to obtain a blurred image, where the first comprehensive loss function includes the second loss function and the first loss function.
  • protection image A and the preprocessing image B are input into the second loss function, and the Loss 2 is minimized to determine whether the input image is a human face.
  • the first comprehensive loss function includes a second loss function and a first loss function.
  • the first comprehensive loss function TotalLoss 1 can be defined:
  • the blurred image is obtained by minimizing the first comprehensive loss function TotalLoss 1.
  • the specific minimization and solution process is the mathematical solution process, which is not expanded here.
  • the quality of the blurred image can be further improved by increasing the loss function for face detection of the protected image, so that the replaced blurred image can more accurately replace the preprocessed image.
  • step S30 further includes:
  • the processor is configured to establish a third loss function to obtain a loss function, and to minimize the third loss function to obtain a blurred image.
  • the input of the third loss function is the noise image T1 and the preprocessing image T0, and the output is the optimization result T generated by the continuous iteration of the intermediate result T2, that is, the blurred image.
  • cos ⁇ is the cosine distance
  • SSIM(T2, T0) is the structural similarity between T2 and T0.
  • the calculation formula of structural dissimilarity is omitted here. The larger the value of SSIM, the more similar T2 is to T0, and the difference between the generated blurred image and the preprocessed image is maximized by minimizing the function. At the same time, the feature vectors of the two images are as close as possible.
  • ⁇ is a penalty term, which is a penalty factor used to control the approximate degree of eigenvectors in the optimization process.
  • the loss function is determined by the above-mentioned third loss function, and the blurred image can be obtained by minimizing the third loss function.
  • the image processing method further includes:
  • y i is the function label label
  • y predict is the probability of prediction
  • n is the number of samples
  • S100 Minimize a second comprehensive loss function to obtain a blurred image, where the second comprehensive loss function includes a second loss function and a third loss function.
  • the processor is configured to minimize a second integrated loss function to obtain a blurred image, the second integrated loss function includes a second loss function and a third loss function.
  • the second comprehensive loss function includes a second loss function and a third loss function.
  • the second comprehensive loss function TotalLoss 2 can be defined:
  • the quality of the blurred image can be further improved by increasing the loss function for face detection on the protection image, so that the replaced blurred image can more accurately replace the preprocessed image.
  • the embodiment of the present application further provides an image processing apparatus 10 for privacy protection.
  • the image processing apparatus 10 of the embodiment of the present application includes an acquisition module 11 , a generation module 12 , a processing module 13 , a blurring module 14 and a replacement module 15.
  • the acquiring module 11 is configured to acquire a preprocessed image to perform feature extraction on the preprocessed image to obtain a first feature vector.
  • the generating module 12 is configured to generate a noise image to perform feature extraction on the noise image to obtain a second feature vector, and the noise image corresponds to the preprocessed image.
  • the processing module 13 is configured to establish a loss function according to the structural similarity between the preprocessed image and the noise image, the first feature vector and the second feature vector.
  • the blurring module 14 is used to process the noise image by using the loss function to obtain a blurred image.
  • the replacement module 15 is used to replace the preprocessed image with the blurred image.
  • the acquisition module 11 acquires the preprocessed image.
  • the image is an image for which privacy protection is desired, and may exist in a photo, an image in a TV or a computer screen, or an object in a video that requires privacy protection.
  • the object may include a human face or other areas that require privacy protection, and the protection scope may be the entire image or a region of interest (Region Of Interest, ROI).
  • the acquired preprocessed image is an image that needs to be subjected to privacy protection after preprocessing.
  • the preprocessing includes processing methods such as image interception, feature extraction, and face detection in which the image is a face.
  • the image required for privacy protection is a human face
  • the face region after face detection is obtained as a preprocessing image
  • a fixed area in the image needs privacy protection, such as the watermark in the lower right corner, then the lower right corner area of a certain size of the image is intercepted as the preprocessed image.
  • feature extraction is performed on the preprocessed image to obtain a first feature vector.
  • different images have different feature vectors, and the feature vectors are the usual meanings in the field of image technology.
  • the feature vector of a face image is a vector of key points of the face, specifically including coordinate points with clear semantics on the face, such as the tip of the nose, the corner of the mouth, the corner of the eye, and the like.
  • the first is used to distinguish subsequent feature vectors.
  • the generating module 12 generates a noise image according to the preprocessed image to perform feature extraction on the noise image to obtain a second feature vector.
  • the noise image may be a noise image satisfying a preset mean value and a preset variance, or a randomly generated random noise image.
  • the feature extraction method for the noise image can be the same as the feature extraction method for the preprocessed image, and the feature vector can also be the same.
  • the processing module 13 establishes a loss function according to the structural similarity between the preprocessed image and the noise image, the first feature vector and the second feature vector.
  • the blurring module 14 uses the loss function to process the noise image to obtain a blurred image.
  • the loss function can be established according to different parameters.
  • the loss function is established by extracting the structural similarity and feature vector between the preprocessed image and the noise image.
  • the structural similarity represents the similarity between the preprocessed image and the output image.
  • structural similarity By calculating the structural similarity of the input and output images, the structural similarity can be minimized, so that the difference between the generated blurred image and the preprocessed image can be maximized as much as possible.
  • structural dissimilarity may also be utilized.
  • the loss function includes a classification loss function such as a cross-entropy loss function to determine the closeness of the actual output to the expected output, and/or a quality loss function to determine the quality difference between the output image and the expected output, and/or a distance loss function to determine the distance of the feature vector.
  • a classification loss function such as a cross-entropy loss function to determine the closeness of the actual output to the expected output
  • a quality loss function to determine the quality difference between the output image and the expected output
  • a distance loss function to determine the distance of the feature vector.
  • one or more loss functions can be used for multi-objective optimization.
  • the established loss function is optimized such as minimizing the loss function so that the output image satisfies the maximum difference from the preprocessed image as much as possible, while the feature vectors are as close as possible.
  • a loss function such as Euclidean distance
  • the distance of the feature vectors can be minimized to make the feature vectors of the generated blurred image and the preprocessed image as close as possible.
  • the noise image is input into the loss function, and the desired blurred image is output by optimizing the loss function.
  • the established loss function is minimized and solved to optimize the intermediate parameters, so that the desired blurred image is output.
  • a cross-entropy loss function may be established to maximize blurring of the output blurred image from the preprocessed image.
  • a cross-entropy loss function may be established such that the output blurred image is the most blurred from the preprocessed image, and at the same time, the output blurred image is closest to the feature vector of the preprocessed image.
  • the cross-entropy loss function and the distance loss function can be established to maximize the blurring of the output blurred image and the preprocessed image, and at the same time, the output blurred image is closest to the feature vector of the preprocessed image.
  • the replacement module 15 replaces the preprocessed image with the blurred image.
  • the blurred image is obtained, the blurred image is replaced with the pre-processed image, or the blurred image is restored to the pre-processed image.
  • the loss function is used to blur the face, and then the blurred image is restored to the real-time monitoring screen, so that only the blurred face can be displayed in the monitoring screen instead of the clear face. .
  • the blurred image can be resized and then restored to the real-time monitoring screen.
  • the image is face information, and subsequent applications such as face recognition may be further performed on the face.
  • the above-mentioned image processing apparatus for privacy protection obtains a preprocessed image through the acquisition module 11 to perform feature extraction on the preprocessed image to obtain a first feature vector, and the generation module 12 generates a noise image according to the preprocessed image to perform feature extraction on the noise image.
  • the second feature vector, the processing module 13 establishes a loss function according to the structural similarity between the preprocessed image and the noise image, the first feature vector and the second feature vector, and the blurring module 14 uses the loss function to process the noise image to obtain a blurred image, and
  • the replacement module 15 replaces the preprocessed image with the blurred image.
  • the subsequent face recognition model can be more accurately restored according to the blurred image, and at the same time, the accuracy of face recognition and authentication can be effectively improved. Further, the application of subsequent services such as face recognition and authentication can be carried out through the face recognition model, so that services such as identification and authentication can be realized while effectively protecting the privacy of images.
  • the acquisition module 11 includes an extraction unit 111 .
  • the extraction unit 111 is configured to perform face detection on a scene image including a face feature to obtain a face region, and determine the face region as a preprocessed image.
  • the face region can be extracted through the face detection model, so that the privacy protection of the face information can be fuzzed in the business application scenarios related to face information.
  • the image processing apparatus 10 further includes an identification module 16 .
  • the recognition module 16 is used to perform face recognition on the preprocessed face image to authenticate the face.
  • the human face can be authenticated on the basis of fuzzing the human face to protect privacy.
  • the face authentication-related business can effectively protect the privacy of the face information and improve the security of the face authentication.
  • the extraction unit 111 further includes a size transformation sub-unit 1111
  • the replacement module 15 includes a size replacement unit 151 .
  • the size transforming subunit 1112 is used to perform size transform on the face region to transform the original first size into the second size to obtain a preprocessed image.
  • the size replacement unit 151 is used to transform the size of the blurred image into the first size to replace the preprocessed image.
  • each extracted face region can be unified through size transformation, which effectively improves the efficiency of subsequent blurring processing.
  • the noise image is a random noise image
  • the size of the noise image is the second size
  • the processing module 13 includes a first loss function unit 131 .
  • the first loss function unit 131 is used to establish an optimized first loss function To determine the loss function and minimize the first loss function to obtain a blurred image, where SSIM is the structural similarity, T2 is the intermediate iterative image, T0 is the preprocessed image, E2 is the second feature vector, and E1 is the The first eigenvector, ⁇ is the penalty term.
  • the loss function is determined by the above-mentioned first loss function, and the blurred image can be obtained by minimizing the first loss function.
  • the image processing apparatus 10 further includes a first comprehensive loss function module 17 .
  • the quality of the blurred image can be further improved by increasing the loss function for face detection on the protection image, so that the replaced blurred image can more accurately replace the preprocessed image.
  • the loss function is determined by the above-mentioned third loss function, and the blurred image can be obtained by minimizing the third loss function.
  • the image processing apparatus 10 further includes a second comprehensive loss function module 18 .
  • the quality of the blurred image can be further improved by increasing the loss function for face detection on the protection image, so that the replaced blurred image can more accurately replace the preprocessed image.
  • the embodiments of the present application also provide a computer-readable storage medium.
  • One or more non-volatile computer-readable storage media storing a computer program, when the computer program is executed by one or more processors, implements the image processing method of any one of the above embodiments.
  • the image processing method for privacy protection, the image processing device, the electronic device, and the storage medium obtain a first feature vector by acquiring a preprocessed image to perform feature extraction on the preprocessed image, and according to the preprocessed image Generate a noise image to perform feature extraction on the noise image to obtain a second feature vector.
  • the first feature vector and the second feature vector establish a loss function, and use the loss function to process the noise image to obtain a loss function.
  • a blurred image is obtained, and the preprocessed image is replaced by the blurred image.
  • the blurred image is replaced with the original image for storage and transmission, which avoids the risk of leakage or copying to a certain extent.
  • the application of subsequent services such as face recognition authentication can be carried out, so that the image privacy protection can be effectively performed, and the recognition authentication and other services can be realized.
  • face recognition is performed on the basis of fuzzing, so that the face can be authenticated on the basis of fuzzing to protect privacy.
  • the face authentication-related business can effectively protect the privacy of the face information and improve the security of the face authentication.
  • Multi-objective optimization through multiple loss functions can further improve the quality of blurred images.
  • first and second are only used for descriptive purposes, and should not be construed as indicating or implying relative importance or implying the number of indicated technical features. Thus, features delimited with “first”, “second” may expressly or implicitly include at least one of said features. In the description of the present application, “plurality” means at least two, such as two, three, unless expressly and specifically defined otherwise.
  • any description of a process or method in the flowcharts or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing a specified logical function or step of the process , and the scope of the preferred embodiments of the present application includes alternative implementations in which the functions may be performed out of the order shown or discussed, including performing the functions substantially concurrently or in the reverse order depending upon the functions involved, which should It is understood by those skilled in the art to which the embodiments of the present application belong.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Multimedia (AREA)
  • Image Processing (AREA)

Abstract

一种隐私保护的图像处理方法、装置、电子设备、及存储介质，所述方法包括：获取预处理图像以对预处理图像进行特征提取得到第一特征向量（S10），根据预处理图像生成噪声图像以对噪声图像进行特征提取得到第二特征向量（S20），根据预处理图像与噪声图像的结构相似性，第一特征向量及第二特征向量建立损失函数（S30），利用损失函数对噪声图像进行处理以得到模糊图像（S40），及利用模糊图像替换预处理图像（S50）。

Description

图像处理方法及装置、电子设备及存储介质 技术领域
本申请涉及影像技术领域，特别涉及一种隐私保护的图像处理方法、图像处理装置、电子设备、及存储介质。
背景技术
随着人工智能时代的到来，人脸信息在现实生活中有了越来越多的应用或采集如人脸识别、视频监控等。但在这些应用过程中，如果不对人脸信息进行有效地保护会导致隐私泄露而侵犯其隐私权及肖像权。在现有技术中，有的采用图片处理的形式如置乱，模糊，遮掩的方法保护个人的身份信息，但这些处理方式不利于后续的人脸识别鉴权等业务应用。还有的采用传统加密方法进行加密，或者在变换域对隐私内容进行加密，但此类方法由于存在加解密的秘钥，所以存在未授权进行解密的风险。
发明内容
本申请实施方式提供一种隐私保护的图像处理方法、图像处理装置、电子设备、及存储介质。
本申请实施方式的一种隐私保护的图像处理方法，包括：获取预处理图像以对所述预处理图像进行特征提取得到第一特征向量；根据所述预处理图像生成噪声图像以对所述噪声图像进行特征提取得到第二特征向量；根据所述预处理图像与所述噪声图像的结构相似性，所述第一特征向量及所述第二特征向量建立损失函数；利用所述损失函数对所述噪声图像进行处理以得到模糊图像；利用所述模糊图像替换所述预处理图像。
本申请实施方式的一种隐私保护的图像处理装置，包括：获取模块，用于获取预处理图像以对所述预处理图像进行特征提取得到第一特征向量；生成模块，用于生成噪声图像以对所述噪声图像进行特征提取得到第二特征向量，所述噪声图像与所述预处理图像对应；处理模块，用于根据所述预处理图像与所述噪声图像的结构相似性，所述第一特征向量及所述第二特征向量建立所述损失函数；模糊模块，用于利用所述损失函数对所述噪声图像进行处理以得到模糊图像；替换模块，用于利用所述模糊图像替换所述预处理图像。
本申请实施方式的一种电子设备，所述电子设备包括存储器和处理器，所述存储器中存储有计算机程序，所述计算机程序被所述处理器执行时，实现上述所述的图像处理方法。
本申请实施方式的一种计算机程序的非易失性计算机可读存储介质，其特征在于，当所述计算机程序被一个或多个处理器执行时，实现上述所述的图像处理方法。
本申请实施方式的附加方面和优点将在下面的描述中部分给出，部分将从下面的描述中变得明显，或通过本申请的实践了解到。
附图说明
本申请的上述和/或附加的方面和优点可以从结合下面附图对实施方式的描述中将变得明显和容易理解，其中：
图1是本申请实施方式的图像处理方法的流程示意图；
图2是本申请实施方式的图像处理方法的流程示意图；
图3是本申请实施方式的图像处理方法的流程示意图；
图4是本申请实施方式的图像处理方法的流程示意图；
图5是本申请实施方式的图像处理方法的流程示意图；
图6是本申请实施方式的图像处理方法的流程示意图；
图7是本申请实施方式的图像处理方法的流程示意图；
图8是本申请实施方式的图像处理方法的流程示意图；
图9是本申请实施方式的图像处理装置模块图；
图10是本申请实施方式的图像处理装置模块图；
图11是本申请实施方式的图像处理装置模块图；
图12是本申请实施方式的图像处理装置模块图。
具体实施方式
下面详细描述本申请的实施方式，实施方式的示例在附图中示出，其中，相同或类似的标号自始至终表示相同或类似的元件或具有相同或类似功能的元件。下面通过参考附图描述的实施方式是示例性的，仅用于解释本申请，而不能理解为对本申请的限制。
请参阅图1，本申请提供一种隐私保护的图像处理方法，其特征在于，包括：
S10：获取预处理图像以对预处理图像进行特征提取得到第一特征向量；
S20：根据预处理图像生成噪声图像以对噪声图像进行特征提取得到第二特征向量；
S30：根据预处理图像与噪声图像的结构相似性，第一特征向量及第二特征向量建立损失函数；
S40：利用损失函数对噪声图像进行处理以得到模糊图像；
S50：利用模糊图像替换预处理图像。
本申请实施方式还提供了一种电子设备。电子设备包括存储器和处理器。存储器中存储有计算机程序，处理器用于获取预处理图像以对预处理图像进行特征提取得到第一特征向量，根据预处理图像生成噪声图像以对噪声图像进行特征提取得到第二特征向量，根据预处理图像与噪声图像的结构相似性，第一特征向量及第二特征向量建立损失函数，利用损失函数对噪声图像进行处理以得到模糊图像，利用模糊图像替换预处理图像。
在步骤S10中，获取预处理图像。其中，图像为想要进行隐私保护的图像，可存在于照片中、电视中或计算机屏幕中的图像、或视频中需要隐私保护的对象。同时，对象可包括人脸或其它需要进行隐私保护的区域，其保护范围可为整幅图像或感兴趣区域(Region Of Interest，ROI)。
具体地，获取的预处理图像为所需进行隐私保护的图像进行预处理之后的图像。其中，预处理包括对图像的截取、特征提取、图像为人脸的人脸检测等处理方式。
在某些实施方式中，所需进行隐私保护的图像为人脸，获取经过人脸检测后的人脸区域作为预处理图像。
在某些实施方式中，图像中的固定区域需要隐私保护如右下角的水印处，则截取图像某一尺寸的右下角区域作为预处理图像。
进一步地，对预处理图像进行特征提取得到第一特征向量。其中，根据实际情况，不同的图像具有不同的特征向量，特征向量为图像技术领域中的通常含义。例如人脸图像的特征向量为人脸关键点的向量，具体包括人脸上具有明确语义的坐标点，如鼻尖、嘴角、眼角等。第一用于区别后续的特征向量。
在步骤S20中，根据预处理图像生成噪声图像以对噪声图像进行特征提取得到第二特征向量。其中，噪声图像可为满足预设均值和预设方差的噪声图像，或随机产生的随机噪声图像。对噪声图像的特征提取方式可与对预处理图像进行特征提取的方式相同，其特征向量也可相同。
进一步地，在步骤S30及S40中，根据预处理图像与噪声图像的结构相似性，第一特征向量及第二特征向量建立损失函数，并利用损失函数对噪声图像进行处理以得到模糊图像。
可以理解的是，损失函数可根据不同的参数建立，本实施例中，通过提取预处理图像与噪声图像的结构相似性及特征向量建立损失函数。其中，结构相似性表征预处理图像及输出图像的相似性，通过对输入输出图像结构相似性的计算可使得结构相似性最小化，从而使得生成的模糊图像和预处理图像尽可能地差异最大化。在某些实施方式中，还可利用结构相异性。
其中，损失函数包括分类损失函数如利用交叉熵损失函数来判定实际的输出与期望的输出的接近程度，和/或质量损失函数判定输出图像与期望的输出的质量差异，和/或距离损失函数来判定特征向量的距离。同时，可利用一个或多个损失函数进行多目标的优化。进一步地，通过优化建立的损失函数如最小化损失函数以使得输出图像满足与预处理图像尽可能地差异最大化，同时特征向量尽可能地接近。
具体地，可根据第一特征向量及第二特征向量的距离损失建立损失函数，如欧式距离。同时可最小化特征向量的距离以使得生成的模糊图像和预处理图像的特征向量尽可能地接近。损失函数中输入噪声图像，并通过对损失函数进行优化使得输出期望的模糊图像，如对建立的损失函数进行最小化求解以优化中间参数，使得输出期望的模糊图像。
在某些实施方式中，可建立交叉熵损失函数以使得输出的模糊图像与预处理图像模糊化程度最大。
在某些实施方式中，可建立交叉熵损失函数以使得输出的模糊图像与预处理图像模糊化程度最大，同时，输出的模糊图像与预处理图像的特征向量最接近。
在某些实施方式中，可建立交叉熵损失函数及距离损失函数以使得输出的模糊图像与预处理图像模糊化程度最大，同时，输出的模糊图像与预处理图像的特征向量最接近。
在步骤S50中，利用模糊图像替换预处理图像。当得到模糊图像后，将模糊图像替换预处理图像，或者说将模糊图像还原到预处理图像中。例如，在实时监控画面中，通过损失函数对人脸进行模糊化处理，然后将模糊化的图像还原到实时监控画面中，使得监控画面中仅能显示模糊化的人脸而非清晰的人脸。
在某些实施方式中，可对模糊化的图像进行尺寸调整等变换后再还原到实时监控画面中。
在某些实施方式中，图像为人脸信息，则还可进一步地对人脸进行人脸识别等后续应用。
如此，上述隐私保护的图像处理方法，通过获取预处理图像以对预处理图像进行特征提取得到第一特征向量，根据预处理图像生成噪声图像以对噪声图像进行特征提取得到第二特征向量，根据预处理图像与噪声图像的结构相似性，第一特征向量及第二特征向量建立损失函数，利用损失函数对噪声图像进行处理以得到模糊图像，利用模糊图像替换预处理图像。可使得对需要进行隐私保护的图像进行模糊化处理，并将模糊图像替换原始图像，使得实时应用类的业务场景可以输出隐私保护区域的模糊画面，从而最小化信息的采集，有效地降低了隐私的泄露。同时，将模糊图像替换原始图像进行存储及传输，在一定程度上避免了泄露或被拷贝的风险。再者，根据结构相似性及特征向量建立损失函数并优化损失函数以生成模糊图像，可以使得对图像模糊化的同时，保障其特征向量尽可能地接近，即就是输出的图像的特征尽可能接近预处理图像。在一定程度上提高了模糊图像的质量。进一步地，后续的人脸识别模型可根据模糊图像进行更为准确的还原，同时有效地提高对人脸进行识别鉴权的精确度。进一步地，通过人脸识别模型可进行人脸识别鉴权等后续业务的应用，使得有效地对图像进行隐私保护的同时可实现识别鉴权等业务。
请参阅图2，在某些实施方式中，获取预处理图像包括：
S11：对包括人脸特征的场景图像进行人脸检测得到人脸区域，并将人脸区域确定为预处理图像。
在某些实施方式中，处理器用于对包括人脸特征的场景图像进行人脸检测得到人脸区域，并将人脸区域确定为预处理图像。
具体地，可通过人脸检测模型如多任务卷积神经网络模型(Multi task Convolutional Neural Network，MTCNN)对场景图像进行检测以得到人脸区域，然后将人脸区域作为预处理人脸图像进行后续模糊化处理，具体过程同上述实施方式。
如此，可通过人脸检测模型对人脸区域进行提取，以使得在人脸信息相关业务应用场景中可对人脸信息进行模糊化的隐私保护。
请参阅图3，在某些实施方式中，图像处理方法还包括：
S60：对预处理人脸图像进行人脸识别以对人脸进行鉴权。
在某些实施方式中，处理器用于对预处理人脸图像进行人脸识别以对人脸进行鉴权。
具体地，对预处理人脸图像进行人脸识别以对人脸进行鉴权。可利用多种人脸识别模型对人脸进行识别，例如在特征提取中可利用欧式距离或余弦距离等。
欧氏距离衡量的是两幅图像中各个点之间的绝对距离，通过欧氏距离对人脸进行特征提取，再通过欧氏距离与阈值的比较以对人脸进行识别。具体欧式距离计算公式此处略过，本申请可利用欧氏距离对预处理人脸图像进行人脸识别。
在某些实施方式中，可利用余弦距离对场景图像进行人脸识别，余弦距离用向量空间中两个向量夹角的余弦值作为衡量两幅图像差异的大小，通过余弦距离对人脸进行特征提取，再通过余弦距离与阈值的比较以对人脸进行识别。具体余弦距离计算公式此处略过，本申请可利用余弦距离对预处理人脸图像进行人脸识别。
需要说明的是，人脸识别与模糊化处理、替换图像没有先后顺序的区分，可按照实际业务情况进行程序设计。
如此，通过同时对人脸进行识别，使得人脸在进行模糊化处理保护隐私的基础上可对人脸进行鉴权。或者说，通过增加对人脸进行模糊化的处理再识别人脸进行鉴权，可使得人脸鉴权相关业务可有效地对人脸信息进行隐私保护，提高了人脸鉴权的安全性。
请参阅图4，在某些实施方式中，步骤S11还包括：
S111：将人脸区域进行尺寸变换以将原始的第一尺寸变换为第二尺寸以得到预处理图像。
同时，步骤S50包括：
S51：将模糊图像的尺寸变换为第一尺寸替换预处理图像。
在某些实施方式中，处理器用于将人脸区域进行尺寸变换以将原始的第一尺寸变换为第二尺寸以得到预处理图像，及将模糊图像的尺寸变换为第一尺寸替换预处理图像。
具体地，将人脸提取后的人脸区域进行如线性差值变换的尺寸变换以将原始的第一尺寸变换为第二尺寸以得到预处理图像。当对预处理图像进行模糊化处理后得到模糊图像，需将模糊图像还原为第一尺寸替换预处理图像。
如此，通过尺寸变换可统一各提取的人脸区域尺寸，同时有效地提高了后续的模糊化处理的效率。
在某些实施方式中，噪声图像为随机噪声图像，噪声图像的尺寸为第二尺寸。
请参阅图5，在某些实施方式中，步骤S30包括：
S31：建立第一损失函数
Figure PCTCN2021085651-appb-000001
以确定损失函数，并最小化第一损失函数以得到模糊图像，其中，SSIM为结构相似性，T2为中间迭代图像，T0为预处理图像，E2为第二特征向量，E1为第一特征向量，λ为惩罚项。
在某些实施方式中，处理器用于建立第一损失函数以得到损失函数，并最小化第一损失函数以得到模糊图像。
具体地，建立如下第一损失函数：
Figure PCTCN2021085651-appb-000002
第一损失函数的输入为噪声图像T1及预处理图像T0，输出为对中间结果T2不断迭代产生的最优化结果T，即模糊图像。
其中，
Figure PCTCN2021085651-appb-000003
为L2范数，SSIM(T2,T0)为T2与T0的结构相似性。此处省略结构相似性的计算公式。SSIM(T2,T0)的值越大，说明T2与T0越相似，通过最小化函数使得生成的模糊图像和预处理图像尽可能地差异最大化。同时两个图像的特征向量尽可能地接近。
另外，λ是损失函数的惩罚项，是优化过程中用于控制特征向量近似程度的惩罚因子。λ值越大，对特征向量的近似程度要求越高，根据实际情况进行设置，在计算过程中可进行相应调整。
如此，通过上述第一损失函数确定损失函数，并最小化第一损失函数可得到模糊图像。
请参阅图6，在某些实施方式中，图像处理方法还包括：
S70：建立第二损失函数
Figure PCTCN2021085651-appb-000004
其中，y i为函数标签label，y predict为预测的概率，n为样本数量，标签label为若保护图像与场景图像的欧式距离小于预定阈值，则label＝1；
S80：最小化第一综合损失函数以得到模糊图像，第一综合损失函数包括第二损失函数及第一损失函数。
在某些实施方式中，处理器用于建立第二损失函数，并最小化第一综合损失函数以得到模糊图像，第一综合损失函数包括第二损失函数及第一损失函数。
具体地，建立如下第二损失函数：
Figure PCTCN2021085651-appb-000005
将模糊图像还原到预处理图像中得到保护图像，对保护图像通过人脸检测模型进行检测，如果检测到人脸，且人脸的位置信息与预处理图像的位置信息的欧氏距离小于给定的阈值，则给定标签为label＝1，否则标签为label＝0。例如，保护图像A中的左上(x3,y3)和右下坐标(x,y4)与预处理图像B中的左上(x1,y1)和右下(x2,y2)的欧式距离dist(A,B)小于给定阈值，则给定标签为label＝1，否则标签为label＝0。
进一步地，在第二损失函数中输入保护图像A与预处理图像B，并对Loss 2进行最小化求解，可以判断输入图像是否是人脸。
进一步地，第一综合损失函数包括第二损失函数及第一损失函数。即可定义第一综合损失函数TotalLoss 1：
TotalLoss 1＝Loss 1+Loss 2
通过最小化第一综合损失函数TotalLoss 1得到模糊图像，具体最小化求解过程为数学求解过程不在此处展开。
如此，通过增加对保护图像的人脸检测的损失函数可进一步提高模糊图像的质量，使得替换的模糊 图像可更准确地替换预处理图像。
请参阅图7，在某些实施方式中，步骤S30还包括：
S32：建立第三损失函数Loss3＝min(SSIM(T2,T0)+λcosθ)以确定损失函数，并最小化第三损失函数以得到模糊图像，其中，SSIM为结构相似性，T2为中间迭代图像，T0为所述预处理图像，cosθ为余弦距离。
在某些实施方式中，处理器用于建立第三损失函数以得到损失函数，并最小化第三损失函数以得到模糊图像。
具体地，建立如下第三损失函数：
Loss3＝min(SSIM(T2,T0)+λcosθ)
第三损失函数的输入为噪声图像T1及预处理图像T0，输出为对中间结果T2不断迭代产生的最优化结果T，即模糊图像。
其中，cosθ为余弦距离，SSIM(T2,T0)为T2与T0的结构相似性。此处省略结构相异性的计算公式。SSIM的值越大，说明T2与T0越相似，通过最小化函数使得生成的模糊图像和预处理图像尽可能地差异最大化。同时两个图像的特征向量尽可能地接近。
相较于上述实施例Loss1中的
Figure PCTCN2021085651-appb-000006
本实施例将其欧式距离的计算替换为余弦距离cosθ，即就是计算两个向量间的夹角的余弦值计算：
Figure PCTCN2021085651-appb-000007
另外，λ是惩罚项，是优化过程中用于控制特征向量近似程度的惩罚因子。λ值越大，对特征向量的近似程度要求越高，根据实际情况进行设置，在计算过程中可进行相应调整。
如此，通过上述第三损失函数确定损失函数，并最小化第三损失函数可得到模糊图像。
请参阅图8，在某些实施方式中，图像处理方法还包括：
S90：建立第二损失函数
Figure PCTCN2021085651-appb-000008
其中，y i为函数标签label，y predict为预测的概率，n为样本数量，标签label为若保护图像与场景图像的欧式距离小于预定阈值，则label＝1；
S100：最小化第二综合损失函数以得到模糊图像，第二综合损失函数包括第二损失函数及第三损失函数。
在某些实施方式中，处理器用于最小化第二综合损失函数以得到模糊图像，第二综合损失函数包括第二损失函数及第三损失函数。
具体地，第二综合损失函数包括第二损失函数及第三损失函数。即可定义第二综合损失函数TotalLoss 2：
TotalLoss 2＝Loss 2+Loss 3
最小化第一综合损失函数TotalLoss 2以得到模糊图像，具体最小化求解过程为数学求解过程不在此处展开。
如此，通过增加对保护图像的人脸检测的损失函数可进一步提高模糊图像的质量，使得替换的模糊图像可更准确地替换预处理图像。
请参阅图9，本申请实施方式还提供了一种隐私保护的图像处理装置10，本申请实施方式的图像处理装置10包括获取模块11、生成模块12、处理模块13、模糊模块14和替换模块15。获取模块11用于获取预处理图像以对预处理图像进行特征提取得到第一特征向量。生成模块12用于生成噪声图像以对噪声图像进行特征提取得到第二特征向量，噪声图像与预处理图像对应。处理模块13用于根据预处理图像与噪声图像的结构相似性，第一特征向量及第二特征向量建立损失函数。模糊模块14用于利用损失函数对噪声图像进行处理以得到模糊图像。替换模块15用于利用模糊图像替换预处理图像。
获取模块11获取预处理图像。其中，图像为想要进行隐私保护的图像，可存在于照片中、电视中或计算机屏幕中的图像、或视频中需要隐私保护的对象。同时，对象可包括人脸或其它需要进行隐私保护的区域，其保护范围可为整幅图像或感兴趣区域(Region Of Interest，ROI)。
具体地，获取的预处理图像为所需进行隐私保护的图像进行预处理之后的图像。其中，预处理包括对图像的截取、特征提取、图像为人脸的人脸检测等处理方式。
在某些实施方式中，所需进行隐私保护的图像为人脸，获取经过人脸检测后的人脸区域作为预处理 图像。
在某些实施方式中，图像中的固定区域需要隐私保护如右下角的水印处，则截取图像某一尺寸的右下角区域作为预处理图像。
进一步地，对预处理图像进行特征提取得到第一特征向量。其中，根据实际情况，不同的图像具有不同的特征向量，特征向量为图像技术领域中的通常含义。例如人脸图像的特征向量为人脸关键点的向量，具体包括人脸上具有明确语义的坐标点，如鼻尖、嘴角、眼角等。第一用于区别后续的特征向量。
生成模块12根据预处理图像生成噪声图像以对噪声图像进行特征提取得到第二特征向量。其中，噪声图像可为满足预设均值和预设方差的噪声图像，或随机产生的随机噪声图像。对噪声图像的特征提取方式可与对预处理图像进行特征提取的方式相同，其特征向量也可相同。
进一步地，处理模块13根据预处理图像与噪声图像的结构相似性，第一特征向量及第二特征向量建立损失函数。模糊模块14并利用损失函数对噪声图像进行处理以得到模糊图像。
可以理解的是，损失函数可根据不同的参数建立，本实施例中，通过提取预处理图像与噪声图像的结构相似性及特征向量建立损失函数。其中，结构相似性表征预处理图像及输出图像的相似性，通过对输入输出图像结构相似性的计算可使得结构相似性最小化，从而使得生成的模糊图像和预处理图像尽可能地差异最大化。在某些实施方式中，还可利用结构相异性。
其中，损失函数包括分类损失函数如利用交叉熵损失函数来判定实际的输出与期望的输出的接近程度，和/或质量损失函数判定输出图像与期望的输出的质量差异，和/或距离损失函数来判定特征向量的距离。同时，可利用一个或多个损失函数进行多目标的优化。进一步地，通过优化建立的损失函数如最小化损失函数以使得输出图像满足与预处理图像尽可能地差异最大化，同时特征向量尽可能地接近。
具体地，可根据第一特征向量及第二特征向量的距离损失建立损失函数，如欧式距离。同时可最小化特征向量的距离以使得生成的模糊图像和预处理图像的特征向量尽可能地接近。损失函数中输入噪声图像，并通过对损失函数进行优化使得输出期望的模糊图像，如对建立的损失函数进行最小化求解以优化中间参数，使得输出期望的模糊图像。
在某些实施方式中，可建立交叉熵损失函数以使得输出的模糊图像与预处理图像模糊化程度最大。
在某些实施方式中，可建立交叉熵损失函数以使得输出的模糊图像与预处理图像模糊化程度最大，同时，输出的模糊图像与预处理图像的特征向量最接近。
在某些实施方式中，可建立交叉熵损失函数及距离损失函数以使得输出的模糊图像与预处理图像模糊化程度最大，同时，输出的模糊图像与预处理图像的特征向量最接近。
替换模块15利用模糊图像替换预处理图像。当得到模糊图像后，将模糊图像替换预处理图像，或者说将模糊图像还原到预处理图像中。例如，在实时监控画面中，通过损失函数对人脸进行模糊化处理，然后将模糊化的图像还原到实时监控画面中，使得监控画面中仅能显示模糊化的人脸而非清晰的人脸。
在某些实施方式中，可对模糊化的图像进行尺寸调整等变换后再还原到实时监控画面中。
在某些实施方式中，图像为人脸信息，则还可进一步地对人脸进行人脸识别等后续应用。
如此，上述隐私保护的图像处理装置，通过获取模块11获取预处理图像以对预处理图像进行特征提取得到第一特征向量，生成模块12根据预处理图像生成噪声图像以对噪声图像进行特征提取得到第二特征向量，处理模块13根据预处理图像与噪声图像的结构相似性，第一特征向量及第二特征向量建立损失函数，模糊模块14利用损失函数对噪声图像进行处理以得到模糊图像，及替换模块15利用模糊图像替换预处理图像。可使得对需要进行隐私保护的图像进行模糊化处理，并将模糊图像替换原始图像，使得实时应用类的业务场景可以输出隐私保护区域的模糊画面，从而最小化信息的采集，有效地降低了隐私的泄露。同时，将模糊图像替换原始图像进行存储及传输，在一定程度上避免了泄露或被拷贝的风险。再者，根据结构相似性及特征向量建立损失函数并优化损失函数以生成模糊图像，可以使得对图像模糊化的同时，保障其特征向量尽可能地接近，即就是输出的图像的特征尽可能接近预处理图像。在一定程度上提高了模糊图像的质量。进一步地，后续的人脸识别模型可根据模糊图像进行更为准确的还原，同时有效地提高对人脸进行识别鉴权的精确度。进一步地，通过人脸识别模型可进行人脸识别鉴权等后续业务的应用，使得有效地对图像进行隐私保护的同时可实现识别鉴权等业务。
请参阅图10，在某些实施方式中，获取模块11包括提取单元111。提取单元111用于对包括人脸特征的场景图像进行人脸检测得到人脸区域，并将人脸区域确定为预处理图像。
如此，可通过人脸检测模型对人脸区域进行提取，以使得在人脸信息相关业务应用场景中可对人脸 信息进行模糊化的隐私保护。
请再次参阅图9，本申请实施方式的图像处理装置10还包括识别模块16。识别模块16用于对预处理人脸图像进行人脸识别以对人脸进行鉴权。
如此，通过同时对人脸进行识别，使得人脸在进行模糊化处理保护隐私的基础上可对人脸进行鉴权。或者说，通过增加对人脸进行模糊化的处理再识别人脸进行鉴权，可使得人脸鉴权相关业务可有效地对人脸信息进行隐私保护，提高了人脸鉴权的安全性。
请再次参阅图10，在某些实施方式中，提取单元111还包括尺寸变换子单元1111，替换模块15包括尺寸替换单元151。尺寸变换子单元1112用于将人脸区域进行尺寸变换以将原始的第一尺寸变换为第二尺寸以得到预处理图像。尺寸替换单元151用于将模糊图像的尺寸变换为第一尺寸以替换预处理图像。
如此，通过尺寸变换可统一各提取的人脸区域尺寸，有效地提高了后续的模糊化处理的效率。
在某些实施方式中，噪声图像为随机噪声图像，噪声图像的尺寸为第二尺寸。
请再次参阅图10，在某些实施方式中，处理模块13包括第一损失函数单元131。第一损失函数单元131用于建立优化第一损失函数
Figure PCTCN2021085651-appb-000009
以确定损失函数，并最小化以使得第一损失函数最小化以得到模糊图像，其中，SSIM为结构相似性，T2为中间迭代图像，T0为预处理图像，E2为第二特征向量，E1为第一特征向量，λ为惩罚项。
如此，通过上述第一损失函数确定损失函数，并最小化第一损失函数可得到模糊图像。
请再次参阅图9，某些实施方式中，图像处理装置10还包括第一综合损失函数模块17。第一综合损失函数模块17用于建立第二损失函数
Figure PCTCN2021085651-appb-000010
其中，y i为函数标签label，y predict为预测的概率，n为样本数量，标签label为若保护图像与场景图像的欧式距离小于预定阈值，则label＝1，及最小化第一综合损失函数以得到模糊图像，第一综合损失函数包括第二损失函数及第一损失函数。
如此，通过增加对保护图像的人脸检测的损失函数可进一步提高模糊图像的质量，使得替换的模糊图像可更准确地替换预处理图像。
请参阅图11，在某些实施方式中，处理模块13还包括第三损失函数单元132，第三损失函数单元132用于建立第三损失函数Loss3＝min(SSIM(T2,T0)+λcosθ)以确定损失函数，并最小化第三损失函数以得到模糊图像，其中，SSIM为结构相似性，T2为中间迭代图像，T0为预处理图像，cosθ为余弦距离。
如此，通过上述第三损失函数确定损失函数，并最小化第三损失函数可得到模糊图像。
请再次参阅图12，在某些实施方式中，图像处理装置10还包括第二综合损失函数模块18。第二综合损失函数模块18用于建立第二损失函数
Figure PCTCN2021085651-appb-000011
其中，y i为函数标签label，y predict为预测的概率，n为样本数量，标签label为若保护图像与场景图像的欧式距离小于预定阈值，则label＝1；及最小化第二综合损失函数以得到模糊图像，第二综合损失函数包括第二损失函数及第三损失函数。
如此，通过增加对保护图像的人脸检测的损失函数可进一步提高模糊图像的质量，使得替换的模糊图像可更准确地替换预处理图像。
本申请实施方式还提供了一种计算机可读存储介质。一个或多个存储有计算机程序的非易失性计算机可读存储介质，当计算机程序被一个或多个处理器执行时，实现上述任一实施方式的图像处理方法。
综上所述，本申请实施方式的隐私保护的图像处理方法，图像处理装置、电子设备及存储介质，通过获取预处理图像以对预处理图像进行特征提取得到第一特征向量，根据预处理图像生成噪声图像以对噪声图像进行特征提取得到第二特征向量，根据预处理图像与噪声图像的结构相似性，第一特征向量及第二特征向量建立损失函数，利用损失函数对噪声图像进行处理以得到模糊图像，利用模糊图像替换预处理图像。至少具有以下有益效果：
一、可使得对需要进行隐私保护的图像进行模糊化处理
二、将模糊图像替换原始图像，使得实时应用类的业务场景可以输出隐私保护区域的模糊画面，从而最小化信息的采集，有效地降低了隐私的泄露。
三、将模糊图像替换原始图像进行存储及传输，在一定程度上避免了泄露或被拷贝的风险。
四、进一步地，通过人脸识别模型可进行人脸识别鉴权等后续业务的应用，使得有效地对图像进行 隐私保护的同时可实现识别鉴权等业务。
五、对于人脸信息领域，在模糊处理的基础上同时对人脸进行识别，使得人脸在进行模糊化处理保护隐私的基础上可对人脸进行鉴权。或者说，通过增加对人脸进行模糊化的处理再识别人脸进行鉴权，可使得人脸鉴权相关业务可有效地对人脸信息进行隐私保护，提高了人脸鉴权的安全性。
六、根据结构相异性或结构相似性及特征向量建立损失函数并优化损失函数以生成模糊图像，可以使得对图像模糊化的同时，保障其特征向量尽可能地接近，即就是输出的图像的特征尽可能接近预处理图像，而不至于差异或变形过大，如模糊成非人脸图像。在一定程度上提高了模糊图像的质量。进一步地，后续的人脸识别模型可根据模糊图像进行更为准确的还原，同时有效地提高对人脸进行识别鉴权的精确度。
七、通过多个损失函数进行多目标的优化，可进一步提高模糊图像的质量。
在本说明书的描述中，参考术语“一个实施方式”、“一些实施方式”、“示意性实施方式”、“示例”、“具体示例”或“一些示例”等的描述意指结合所述实施方式或示例描述的具体特征、结构、材料或者特点包含于本申请的至少一个实施方式或示例中。在本说明书中，对上述术语的示意性表述不一定指的是相同的实施方式或示例。而且，描述的具体特征、结构、材料或者特点可以在任何的一个或多个实施方式或示例中以合适的方式结合。此外，在不相互矛盾的情况下，本领域的技术人员可以将本说明书中描述的不同实施例或示例以及不同实施例或示例的特征进行结合和组合。
此外，术语“第一”、“第二”仅用于描述目的，而不能理解为指示或暗示相对重要性或者隐含指明所指示的技术特征的数量。由此，限定有“第一”、“第二”的特征可以明示或者隐含地包括至少一个所述特征。在本申请的描述中，“多个”的含义是至少两个，例如两个，三个，除非另有明确具体的限定。
流程图中或在此以其他方式描述的任何过程或方法描述可以被理解为，表示包括一个或更多个用于实现特定逻辑功能或过程的步骤的可执行指令的代码的模块、片段或部分，并且本申请的优选实施方式的范围包括另外的实现，其中可以不按所示出或讨论的顺序，包括根据所涉及的功能按基本同时的方式或按相反的顺序，来执行功能，这应被本申请的实施例所属技术领域的技术人员所理解。
尽管上面已经示出和描述了本申请的实施方式，可以理解的是，上述实施方式是示例性的，不能理解为对本申请的限制，本领域的普通技术人员在本申请的范围内可以对上述实施方式进行变化、修改、替换和变型。

Claims (20)

  1. 一种隐私保护的图像处理方法，其特征在于，包括：
    获取预处理图像以对所述预处理图像进行特征提取得到第一特征向量；
    根据所述预处理图像生成噪声图像以对所述噪声图像进行特征提取得到第二特征向量；
    根据所述预处理图像与所述噪声图像的结构相似性，所述第一特征向量及所述第二特征向量建立损失函数；
    利用所述损失函数对所述噪声图像进行处理以得到模糊图像；
    利用所述模糊图像替换所述预处理图像。
  2. 根据权利要求1所述的图像处理方法，其特征在于，所述获取预处理图像包括：
    对包括人脸特征的场景图像进行人脸检测得到人脸区域，并将所述人脸区域确定为所述预处理图像。
  3. 根据权利要求2所述的图像处理方法，其特征在于，所述图像处理方法还包括：
    对所述预处理图像进行人脸识别以对人脸进行鉴权。
  4. 根据权利要求2所述的图像处理方法，其特征在于，所述对场景图像进行人脸检测得到人脸区域，并将所述人脸区域确定为所述预处理图像还包括：
    将所述人脸区域进行尺寸变换以将原始的第一尺寸变换为第二尺寸以得到所述预处理图像；
    所述利用所述模糊图像替换所述预处理图像包括：
    将所述模糊图像的尺寸变换为所述第一尺寸以替换所述预处理图像。
  5. 根据权利要求4所述的图像处理方法，其特征在于，所述噪声图像为随机噪声图像，所述噪声图像的尺寸为所述第二尺寸。
  6. 根据权利要求1所述的图像处理方法，其特征在于，所述根据所述预处理图像与所述噪声图像的结构相异性或结构相似性，所述第一特征向量及所述第二特征向量建立所述损失函数包括：
    建立第一损失函数
    Figure PCTCN2021085651-appb-100001
    以确定所述损失函数，并最小化所述第一损失函数以得到所述模糊图像，其中，SSIM为结构相似性，T2为中间迭代图像，T0为所述预处理图像，E2为所述第二特征向量，E1为所述第一特征向量，λ为惩罚项。
  7. 根据权利要求6所述的图像处理方法，其特征在于，所述图像处理方法还包括：
    建立第二损失函数
    Figure PCTCN2021085651-appb-100002
    其中，y i为函数标签label，y predict为预测的概率，n为样本数量，所述标签label为若保护图像与场景图像的欧式距离小于预定阈值，则label＝1；
    最小化第一综合损失函数以得到所述模糊图像，所述第一综合损失函数包括所述第二损失函数及所述第一损失函数。
  8. 根据权利要求1所述的图像处理方法，其特征在于，所述通过优化所述损失函数对所述噪声图像进行处理以得到所述模糊图像还包括：
    建立第三损失函数Loss3＝min(SSIM(T2,T0)+λcosθ)以确定所述损失函数，并最小化所述第三损失函数以得到所述模糊图像，其中，SSIM为结构相似性，T2为中间迭代图像，T0为所述预处理图像，cosθ为余弦距离。
  9. 根据权利要求8所述的图像处理方法，其特征在于，所述图像处理方法还包括：
    建立所述第二损失函数
    Figure PCTCN2021085651-appb-100003
    其中，y i为函数标签label，y predict为预测的概率，n为样本数量，所述标签label为若保护图像与场景图像的欧式距离小于预定阈值，则label＝1；
    最小化第二综合损失函数以得到所述模糊图像，所述第二综合损失函数包括所述第二损失函数及所述第三损失函数。
  10. 一种隐私保护的图像处理装置，其特征在于，包括：
    获取模块，用于获取预处理图像以对所述预处理图像进行特征提取得到第一特征向量；
    生成模块，用于生成噪声图像以对所述噪声图像进行特征提取得到第二特征向量，所述噪声图像与所述预处理图像对应；
    处理模块，用于根据所述预处理图像与所述噪声图像的结构相似性，所述第一特征向量及所述第二特征向量建立所述损失函数；
    模糊模块，用于利用所述损失函数对所述噪声图像进行处理以得到模糊图像；
    替换模块，用于利用所述模糊图像替换所述预处理图像。
  11. 根据权利要求10所述的图像处理装置，其特征在于，所述获取模块包括：
    提取单元，用于对包括人脸特征的场景图像进行人脸检测得到人脸区域，并将所述人脸区域确定为所述预处理图像。
  12. 根据权利要求11所述的图像处理装置，其特征在于，所述图像处理装置还包括：
    识别模块，用于对所述预处理图像进行人脸识别以对人脸进行鉴权。
  13. 根据权利要求11所述的图像处理装置，其特征在于，所述提取单元还包括：
    尺寸变换子单元，用于将所述人脸区域进行尺寸变换以将原始的第一尺寸变换为第二尺寸以得到所述预处理图像；
    所述替换模块包括：
    尺寸替换单元，用于将所述模糊图像的尺寸变换为所述第一尺寸以替换所述预处理图像。
  14. 根据权利要求13所述的图像处理装置，其特征在于，所述噪声图像为随机噪声图像，所述噪声图像的尺寸为所述第二尺寸。
  15. 根据权利要求10所述的图像处理装置，其特征在于，所述处理模块包括：
    第一损失函数单元，用于建立优化第一损失函数
    Figure PCTCN2021085651-appb-100004
    以确定所述损失函数，并最小化以使得所述第一损失函数最小化以得到所述模糊图像，其中，SSIM为结构相似性，T2为中间迭代图像，T0为所述预处理图像，E2为所述第二特征向量，E1为所述第一特征向量，λ为惩罚项。
  16. 根据权利要求15所述的图像处理装置，其特征在于，所述图像处理装置还包括：
    第一综合损失函数模块，用于建立第二损失函数
    Figure PCTCN2021085651-appb-100005
    其中，y i为函数标签label，y predict为预测的概率，n为样本数量，所述标签label为若保护图像与场景图像的欧式距离小于预定阈值，则label＝1；及
    最小化第一综合损失函数以得到所述模糊图像，所述第一综合损失函数包括所述第二损失函数及所述第一损失函数。
  17. 根据权利要求10所述的图像处理装置，其特征在于，所述处理模块还包括：
    第三损失函数单元，用于建立第三损失函数Loss3＝min(SSIM(T2,T0)+λcosθ)以确定所述损失函数，并最小化所述第三损失函数以得到所述模糊图像，其中，SSIM为结构相似性，T2为中间迭代图像，T0为所述预处理图像，cosθ为余弦距离。
  18. 根据权利要求17所述的图像处理装置，其特征在于，所述图像处理装置还包括：
    第二综合损失函数模块，用于建立第二损失函数
    Figure PCTCN2021085651-appb-100006
    其中，y i为函数标签label，y predict为预测的概率，n为样本数量，所述标签label为若保护图像与场景图像的欧式距离小于预定阈值，则label＝1；及
    最小化第二综合损失函数以得到所述模糊图像，所述第二综合损失函数包括所述第二损失函数及所述第三损失函数。
  19. 一种电子设备，其特征在于，所述电子设备包括存储器和处理器，所述存储器中存储有计算机程序，所述计算机程序被所述处理器执行时，实现权利要求1-9任一项所述的图像处理方法。
  20. 一种计算机程序的非易失性计算机可读存储介质，其特征在于，当所述计算机程序被一个或多个处理器执行时，实现权利要求1-9任一项所述的图像处理方法。
PCT/CN2021/085651 2021-04-06 2021-04-06 图像处理方法及装置、电子设备及存储介质 WO2022213261A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN202180096597.2A CN117121048A (zh) 2021-04-06 2021-04-06 图像处理方法及装置、电子设备及存储介质
PCT/CN2021/085651 WO2022213261A1 (zh) 2021-04-06 2021-04-06 图像处理方法及装置、电子设备及存储介质

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2021/085651 WO2022213261A1 (zh) 2021-04-06 2021-04-06 图像处理方法及装置、电子设备及存储介质

Publications (1)

Publication Number Publication Date
WO2022213261A1 true WO2022213261A1 (zh) 2022-10-13

Family

ID=83545897

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/085651 WO2022213261A1 (zh) 2021-04-06 2021-04-06 图像处理方法及装置、电子设备及存储介质

Country Status (2)

Country Link
CN (1) CN117121048A (zh)
WO (1) WO2022213261A1 (zh)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107886064A (zh) * 2017-11-06 2018-04-06 安徽大学 一种基于卷积神经网络的人脸识别场景适应的方法
CN110175961A (zh) * 2019-05-22 2019-08-27 艾特城信息科技有限公司 一种基于人脸图像分割对抗思想的去网纹方法
CN111160269A (zh) * 2019-12-30 2020-05-15 广东工业大学 一种人脸关键点检测方法及装置
CN112287870A (zh) * 2020-11-11 2021-01-29 合肥的卢深视科技有限公司 一种人脸识别方法、装置及电子设备

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107886064A (zh) * 2017-11-06 2018-04-06 安徽大学 一种基于卷积神经网络的人脸识别场景适应的方法
CN110175961A (zh) * 2019-05-22 2019-08-27 艾特城信息科技有限公司 一种基于人脸图像分割对抗思想的去网纹方法
CN111160269A (zh) * 2019-12-30 2020-05-15 广东工业大学 一种人脸关键点检测方法及装置
CN112287870A (zh) * 2020-11-11 2021-01-29 合肥的卢深视科技有限公司 一种人脸识别方法、装置及电子设备

Also Published As

Publication number Publication date
CN117121048A (zh) 2023-11-24

Similar Documents

Publication Publication Date Title
US11288504B2 (en) Iris liveness detection for mobile devices
US11449971B2 (en) Method and apparatus with image fusion
Joseph et al. Retracted article: a multimodal biometric authentication scheme based on feature fusion for improving security in cloud environment
Peng et al. Face presentation attack detection using guided scale texture
Chakraborty et al. An overview of face liveness detection
US11790494B2 (en) Facial verification method and apparatus based on three-dimensional (3D) image
KR102415509B1 (ko) 얼굴 인증 방법 및 장치
WO2015149534A1 (zh) 基于Gabor二值模式的人脸识别方法及装置
CN111144366A (zh) 一种基于联合人脸质量评估的陌生人脸聚类方法
Rostami et al. Detection and continual learning of novel face presentation attacks
Sun et al. A face spoofing detection method based on domain adaptation and lossless size adaptation
Deep Kaur et al. An analysis of image forgery detection techniques
Yin et al. A face anti-spoofing method based on optical flow field
Galdi et al. Exploring new authentication protocols for sensitive data protection on smartphones
Hao et al. Robustness analysis of face obscuration
Nguyen et al. Face presentation attack detection based on a statistical model of image noise
Beham et al. Anti-spoofing enabled face recognition based on aggregated local weighted gradient orientation
WO2022213261A1 (zh) 图像处理方法及装置、电子设备及存储介质
Teja Real-time live face detection using face template matching and DCT energy analysis
Abdelouahed et al. A comparative study of anti-spoofing detection based on deep learning
Kisku et al. Multithread face recognition in cloud
Abdullah et al. 2.5 d facial analysis via bio-inspired active appearance model and support vector machine for forensic application
Patel et al. Review of digital image forgery detection
Diviya et al. A novel approach for detecting facial image spoofing using local ternary pattern
Lin et al. Liveness detection using texture and 3d structure analysis

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21935498

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21935498

Country of ref document: EP

Kind code of ref document: A1