WO2022194079A1 - Procédé et appareil de segmentation de région du ciel, dispositif informatique et support de stockage - Google Patents

Procédé et appareil de segmentation de région du ciel, dispositif informatique et support de stockage Download PDF

Info

Publication number
WO2022194079A1
WO2022194079A1 PCT/CN2022/080598 CN2022080598W WO2022194079A1 WO 2022194079 A1 WO2022194079 A1 WO 2022194079A1 CN 2022080598 W CN2022080598 W CN 2022080598W WO 2022194079 A1 WO2022194079 A1 WO 2022194079A1
Authority
WO
WIPO (PCT)
Prior art keywords
sky
panoramic
images
frames
image
Prior art date
Application number
PCT/CN2022/080598
Other languages
English (en)
Chinese (zh)
Inventor
贾配洋
林晓帆
Original Assignee
影石创新科技股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 影石创新科技股份有限公司 filed Critical 影石创新科技股份有限公司
Publication of WO2022194079A1 publication Critical patent/WO2022194079A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/136—Segmentation; Edge detection involving thresholding
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/60—Analysis of geometric attributes
    • G06T7/62—Analysis of geometric attributes of area, perimeter, diameter or volume
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning

Definitions

  • the present application relates to the technical field of image processing, and in particular, to a sky area segmentation method, apparatus, computer equipment and storage medium.
  • sky segmentation technology which can distinguish sky pixels and non-sky pixels in an image, so as to realize the special effect of sky replacement for the image.
  • the camera or mobile phone when performing sky segmentation to achieve sky replacement, can send the plane image taken to the cloud, and the cloud uses the pre-stored sky segmentation algorithm to perform sky segmentation on the plane image, and returns the segmented result. to the camera or mobile phone; the camera or mobile phone can also perform sky segmentation on the plane image through the locally stored sky segmentation algorithm.
  • the processing speed of the sky segmentation algorithm in the prior art is relatively slow, and the processing accuracy of edge segmentation is low; and the current sky segmentation algorithm processes only plane images, and the sky segmentation processing effect for panoramic images or panoramic videos is relatively low. Difference.
  • a method for segmenting a sky area comprising:
  • the proportion of sky elements in the target panoramic video determine whether to perform sky segmentation processing on the target panoramic video
  • sky segmentation processing is performed on the panoramic image corresponding to the mask image among the M frames of panoramic images.
  • judging whether to perform sky segmentation processing on the target panoramic video including:
  • determining whether to perform sky segmentation processing on the target panoramic video according to the proportion of sky pixels in the N mask images includes:
  • judging whether to perform sky segmentation processing on the target panoramic video including:
  • the N frames of panoramic images are identified by the second preset model, and N output results are obtained;
  • the input of the second preset model is a panoramic image, and the output is that the panoramic image contains the sky or the panoramic image does not contain the sky;
  • judging whether to perform sky segmentation processing on the target panoramic video including:
  • the proportion of the panoramic image including the sky and the second preset threshold it is determined whether to perform sky segmentation processing on the target panoramic video.
  • the training sample set of the preset model includes multiple frames of panoramic images and a reference image corresponding to each frame of panoramic images in the multiple frames of panoramic images, where the reference image is used to indicate the panoramic image corresponding to the reference image sky area.
  • the training process of the preset model includes:
  • the panoramic images in the training sample set are input into a neural network model, and the parameters of the neural network model are adjusted according to the output of the neural network model and the loss value of the reference image to obtain the preset model.
  • the training process of the preset model further includes:
  • the neural network model is processed with a model compression algorithm to update the neural network model.
  • a device for segmenting a sky area comprising:
  • the judgment module is used to judge whether to perform sky segmentation processing on the target panoramic video according to the proportion of sky elements in the target panoramic video;
  • the acquisition module is used to extract M frames of panoramic images of the target panoramic video when it is judged to perform sky segmentation processing on the target panoramic video, input each frame of panoramic images in the M frames of panoramic images into a preset model, and obtain The mask image of each frame of panoramic image in the M frames of panoramic images; the mask image includes sky area and non-sky area;
  • a segmentation module configured to perform sky segmentation processing on the panoramic image corresponding to the mask image in the M frames of panoramic images according to the mask image.
  • a computer device comprising a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:
  • the proportion of sky elements in the target panoramic video determine whether to perform sky segmentation processing on the target panoramic video
  • Extracting M frames of panoramic images of the target panoramic video inputting each frame of panoramic images in the M frames of panoramic images into a preset model, and obtaining a mask image of each frame of panoramic images in the M frames of panoramic images; the mask image including sky areas and non-sky areas;
  • sky segmentation processing is performed on the panoramic image corresponding to the mask image among the M frames of panoramic images.
  • a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:
  • the proportion of sky elements in the target panoramic video determine whether to perform sky segmentation processing on the target panoramic video
  • Extracting M frames of panoramic images of the target panoramic video inputting each frame of panoramic images in the M frames of panoramic images into a preset model, and obtaining a mask image of each frame of panoramic images in the M frames of panoramic images; the mask image including sky areas and non-sky areas;
  • sky segmentation processing is performed on the panoramic image corresponding to the mask image among the M frames of panoramic images.
  • the computer equipment determines whether to perform sky segmentation processing on the target panoramic video according to the proportion of sky elements in the target panoramic video; after determining whether to perform sky segmentation processing on the target panoramic video Under the situation, extract the M frames of panoramic images of this target panoramic video, input each frame of panoramic images in this M frames of panoramic images into a preset model, and obtain the mask image of each frame of panoramic images in this M frames of panoramic images; Further, sky segmentation processing may be performed on the panoramic image corresponding to the mask image among the M frames of panoramic images according to the mask image, wherein the mask image includes a sky area and a non-sky area.
  • the target panoramic video before the sky segmentation processing of the target panoramic video is performed, the target panoramic video is first judged to determine whether the target panoramic video is suitable for sky segmentation, and when the target panoramic video is suitable for sky segmentation, the The preset model performs sky segmentation processing on the target panoramic video; this can avoid the problem of poor sky segmentation effect caused by still performing sky segmentation when the panoramic video contains less sky areas in the prior art.
  • the scenes with less sky area that are not suitable for sky segmentation can be effectively filtered, thereby improving the sky segmentation effect; further, each frame of panoramic image can be accurately obtained through the preset model in this embodiment.
  • Respectively corresponding mask images improve the edge processing accuracy of panoramic images, and improve the processing speed of sky segmentation, realize accurate recognition of panoramic images and panoramic videos, and greatly improve the accuracy of panoramic image and panoramic video recognition. Improve the segmentation effect of panoramic images and panoramic videos. Therefore, by using the sky region segmentation method in this embodiment, the sky segmentation effect on panoramic images and panoramic videos can be greatly improved.
  • Fig. 1 is the application environment diagram of the sky area segmentation method in one embodiment
  • FIG. 2 is a schematic flowchart of a method for segmenting a sky region in one embodiment
  • FIG. 3 is a schematic flowchart of a method for segmenting a sky region in another embodiment
  • FIG. 4 is a schematic flowchart of a method for segmenting a sky region in another embodiment
  • FIG. 5 is a structural block diagram of an apparatus for dividing a sky area in one embodiment
  • FIG. 6 is a diagram of the internal structure of a computer device in one embodiment.
  • the sky area segmentation method provided in this application can be applied to the computer equipment as shown in FIG. 1 .
  • the computer equipment can be, but is not limited to, any type of terminal capable of image processing or video processing, such as: ordinary cameras, panoramic cameras , smart phones, personal computers, notebook computers, tablet computers, and VR eyes, etc.; the internal structure of the computer equipment is shown in Figure 1, including a processor, memory, communication interface, display screen and input device connected through the system bus .
  • a method for segmenting a sky area is provided, and the method is applied to the computer device in FIG. 1 as an example for description, including the following steps:
  • Step 201 according to the proportion of sky elements in the target panoramic video, determine whether to perform sky segmentation processing on the target panoramic video.
  • the target panoramic video is a panoramic video to be processed that needs to be divided into the sky.
  • the computer equipment When the computer equipment performs sky segmentation on the target panoramic video, it needs to first determine whether the target panoramic video is suitable for sky segmentation, that is to say, only when the target panoramic video is suitable for sky segmentation, the target panoramic video is processed.
  • Sky segmentation operation optionally, it can be determined whether to perform sky segmentation processing on the target panoramic video according to the proportion of sky elements in the target panoramic video; wherein, the target panoramic video may include multiple frames of panoramic images, each frame of panoramic video
  • the image can include sky elements and non-sky elements; optionally, the color panoramic image can be converted into a grayscale image through operations such as preprocessing, normalization, subtraction of mean difference, and division of variance, so as to reduce the amount of data processing.
  • image processing technology can be used to mark the sky area and non-sky area of each frame of panoramic image, and then the target panoramic video can be determined.
  • the total area of the sky area that is, the sum of the area of the sky area of each frame of panoramic image
  • the total area of the target panoramic video that is, the sum of the area of each frame of panoramic image; then, according to the sky of the target panoramic video
  • the proportion of sky elements in the target panoramic video can be obtained by dividing the total area of the area by the total area of the target panoramic video.
  • the proportion of sky elements in the target panoramic video and the size of the preset threshold can be used. relationship to determine whether to perform sky segmentation processing on the target panoramic video; when the proportion of sky elements in the target panoramic video is greater than the preset threshold, it can be determined to perform sky segmentation processing on the target panoramic video; in the target panoramic video When the proportion of the sky elements is not greater than the preset threshold, the sky segmentation process is not performed on the target panoramic video; Split processing.
  • Step 202 if yes, then extract M frames of panoramic images of the target panoramic video, input each frame of panoramic images in the M frames of panoramic images into a preset model, and obtain the mask of each frame of panoramic images in the M frames of panoramic images image; the mask image includes sky and non-sky areas.
  • the M frames of panoramic images of the target panoramic video are all corresponding panoramic images after converting the target panoramic video into video frame images according to a preset video frame conversion ratio, that is, M is the total number of frames of panoramic images.
  • the preset model includes a panoramic sky segmentation algorithm for accurately identifying panoramic images.
  • the input is a panoramic image
  • the output is a mask image corresponding to the panoramic image.
  • the mask image includes sky areas and non-sky areas. area.
  • the computer device can extract the target panoramic video when it determines that the target panoramic video is subjected to sky segmentation processing, that is, the target panoramic video has a large proportion of sky elements and is suitable for sky segmentation processing of the target panoramic video.
  • M frames of panoramic images of the video and input each frame of the panoramic images of the M frames of panoramic images into the preset model to obtain the mask images corresponding to each frame of the panoramic images in the M frames of panoramic images, that is, Get M mask images.
  • Step 203 Perform sky segmentation processing on the panoramic image corresponding to the mask image among the M frames of panoramic images according to the mask image.
  • the frame of the panoramic image corresponding to the mask image can be subjected to sky segmentation processing according to the mask image, that is, to remove the frame of the panoramic image and the mask image.
  • the non-sky area image obtained by dividing the sky area can be obtained.
  • the computer device determines whether to perform sky segmentation processing on the target panoramic video according to the proportion of sky elements in the target panoramic video; if it is determined to perform sky segmentation processing on the target panoramic video, extract the target panoramic video M frames of panoramic images, input each frame of panoramic images in the M frames of panoramic images into a preset model, and obtain a mask image of each frame of panoramic images in the M frames of panoramic images; The panoramic image corresponding to the mask image in the M frames of panoramic images is subjected to sky segmentation processing; wherein, the mask image includes a sky area and a non-sky area.
  • the target panoramic video before the sky segmentation processing of the target panoramic video is performed, the target panoramic video is first judged to determine whether the target panoramic video is suitable for sky segmentation, and when the target panoramic video is suitable for sky segmentation, the The preset model performs sky segmentation processing on the target panoramic video; this can avoid the problem of poor sky segmentation effect caused by still performing sky segmentation when the panoramic video contains less sky areas in the prior art.
  • the scenes with less sky area that are not suitable for sky segmentation can be effectively filtered, thereby improving the sky segmentation effect; further, each frame of panoramic image can be accurately obtained through the preset model in this embodiment.
  • Respectively corresponding mask images improve the edge processing accuracy of panoramic images, and improve the processing speed of sky segmentation, realize accurate recognition of panoramic images and panoramic videos, and greatly improve the accuracy of panoramic image and panoramic video recognition. Improve the segmentation effect of panoramic images and panoramic videos. Therefore, by using the sky region segmentation method in this embodiment, the sky segmentation effect on panoramic images and panoramic videos can be greatly improved.
  • FIG. 3 is a schematic flowchart of a method for segmenting a sky region in another embodiment.
  • This embodiment relates to an optional implementation process of judging whether to perform sky segmentation processing on the target panoramic video according to the proportion of sky elements in the target panoramic video; on the basis of the above embodiment, as shown in FIG. 3 , the above step 201 includes:
  • Step 301 Determine N frames of panoramic images from the M frames of panoramic images.
  • a partial panoramic image may be selected from all frames of panoramic images corresponding to the target panoramic video, and according to the partial panoramic video
  • the proportion of sky elements in the image is used to determine whether to perform sky segmentation processing on the target panoramic video, so as to reduce the amount of data processing of computer equipment and the amount of calculation of memory. That is, N frames of panoramic images can be determined from the M frames of panoramic images; alternatively, continuous N frames of panoramic images can be randomly selected from the M frames of panoramic images, or different frames of panoramic images can be randomly selected from the M frames of panoramic images. Consecutive N frames of panoramic images. The manner of acquiring N frames of panoramic images is not limited in this embodiment.
  • Step 302 Input the N frames of panoramic images into the preset model to obtain N mask images corresponding to the N frames of panoramic images.
  • Step 303 according to the proportion of sky pixels in the N mask images, determine whether to perform sky segmentation processing on the target panoramic video.
  • the mask image includes a sky area and a non-sky area, that is, the sky area and the non-sky area Pixels are marked by binary values of 0 and 1, or 0 and 255; further, it can be judged whether to perform sky segmentation processing on the target panoramic video according to the proportion of sky pixels in the N mask images; optional
  • the proportion of the sky pixels can be determined according to the number of sky pixels in the N mask images and the number of non-sky pixels in the N mask images; and according to the proportion of the sky pixels and the first preset The size relationship of the threshold value determines whether to perform sky segmentation processing on the target panoramic video.
  • the number of sky pixels in the N mask images can be expressed as sky_pixels
  • the number of non-sky pixels in the N mask images can be expressed as background_pixels
  • the proportion sky_ratio is greater than the first preset threshold, it can be judged that the target panoramic video is subjected to sky segmentation processing; when the proportion sky_ratio of the sky pixels is not greater than the first preset threshold, it can be judged that the target panoramic video is not to be processed. Perform sky segmentation processing.
  • the computer device determines N frames of panoramic images from the M frames of panoramic images, and inputs the N frames of panoramic images into the preset model to obtain N mask images corresponding to the N frames of panoramic images; then, according to The proportion of sky pixels in the N mask images determines whether to perform sky segmentation processing on the target panoramic video; that is, in this embodiment, the computer device selects a part of the panoramic image and divides the sky pixels of the part of the panoramic image.
  • the proportion of the target panoramic video is used as the judgment basis for the target panoramic video to determine whether to perform sky segmentation processing on the target panoramic video, which can reduce the data processing volume of the computer equipment; in addition, by inputting the selected N panoramic images into the above-mentioned preset model.
  • the preset model can accurately identify the sky area and non-sky area in the panoramic image, therefore, according to the proportion of sky pixels obtained from the N mask images, and then according to the sky area
  • the proportion of pixels determines whether to perform sky segmentation processing on the target panoramic video, which greatly improves the accuracy of judging the target panoramic video.
  • FIG. 4 is a schematic flowchart of a method for segmenting a sky region in another embodiment.
  • This embodiment relates to another optional implementation process of judging whether to perform sky segmentation processing on the target panoramic video according to the proportion of sky elements in the target panoramic video; on the basis of the above embodiment, as shown in FIG. 4
  • the above step 201 includes:
  • Step 401 Determine N frames of panoramic images from the M frames of target panoramic images.
  • step 301 details are not repeated here.
  • Step 402 using the second preset model to identify the N frames of panoramic images to obtain N output results.
  • the input of the second preset model is a panoramic image
  • the output is that the panoramic image contains the sky or the panoramic image does not contain the sky.
  • a positive sample image containing a sky area and a negative sample image not containing a sky area can be input into any traditional classification algorithm model, and the sky classification algorithm model can be obtained by training, and the sky classification algorithm model can be used as the second classification algorithm model.
  • any traditional classification algorithm model can be based on a support vector machine (Support Vector Machine) Machine, referred to as SVM), adjacent algorithm (K-NearestNeighbor, referred to as KNN), and deep learning convolutional neural network (Convolutional Neural Networks, CNN for short) and other classification algorithms; among them, deep learning CNN algorithms may include: Residual Neural Networks (Residual Neural Networks) Network, referred to as ResNet), high-resolution network (High-Resoultion Net, referred to as HRNet), and high-efficiency models for mobile and embedded vision applications (referred to as MobileNet), etc.; this embodiment uses which traditional classification algorithm model training The obtained second preset model is not limited.
  • SVM Support Vector Machine
  • KNN K-NearestNeighbor
  • CNN deep learning convolutional neural network
  • ResNet Residual Neural Networks
  • HRNet high-resolution network
  • MobileNet high-efficiency models for mobile and embedded vision applications
  • the N frames of panoramic images may be respectively input into the second preset model, and the N frames of panoramic images may be input by using the second preset model.
  • Each frame of panoramic image is identified separately, and the corresponding output results of each frame of panoramic image are obtained, that is, N output results are obtained; wherein, each output result indicates whether the corresponding frame of panoramic image is a panoramic image containing the sky, or does not contain the sky. panoramic image.
  • Step 403 according to the proportion of panoramic images including the sky in the N output results, determine whether to perform sky segmentation processing on the target panoramic video.
  • N output results after N output results are obtained, it may be determined whether to perform sky segmentation on the target panoramic video according to the proportion of panoramic images containing the sky in the N output results processing; optionally, according to the number of panoramic images that contain the sky in the N output results and the number of panoramic images that do not contain the sky in the N output results, determine the proportion of the panoramic images that contain the sky; and then according to the Whether to perform sky segmentation processing on the target panoramic video is determined according to the relationship between the proportion of panoramic images including the sky and the second preset threshold.
  • the number of panoramic images containing the sky in the N output results can be expressed as sky_nums
  • the number of panoramic images that do not contain the sky in the N output results can be expressed as background_nums
  • the computer device determines N frames of panoramic images from the M frames of target panoramic images, and uses the second preset model to identify the N frames of panoramic images to obtain N output results, and then, according to the N outputs
  • the proportion of the panoramic images containing the sky in the result determines whether to perform sky segmentation processing on the target panoramic video; that is, in this embodiment, the computer equipment selects a part of the panoramic image and divides the panoramic image of the part of the panoramic image including the sky.
  • the proportion of images is used as the basis for judging the target panoramic video to determine whether to perform sky segmentation processing on the target panoramic video, which can reduce the data processing volume of computer equipment; in addition, pre-trained can accurately identify whether the panoramic image contains sky.
  • the second preset model is to identify the part of the panoramic images respectively to obtain the corresponding output results, and then determine the proportion of the panoramic images containing the sky according to the output results and determine whether to perform sky segmentation processing on the target panoramic video, which can improve the The accuracy of the target panoramic video judgment.
  • the training sample set of the preset model may include multiple frames of panoramic images and a reference image corresponding to each frame of panoramic images in the multiple frames of panoramic images, where the reference image is used to indicate the The sky area of the panoramic image corresponding to the reference image, that is to say, the sky area and the non-sky area in the panoramic image corresponding to the reference image can be distinguished according to the reference image; in this embodiment, the panoramic image can be in any form.
  • the sky area is marked to obtain a reference image that can distinguish the sky area in the panoramic image.
  • the reference image can be a mask image corresponding to each frame of panoramic image, and the mask image can be marked by relevant software, for example: a frame of panoramic image can be marked by PS software to distinguish the sky.
  • the multi-frame panoramic images may include multi-frame panoramic images corresponding to different scenes, may also include panoramic images corresponding to different resolutions, and may also include any panoramic image with distortion or an indistinct boundary between sky and non-sky etc.; this embodiment does not limit it.
  • the training process of the preset model includes: inputting the panoramic images in the training sample set into a neural network model, and adjusting the neural network model according to the output of the neural network model and the loss value of the reference image parameters of the neural network model to obtain the preset model.
  • a partial panoramic image in the training sample set and a reference image corresponding to each panoramic image in the partial panoramic image can be input into a neural network model, and model training can be performed to obtain a panoramic sky segmentation algorithm; Input another part of the panoramic image in the training sample set into the trained neural network model, test the panoramic sky segmentation algorithm, and obtain the output of the neural network model, that is, obtain the mask image corresponding to each panoramic image.
  • the neural network model of the accurate panoramic sky segmentation algorithm is used as the preset model; wherein, the loss value of the reference image is the loss between the output of the neural network model, that is, the mask image, and the reference image, and the loss value is The larger the value is, the larger the error between the mask image and the reference image is; the neural network model is further optimized according to the loss value until the loss value reaches the minimum, that is, the neural network model The output mask image has the smallest error with the corresponding reference image, which makes the output of the neural network model more accurate.
  • the neural network model can use a fully convolutional network (Fully Convolutional Networks (FCN for short), Deep Convolutional Network (Deeplab), Pyramid Scene Parsing Network (PSPNet), High-Resoultion Net (HRNet) and other deep learning-based convolutional neural networks Network model; this is not limited in this embodiment of the present application.
  • FCN Fully Convolutional Networks
  • Deeplab Deep Convolutional Network
  • PSPNet Pyramid Scene Parsing Network
  • HRNet High-Resoultion Net
  • other deep learning-based convolutional neural networks Network model this is not limited in this embodiment of the present application.
  • the performance evaluation of the preset model can be performed, the accuracy of the preset model can be calculated, and the actual measurement of the preset model can be performed.
  • the processing speed on the computer equipment optionally, any number of panoramic images can be input into the preset model, respectively, the mask images corresponding to the multiple panoramic images are output, and the mask images are calculated according to the multiple mask images.
  • the accuracy of the preset model, the accuracy of the preset model may include calculating the average intersection ratio of the preset model (mean Intersection over Union, referred to as mIoU), and calculate the pixel accuracy of the preset model (Accuracy, referred to as acc), etc.; and calculate the average time-consuming of the preset model to process a panoramic image, that is, the processing speed of the model.
  • mIoU mean Intersection over Union
  • acc pixel accuracy of the preset model
  • a model compression algorithm may also be used to process the neural network model to update the neural network model.
  • model compression can be performed on the neural network model through a model compression algorithm to reduce the data processing volume of the model; and then according to the neural network model after model compression.
  • the training of the panoramic sky segmentation algorithm model is carried out to improve the processing speed of the computer equipment to execute the panoramic sky segmentation algorithm and achieve the effect of real-time processing.
  • the model compression algorithm may include model pruning, reducing the number of channels, reducing the number of network layers, reducing the input resolution, knowledge distillation, ablation experiments, and replacing the neural network model with more lightweight feature extraction.
  • the processing speed of the model is improved to meet the requirements of real-time processing.
  • the model compression algorithm is not specifically limited in this embodiment, as long as the processing speed of the neural network model can be improved under the condition that the accuracy of the neural network model is not significantly reduced.
  • steps in the flowcharts of FIGS. 2-4 are shown in sequence according to the arrows, these steps are not necessarily executed in the sequence shown by the arrows. Unless explicitly stated herein, the execution of these steps is not strictly limited to the order, and these steps may be performed in other orders. Moreover, at least a part of the steps in FIGS. 2-4 may include multiple steps or multiple stages. These steps or stages are not necessarily executed and completed at the same time, but may be executed at different times. The execution of these steps or stages The order is also not necessarily sequential, but may be performed alternately or alternately with other steps or at least a portion of the steps or phases within the other steps.
  • a sky area segmentation device including: a judgment module 501, an acquisition module 502 and a segmentation module 503, wherein:
  • the judgment module 501 is configured to judge whether to perform sky segmentation processing on the target panoramic video according to the proportion of sky elements in the target panoramic video.
  • the acquisition module 502 is configured to extract M frames of panoramic images of the target panoramic video when it is judged to perform sky segmentation processing on the target panoramic video, and input each frame of panoramic images in the M frames of panoramic images into a preset model, A mask image of each frame of panoramic images in the M frames of panoramic images is obtained; the mask image includes a sky area and a non-sky area.
  • the segmentation module 503 is configured to perform sky segmentation processing on the panoramic image corresponding to the mask image in the M frames of panoramic images according to the mask image.
  • the above judgment module 501 is specifically configured to determine N frames of panoramic images from the M frames of panoramic images before determining to perform sky segmentation processing on the target panoramic video according to the proportion of sky elements in the target panoramic video image; input the N frames of panoramic images into the preset model, and obtain N mask images corresponding to the N frames of panoramic images; according to the proportion of sky pixels in the N mask images, determine whether to perform Sky segmentation processing.
  • the above judgment module 501 is specifically configured to determine the proportion of the sky pixels according to the number of sky pixels in the N mask images and the number of non-sky pixels in the N mask images; The relationship between the proportion of the sky pixels and the first preset threshold determines whether to perform sky segmentation processing on the target panoramic video.
  • the above judgment module 501 is further configured to determine N frames from the M frames of target panoramic images before determining to perform sky segmentation processing on the target panoramic video according to the proportion of sky elements in the target panoramic video Panoramic image; use the second preset model to identify the N frames of panoramic images, and obtain N output results; the input of the second preset model is a panoramic image, and the output is that the panoramic image contains the sky or the panoramic image does not contain the sky ; According to the proportion of panoramic images containing the sky in the N output results, determine whether to perform sky segmentation processing on the target panoramic video.
  • the above judgment module 501 is specifically configured to determine the number of panoramic images containing the sky according to the number of panoramic images containing the sky in the N output results and the number of panoramic images that do not contain the sky in the N output results.
  • the proportion of the panoramic image according to the relationship between the proportion of the panoramic image including the sky and the second preset threshold, it is determined whether to perform sky segmentation processing on the target panoramic video.
  • the training sample set of the preset model includes multiple frames of panoramic images and a reference image corresponding to each frame of panoramic images in the multiple frames of panoramic images, where the reference image is used to indicate the panoramic image corresponding to the reference image sky area.
  • the apparatus further includes: a model training module; the model training module is used for inputting the panoramic images in the training sample set into a neural network model, and according to the output of the neural network model and the loss of the reference image value to adjust the parameters of the neural network model to obtain the preset model.
  • the model training module is further configured to process the neural network model by using a model compression algorithm to update the neural network model.
  • sky area segmentation device For the specific limitation of the sky area segmentation device, reference may be made to the definition of the sky area segmentation method above, which will not be repeated here. All or part of the modules in the above-mentioned sky area segmentation device can be implemented by software, hardware and combinations thereof.
  • the above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
  • a computer device is provided, and the computer device may be the above-mentioned terminal, and its internal structure diagram may be as shown in FIG. 6 .
  • the computer equipment includes a processor, memory, a communication interface, a display screen, and an input device connected by a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities.
  • the memory of the computer device includes a non-volatile storage medium, an internal memory.
  • the nonvolatile storage medium stores an operating system and a computer program.
  • the internal memory provides an environment for the execution of the operating system and computer programs in the non-volatile storage medium.
  • the communication interface of the computer device is used for wired or wireless communication with an external terminal, and the wireless communication can be realized by WIFI, operator network, NFC (Near Field Communication) or other technologies.
  • the computer program when executed by a processor, implements a sky region segmentation method.
  • the display screen of the computer equipment may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment may be a touch layer covered on the display screen, or a button, a trackball or a touchpad set on the shell of the computer equipment , or an external keyboard, trackpad, or mouse.
  • FIG. 6 is only a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer equipment to which the solution of the present application is applied. Include more or fewer components than shown in the figures, or combine certain components, or have a different arrangement of components.
  • a computer device including a memory and a processor, a computer program is stored in the memory, and the processor implements the following steps when executing the computer program:
  • the proportion of sky elements in the target panoramic video determine whether to perform sky segmentation processing on the target panoramic video
  • sky segmentation processing is performed on the panoramic image corresponding to the mask image among the M frames of panoramic images.
  • the processor further implements the following steps when executing the computer program: determining N frames of panoramic images from the M frames of panoramic images; inputting the N frames of panoramic images into the preset model to obtain the corresponding N frames of panoramic images. N mask images; according to the proportion of sky pixels in the N mask images, determine whether to perform sky segmentation processing on the target panoramic video.
  • the processor further implements the following steps when executing the computer program: determining the proportion of the sky pixels according to the number of sky pixels in the N mask images and the number of non-sky pixels in the N mask images ; According to the relationship between the proportion of the sky pixels and the first preset threshold, it is judged whether to perform sky segmentation processing on the target panoramic video.
  • the processor further implements the following steps when executing the computer program: determining N frames of panoramic images from the M frames of target panoramic images; identifying the N frames of panoramic images by using a second preset model to obtain N outputs Result; the input of the second preset model is a panoramic image, and the output is that the panoramic image contains the sky or the panoramic image does not contain the sky; according to the proportion of the panoramic images containing the sky in the N output results, it is judged whether the target The panoramic video is processed by sky segmentation.
  • the following steps are further implemented: according to the number of panoramic images including the sky in the N output results and the number of panoramic images that do not include the sky in the N output results, determine the number of the panoramic images including the sky.
  • the proportion of the panoramic image of the sky according to the relationship between the proportion of the panoramic image containing the sky and the second preset threshold, it is determined whether to perform sky segmentation processing on the target panoramic video.
  • the processor further implements the following steps when executing the computer program: the training sample set of the preset model includes multiple frames of panoramic images and a reference image corresponding to each frame of panoramic images in the multiple frames of panoramic images, the reference image It is used to indicate the sky area of the panoramic image corresponding to the reference image.
  • the processor executes the computer program, the following steps are further implemented: input the panoramic images in the training sample set into a neural network model, and adjust the output of the neural network model according to the output of the neural network model and the loss value of the reference image. parameters to obtain the preset model.
  • the processor further implements the following steps when executing the computer program: processing the neural network model with a model compression algorithm to update the neural network model.
  • a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:
  • the proportion of sky elements in the target panoramic video determine whether to perform sky segmentation processing on the target panoramic video
  • sky segmentation processing is performed on the panorama image corresponding to the mask image among the M frames of panorama images.
  • the following steps are further implemented: determining N frames of panoramic images from the M frames of panoramic images; inputting the N frames of panoramic images into the preset model to obtain the corresponding N frames of panoramic images. N mask images of ; according to the proportion of sky pixels in the N mask images, it is judged whether to perform sky segmentation processing on the target panoramic video.
  • the following steps are further implemented: determining the proportion of the sky pixels according to the number of sky pixels in the N mask images and the number of non-sky pixels in the N mask images ratio; according to the relationship between the proportion of the sky pixels and the first preset threshold, it is judged whether to perform sky segmentation processing on the target panoramic video.
  • the following steps are further implemented: determine N frames of panoramic images from the M frames of target panoramic images; identify the N frames of panoramic images by using the second preset model, and obtain N frames of panoramic images.
  • the output result; the input of the second preset model is a panoramic image, and the output is that the panoramic image contains the sky or the panoramic image does not contain the sky; according to the proportion of the panoramic images containing the sky in the N output results, determine whether to The target panoramic video is subjected to sky segmentation processing.
  • the following steps are further implemented: according to the number of panoramic images including the sky in the N output results and the number of panoramic images not including the sky in the N output results, determine the The proportion of the panoramic image including the sky; according to the relationship between the proportion of the panoramic image including the sky and the second preset threshold, it is determined whether to perform sky segmentation processing on the target panoramic video.
  • the training sample set of the preset model includes multiple frames of panoramic images and a reference image corresponding to each frame of panoramic images in the multiple frames of panoramic images, the reference The image is used to indicate the sky area of the panoramic image corresponding to the reference image.
  • the computer program further implements the following steps when executed by the processor: inputting the panoramic images in the training sample set into a neural network model, and adjusting the neural network model according to the output of the neural network model and the loss value of the reference image parameters to obtain the preset model.
  • the computer program when executed by the processor, further implements the step of: processing the neural network model with a model compression algorithm to update the neural network model.
  • Non-volatile memory may include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory or optical memory, etc.
  • Volatile memory may include random access memory (Random Access Memory, RAM) or external cache memory.
  • RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (Dynamic Random Access Memory). Access Memory, DRAM), etc.

Abstract

L'invention concerne un procédé et un appareil de segmentation de région du ciel, un dispositif informatique et un support de stockage. Un dispositif informatique détermine, en fonction de la proportion d'éléments du ciel dans une vidéo panoramique cible, s'il faut effectuer une segmentation du ciel sur la vidéo panoramique cible ; si oui, il extrait M trames d'image panoramique de la vidéo panoramique cible, et entre chacune des M trames d'image panoramique dans un modèle prédéfini pour obtenir une image de masque de chacune des M trames d'image panoramique ; et effectue, en fonction de l'image de masque, une segmentation du ciel sur la trame d'image panoramique dans les M trames d'image panoramique correspondant à l'image de masque. Avant d'effectuer une segmentation du ciel sur la vidéo panoramique cible, une détermination est effectuée sur la vidéo panoramique cible en premier, et en tenant compte de la proportion de ciel de la vidéo panoramique, des scénarios qui ont des régions de ciel relativement petites et qui sont inappropriés pour la segmentation du ciel peuvent être efficacement filtrés, ce qui permet d'améliorer l'effet de segmentation du ciel ; de plus, la reconnaissance précise de la vidéo panoramique par le modèle prédéfini peut améliorer l'effet de segmentation du ciel pour la vidéo panoramique.
PCT/CN2022/080598 2021-03-19 2022-03-14 Procédé et appareil de segmentation de région du ciel, dispositif informatique et support de stockage WO2022194079A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110295509.3A CN113034514A (zh) 2021-03-19 2021-03-19 天空区域分割方法、装置、计算机设备和存储介质
CN202110295509.3 2021-03-19

Publications (1)

Publication Number Publication Date
WO2022194079A1 true WO2022194079A1 (fr) 2022-09-22

Family

ID=76471741

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/080598 WO2022194079A1 (fr) 2021-03-19 2022-03-14 Procédé et appareil de segmentation de région du ciel, dispositif informatique et support de stockage

Country Status (2)

Country Link
CN (1) CN113034514A (fr)
WO (1) WO2022194079A1 (fr)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113034514A (zh) * 2021-03-19 2021-06-25 影石创新科技股份有限公司 天空区域分割方法、装置、计算机设备和存储介质
CN113808073A (zh) * 2021-08-03 2021-12-17 北京中科慧眼科技有限公司 基于双目立体匹配算法的去天空方法、系统和智能终端
CN115705666A (zh) * 2021-08-09 2023-02-17 北京字跳网络技术有限公司 图像处理方法、装置、设备及存储介质

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170236287A1 (en) * 2016-02-11 2017-08-17 Adobe Systems Incorporated Object Segmentation, Including Sky Segmentation
US20170294000A1 (en) * 2016-04-08 2017-10-12 Adobe Systems Incorporated Sky editing based on image composition
CN111127307A (zh) * 2019-12-09 2020-05-08 上海传英信息技术有限公司 图像处理方法、装置、电子设备及计算机可读存储介质
CN111210434A (zh) * 2019-12-19 2020-05-29 上海艾麒信息科技有限公司 基于天空识别的图像替换方法及系统
CN111428582A (zh) * 2020-03-05 2020-07-17 南京大学 一种利用互联网街景照片计算城市天空开阔度的方法
CN111464834A (zh) * 2020-04-07 2020-07-28 腾讯科技（深圳）有限公司 一种视频帧处理方法、装置、计算设备及存储介质
CN112258380A (zh) * 2019-07-02 2021-01-22 北京小米移动软件有限公司 图像处理方法、装置、设备及存储介质
CN113034514A (zh) * 2021-03-19 2021-06-25 影石创新科技股份有限公司 天空区域分割方法、装置、计算机设备和存储介质

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109003237A (zh) * 2018-07-03 2018-12-14 深圳岚锋创视网络科技有限公司 全景图像的天空滤镜方法、装置及便携式终端
CN110944163A (zh) * 2019-11-21 2020-03-31 维沃移动通信有限公司 一种图像处理方法及电子设备
CN111489322B (zh) * 2020-04-09 2023-05-26 广州光锥元信息科技有限公司 给静态图片加天空滤镜的方法及装置
CN111833372A (zh) * 2020-07-23 2020-10-27 浙江大华技术股份有限公司 一种前景目标提取方法及装置

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170236287A1 (en) * 2016-02-11 2017-08-17 Adobe Systems Incorporated Object Segmentation, Including Sky Segmentation
US20170294000A1 (en) * 2016-04-08 2017-10-12 Adobe Systems Incorporated Sky editing based on image composition
CN112258380A (zh) * 2019-07-02 2021-01-22 北京小米移动软件有限公司 图像处理方法、装置、设备及存储介质
CN111127307A (zh) * 2019-12-09 2020-05-08 上海传英信息技术有限公司 图像处理方法、装置、电子设备及计算机可读存储介质
CN111210434A (zh) * 2019-12-19 2020-05-29 上海艾麒信息科技有限公司 基于天空识别的图像替换方法及系统
CN111428582A (zh) * 2020-03-05 2020-07-17 南京大学 一种利用互联网街景照片计算城市天空开阔度的方法
CN111464834A (zh) * 2020-04-07 2020-07-28 腾讯科技（深圳）有限公司 一种视频帧处理方法、装置、计算设备及存储介质
CN113034514A (zh) * 2021-03-19 2021-06-25 影石创新科技股份有限公司 天空区域分割方法、装置、计算机设备和存储介质

Also Published As

Publication number Publication date
CN113034514A (zh) 2021-06-25

Similar Documents

Publication Publication Date Title
US10453204B2 (en) Image alignment for burst mode images
WO2020192483A1 (fr) Procédé et dispositif d'affichage d'image
WO2022199583A1 (fr) Procédé et appareil de traitement d'image, dispositif informatique et support de stockage
CN109344742B (zh) 特征点定位方法、装置、存储介质和计算机设备
WO2021073418A1 (fr) Procédé et appareil de reconnaissance faciale, dispositif et support de stockage
WO2022194079A1 (fr) Procédé et appareil de segmentation de région du ciel, dispositif informatique et support de stockage
EP2864933B1 (fr) Procédé, appareil et produit de programme informatique d'extraction de caractéristiques de visage humain
US11900676B2 (en) Method and apparatus for detecting target in video, computing device, and storage medium
WO2021115242A1 (fr) Procédé de traitement d'image à très haute résolution et appareil associé
CN111275034B (zh) 从图像中提取文本区域的方法、装置、设备和存储介质
CN113469092B (zh) 字符识别模型生成方法、装置、计算机设备和存储介质
WO2022206680A1 (fr) Procédé et appareil de traitement d'image, dispositif informatique et support d'enregistrement
CN110930296A (zh) 图像处理方法、装置、设备及存储介质
CN114511041B (zh) 模型训练方法、图像处理方法、装置、设备和存储介质
CN111489322A (zh) 给静态图片加天空滤镜的方法及装置
CN111915541B (zh) 基于人工智能的图像增强处理方法、装置、设备及介质
CN114445651A (zh) 一种语义分割模型的训练集构建方法、装置及电子设备
CN112101386A (zh) 文本检测方法、装置、计算机设备和存储介质
WO2024041108A1 (fr) Procédé et appareil d'entraînement de modèle de correction d'image, procédé et appareil de correction d'image, et dispositif informatique
CN110689478B (zh) 图像风格化处理方法、装置、电子设备及可读介质
CN111160240A (zh) 图像对象的识别处理方法、装置及智能设备、存储介质
CN109934783B (zh) 图像处理方法、装置、计算机设备和存储介质
WO2022206679A1 (fr) Procédé et appareil de traitement d'image, dispositif informatique et support de stockage
CN116403200A (zh) 基于硬件加速的车牌实时识别系统
CN116052233A (zh) 一种神经网络优化方法、装置、计算设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22770434

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22770434

Country of ref document: EP

Kind code of ref document: A1