WO2023193648A1

WO2023193648A1 - Image processing method and apparatus, electronic device, and storage medium

Info

Publication number: WO2023193648A1
Application number: PCT/CN2023/084967
Authority: WO
Inventors: 郭士嘉; 龙良曲
Original assignee: 影石创新科技股份有限公司
Priority date: 2022-04-08
Filing date: 2023-03-30
Publication date: 2023-10-12
Also published as: CN116958164A

Abstract

Embodiments of the present application disclose an image processing method and apparatus, an electronic device, and a storage medium. According to the embodiments of the present application, the method comprises: obtaining a panoramic image; segmenting the panoramic image by using a preset segmentation model so as to obtain a first local image; performing feature recognition on the first local image to obtain image features of the first local image; determining a target feature element according to a plurality of feature elements, and determining the position of the target feature element in the first local image; according to the preset segmentation model, determining a position relationship between the first local image and the panoramic image, and the position of the target feature element in the first local image, and determining the position of the target feature element in the panoramic image; and performing distortion correction on the panoramic image on the basis of the position of the target feature element in the panoramic image so as to obtain a corrected panoramic image. Therefore, a photographed target in the panoramic image can be quickly recognized, thereby alleviating the problem of slow processing speeds caused by existing recognition processing processes that are relatively complicated.

Description

An image processing method, device, electronic equipment and storage medium

Technical field

The present application relates to the field of image processing, and specifically relates to an image processing method, device, electronic equipment and storage medium.

Background technique

A panoramic image is a wide-angle image that can show as much of the surrounding environment as possible through wide-angle expression methods, photos, videos, etc. A panoramic image is an image that can surround 0 to 360° in the horizontal direction and 0 to 180° in the vertical direction. Currently, when shooting a panoramic image, the panoramic image needs to be recognized and processed to determine the shooting target in the panoramic image, so as to obtain an image with the shooting target as the focus.

Contents of the invention

Embodiments of the present application provide an image processing method, device, electronic device, and storage medium, which can quickly identify shooting targets in panoramic images, and improve the problem that the existing identification processing process is cumbersome, resulting in slow processing speed.

On the one hand, embodiments of the present application provide an image processing method, including:

Get a panoramic image;

Segment the panoramic image using a preset segmentation model to obtain a first partial image and a second partial image, wherein the degree of distortion of the first partial image is lower than that of the second partial image;

Perform feature recognition processing on the first partial map to obtain image features of the first partial map, where the image features of the first partial map are composed of a plurality of feature elements;

Determine a target feature element based on a plurality of the feature elements, and determine the position of the target feature element in the first partial map;

According to the preset segmentation model, the positional relationship between the first partial image and the panoramic image is determined, as well as the position of the target feature element in the first partial image, and the location of the target feature element is determined. The position in the panoramic image;

Based on the position of the target feature element in the panoramic image, distortion correction processing is performed on the panoramic image to obtain a corrected panoramic image.

On the other hand, embodiments of the present application also provide an image processing device, including:

Acquisition unit, used to acquire panoramic images;

A segmentation processing unit configured to segment the panoramic image using a preset segmentation model to obtain a first partial image and a second partial image, wherein the degree of distortion of the first partial image is lower than that of the second partial image. ;

A feature recognition processing unit, configured to perform feature recognition processing on the first partial map to obtain image features of the first partial map, wherein the image features of the first partial map are composed of a plurality of feature elements;

A first determination unit configured to determine a target feature element based on a plurality of the feature elements, and determine the position of the target feature element in the first partial map;

A second determination unit, configured to determine the positional relationship between the first partial image and the panoramic image and the position of the target feature element in the first partial image according to the preset segmentation model, and determine The position of the target feature element in the panoramic image;

A correction unit configured to perform distortion correction processing on the panoramic image based on the position of the target feature element in the panoramic image to obtain a corrected panoramic image.

On the other hand, embodiments of the present application also provide an electronic device, including a processor and a memory, the memory stores a plurality of instructions; the processor loads instructions from the memory to execute the instructions provided by the embodiments of the present application. steps in any image processing method.

On the other hand, embodiments of the present application also provide a computer-readable storage medium that stores a plurality of instructions, and the instructions are suitable for loading by the processor to execute the methods provided by the embodiments of the present application. The steps in any image processing method.

In this application, a preset segmentation model is used to segment the panoramic image to obtain a first partial image with a low degree of distortion, and then feature recognition processing is performed on the image features of the first partial image to determine the image of the first partial image. The target feature element in the feature, and the position of the target feature element in the first partial image, and according to the position change relationship corresponding to the preset segmentation model, determine the position of the target feature element in the panoramic image, so that the target feature element is in the panoramic image location, panorama The image is subjected to distortion correction processing to obtain a corrected panoramic image. As a result, the panoramic image is quickly processed through the preset segmentation model, thereby processing the acquired panoramic image and improving the problem of large distortion of some important scenery or people in the image.

Description of the drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can also be obtained based on these drawings without exerting creative efforts.

Figure 1 is a schematic scene diagram of an image processing method provided by an embodiment of the present application;

Figure 2 is a schematic flowchart of an image processing method provided by an embodiment of the present application;

Figure 2a is a schematic diagram of a panoramic image segmented according to a preset segmentation model provided by an embodiment of the present application;

Figure 2b is a schematic diagram of a sub-image obtained by extracting the first partial image provided by the embodiment of the present application;

Figure 3 is a schematic flowchart of a method for obtaining image features of a first partial image provided by an embodiment of the present application;

Figure 4 is a schematic flowchart of a method for determining a target feature element and the position of the target feature element in the first partial diagram provided by an embodiment of the present application;

Figure 5 is a schematic flowchart of a method for obtaining a corrected panoramic image of the next frame provided by an embodiment of the present application;

Figure 6 is a schematic diagram of the image processing method provided by the embodiment of the present application applied in a server scenario;

Figure 7 is a first structural schematic diagram of an image processing device provided by an embodiment of the present application;

FIG. 8 is a schematic structural diagram of an image processing device provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, and Not all examples. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts fall within the scope of protection of this application.

Embodiments of the present application provide an image processing method, device, storage medium, and storage medium.

The image processing device may be integrated into an electronic device, and the electronic device may be a terminal, a server, or other equipment. Among them, the terminal can be a mobile phone, a tablet, a smart Bluetooth device, a laptop, or a personal computer (PC); the server can be a single server or a server cluster composed of multiple servers.

In some embodiments, the image processing device can also be integrated in multiple electronic devices. For example, the image processing device can be integrated in multiple servers, and the image processing method of the present application is implemented by multiple servers.

In some embodiments, the server can also be implemented in the form of a terminal.

For example, referring to Figure 1, the electronic device can be a server, and the server is integrated with an image processing device. The server in the embodiment of the present application is used to obtain a panoramic image; the panoramic image is segmented using a preset segmentation model to obtain the first local map and second local map, the degree of distortion of the first local map is lower than that of the second local map; feature recognition processing is performed on the first local map to obtain the image features of the first local map, so The image features of the first partial map are composed of a plurality of feature elements; perform extraction processing on the feature elements in the image features of the first partial map, determine the target feature element in the feature elements, and determine the target feature The position of the element in the first partial map; determine the position change relationship corresponding to the preset segmentation model; determine based on the position change relationship of the target feature element in the first partial map based on the position change relationship The position of the target feature element in the panoramic image; based on the position of the target feature element in the panoramic image, perform distortion correction processing on the panoramic image to obtain a corrected panoramic image.

Each is explained in detail below. It should be noted that the serial numbers of the following embodiments are not used to limit the preferred order of the embodiments.

In this embodiment, an image processing method is provided, as shown in Figure 2. The specific process of the image processing method can be as follows:

S110. Obtain a panoramic image.

Panoramic image is a kind of wide-angle image, which can be expressed through wide-angle means and photos, videos and other forms. style, showing as much of the surrounding environment as possible. A panoramic image is an image that can surround 0 to 360° in the horizontal direction and 0 to 180° in the vertical direction. Panoramic images can be captured with an ordinary camera and then synthesized, or they can be captured directly with a panoramic camera.

The panoramic image obtained may be the current frame panoramic image extracted from the panoramic video, or it may be a pre-photographed panoramic image.

In some embodiments, the acquired panoramic image may be a rectangular image with an aspect ratio of 2:1.

S120. Use a preset segmentation model to segment the panoramic image to obtain a first partial image and a second partial image, where the degree of distortion of the first partial image is lower than that of the second partial image.

The preset segmentation model may refer to dividing image areas with a high degree of image distortion and image areas with a low degree of image distortion in the panoramic image through machine learning or other methods, so as to extract image areas with a low degree of image distortion in the panoramic image. Among them, image distortion refers to problems such as blurring, stretching, and deformation of characters or scenes in the image that cannot reflect the real situation of the characters or scenes during imaging. The degree of distortion can be judged through human experience, or can be confirmed by comparing the coordinates of the distortion position with the coordinates under normal conditions. Region division may refer to dividing the panoramic image into several regional parts according to the degree of distortion. In some embodiments, the panoramic image may be divided into two parts according to the degree of distortion. For example, the panoramic image has a low degree of distortion. The area with a preset degree of distortion is a low-distortion area, which is the area where the first partial image is located. The area on the panoramic image where the degree of distortion is higher than the preset distortion level is a high-distortion area, which is the area where the second partial image is located.

In this embodiment of the present application, the preset segmentation model may be set by dividing low-distortion areas and high-distortion areas of the panoramic image based on human experience. For example, in the panoramic image as shown in Figure 2a, the low-distortion area of the panoramic image It can be the 80% image area in the middle of the image along the width direction, that is, the middle rectangular frame part of the panoramic image shown in Figure 2a. This image area is located in the middle of the panoramic image. The imaging effect is good and the panoramic image has high distortion. The area can be 10% of the image area in the upper and lower parts of the image, that is, the upper rectangular frame part and the lower rectangular frame part of the panoramic image shown in Figure 2a. This image area is located at the upper edge and lower edge of the panoramic image, imaging Less effective. Panoramic images can be processed quickly through preset segmentation models.

The segmentation process can be to divide the panoramic image into several segments according to the division results of the preset segmentation model. Images with different degrees of distortion. For example, in the embodiment of the present application, the preset segmentation model segments the panoramic image into low distortion areas and high distortion areas, and the segmentation process is to divide the panoramic image into the first partial image in the low distortion area and the third partial image in the high distortion area. Two partial pictures.

The first partial map and the second partial map may be obtained through sliding window sampling, or the panoramic image may be extracted by cutting. Wherein, the number of the first partial images and the number of the second partial images obtained may be arbitrary; there may be no overlapping area between the first partial images and the second partial images; the first partial images and the second partial images may be Stitching creates a panoramic image.

S130. Perform feature recognition processing on the first partial map to obtain image features of the first partial map, where the image features of the first partial map are composed of multiple feature elements.

The feature recognition process may refer to performing pixel-level image classification on the first partial image, and labeling to determine the object category to which each pixel in the first partial image belongs, thereby obtaining images with different object category labels. Among them, object categories can include landscapes, buildings, and people. After labeling the corresponding object categories on the first partial map, the resulting images with different labels are image features. In some embodiments, the image features may include color features, texture features, shape features and spatial relationship features. In some embodiments, the image feature may be composed of multiple feature elements, wherein the feature elements constituting the image feature may be pixels. Pixel refers to the basic coding of basic original pigments and their grayscales. The pixel category is the pixel type. Different images have different pixel types. However, for different pixel types, different values need to be passed in the template parameters. For example, the data types of pixels include CV_32U, CV_32S, CV_32F, CV_8U, CV_8UC3, etc.

For example, in the embodiment of the present application, through feature recognition processing, the scenery, buildings, and people in the first partial picture can be identified, and by labeling and segmenting the scenery, buildings, and people in the first partial picture, it is determined that The background part and foreground part of the first partial image can be represented in the form of a mask image, where the mask image refers to the background and foreground of the first partial image (the foreground can include buildings and people) through Images with different grayscales are presented.

As shown in Figure 3, in some embodiments, the method of obtaining the image features of the first local map includes:

S131. Determine the size information of the first partial image.

The size information may include the length, width, and angular field of view of the first partial image. Among them, the field of view (Field of view, FOV) is also called the field of view in optical engineering. The size of the field of view determines the field of view of the optical instrument. In the embodiment of the present application, when the captured image is a panoramic image, the field of view angle of the panoramic image in the horizontal direction is 0 to 360°, and the field of view angle in the vertical direction is 0 to 180°.

The size information of the first partial image can be determined by measuring the first partial image or by determining the segmentation ratio of the preset segmentation model.

S132. Determine the set size, sliding step length and sliding direction of the sliding window based on the size information of the first partial image.

The sliding window is a sliding window. In order to improve the accuracy of the data, the value of a certain point is expanded to the area containing the point, and the area is used for judgment. This area is the window. The sliding window is able to frame the time series according to the specified unit length, thereby calculating the statistical indicators within the frame. It is equivalent to sliding a slider with a specified length on the scale, and the image data in the slider can be fed back every time it slides one unit.

The size of the sliding window is the range of the sliding window for image segmentation on the first partial image. The sliding window can be of any shape. In some embodiments, the sliding window can be a rectangular frame. The size of the sliding window can include the length of the rectangular frame and width. In some embodiments, the sliding window may be a square frame, and the size of the sliding window may include the side length of the square frame.

The sliding direction of the sliding window is the direction in which the sliding window slides in the first partial graph, and the sliding direction of the sliding window can be arbitrary. In some embodiments, the sliding direction of the sliding window is sliding along the length direction of the first partial image.

The sliding step size of the sliding window is the distance of each sliding when the sliding window slides on the first partial graph. The sliding step size of the sliding window can be set according to the size information of the first partial graph. In some embodiments, when the sliding direction of the sliding window is to slide along the length direction of the first partial graph, the sliding step size of the sliding window is equal to the first partial graph. The length of a partial graph is divided by the number of times the sliding window slides.

S133. Segment the first partial image according to the size, sliding step and sliding direction of the sliding window to obtain multiple sub-images.

Segmentation processing refers to intercepting the image within the position of the sliding window from the first partial image.

Segmenting the first partial image may be when the sliding window moves along the preset direction, and each time the sliding window slides through a sliding step, the image in the sliding window is intercepted to obtain a separate image. Images are denoted as sub-images. When the sliding window slides through n sliding steps, there are n sub-images obtained. Among them, the sub-images can be set alternately, separated and partially overlapped.

For example, in the embodiment of the present application, the preset direction is along the length direction of the first partial image, the sliding window slides through two sliding steps, and there are two sub-images obtained. The sum of the lengths of the two sub-images is greater than the length of the first partial image, that is, as shown in Figure 2b. In the embodiment of the present application, the first partial image located in the middle of the panoramic image is divided into two sub-images, left and right.

S134. Perform feature recognition processing on the sub-image to obtain image features of the sub-image.

In the embodiment of the present application, through feature recognition processing, the scenery, buildings, and people in the sub-image can be identified, and by labeling and segmenting the scenery, buildings, and people in the sub-image, the background part and the background part of the sub-image can be determined. The foreground part can be represented in the form of a mask image, where the mask image refers to presenting the background and foreground of the sub-image (the foreground can include buildings and people) through images with different grayscales.

Among them, in some embodiments, feature recognition processing is performed on the sub-image, and the method for determining the image features of the sub-image includes:

Determine the sub-image and the shape of the sub-image;

Compare the shape of the sub-image to the preset shape:

When the shape of the sub-image is the same as the preset shape, the sub-image is determined to be the target sub-image;

When the shape of the sub-image is different from the preset shape, adjust the shape of the sub-image to obtain a target sub-image that conforms to the preset shape;

Perform feature recognition processing on the target sub-image to obtain the image features of the target sub-image;

According to the image characteristics of the target sub-image, the image characteristics of the sub-image are determined.

Among them, in some embodiments, according to the target sub-image, the method of determining the image characteristics of the sub-image includes:

Encode and decode the target sub-image to obtain a mask image of the target sub-image;

According to the mask map of the target sub-image, determine the image characteristics of the target sub-image;

Determine the proportional relationship between the sub-image and the target sub-image;

According to the proportional relationship between the sub-image and the target sub-image, the image characteristics of the sub-image are determined.

The preset shape refers to the shape that meets the input image shape requirements during encoding and decoding processing, for example For example, in the embodiment of the present application, the preset shape is a square. Comparing the shape of the sub-image with a preset shape refers to determining whether the shape of the sub-image is a square. When the shape of the sub-image is a square, the sub-image is determined to be the target sub-image. When the shape of the sub-image is not a square, Adjust the shape of the sub-image so that the sub-image is a target sub-image that conforms to a preset shape.

In some embodiments, the method of performing shape adjustment may include:

Get the size information of the subimage. Among them, the sub-image is generally a rectangular image, and the size information of the sub-image includes the length and width of the sub-image. Determining the size information of the sub-image can be obtained by measuring the sub-image.

According to the size information of the sub-image, the sub-image is resized. Among them, resizing the sub-image means shortening the rectangular picture along the length direction or elongating it along the width direction, so that the length and width of the sub-image are the same, and a target sub-image with the same length and width is obtained, and then the feature recognition process is performed When the target sub-image has the same length and width, it can be conveniently processed by convolution and pooling, so as to better identify the features in the target sub-image.

The encoding and decoding process may refer to feature extraction and upsampling of the target sub-image, thereby obtaining a mask map of the target sub-image. Among them, the feature extraction of the target sub-image can be performed by performing convolution and pooling processing on the target sub-images with the same length and width; the up-sampling of the target sub-image can be performed by deconvolution of the target sub-image after feature extraction. , thereby obtaining the mask map of the target sub-image. Among them, the mask image of the target sub-image includes a foreground part and a background part, where the gray value of the pixels in the foreground part is greater than the gray value of the pixels in the background part, and the foreground part is the image feature of the target sub-image.

The proportional relationship between the sub-image and the target sub-image refers to the transformation relationship between the target sub-image after shape adjustment and the sub-image before shape adjustment. According to the transformation relationship, the relationship between the sub-image and the target sub-image can be directly determined. The proportional relationship between them can then quickly determine the image characteristics of the sub-image based on the image characteristics of the target sub-image.

S135. Determine the image features of the first partial image according to the image features of the sub-image.

Among them, the method of determining the image characteristics of the first partial map includes:

Determine the image characteristics of the sub-image and the position of the sub-image in the first partial image;

According to the image characteristics of the sub-image and the position of the sub-image in the first partial diagram, the image characteristics of the first partial diagram are determined.

Determining the position of the sub-image in the first partial image may refer to determining the coordinate position of the sub-image on the first partial image. For example, in some embodiments, a coordinate system can be established on the first local graph, the coordinate position of the sliding window after each sliding step is determined, and then the sub-image is determined based on the coordinate position after each sliding step. The coordinate position on the first partial map.

After determining the coordinate position of the sub-image on the first partial map, by determining the position of the image feature of the sub-image in the sub-image, determining the position of the image feature of the sub-image in the first partial map, thereby determining the first partial map image features.

In some embodiments, when the overlapping area of adjacent sub-images includes overlapping image features, the image features in the overlapping area can be determined by determining the feature intensity of the overlapping image features in the overlapping area. The characteristic intensity may be the pixel intensity, that is, the brightness value of the pixel. The brightness value of the pixel is between 0 and 255. The brightness value close to 255 is high and the brightness close to 0 is low.

For example, in some embodiments, the sub-image includes a first sub-image and a second sub-image, and the first sub-image and the second sub-image include an overlapping area that overlaps each other, wherein the first sub-image in the overlapping area The image feature is the first image feature, and the image feature of the second sub-image in the overlapping area is the second image feature;

The method of performing feature recognition processing on the first partial image and obtaining the image features of the first partial image also includes:

When the first image feature overlaps with the second image feature, calculating a first feature intensity of the first image feature and a second feature intensity of the second image feature;

Compare the first feature strength with the second feature strength:

When the first feature intensity is greater than the second feature intensity, select the first image feature as the image feature of the first local map;

When the first feature intensity is less than the second feature intensity, select the second image feature as the image feature of the first local map;

When the first feature intensity is equal to the second feature intensity, the first image feature or the second image feature is selected as the image feature of the first local map.

S140. Determine the target characteristic element based on multiple characteristic elements, and determine whether the target characteristic element is in A location in a part of the graph.

Determining the position of the target feature element in the first partial image can be determined by extracting the feature elements. Extracting the feature elements means determining the more significant feature elements based on the pixel values of the feature elements, where , the more significant feature element refers to the target feature element whose pixel value is greater than the preset pixel threshold. Among them, the pixel value can be the pixel intensity, and the pixel intensity is between 0 and 255. The pixel threshold can be set manually. For example, the pixel preset can be set to 150 or 200 as needed. The target feature element means that the pixel value is greater than 150 or 250. 200 characteristic elements.

As shown in Figure 4, in some embodiments, a method for determining a target feature element based on multiple feature elements and determining the position of the target feature element in the first partial map includes:

S141. Determine the pixel values of multiple feature elements in the first partial image.

The feature elements are the pixels that make up the image features, and the pixel value of each pixel can be determined, and the pixel value can be the gray value of the pixel. The method of determining the pixel value of the feature element of the image feature in the first partial image may include obtaining the grayscale value of the pixel point through the impixel function.

S142. Compare the pixel values of multiple feature elements in the first local image with the preset pixel threshold, and determine respectively the first feature element whose pixel value is greater than the pixel threshold, and the first feature element whose pixel value is less than the pixel threshold. a second characteristic element, wherein the first characteristic element is the target characteristic element;

S143. Binarize the pixel value of the first feature element and the pixel value of the second feature element to obtain a binarized first local map.

Binarization processing refers to increasing the pixel value of the first feature element and reducing the pixel value of the second feature element based on the intensity comparison result, so that the difference between the first feature element and the second feature element is more obvious. . For example, in some embodiments, the gray value of a pixel ranges from 0 to 255. Increasing the pixel value of the first feature element may mean adjusting the pixel value of the first feature element to 255, and increasing the pixel value of the second feature element. Decreasing the pixel value of the feature element may refer to adjusting the pixel value of the second feature element to 0.

S144. Determine the position of the target feature element in the first partial map according to the binarized first partial map.

The position of the target feature element in the first partial diagram can be determined by comparing the target feature element with the first The positional relationship of the local graph is determined.

S150. According to the preset segmentation model, determine the positional relationship between the first partial image and the panoramic image, as well as the position of the target feature element in the first partial image, and determine the position of the target feature element in the panoramic image.

Determining the positional relationship between the first partial image and the panoramic image refers to determining the size and position of the first partial image in the panoramic image when segmenting the panoramic image according to the preset segmentation model, where, since the preset segmentation model is preset When determining the position relationship, it can be obtained directly and quickly based on the corresponding position change relationship.

Determining the position of the target feature element in the panoramic image means that since the positional relationship between the first partial image and the panoramic image is determined, the position of the target feature element in the first partial image is also determined. Therefore, it can be quickly and accurately determined through coordinate conversion. Determine the position of the target feature element in the panoramic image.

S160. Based on the position of the target feature element in the panoramic image, perform distortion correction processing on the panoramic image to obtain a corrected panoramic image.

Distortion correction processing refers to changing the distance between the lens and the imaging surface when acquiring an image to make the image of the subject clear. That is, the position of the target feature element in the panoramic image is used as the position for distortion correction processing to correct the image, that is, the image is obtained. The position of the target feature element in the corrected panoramic image has the clearest image and smaller distortion. The correction method may include changing the focus point so that the position of the focus point is aligned with the location of the target feature element. Focus refers to changing the distance between the lens and the imaging surface to make the subject image clear when acquiring an image. The focus point refers to the photographed object characterized by the desired target characteristic elements.

As shown in Figure 5, after obtaining the corrected panoramic image, the image processing method of the present application also includes:

S170. Obtain the position of the target feature element in the modified panoramic image;

S180. Based on the position of the target feature element in the corrected panoramic image, obtain the next frame of panoramic image, and perform distortion correction processing on the next frame of panoramic image to obtain the corrected next frame of panoramic image.

The next frame of panoramic image can be the next frame of panoramic image in the panoramic video, or the next frame of panoramic image during continuous shooting. The method of obtaining the next frame of panoramic image can be through shooting, Pre-stored panoramic images can also be retrieved. Performing distortion correction processing on the acquired panoramic image of the next frame refers to shooting the panoramic image of the next frame based on the modified position of the target feature element of the currently captured panoramic image, and using the target feature element of the modified panoramic image. Perform distortion correction processing on the clearest and less distorted position in the panoramic image captured in the next frame to obtain the corrected panoramic image of the next frame.

The image processing method in the embodiment of the present invention is described below in conjunction with a specific application scenario.

Please refer to Figure 6, which is a schematic flow chart of an image processing method applied in an experimental scenario according to an embodiment of the present invention. The image processing method is applied to a server. The image processing method includes:

S210. Obtain the panoramic image.

The horizontal field of view angle of the panoramic image is 0~360°, and the vertical field of view angle is 0~180°.

S220. Segment the panoramic image to obtain a first partial image and a second partial image. The degree of distortion of the first partial image is lower than that of the second partial image.

Divide 10% of the area from the upper and lower parts of the panoramic image, where the 10% area from the upper and lower parts is the second partial image, and the remaining 80% of the area is the first partial image. For example, the panoramic image has a length of 1000 and a width of 500, and the first partial image after excision has a length of 1000 and a width of 400.

S230. Extract the first partial image to obtain the sub-image.

The first partial image is extracted through a sliding window to obtain two left and right sub-images, in which there is an overlapping area between the two sub-images. For example, the length of the first sub-image is 600 and the width is 400, and the length of the second sub-image is 600 and the width is 400. Among them, the length of the overlapping area between the first sub-image and the second sub-image is 200, width is 400.

S240. Perform feature recognition processing on the sub-image to obtain image features of the sub-image.

U ² net is used to perform feature recognition processing on the sub-image. The sub-image is input into the encoder-decoder network structure model to obtain 6 mask images with the same size as the input sub-image. The 6 mask images are The intensity is averaged and output to obtain the mask map of the sub-image.

S250. Determine the position of the sub-image in the first partial map, and determine the image features of the first partial map based on the position of the sub-image in the first partial map. The image features of the first partial map are composed of multiple feature elements.

Establish a coordinate system on the first partial graph, where the coordinates of the four corners of the first partial graph are (0,0), (1000,0), (1000,400), (0,400), from which the coordinates of the four corners of the first sub-image can be determined to be (0,0), (600,0 ), (600, 400), (0, 400), the coordinates of the four corners of the second sub-image are (400, 0), (1000, 0), (1000, 400), (400, 400), According to the coordinate relationship between the sub-image and the first partial map, the positional relationship between the sub-image and the first partial map can be determined, and then based on the positional relationship between the mask map of the sub-image and the first partial map, the positional relationship used to represent the first partial map can be determined. A mask map of the image features of a partial graph.

S260. Extract the feature elements of the image features of the first partial map, determine the target feature element among the feature elements, and determine the position of the target feature element in the first partial map.

Determine the pixel value of each pixel in the mask map used to characterize the image features of the first local map, perform binarization processing on the mask map used to characterize the image features of the first local map, and convert all the image features into The pixel value of a pixel whose pixel intensity value is greater than the preset threshold is adjusted to 255, and the pixel value of a pixel whose pixel intensity value is less than the preset threshold is adjusted to 0. Thus, a binary image of the first partial map is obtained, which facilitates the identification and selection of the target feature element and obtains the position of the target feature element in the first partial map.

S270. Determine the position of the target feature element in the panoramic image according to the positional relationship between the first partial image and the panoramic image.

The positional relationship between the first partial image and the panoramic image is determined based on the segmentation ratio. Since the segmentation ratio between the first partial image and the panoramic image is determined during the segmentation process, it is possible to quickly determine the position of the target feature element in the panorama. location in the image.

S280. Based on the position of the target feature element in the panoramic image, perform distortion correction processing on the panoramic image to obtain a corrected panoramic image.

In the embodiment of the present application, the panoramic image is segmented to quickly determine the first partial image with a lower degree of distortion, and the first partial image is determined by performing feature recognition and extraction processing on the first partial image. The target feature element, according to the segmentation ratio of the segmentation process, quickly determines the position of the target feature element of the first partial image in the panoramic image, and performs distortion correction on the panoramic image according to the position of the target feature element in the panoramic image to obtain the correction panoramic image after. As a result, the speed of processing panoramic images can be increased, and the problem that the existing recognition processing process is cumbersome, resulting in slow processing speed, can be improved.

In order to better implement the above method, embodiments of the present application also provide an image processing device. The image processing device can be integrated in an electronic device. The electronic device can be a terminal, a server, and other equipment. Among them, the terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop, personal computer and other devices; the server can be a single server or a server cluster composed of multiple servers.

For example, in this embodiment, the image processing device is specifically integrated in a server as an example to describe the method in the embodiment of the present application in detail.

For example, as shown in Figure 7, the image processing device may include:

Acquisition unit 301, used to acquire panoramic images;

The segmentation processing unit 302 is configured to segment the panoramic image using a preset segmentation model to obtain a first partial image and a second partial image, wherein the degree of distortion of the first partial image is lower than that of the second partial image;

The feature recognition processing unit 303 is configured to determine a target feature element based on a plurality of the feature elements, and determine the position of the target feature element in the first partial map;

The first determination unit 304 is configured to determine the positional relationship between the first partial image and the panoramic image and the position of the target feature element in the first partial image according to the preset segmentation model, and Determine the position of the target feature element in the panoramic image;

The second determination unit 305 is used to determine the positional relationship between the first partial image and the panoramic image according to the preset segmentation model; according to the positional relationship between the first partial image and the panoramic image, and the position of the target feature element in the first partial image. , determine the position of the target feature element in the panoramic image;

The correction unit 306 is configured to perform distortion correction processing on the panoramic image based on the position of the target feature element in the panoramic image to obtain a corrected panoramic image.

In some embodiments of the present application, the feature recognition processing unit 303 is also specifically used to:

Determine the size information of the first partial image;

According to the size information of the first partial graph, determine the size, sliding step length and sliding direction of the sliding window;

According to the size of the sliding window, the sliding step and the sliding direction, the first partial image is segmented to obtain multiple sub-images;

Perform feature recognition processing on the sub-image to obtain the image features of the sub-image;

According to the image characteristics of the sub-image, the image characteristics of the first partial image are determined.

Determine the sub-image and the shape of the sub-image;

Compare the shape of the sub-image to the preset shape:

When the shape of the sub-image is the same as the preset shape, the sub-image is determined to be the target sub-image; when the shape of the sub-image is different from the preset shape, the shape of the sub-image is adjusted to obtain a target that conforms to the preset shape. subimage;

The sub-image includes a first sub-image and a second sub-image, and the first sub-image and the second sub-image include an overlapping area that overlaps each other, wherein the image feature of the first sub-image in the overlapping area is the first image feature, The image features of the second sub-image in the overlapping area are the second image features;

Compare the first feature strength with the second feature strength:

In some embodiments of this application, the first determining unit 304 is also specifically used to:

Determine pixel values of multiple feature elements in the first local map;

Compare the pixel values of multiple feature elements in the first local map with a preset pixel threshold, and determine the first feature element whose pixel value is greater than the pixel threshold, and the second feature element whose pixel value is less than the pixel threshold. Feature elements, where the first feature element is the target feature element;

Binarize the pixel value of the first feature element and the pixel value of the second feature element to obtain the first local image after binarization;

According to the binarized first local map, the position of the target feature element in the first local map is determined.

In some embodiments of this application, the correction unit 306 is also specifically used to:

Obtain the position of the target feature element in the corrected panoramic image;

Based on the position of the target feature element in the corrected panoramic image, the next frame of panoramic image is obtained, and distortion correction processing is performed on the next frame of panoramic image to obtain the corrected next frame of panoramic image.

During specific implementation, each of the above units can be implemented as an independent entity, or can be combined in any way to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, and will not be described again here.

It can be seen from the above that the image processing device of this embodiment includes an acquisition unit 301 for acquiring a panoramic image; a segmentation processing unit 302 for segmenting the panoramic image using a preset segmentation model to obtain the first partial image and the second partial image. Figure, where the degree of distortion of the first partial figure is lower than that of the second partial figure; the feature recognition processing unit 303 is used to perform feature recognition processing on the first partial figure to obtain the first The image features of the local map, where the image features of the first local map are composed of multiple feature elements; the first determination unit 304 is used to determine the target feature element based on the multiple feature elements, and determine whether the target feature element is in The position in the first partial image; the second determination unit 305 is used to determine the positional relationship between the first partial image and the panoramic image according to the preset segmentation model, and the location of the target feature element. The position in the first partial image is determined, and the position of the target feature element in the panoramic image is determined; the correction unit 306 is used to perform distortion correction processing on the panoramic image based on the position of the target feature element in the panoramic image, Get the corrected panoramic image. Therefore, embodiments of the present application can increase the speed of processing panoramic images, and improve the problem that the existing recognition processing process is cumbersome, resulting in slow processing speed.

An embodiment of the present application also provides an electronic device, which may be a terminal, a server, or other devices. Among them, the terminal can be a mobile phone, a tablet, a smart Bluetooth device, a laptop, a personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers, etc.

In this embodiment, a detailed description will be given taking the electronic device of this embodiment as an image processing device as an example. For example, as shown in Figure 8, it shows a schematic structural diagram of the image processing device involved in the embodiment of the present application. Specifically:

The image processing device may include components such as a processor 401 of one or more processing cores, a memory 402 of one or more computer-readable storage media, a power supply 403, an input module 404, and a communication module 405. Those skilled in the art can understand that the structure of the image processing device shown in FIG. 8 does not constitute a limitation on the image processing device, and may include more or fewer components than shown, or some components may be combined, or different components may be used. layout. in:

The processor 401 is the control center of the image processing device, using various interfaces and lines to connect various parts of the entire image processing device, by running or executing software programs and/or modules stored in the memory 402, and calling the software programs and/or modules stored in the memory 402. The data in the image processing device performs various functions and processes the data. In some embodiments, processor 401 may include one or more processing cores Center; In some embodiments, the processor 401 can integrate an application processor and a modem processor, where the application processor mainly processes the operating system, user interface, application programs, etc., and the modem processor mainly processes wireless communications. It can be understood that the above modem processor may not be integrated into the processor 401.

The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402 . The memory 402 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the storage data area may store data based on Data created using image processing devices, etc. In addition, memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402 .

The image processing device also includes a power supply 403 that supplies power to various components. In some embodiments, the power supply 403 can be logically connected to the processor 401 through a power management system, thereby realizing functions such as managing charging, discharging, and power consumption management through the power management system. . The power supply 403 may also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

The image processing device may further include an input module 404 that may be used to receive input numeric or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

The image processing device may also include a communication module 405. In some embodiments, the communication module 405 may include a wireless module. The image processing device may perform short-distance wireless transmission through the wireless module of the communication module 405, thereby providing users with wireless broadband. Internet access. For example, the communication module 405 can be used to help users send and receive emails, browse web pages, access streaming media, etc.

Although not shown, the image processing device may also include a display unit and the like, which will not be described again here. Specifically, in this embodiment, the processor 401 in the image processing device will load the executable files corresponding to the processes of one or more application programs into the memory 402 according to the following instructions, and use The processor 401 runs application programs stored in the memory 402 to implement various functions.

In some embodiments, a computer program product is also proposed, including a computer program or instructions that implement the steps in any of the above image processing methods when executed by a processor.

For the specific implementation of each of the above operations, please refer to the previous embodiments and will not be described again here.

Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling relevant hardware through instructions. The instructions can be stored in a computer-readable storage medium, and loaded and executed by the processor.

To this end, embodiments of the present application provide a computer-readable storage medium in which a plurality of instructions are stored, and the instructions can be loaded by the processor to execute the steps in any image processing method provided by the embodiments of the present application. .

Among them, the storage medium may include: read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk, etc.

According to one aspect of the present application, a computer program product or computer program is provided, which computer program product or computer program includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the method provided in the various optional implementations of image processing provided in the above embodiments.

Since the instructions stored in the storage medium can execute the steps in any image processing method provided by the embodiments of this application, therefore, it is possible to achieve what can be achieved by any image processing method provided by the embodiments of this application. The beneficial effects can be found in the previous embodiments for details and will not be described again here.

The above is a detailed introduction to an image processing method, device, terminal, storage medium and computer-readable storage medium provided by the embodiments of the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The above The description of the embodiments is only used to help understand the method and the core idea of the present application; at the same time, for those skilled in the art, there will be changes in the specific implementation and application scope based on the ideas of the present application. In summary, , the content of this description should not be understood as a limitation of this application.

Claims

An image processing method, characterized by including:

Get a panoramic image;

Segment the panoramic image using a preset segmentation model to obtain a first partial image and a second partial image, wherein the degree of distortion of the first partial image is lower than that of the second partial image;

Perform feature recognition processing on the first partial map to obtain image features of the first partial map, where the image features of the first partial map are composed of a plurality of feature elements;

Determine a target feature element based on a plurality of the feature elements, and determine the position of the target feature element in the first partial map;

According to the preset segmentation model, the positional relationship between the first partial image and the panoramic image is determined, as well as the position of the target feature element in the first partial image, and the location of the target feature element is determined. The position in the panoramic image;

Based on the position of the target feature element in the panoramic image, distortion correction processing is performed on the panoramic image to obtain a corrected panoramic image.
The image processing method according to claim 1, characterized in that after obtaining the corrected panoramic image, the method includes:

Obtain the position of the target feature element in the corrected panoramic image;

Based on the position of the target feature element in the corrected panoramic image, obtain the next frame of panoramic image, and perform distortion correction processing on the next frame of panoramic image to obtain the corrected next frame of panoramic image. .
The image processing method according to claim 1, characterized in that, performing feature recognition processing on the first partial image to obtain image features of the first partial image includes:

Determine the size information of the first partial image;

Determine the size, sliding step length and sliding direction of the sliding window according to the size information of the first partial graph;

According to the size, sliding step size and sliding direction of the sliding window, segment the first partial image to obtain multiple sub-images;

Perform feature recognition processing on the sub-image to obtain image features of the sub-image;

According to the image characteristics of the sub-image, the image characteristics of the first partial image are determined.
The image processing method according to claim 3, wherein determining the image characteristics of the first local image according to the image characteristics of the sub-image includes:

Determine the image characteristics of the sub-image and the position of the sub-image in the first partial image;

The image characteristics of the first partial map are determined according to the image characteristics of the sub-image and the position of the sub-image in the first partial map.
The image processing method according to claim 3, characterized in that, performing feature recognition processing on the sub-image and determining image features of the sub-image includes:

determining the sub-image and the shape of the sub-image;

Compare the shape of the sub-image with the preset shape:

When the shape of the sub-image is the same as the preset shape, determine the sub-image to be the target sub-image;

When the shape of the sub-image is different from the preset shape, perform shape adjustment on the sub-image to obtain a target sub-image that conforms to the preset shape;

Perform feature recognition processing on the target sub-image to obtain image features of the target sub-image;

According to the image characteristics of the target sub-image, the image characteristics of the sub-image are determined.
The image processing method according to claim 5, characterized in that the method of determining the image characteristics of the sub-image includes:

Perform encoding and decoding processing on the target sub-image to obtain a mask image of the target sub-image;

Determine the image characteristics of the target sub-image according to the mask image of the target sub-image;

Determine a proportional relationship between the sub-image and the target sub-image;

Image features of the sub-image are determined based on the proportional relationship between the sub-image and the target sub-image.
The image processing method according to claim 3, characterized in that the sub-image includes a first sub-image and a second sub-image, and the first sub-image and the second sub-image include overlapping overlaps. area, wherein the image feature of the first sub-image in the overlapping area is the first image feature, and the image feature of the second sub-image in the overlapping area is the second image feature. sign; sign

Perform feature recognition processing on the first partial map, and the method of obtaining image features of the first partial map further includes:

When the first image feature overlaps with the second image feature, calculating a first feature intensity of the first image feature and a second feature intensity of the second image feature;

Compare the first characteristic intensity and the second characteristic intensity:

When the first feature intensity is greater than the second feature intensity, select the first image feature as the image feature of the first local map;

When the first feature intensity is less than the second feature intensity, select the second image feature as the image feature of the first local map;

When the first feature intensity is equal to the second feature intensity, the first image feature or the second image feature is selected as the image feature of the first local map.
The image processing method according to claim 1, wherein determining a target feature element based on a plurality of the feature elements and determining the position of the target feature element in the first partial image includes:

Determine pixel values of a plurality of the feature elements in the first local map;

Compare the pixel values of multiple feature elements in the first local image with a preset pixel threshold, and determine the first feature element whose pixel value is greater than the pixel threshold, and the feature A second feature element whose pixel value is less than the pixel threshold, wherein the first feature element is a target feature element;

Perform a binarization process on the pixel value of the first feature element and the pixel value of the second feature element to obtain the binarized first local map;

According to the binarized first local map, the position of the target feature element in the first local map is determined.
An image processing device, characterized in that it includes:

Acquisition unit, used to acquire panoramic images;

A segmentation processing unit configured to segment the panoramic image using a preset segmentation model to obtain a first partial image and a second partial image, wherein the degree of distortion of the first partial image is lower than that of the first partial image. The second partial picture;

A feature recognition processing unit, configured to perform feature recognition processing on the first partial map to obtain image features of the first partial map, wherein the image features of the first partial map are composed of a plurality of feature elements;

A first determination unit configured to determine a target feature element based on a plurality of the feature elements, and determine the position of the target feature element in the first partial map;

A second determination unit, configured to determine the positional relationship between the first partial image and the panoramic image and the position of the target feature element in the first partial image according to the preset segmentation model, and determine The position of the target feature element in the panoramic image;

A correction unit configured to perform distortion correction processing on the panoramic image based on the position of the target feature element in the panoramic image to obtain a corrected panoramic image.
An electronic device, characterized in that it includes a processor and a memory, the memory stores a plurality of instructions; the processor loads instructions from the memory to execute the method as described in any one of claims 1 to 8 Steps in image processing methods.
A computer-readable storage medium, characterized in that the computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for loading by a processor to perform the image processing described in any one of claims 1 to 8 steps in the method.