CN117058012A - Image processing method, device, electronic equipment and storage medium - Google Patents

Abstract

The application provides an image processing method, an image processing device, electronic equipment and a storage medium, wherein the method comprises the steps of obtaining a plane image, wherein the plane image is determined by projecting a first spherical image, and the first spherical image is an image obtained by shooting the content to be shot in 360 degrees; inputting the planar image into a trained saliency model, generating a second spherical image corresponding to the planar image through the saliency model based on the planar image, and determining spherical pixel point coordinates corresponding to the second spherical image; determining the plane pixel point coordinates corresponding to the plane image through the saliency model based on the second spherical image and the spherical pixel point coordinates; based on the coordinates of the plane pixel points, outputting a target image with a salient region corresponding to the plane image through a salient model, solving the technical problem that the visual field prediction of the plane image obtained after the projection of the spherical image is inaccurate in the prior art, and achieving the purpose of accurately determining the salient region of the plane image.

Description

Image processing methods, devices, electronic equipment and storage media

Technical field

The present application relates to the field of data processing technology, and in particular to an image processing method, device, electronic equipment and storage medium.

Background technique

With the commercialization of 5G and the rapid development of new multimedia technologies, virtual reality videos (such as panoramic videos, 360-degree videos) have become increasingly popular in recent years. Unlike traditional videos, virtual reality videos allow users to watch 360-degree video content, so transmitting virtual reality videos requires a large amount of bandwidth. However, users usually watch areas of greater interest when watching virtual reality videos. In order to reduce bandwidth consumption, the video area of the user's area of interest can be transmitted at high resolution, and the remaining video areas are transmitted at lower resolution. , Therefore, it is very important to accurately determine the video area that the user is interested in.

In the existing technology, virtual reality videos are generally projected from a spherical image into a flat image and then transmitted. However, the projected flat image usually suffers from image distortion and pixel distortion, and the image distortion ranges from the equatorial plane corresponding to the spherical image to the north and south poles. It will become more and more serious. The traditional model cannot accurately learn the characteristics of the plane image, which leads to the inaccurate salience area determined by the traditional model.

Contents of the invention

In view of this, the purpose of this application is to propose an image processing method, device, electronic device and storage medium to overcome all or part of the shortcomings of the prior art.

Based on the above purpose, the present application provides an image processing method, including: acquiring a plane image, wherein the plane image is determined by projection of a first spherical image, and the first spherical image is obtained by shooting a 360-degree shot of the content to be shot. The resulting image; input the planar image to a trained saliency model, generate a second spherical image corresponding to the planar image through the saliency model based on the planar image, and determine the second spherical shape The spherical pixel point coordinates corresponding to the image; based on the second spherical image and the spherical pixel point coordinates, determine the plane pixel point coordinates corresponding to the plane image through the saliency model; based on the plane pixel point coordinates, through The saliency model outputs a target image with a saliency area corresponding to the plane image.

Optionally, based on the second spherical image and the spherical pixel point coordinates, determining the plane pixel point coordinates corresponding to the plane image includes: determining any one of a plurality of tangent points corresponding to the second spherical image tangent point, and based on the tangent point, determine a tangent plane of preset size centered on the tangent point; based on the spherical pixel point coordinates, project the spherical pixel point coordinates onto the tangent plane to determine The projected coordinates of the spherical pixel point coordinates in the tangent plane; based on the projected coordinates, the plane pixel point coordinates are determined.

Optionally, based on the spherical pixel point coordinates, projecting the spherical pixel point coordinates to the tangent plane to determine the projected coordinates of the spherical pixel point coordinates in the tangent plane includes: Taking the tangent point as the center, establish a coordinate system corresponding to the tangent plane, divide the tangent plane into regions based on the coordinate system corresponding to the tangent plane, and calculate the unit coordinates of each region; determine the spherical pixel points The coordinates correspond to the area in the tangent plane; the projection coordinates are calculated based on the spherical pixel point coordinates and the unit coordinates of the area corresponding to the spherical pixel point coordinates.

Optionally, determining the plane pixel point coordinates based on the projection coordinates includes: determining the plane pixel point coordinates in the tangent plane through the following formula:

Among them, Γ _x (φ, θ) is the abscissa coordinate of the plane pixel point in the tangent plane, Γ _y (φ, θ) is the ordinate coordinate of the plane pixel point in the tangent plane, θ is the abscissa coordinate of the spherical pixel point, φ is the ordinate of the spherical pixel point, θ _γ is the abscissa in the projection coordinates, and φ _γ is the ordinate in the projection coordinates.

Optionally, the loss function used to train the saliency model is determined by the following formula: i=L _S-MSE (S, Q) + L _CC (S, Q) + L _KL (S, Q), Among them, ι is the loss function, L _S-MSE (S, Q) is the weight, L _CC (S, Q) represents the linear correlation relationship, L _KL (S, Q) represents the difference relationship, and S is the target image , Q is the labeled sample image.

Optionally, the linear correlation relationship is determined by the following formula: L _CC (S, Q) = 1-CC (S, Q), Among them, L _CC (S, Q) is the linear correlation relationship, CC (S, Q) is the linear correlation coefficient, Cov (S, Q) is the covariance, σ (S) is the standard deviation of the target image, σ(Q) is the standard deviation of the labeled sample image, S is the target image, and Q is the labeled sample image; the difference relationship is determined by the following formula: L _KL (S, Q) = KL(S,Q),/> Among them, L _KL (S, Q) is the difference relationship, KL (S, Q) is the difference between the target image and the annotated sample image in the case of information loss, and S is the target Image, Q is the labeled sample image, ε is the regularization constant, n is the total number of initial plane pixels, and i is the current pixel.

Optionally, the saliency model is a convolutional neural network model. Based on the plane pixel point coordinates, the saliency model outputs a target image with a saliency area corresponding to the plane image, including: response After determining that the plane image has a preset calibration image corresponding to it, the preset calibration image is input to the saliency model; based on the plane pixel point coordinates and the preset calibration image, using the saliency The convolutional layer of the model respectively extracts the first feature corresponding to the plane pixel point coordinates and the second feature corresponding to the preset calibration image; based on the first feature and the second feature, through the saliency model Output the target image.

Based on the same inventive concept, the present application also provides an image processing device, including: an acquisition module configured to acquire a planar image, wherein the planar image is determined by projection of a first spherical image, and the first spherical image is An image obtained by 360-degree shooting of the content to be photographed; the first determination module is configured to input the planar image to a trained saliency model, and based on the planar image, generate the same image as the saliency model through the saliency model a second spherical image corresponding to the plane image, and determine the spherical pixel point coordinates corresponding to the second spherical image; the second determination module is configured to determine based on the second spherical image and the spherical pixel point coordinates through the The saliency model determines the plane pixel point coordinates corresponding to the plane image; the output determination module is configured to output the target image with the saliency area corresponding to the plane image through the saliency model based on the plane pixel point coordinates. .

Based on the same inventive concept, this application also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable by the processor. When the processor executes the computer program, method as described above.

Based on the same inventive concept, the present application also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to cause the computer to execute the method as described above.

It can be seen from the above that the image processing method, device, electronic device and storage medium provided by this application include: acquiring a planar image, wherein the planar image is determined by projection of the first spherical image, and the The first spherical image is an image obtained by taking a 360-degree shot of the content to be shot; the plane image is input to the trained saliency model, and based on the plane image, the saliency model is used to generate the plane image corresponding to the plane image. The second spherical image is obtained, and the coordinates of the spherical pixel points corresponding to the second spherical image are determined, and the second spherical image is represented by specific numerical values in the form of coordinates, so as to establish an accurate association with the planar image later. Based on the second spherical image and the spherical pixel point coordinates, the plane pixel point coordinates corresponding to the plane image are determined through the saliency model, so that the correlation between the plane image and the second spherical image is more accurate. Based on the plane pixel point coordinates, the saliency model outputs the target image with the saliency area corresponding to the plane image. The saliency model can accurately extract the characteristics of the plane image, thereby accurately determining the plane image through the saliency model. The purpose of the salient area.

Description of the drawings

In order to more clearly illustrate the technical solutions in this application or related technologies, the drawings needed to be used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings in the following description are only for the purposes of this application. Embodiments, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without exerting creative efforts.

Figure 1 is a schematic flow chart of an image processing method according to an embodiment of the present application;

Figure 2 is a schematic structural diagram of an image processing device according to an embodiment of the present application;

Figure 3 is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present application more clear, the present application will be further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of this application should have the usual meanings understood by those with ordinary skills in the field to which this application belongs. The "first", "second" and similar words used in the embodiments of this application do not indicate any order, quantity or importance, but are only used to distinguish different components. Words such as "include" or "comprising" mean that the elements or things appearing before the word include the elements or things listed after the word and their equivalents, without excluding other elements or things. Words such as "connected" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right", etc. are only used to express relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

As mentioned in the background art section, with the commercialization of 5G and the rapid development of new multimedia technologies, virtual reality videos (such as panoramic videos, 360-degree videos) have become increasingly popular in recent years. Virtual reality video allows users to watch 360-degree video content. Therefore, virtual reality video consumes different bandwidth than traditional video. For example, a 4k panoramic video is transmitted to the client and allows users to watch it in all directions. The required data rate is 400Mb/s, while traditional 4k video streaming only requires a data rate of 25Mb/s. Users usually watch areas of interest when watching virtual reality videos, and there may also be cases where users can only see 20% to 30% of the full screen due to the field of view limitation of the head-mounted device. Content, the video content in other areas transmitted is not watched by the user and is completely wasted. Therefore, it is important to accurately determine the video area that is of interest to the user.

Virtual reality videos are generally projected from a spherical image into a flat image and then transmitted. However, the projected flat image usually has image distortion and pixel distortion, and the image distortion will become more and more serious from the equatorial plane corresponding to the spherical image to the north and south poles. , the traditional model cannot accurately learn the characteristics of the plane image, which leads to the inaccurate salience area determined by the traditional model.

In view of this, the embodiment of the present application proposes an image processing method. Referring to Figure 1, it includes the following steps:

Step 101: Obtain a planar image, wherein the planar image is determined by projection of a first spherical image, and the first spherical image is an image obtained by taking a 360-degree shot of the content to be photographed.

In this step, in order to improve the user's viewing experience, the first spherical image is usually projected into a flat image, and the above projection step is usually completed during the image capturing stage. However, the flat image projected from the first spherical image usually suffers from image distortion and pixel distortion. A planar image determined by projection of the first spherical image is obtained. The sources of obtaining the planar image are diverse. For example, the planar image can be obtained from a panoramic video, or a panoramic image captured by a user can be determined as a planar image.

Step 102: Input the plane image into a trained saliency model, generate a second spherical image corresponding to the plane image through the saliency model based on the plane image, and determine the second spherical image The corresponding spherical pixel coordinates.

In this step, the traditional saliency model can determine the saliency area corresponding to the image by extracting the features of the input image. The saliency area is the area where the user's line of sight is focused. However, due to image distortion and pixel distortion in flat images, the traditional saliency model cannot accurately extract the features of flat images. In order to solve the above problems, the saliency model used in this embodiment is a spherical convolutional neural network model. The spherical convolutional neural network model can divide the input image into multiple areas, and obtain the model through model training in each area. Different weights are used to extract the features of each region separately. When the trained spherical convolutional neural network model extracts features from spherical images, the output results will have high accuracy. Therefore, it is necessary to associate the flat image with the spherical image that the spherical convolutional neural network model can process. A second spherical image corresponding to the planar image is generated through the saliency model, where the relevant parameters of the second spherical image can be set according to the user's needs. For example, based on the user's needs, the radius of the second spherical image can be set to the planar image. half the length. It is also necessary to determine the coordinates of the second spherical image. For example, the coordinates of the spherical pixel points corresponding to the second spherical image can be determined by establishing a coordinate system. Represent the second spherical image with specific numerical values in the form of coordinates so that it can be accurately associated with the flat image later.

It should be noted that the saliency model used in this embodiment may be a spherical convolutional neural network model with the following structure. The spherical convolutional neural network model consists of an encoder (Encoder) and a decoder (Decoder). The Encoder serves as the backbone feature extraction network and can obtain feature layers one after another through the contraction path. The Encoder includes four spherical convolution layers and four There are three spherical pooling operations between the spherical convolution layer and the ReLU activation layer. As an enhanced feature extraction network, Decoder uses the preliminary effective feature layer obtained by the backbone feature extraction network to perform feature fusion through the expansion path to obtain the final enhanced feature. The Decoder includes three spherical convolution layers, followed by the corresponding ReLU activation function layer. There are three upsampling layers between the spherical convolution layer and the ReLU activation layer.

Step 103: Based on the second spherical image and the spherical pixel point coordinates, determine the plane pixel point coordinates corresponding to the plane image through the saliency model.

In this step, the plane pixel coordinates corresponding to the plane image are determined through the second spherical image that can be processed by the spherical convolutional neural network model and the spherical pixel coordinates corresponding to the second spherical image. The second spherical image and the planar image are numerically associated, and the association between the planar image and the second spherical image is digitized, so that the association between the planar image and the second spherical image is more accurate.

Step 104: Based on the plane pixel coordinates, use the saliency model to output a target image with a saliency area corresponding to the plane image.

In this step, the features of the plane image are extracted through a saliency model, where the saliency model is a spherical convolutional neural network model. The spherical convolutional neural network model can identify the equatorial plane and the north and south poles corresponding to the spherical image. Therefore, after establishing an association between the spherical image and the planar image, the spherical convolutional neural network model can also identify the first spherical image in the planar image. Corresponding equatorial plane to the north and south poles. Since the image distortion of the plane image will become more and more serious from the equatorial plane corresponding to the first spherical image to the north and south poles, the spherical convolutional neural network model corresponds to the coordinates near the polar areas of the first spherical image that exist in the plane image. The weight of is relatively small, and the weight corresponding to the coordinates near the equatorial plane area of the first spherical image existing in the plane image is relatively large. The coordinates of the undistorted area in the plane image can be highlighted through the saliency model. The saliency model can accurately extract the features of the plane image, thereby achieving the purpose of accurately determining the saliency area of the plane image through the saliency model. Subsequently, the salient area of the plane image can be transmitted at high resolution, and the remaining areas at lower resolution, which not only improves the user experience, but also reduces the consumption of communication bandwidth.

Through the above solution, a planar image is obtained, wherein the planar image is determined by projection of a first spherical image, which is an image obtained by taking a 360-degree shot of the content to be shot; the planar image is input to The trained saliency model, based on the plane image, uses the saliency model to generate a second spherical image corresponding to the plane image, and determines the spherical pixel point coordinates corresponding to the second spherical image, and converts the second spherical image into The image is represented by specific numerical values in the form of coordinates so that it can be accurately associated with the plane image later. Based on the second spherical image and the spherical pixel point coordinates, the plane pixel point coordinates corresponding to the plane image are determined through the saliency model, so that the correlation between the plane image and the second spherical image is more accurate. Based on the plane pixel point coordinates, the saliency model outputs the target image with the saliency area corresponding to the plane image. The saliency model can accurately extract the characteristics of the plane image, thereby accurately determining the plane image through the saliency model. The purpose of the salient area.

In some embodiments, based on the second spherical image and the spherical pixel point coordinates, determining the plane pixel point coordinates corresponding to the plane image includes: determining a plurality of tangent points corresponding to the second spherical image. Any tangent point, and based on the tangent point, determine a tangent plane of a preset size centered on the tangent point; based on the spherical pixel point coordinates, project the spherical pixel point coordinates to the tangent plane, To determine the projected coordinates of the spherical pixel point coordinates in the tangent plane; and determine the plane pixel point coordinates based on the projected coordinates.

In this embodiment, in order to establish a relationship between the spherical pixel point coordinates and the plane pixel point coordinates, other planes may be used to display the spherical pixel point coordinates and the plane pixel point coordinates. Since the second spherical image has a tangent point, the tangent plane can be established with the help of the tangent point of the second spherical image, where the preset size can be determined according to actual needs. For example, in order to quickly determine the size of the tangent plane, the plane image can be The dimensions are determined as the dimensions of the tangent plane. Project the spherical pixels to the tangent plane to obtain the projection coordinates. First establish the relationship between the tangent plane and the spherical pixels. Then determine the plane pixel coordinates through the projected coordinates, and establish the relationship between the plane pixel coordinates and the projected coordinates. Since the trained spherical convolutional neural network model will have higher accuracy when extracting features from a spherical image, therefore, after establishing the numerical correlation between the flat image and the second spherical image, the trained spherical convolutional neural network model will The spherical convolutional neural network model extracts the features of plane pixels, and the output results also have high accuracy.

In some embodiments, projecting the spherical pixel coordinates to the tangent plane based on the spherical pixel coordinates to determine the projected coordinates of the spherical pixel coordinates in the tangent plane includes: Taking the tangent point as the center, establish a coordinate system corresponding to the tangent plane, divide the tangent plane into regions based on the coordinate system corresponding to the tangent plane, and calculate the unit coordinates of each region; determine the spherical surface The area corresponding to the pixel point coordinates in the tangent plane; the projection coordinates are calculated based on the spherical pixel point coordinates and the unit coordinates of the area corresponding to the spherical pixel point coordinates.

In this embodiment, since the second spherical image and the planar image need to be associated with other planes, the projection coordinates of the spherical pixel point coordinates of the second spherical image in the tangent plane are first determined. The tangent plane is centered on the tangent point, and the coordinate system corresponding to the tangent plane is established. In order to improve the efficiency of determining the projection coordinates, the tangent plane can be divided into regions. For example, for the convenience of description, it can be regarded as dividing the horizontal axis with the center as the dividing line, the horizontal axis at the left part of the center is the first horizontal axis, and the horizontal axis at the right part of the center is the second horizontal axis; it can be regarded as In order to divide the vertical axis with the center as the dividing line, the vertical axis of the upper part of the center is the first vertical axis, the vertical axis of the lower part of the center is the second vertical axis, and the center of the tangent plane is the center of the spherical pixel point coordinates. . Divide the tangent plane into eight regions, where the first region is the first horizontal axis, and the unit coordinate of the first region is p _{γ (-1,0)} = (-tanΔ _θ ,0); the second region is the second The horizontal axis, the unit coordinate of the second region is p _{γ (1,0)} = (tanΔ _θ ,0); the third region is the first vertical axis, and the unit coordinate of the third region is p _{γ (0,1)} = ( 0,tanΔ _φ ); the fourth region is the second vertical axis, and the unit coordinate of the fourth region is p _γ(0,-1) = (0,-tanΔ _φ ); the fifth region is the first horizontal axis and the first In the area enclosed by the vertical axis, the unit coordinates of the fifth area are p _γ(-1,+1) =(-tanΔ _θ ,+secΔ _θ tanΔ _φ ); the sixth area is the second horizontal axis and the first vertical axis. _{In the} area _{enclosed by} The unit coordinates of the seventh area are p _{γ (+1,-1)} = (+tanΔ _θ ,-secΔ _θ tanΔ _φ ); the eighth area is the area surrounded by the second vertical axis and the second horizontal axis. The unit coordinates of the area are p _{γ (-1,-1)} = (-tanΔ _θ ,-secΔ _θ tanΔ _φ ); where Δ _θ and Δ _φ are the preset step sizes.

Match the positive and negative signs corresponding to the coordinates of the spherical pixel points with the positive and negative signs corresponding to the unit coordinates in each area, and in response to determining the positive and negative signs corresponding to the coordinates of the spherical pixel points and the positive and negative signs corresponding to the unit coordinates of one of the areas. The symbols are the same, and the area is determined as the area corresponding to the spherical pixel point coordinates in the tangent plane. For example, in response to determining that the abscissa of the spherical pixel point is a positive number and the ordinate is a negative number, the unit coordinates of the spherical pixel with a positive abscissa and a negative ordinate are searched for in the unit coordinates corresponding to all areas, and then the spherical pixel is determined. The point coordinates are in the seventh area in the tangent plane. Calculate the projection coordinates based on the spherical pixel coordinates and the unit coordinates of the area corresponding to the spherical pixel coordinates. For example, when the spherical pixel coordinates are (2, -2), you need to use values without positive or negative signs. Multiply the unit coordinates corresponding to the seventh area to determine the projected coordinates, which are (+2tanΔ _θ ,-2secΔ _θ tanΔ _φ ). The relationship between the second spherical image and the tangent plane is digitized in the form of coordinates, so that the relationship between the second spherical image and the tangent plane can be accurately determined.

In some embodiments, determining the plane pixel point coordinates based on the projection coordinates includes: determining the plane pixel point coordinates in the tangent plane through the following formula:

Among them, Γ _x (φ, θ) is the abscissa coordinate of the plane pixel point in the tangent plane, Γ _y (φ, θ) is the ordinate coordinate of the plane pixel point in the tangent plane, θ is the abscissa coordinate of the spherical pixel point, φ is the ordinate of the spherical pixel point, θ _γ is the abscissa in the projection coordinates, and φ _γ is the ordinate in the projection coordinates.

In this embodiment, the plane image needs to be associated with the second spherical image through the tangent plane. Therefore, based on the spherical pixel point coordinates and projection coordinates corresponding to the second spherical image, the plane pixel point coordinates corresponding to the plane image are determined in the tangent plane. . The coordinates of the plane pixels can be determined through formulas. With the help of the tangent plane, the relationship between the second spherical image and the plane image can be digitized in the form of coordinates, and then the relationship between the second spherical image and the plane image can be accurately determined, making subsequent significance The model can also accurately extract features of plane pixels.

In some embodiments, the loss function used to train the saliency model is determined by: i = L _{S - MSE} (S, Q) + L _CC (S, Q) + L _KL (S, Q ), where ι is the loss function, L _S-MSE (S, Q) is the weight, L _CC (S, Q) represents the linear correlation relationship, L _KL (S, Q) represents the difference relationship, and S is the The target image, Q is the annotated sample image.

In this embodiment, the saliency model is a spherical convolutional neural network model. The spherical convolutional neural network model can divide the input image into multiple areas and learn the characteristics of each area by setting different weights. Therefore, the weight is added to the calculation of the loss function, and then the training direction of the saliency model is guided based on the weight corresponding to each region. In order to more fully train the saliency model, linear correlation and difference relationships are also introduced to guide the training direction of the saliency model. Among them, the linear correlation is used to measure the linear correlation coefficient between the target image and the labeled sample image. The larger the correlation coefficient, the closer the two images are; the difference relationship measures the difference between the target image and the annotated sample image in the case of information loss. The smaller the difference value, the smaller the difference between the two images. Through the above loss function, the training direction of the saliency model is made more accurate.

In some embodiments, the linear correlation relationship is determined by the following formula: L _CC (S, Q) = 1 - CC (S, Q), Among them, L _CC (S, Q) is the linear correlation relationship, CC (S, Q) is the linear correlation coefficient, Cov (S, Q) is the covariance, σ (S) is the standard deviation of the target image, σ(Q) is the standard deviation of the labeled sample image, S is the target image, and Q is the labeled sample image; the difference relationship is determined by the following formula: L _KL (S, Q) = KL(S,Q),/> Among them, L _KL (S, Q) is the difference relationship, KL (S, Q) is the difference between the target image and the annotated sample image in the case of information loss, and S is the target Image, Q is the labeled sample image, ε is the regularization constant, n is the total number of initial plane pixels, and i is the current pixel.

In this embodiment, the linear correlation relationship and the difference relationship need to be determined respectively based on the formula, and the above two abstract correlation relationships are digitized to make the determination of the above two relationships more accurate, thereby making the training direction of the saliency model more accurate.

In some embodiments, the saliency model is a convolutional neural network model, and based on the plane pixel coordinates, the saliency model outputs a target image with a saliency area corresponding to the plane image, including : In response to determining that the plane image has a preset calibration image corresponding to it, input the preset calibration image to the saliency model; based on the plane pixel point coordinates and the preset calibration image, use the The convolutional layer of the saliency model respectively extracts the first feature corresponding to the plane pixel point coordinates and the second feature corresponding to the preset calibration image; based on the first feature and the second feature, through the saliency The sexual model outputs the target image.

In this embodiment, considering that there is a situation where the user may continuously view a group of related planar images, therefore, the related planar images are marked. For example, correlation may mean that the plane image and other plane images are obtained from the same source. For example, the plane images that are correlated come from the same panoramic video. Before inputting the plane image into the saliency model, it is detected whether other plane images with the same mark as the plane image to be input have corresponding target images output through the saliency model. In response to determining that the above target image exists, one of the target images can be It is input to the saliency model together with the plane image to be input. The above target image is the preset calibration image. The preset calibration image can correct the output result of the currently input plane image through the saliency model, making the saliency model efficient and accurate. the output target image. In order to make the output result of the above target image more accurate, the above target image can be the target image output by the saliency model at the moment before the plane image to be input is input to the saliency model. It should be noted that the saliency model in this embodiment can also be embedded into other models according to actual needs. For example, in order to further improve the user's viewing effect, the saliency model can be embedded into the FoV model, where FoV The model can accurately represent the image that the human eye or machine vision system can see, and can also analyze the image.

It should be noted that the method in the embodiment of the present application can be executed by a single device, such as a computer or server. The method of this embodiment can also be applied in a distributed scenario, and is completed by multiple devices cooperating with each other. In this distributed scenario, one of the multiple devices can only execute one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete all the steps. method described.

It should be noted that some embodiments of the present application have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the above-described embodiments and still achieve the desired results. Additionally, the processes depicted in the figures do not necessarily require the specific order shown, or sequential order, to achieve desirable results. Multitasking and parallel processing are also possible or may be advantageous in certain implementations.

Based on the same inventive concept and corresponding to any of the above embodiments, this application also provides an image processing device.

Referring to Figure 2, the image processing device includes:

The acquisition module 10 is configured to acquire a planar image, wherein the planar image is determined by projection of a first spherical image, and the first spherical image is an image obtained by taking a 360-degree shot of the content to be shot.

The first determination module 20 is configured to input the planar image to a trained saliency model, generate a second spherical image corresponding to the planar image through the saliency model based on the planar image, and determine The spherical pixel coordinates corresponding to the second spherical image.

The second determination module 30 is configured to determine the plane pixel coordinates corresponding to the plane image through the saliency model based on the second spherical image and the spherical pixel coordinates.

The output module 40 is configured to output a target image with a salient area corresponding to the plane image through the saliency model based on the plane pixel point coordinates.

Through the above device, a planar image is obtained, wherein the planar image is determined by projection of a first spherical image, and the first spherical image is an image obtained by taking a 360-degree shot of the content to be shot; the planar image is input to The trained saliency model, based on the plane image, uses the saliency model to generate a second spherical image corresponding to the plane image, and determines the spherical pixel point coordinates corresponding to the second spherical image, and converts the second spherical image into The image is represented by specific numerical values in the form of coordinates so that it can be accurately associated with the plane image later. Based on the second spherical image and the spherical pixel point coordinates, the plane pixel point coordinates corresponding to the plane image are determined through the saliency model, so that the correlation between the plane image and the second spherical image is more accurate. Based on the plane pixel point coordinates, the saliency model outputs the target image with the saliency area corresponding to the plane image. The saliency model can accurately extract the characteristics of the plane image, thereby accurately determining the plane image through the saliency model. The purpose of the salient area.

In some embodiments, the second determination module 30 is further configured to determine any one of the multiple tangent points corresponding to the second spherical image, and based on the tangent point, determine the tangent point with the tangent point. A tangent plane with a preset size at the center; based on the spherical pixel point coordinates, project the spherical pixel point coordinates to the tangent plane to determine the projection coordinates of the spherical pixel point coordinates in the tangent plane ; Based on the projection coordinates, determine the coordinates of the plane pixel points.

In some embodiments, the second determination module 30 is further configured to establish a coordinate system corresponding to the tangent plane with the tangent point as the center, and calculate the tangent plane based on the coordinate system corresponding to the tangent plane. The plane is divided into areas and the unit coordinates of each area are calculated; the area corresponding to the spherical pixel point coordinates in the tangent plane is determined; the unit of the area corresponding to the spherical pixel point coordinates is based on the spherical pixel point coordinates and the spherical pixel point coordinates. coordinates, calculate the projected coordinates.

In some embodiments, the second determination module 30 is further configured to determine the plane pixel point coordinates based on the projection coordinates, including: determining the plane pixel point coordinates in the tangent plane through the following formula : Among them, Γ _x (φ, θ) is the abscissa coordinate of the plane pixel point in the tangent plane, Γ _y (φ, θ) is the ordinate coordinate of the plane pixel point in the tangent plane, θ is the abscissa coordinate of the spherical pixel point, φ is the ordinate of the spherical pixel point, θ _γ is the abscissa in the projection coordinates, and φ _γ is the ordinate in the projection coordinates.

In some embodiments, a third determination module is further included, and the third determination module is further configured so that the loss function used to train the saliency model is determined by the following formula: i=L _S-MSE (S, Q)+L _CC (S,Q)+L _KL (S,Q), where ι is the loss function, L _S-MSE (S,Q) is the weight, and L _CC (S,Q) represents linear correlation Relationship, L _KL (S, Q) represents the difference relationship, S is the target image, and Q is the labeled sample image.

In some embodiments, the third determination module is further configured to determine the linear correlation relationship through the following formula: L _CC (S, Q) = 1-CC (S, Q), Among them, L _CC (S, Q) is the linear correlation relationship, CC (S, Q) is the linear correlation coefficient, Cov (S, Q) is the covariance, σ (S) is the standard deviation of the target image, σ(Q) is the standard deviation of the labeled sample image, S is the target image, and Q is the labeled sample image; the difference relationship is determined by the following formula: L _KL (S, Q) = KL(S,Q),/> Among them, L _KL (S, Q) is the difference relationship, KL (S, Q) is the difference between the target image and the annotated sample image in the case of information loss, and S is the target Image, Q is the labeled sample image, ε is the regularization constant, n is the total number of initial plane pixels, and i is the current pixel.

In some embodiments, the output module 40 is further configured such that the saliency model is a convolutional neural network model, and in response to determining that the plane image has a preset calibration image corresponding to it, the preset calibration The image is input to the saliency model; based on the plane pixel point coordinates and the preset calibration image, the first feature corresponding to the plane pixel point coordinates and the first feature corresponding to the plane pixel point coordinates are extracted using the convolution layer of the saliency model. A second feature corresponding to the calibration image is preset; based on the first feature and the second feature, the target image is output through the saliency model.

For the convenience of description, when describing the above device, the functions are divided into various modules and described separately. Of course, when implementing this application, the functions of each module can be implemented in the same or multiple software and/or hardware.

The devices of the above embodiments are used to implement the corresponding image processing methods in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be described again here.

Based on the same inventive concept, corresponding to any of the above embodiments, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor When the program is executed, the image processing method described in any of the above embodiments is implemented.

FIG. 3 shows a more specific hardware structure diagram of an electronic device provided by this embodiment. The device may include: a processor 1010, a memory 1020, an input/output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input/output interface 1030 and the communication interface 1040 implement communication connections between each other within the device through the bus 1050.

The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit, central processing unit), a microprocessor, an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute related tasks. program to implement the technical solutions provided by the embodiments of this specification.

The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store operating systems and other application programs. When implementing the technical solutions provided by the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 1020 and called and executed by the processor 1010 .

The input/output interface 1030 is used to connect the input/output module to realize information input and output. The input/output module can be configured in the device as a component (not shown in the figure), or can be externally connected to the device to provide corresponding functions. Input devices can include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices can include monitors, speakers, vibrators, indicator lights, etc.

The communication interface 1040 is used to connect a communication module (not shown in the figure) to realize communication interaction between this device and other devices. The communication module can realize communication through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

Bus 1050 includes a path that carries information between various components of the device (eg, processor 1010, memory 1020, input/output interface 1030, and communication interface 1040).

It should be noted that although the above device only shows the processor 1010, the memory 1020, the input/output interface 1030, the communication interface 1040 and the bus 1050, during specific implementation, the device may also include necessary components for normal operation. Other components. In addition, those skilled in the art can understand that the above-mentioned device may only include components necessary to implement the embodiments of this specification, and does not necessarily include all components shown in the drawings.

The electronic devices of the above embodiments are used to implement the corresponding image processing methods in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be described again here.

Based on the same inventive concept, corresponding to any of the above embodiment methods, the present application also provides a non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions use To cause the computer to execute the image processing method described in any of the above embodiments.

The computer-readable media in this embodiment include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information may be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), and read-only memory. (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, Magnetic tape cassettes, tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium can be used to store information that can be accessed by a computing device.

The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the image processing method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be described again here.

Those of ordinary skill in the art should understand that the discussion of any above embodiments is only illustrative, and is not intended to imply that the scope of the present application (including the claims) is limited to these examples; under the spirit of the present application, the above embodiments or Technical features in different embodiments can also be combined, steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

In addition, to simplify illustration and discussion, and so as not to obscure the embodiments of the present application, well-known power supplies/power supplies with integrated circuit (IC) chips and other components may or may not be shown in the provided figures. Ground connection. Furthermore, devices may be shown in block diagram form in order to avoid obscuring the embodiments of the present application, and this also takes into account the fact that details regarding the implementation of these block diagram devices are highly dependent on the implementation of the embodiments of the present application. platform (i.e., these details should be well within the understanding of those skilled in the art). Where specific details (eg, circuits) are set forth to describe exemplary embodiments of the present application, it will be apparent to those skilled in the art that construction may be accomplished without these specific details or with changes in these specific details. The embodiments of this application are implemented below. Accordingly, these descriptions should be considered illustrative rather than restrictive.

Although the present application has been described in conjunction with specific embodiments thereof, many alternatives, modifications and variations of these embodiments will be apparent to those of ordinary skill in the art from the foregoing description. For example, other memory architectures such as dynamic RAM (DRAM) may use the discussed embodiments.

The present embodiments are intended to embrace all such alternatives, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of this application shall be included in the protection scope of this application.

Claims (10)

1. An image processing method, comprising:

acquiring a plane image, wherein the plane image is determined by projecting a first spherical image, and the first spherical image is an image obtained by shooting 360 degrees of content to be shot;

inputting the planar image into a trained saliency model, generating a second spherical image corresponding to the planar image through the saliency model, and determining spherical pixel point coordinates corresponding to the second spherical image;

determining a plane pixel point coordinate corresponding to the plane image through the saliency model based on the second spherical image and the spherical pixel point coordinate;

and outputting a target image with a salient region corresponding to the planar image through the salient model based on the planar pixel point coordinates.

2. The method of claim 1, wherein determining planar pixel point coordinates corresponding to the planar image based on the second spherical image and the spherical pixel point coordinates comprises:

Determining any one of a plurality of tangent points corresponding to the second spherical image, and determining a tangent plane with a preset size taking the tangent point as a center based on the tangent point;

projecting the spherical pixel point coordinates to the tangent plane based on the spherical pixel point coordinates to determine projection coordinates of the spherical pixel point coordinates in the tangent plane;

and determining the plane pixel point coordinates based on the projection coordinates.

3. The method of claim 2, wherein the projecting the spherical pixel point coordinates to the tangent plane based on the spherical pixel point coordinates to determine projection coordinates of the spherical pixel point coordinates in the tangent plane comprises:

establishing a coordinate system corresponding to the tangent plane by taking the tangent point as the center, dividing the tangent plane into areas based on the coordinate system corresponding to the tangent plane, and calculating unit coordinates of each area;

determining a corresponding region of the spherical pixel point coordinates in the tangential plane;

and calculating the projection coordinates based on the spherical pixel point coordinates and the unit coordinates of the region corresponding to the spherical pixel point coordinates.

4. The method of claim 2, wherein the determining the planar pixel point coordinates based on the projection coordinates comprises:

Determining plane pixel point coordinates in the tangent plane by the following formula:

wherein Γ is _x (phi, theta) is the abscissa, Γ, of the planar pixel point in the tangent plane _y (phi, theta) is the ordinate of the plane pixel point in the tangential plane, theta is the abscissa of the spherical pixel point, phi is the ordinate of the spherical pixel point, theta _γ Phi is the abscissa in the projection coordinates _γ Is the ordinate of the projected coordinates.

5. The method of claim 1, wherein the loss function for training the saliency model is determined by:

ι＝L _S-MSE (S,Q)+L _CC (S,Q)+L _KL (S,Q)，

wherein iota is the loss function, L _S-MSE (S, Q) is a weight, L _CC (S, Q) represents a linear correlation, L _KL (S, Q) represents the difference relation, S is the target image, and Q is the marked sample image.

6. The method of claim 5, wherein the step of determining the position of the probe is performed,

the linear correlation is determined by the following formula:

L _CC (S,Q)＝1-CC(S,Q)，

wherein L is _CC (S, Q) is the linear correlation, CC (S, Q) is a linear correlation coefficient, cov (S, Q) is a covariance, σ (S) is a standard deviation of the target image, σ (Q) is a standard deviation of the noted sample image, S is the target image, and Q is the noted sample image;

The difference relationship is determined by the following formula:

L _KL (S,Q)＝KL(S,Q)，

wherein L is _KL (S, Q) is the difference relation, KL (S, Q) is the difference between the target image and the marked sample image under the condition of information loss, S is the target image, Q is the marked sample image, epsilon is a regularization constant, n is the total number of initial plane pixel points, and i is the current pixel point.

7. The method of claim 1, wherein the saliency model is a convolutional neural network model,

outputting, based on the planar pixel point coordinates, a target image with a salient region corresponding to the planar image through the salient model, including:

in response to determining that the planar image has a preset calibration image corresponding to the planar image, inputting the preset calibration image into the significance model;

based on the plane pixel point coordinates and the preset calibration image, respectively extracting first features corresponding to the plane pixel point coordinates and second features corresponding to the preset calibration image by using a convolution layer of the saliency model;

outputting the target image through the saliency model based on the first feature and the second feature.

8. An image processing apparatus, comprising:

the acquisition module is configured to acquire a plane image, wherein the plane image is determined by projection of a first spherical image, and the first spherical image is an image obtained by shooting 360 degrees of content to be shot;

a first determination module configured to input the planar image into a trained saliency model, generate a second spherical image corresponding to the planar image from the saliency model based on the planar image, and determine spherical pixel point coordinates corresponding to the second spherical image;

the second determining module is configured to determine the plane pixel point coordinates corresponding to the plane image through the saliency model based on the second spherical image and the spherical pixel point coordinates;

and the output module is configured to output a target image with a salient region corresponding to the planar image through the salient model based on the planar pixel point coordinates.

9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of any one of claims 1 to 7 when the program is executed by the processor.

10. A non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the method of any one of claims 1 to 7.