TWI778895B - Saliency detection method and model training method, equipment and computer readable storage medium - Google Patents

Abstract

The embodiment of the application discloses a saliency detection method and a training method, equipment and computer-readable storage medium of the saliency detection model. The training method of the saliency detection model includes: acquiring at least one sample image, wherein, at least one sample image Including the target sample image belonging to the preset image type; filtering the target sample image based on the missing contour of the salient area in the target sample image; using the saliency detection model to detect the filtered sample image to obtain the predicted location information of the salient region in the sample image; adjusting the parameters of the saliency detection model based on the label location information and predicted location information of the saliency area of the sample image. In the above solution, the saliency detection model is trained by filtering the sample images and then using the filtered sample images, which can improve the accuracy of the output results of the saliency detection model.

Description

Saliency detection method and model training method, device and computer-readable storage medium

The present invention relates to the technical field of image processing, and in particular, to a saliency detection method and its model training method, device and computer-readable storage medium.

At present, in the process of training the model, the sample images of certain data are simply obtained from the sample image database, and the model is directly trained using these sample images. However, some sample images themselves have certain defects. If these sample images are used to train the model, the accuracy of the results obtained by the trained model processing the images will be low.

Embodiments of the present invention provide at least a saliency detection method and a model training method, device, and computer-readable storage medium.

Embodiments of the present invention provide training of a saliency detection model The method includes: acquiring at least one sample image, wherein the at least one sample image includes a target sample image belonging to a preset image type; Filter the image; use the saliency detection model to detect the filtered sample image to obtain the predicted position information about the saliency area in the sample image; based on the labeled position information and predicted position of the saliency area in the sample image information, adjust the parameters of the saliency detection model.

Therefore, the target sample image is filtered according to the missing outline of the salient region of the obtained target sample image of the preset image type, so that the salient region in the retained sample image is relatively complete, Further, the saliency detection model is trained by using the retained sample images with higher quality, which can make the result of subsequent image detection by the trained saliency detection model more accurate.

In some embodiments, filtering the target sample image based on the missing contour of the salient region in the target sample image includes: filling the contour of the salient region in the target sample image to obtain a filled sample image; Obtain the difference between the saliency area in the filled sample image and the target sample image; filter the target sample image if the difference meets the preset requirements.

Therefore, by filtering the sample images according to the situation of missing contours, the quality of the contours of the salient regions in the remaining sample images is better. In addition, the missing contour of the salient region can be quickly acquired by acquiring the difference between the saliency region in the fill-in sample image and the target sample image.

In some embodiments, the preset requirement is that the difference is greater than the preset difference value; filling the contours of the salient regions in the target sample image to obtain the filled sample image includes: performing a closing operation on the target sample image to obtain the filled sample image Sample image; obtain the difference between the saliency area in the filled sample image and the target sample image, including: obtaining the first area of the saliency area in the filled sample image, and the first area of the saliency area in the target sample image. Two areas; the difference between the first area and the second area is determined as the difference.

Therefore, if there is a large gap in the contour of the salient region in the target sample image, there may be a large difference in the area of the salient region before and after filling. Whether the contours of salient regions in the target sample image are missing.

In some embodiments, after filtering the target sample image based on the missing contour of the salient region in the target sample image, the method further includes: obtaining the target sample based on the position information of the salient region filling in the sample image Annotated location information about salient regions in the image.

Therefore, by filling in the position information of the salient region of the sample image, and determining the marked position information of the target sample image about the salient region, the integrity of the salient region can be guaranteed.

In some embodiments, the at least one sample image includes multiple image types.

Therefore, the saliency detection model is trained by using sample images of various image types, so that the trained saliency detection model can perform image processing on various types of images, thereby improving the applicability of the saliency detection model. .

In some embodiments, the plurality of image types include at least two of images captured on real objects, hand-drawn drawings, and cartoon drawings.

Therefore, by using the sample images corresponding to common image types for training the image processing model, the image processing model obtained by training is more suitable for daily life or work.

In some embodiments, adjusting the parameters of the saliency detection model based on the labeled location information and predicted location information about the saliency region of the sample image, including: based on the labeled location information and predicted location information about the saliency region of the sample image, Obtain the first loss of each image element in the sample image; weight the first loss of each image element in the sample image to obtain the second loss of the sample image; adjust the parameters of the saliency detection model based on the second loss.

Therefore, by weighting the first loss of each primitive, it is more accurate to use the weighted second loss to adjust the parameters of the saliency detection model.

In some embodiments, the weight of the first loss of the primitive is related to the boundary distance of the primitive, the boundary distance of the primitive is the distance between the primitive and the boundary of the real saliency area, and the real saliency area is the sample image The saliency area defined by the label position information in the .

Therefore, by determining the weight according to the boundary distance of the primitives, it is more accurate to use the weighted second loss to adjust the parameters of the saliency detection model.

In some embodiments, the smaller the boundary distance of the primitive, the greater the weight of the primitive's first loss.

Therefore, the boundary distance of the primitive is proportional to the weight of the first loss of the primitive Negative correlation, making the resulting second loss more accurate.

In some embodiments, the saliency detection model includes at least one of the following: the saliency detection model is the network structure of MobileNetV3, and the saliency detection model includes a feature extraction sub-network, a first detection sub-network and a second detection sub-network network; use the saliency detection model to detect the filtered sample images, and obtain the predicted location information about the salient regions in the sample images, including: using the feature extraction sub-network to perform feature extraction on the sample images to obtain The feature map corresponding to the sample image; use the first detection sub-network to perform initial detection on the feature map to obtain the initial position information about the salient region in the sample image; fuse the feature map and the initial position information to obtain a fusion result; The fusion result is finally detected by the second detection sub-network, and the predicted position information of the sample image is obtained.

Therefore, due to the simple network structure of MobileNetV3, by using the network structure of MobileNetV3, the detection efficiency can be accelerated, and devices with smaller processing capabilities can also use the saliency detection model to achieve saliency detection; After the detection sub-network performs initial detection on the feature map, the second detection sub-network is used to perform final detection on the initial detection result, which can improve the detection accuracy.

In some embodiments, before using the saliency detection model to detect the filtered sample image to obtain the predicted position information about the saliency region in the sample image, the method further includes: Carry out data enhancement; wherein, the data enhancement method includes filling the background area except the salient area in the sample image.

Therefore, by enhancing the data of the sample image, the display can be improved. Applicability of the relevance detection model.

An embodiment of the present invention provides a saliency detection method, including: acquiring an image to be processed; processing the image to be processed by using a saliency detection model to obtain predicted location information about a saliency area in the content of the image to be processed, wherein, The saliency detection model is obtained by training the above-mentioned saliency detection model training method.

Therefore, by using the saliency detection model trained by the saliency detection model training method to detect the to-be-processed image, the accuracy of obtaining the predicted position information about the saliency region can be improved.

In some embodiments, after using the saliency detection model to process the image to be processed to obtain predicted location information about the saliency region in the content of the image to be processed, the method further includes: using the predicted location information, perform a Bone extraction is performed to obtain the target bone; a bone model is selected for the target bone as the source bone; the first animation driving data related to the source bone is transferred to the target bone to obtain the second animation driving data of the target bone.

Therefore, by using the predicted position information to perform skeleton extraction on the salient region, the accuracy of the target skeleton can be improved.

An embodiment of the present invention provides a training device for a saliency detection model, including: a first acquisition module configured to acquire at least one sample image, wherein the at least one sample image includes a target belonging to a preset image type The sample image; the screening module is configured to filter the target sample image based on the absence of the contour of the salient region in the target sample image; the first detection module is configured to use the saliency detection model to filter the filtered sample The image is detected to obtain the predicted position information about the saliency area in the sample image; the adjustment module is configured to adjust the parameters of the saliency detection model based on the labeled position information and predicted position information of the saliency area in the sample image.

In some embodiments, the screening module is configured to filter the target sample image based on the absence of the contour of the salient region in the target sample image, including: filling the contour of the salient region in the target sample image to obtain Fill in the sample image; obtain the difference between the saliency area in the filled sample image and the target sample image; filter the target sample image if the difference meets the preset requirements.

In some embodiments, the preset requirement is that the difference is greater than the preset difference value; the screening module is configured to fill in the contours of the salient regions in the target sample image to obtain the filled sample image, including: performing a step on the target sample image. Closing operation to obtain the filled sample image; obtaining the difference between the saliency region in the filled sample image and the target sample image, including: obtaining the first area of the salient region in the filled sample image, and about the salient region in the target sample image The second area of the significant region; the difference between the first area and the second area is taken as the difference.

In some embodiments, after filtering the target sample image based on the absence of the contour of the salient region in the target sample image, the filtering module is further configured to: fill in the position information of the salient region of the sample image, Obtain the annotation location information about the saliency region of the target sample image.

In some embodiments, the at least one sample image includes multiple image types.

In some embodiments, the plurality of image types include at least two of images captured on real objects, hand-drawn drawings, and cartoon drawings.

In some embodiments, the adjustment module is configured to adjust the parameters of the saliency detection model based on the labeled location information and predicted location information of the saliency region of the sample image, including: obtaining a sample image based on the labeled location information and predicted location information The first loss of each image element in the image; the first loss of each image element in the sample image is weighted to obtain the second loss of the sample image; based on the second loss, the parameters of the saliency detection model are adjusted.

In some embodiments, the weight of the first loss of the primitive is related to the boundary distance of the primitive, the boundary distance of the primitive is the distance between the primitive and the boundary of the real saliency area, and the real saliency area is the sample image The saliency area defined by the label position information in the .

In some embodiments, the smaller the boundary distance of the primitive, the greater the weight of the primitive's first loss.

In some embodiments, the saliency detection model includes at least one of the following: the saliency detection model is the network structure of MobileNetV3, and the saliency detection model includes a feature extraction sub-network, a first detection sub-network and a second detection sub-network The first detection module is configured to use the saliency detection model to detect the filtered sample image, and obtain the predicted position information about the saliency area in the sample image, including: using the feature extraction sub-network to detect the sample image Perform feature extraction on the sample image to obtain the feature map corresponding to the sample image; use the first detection sub-network to perform initial detection on the feature map to obtain the initial position information about the salient region in the sample image; Fusion to get the fusion result; use the second detection subnet to The fusion results are used for final detection, and the predicted position information of the sample image is obtained.

In some embodiments, the first detection module is configured to detect the filtered sample image by using the saliency detection model to obtain the predicted position information about the saliency region in the sample image, the screening module is further configured to The steps are: performing data enhancement on the filtered sample image; wherein, the data enhancement method includes filling in the background area other than the salient area in the sample image.

An embodiment of the present invention provides a saliency detection device, comprising: a second acquisition module, configured to acquire an image to be processed; a second detection module, configured to process the image to be processed by using a saliency detection model, and obtain the to-be-processed image The predicted location information about the saliency region in the image content is processed, wherein the saliency detection model is obtained by training the above-mentioned saliency detection model training method.

In some embodiments, after using the saliency detection model to process the image to be processed to obtain the predicted location information about the saliency region in the content of the image to be processed, the saliency detection device further includes a function module, and the function module is configured It is: using the predicted position information to extract the salient area to obtain the target bone; select a bone model for the target bone as the source bone; transfer the first animation driving data related to the source bone to the target bone to obtain the target bone A second animation driver profile for the bone.

An embodiment of the present invention provides an electronic device including a memory and a processor, where the processor is configured to execute program instructions stored in the memory to implement the above-mentioned training method and/or saliency detection method of a saliency detection model.

Embodiments of the present invention provide a computer-readable storage medium on which program instructions are stored, and when the program instructions are executed by a processor, the above-mentioned training method and/or saliency detection method for a saliency detection model are implemented.

An embodiment of the present invention further provides a computer program, where the computer program includes computer-readable code, and when the computer-readable code is executed in an electronic device, the processor of the electronic device executes any of the above-mentioned embodiments. The training method and/or the saliency detection method of the saliency detection model described above.

The embodiments of the present invention provide at least a saliency detection method and a training method, device and computer-readable storage medium for a saliency detection method. In this case, filter the target sample images so that the salient regions in the retained sample images are relatively complete, and then use the retained higher-quality sample images to train the saliency detection model, so that the training results can be obtained. The saliency detection model of the subsequent image detection results is more accurate.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention.

In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below, and are described in detail as follows in conjunction with the accompanying drawings.

201: Sample image acquisition terminal

202: Internet

203: Control Terminal

30: Training device for saliency detection model

31: The first acquisition module

32: Screening module

33: The first detection module

34: Adjustment Mods

40: Significant detection device

41: Second acquisition module

42: The second detection module

50: Electronics

51: Memory

53: Processor

60: Computer-readable storage medium

61: Program command

S11~S14, S21~S22: Steps

The accompanying drawings, which are incorporated into and constitute a part of this specification, illustrate embodiments consistent with the present invention, and together with the description, serve to explain the technical solutions of the present invention.

1 is a schematic flowchart of an embodiment of a training method for a saliency detection model according to an embodiment of the present invention; FIG. 2 is a schematic diagram of a system architecture that can apply a training method for a saliency detection model according to an embodiment of the present invention; Figure 4 is a schematic diagram of a hand-drawn drawing shown in an embodiment of a training method for a saliency detection model of the present invention; A schematic diagram of a cartoon diagram shown in an embodiment of a training method for a saliency detection model of the present invention; FIG. 6 is a schematic diagram of a hand-drawn drawing showing that the saliency area is missing in an embodiment of the training method for a saliency detection model of the present invention; Figure 6 7 is a schematic diagram showing the filled hand drawing in an embodiment of the training method of the saliency detection model of the present invention; FIG. 8 is a schematic diagram showing a sample image according to an embodiment of the training method of the saliency detection model of the present invention; FIG. 9 is a An embodiment of the training method for a saliency detection model of the present invention shows a schematic diagram of a saliency map; FIG. 10 is a schematic flowchart of an embodiment of the saliency detection method of the present invention; FIG. 11 is an embodiment of the saliency detection method of the present invention showing a mapping relationship Fig. 12 is the second schematic diagram showing the mapping relationship according to an embodiment of the distinctiveness detection method of the present invention; Fig. 13 is the third schematic diagram showing the mapping relationship according to an embodiment of the distinctiveness detection method of the present invention; Fig. 14 is the present invention A schematic structural diagram of an embodiment of a training device for a saliency detection model; Fig. 15 is a schematic structural diagram of an embodiment of a saliency detection device of the present invention; Fig. 16 is a structural schematic diagram of an embodiment of an electronic device of the present invention; Fig. 17 is a computer of the present invention A schematic structural diagram of an embodiment of a readable storage medium.

The solutions of the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

In the following description, for purposes of illustration and not limitation, specific details such as specific system structures, interfaces, techniques, etc. are set forth in order to provide a thorough understanding of the present invention.

The term "and/or" in this article is only an association relationship to describe associated objects, indicating that there can be three kinds of relationships, for example, A and/or B, which can mean that A exists alone, A and B exist at the same time, and B exists alone. three conditions. In addition, the character "/" in this text generally indicates that the related objects are an "or" relationship. Furthermore, "multiple" as used herein means two or more than two. In addition, the term "at least one" herein refers to any combination of any one of a plurality or at least two of a plurality, for example, including at least one of A, B, and C, and may mean including those composed of A, B, and C. Any one or more elements selected in the collection.

The present invention can be applied to a device with image processing capability. In addition, the device may have an image capture or video capture function, for example, the device may include components such as a camera for capturing images or videos. Or the device can obtain the required video stream or image from other devices by data transmission or data interaction with other devices, or access the required video from the storage resources of other devices stream or image, etc. Among them, other devices have image capture or video capture functions, and have a communication connection with the device. For example, the device can perform data transmission or data interaction with other devices through Bluetooth, wireless network, etc., The communication mode between the two is not limited here, and may include but not be limited to the above-mentioned cases. In an implementation manner, the device may include a mobile phone, a tablet computer, an interactive screen, etc., which are not limited herein.

Please refer to FIG. 1. FIG. 1 is a schematic flowchart of an embodiment of a training method for a saliency detection model according to an embodiment of the present invention. The training method of the saliency detection model may include the following steps.

Step S11: Acquire at least one sample image, wherein the at least one sample image includes a target sample image belonging to a preset image type.

At least one can be one or more. There are several ways to obtain sample images. For example, obtain the storage location of the sample image in the execution device that executes the training method, and then obtain the sample image by accessing the storage location image, or obtain sample images from other devices through transmission methods such as Bluetooth, wireless network, etc.

Step S12: Filtering the target sample image based on the lack of contours of the salient region in the target sample image.

Wherein, if the absence of the outline of the salient region in the target sample image satisfies the deletion condition, the target sample image is deleted from the sample image. If the outline of the salient region in the target sample image is missing, the deletion condition is not satisfied, then the target sample image is kept in the sample image. Among them, if the contour loss is more serious, it will be deleted, and if it is relatively slight, it will be retained. Among them, serious or minor determinations can be determined according to specific circumstances, and no specific provisions are made here.

Step S13: Detect the filtered sample image by using the saliency detection model to obtain predicted position information about the saliency area in the sample image.

The saliency detection model can process each sample image at the same time to obtain a batch of prediction results, or it can process each sample image in a time-sharing manner to obtain prediction results corresponding to each sample image respectively.

Step S14: Adjust the parameters of the saliency detection model based on the marked position information and predicted position information of the saliency region of the sample image.

The parameters of the saliency detection model can be adjusted according to the loss between the marked position information of the saliency region and the predicted position information.

The above scheme filters the target sample images according to the missing contours of the salient regions of the acquired target sample images of the preset image type, so that the salient regions in the retained sample images are relatively small. In order to be complete, the saliency detection model is trained by using the retained sample images of higher quality, so that the result of subsequent image detection by the trained saliency detection model can be more accurate.

FIG. 2 is a schematic diagram of a system architecture to which a training method of a saliency detection model according to an embodiment of the present invention can be applied; as shown in FIG. In order to support an exemplary application, the sample image acquisition terminal 201 and the control terminal 203 establish a communication connection through the network 202. The sample image acquisition terminal 201 reports at least one sample image to the control terminal 203 through the network 202, and the control terminal 203 Respond to the target sample image in at least one sample image, and filter the target sample image based on the missing contour of the salient region in the target sample image, and then use the saliency detection model to filter all the filtered images. The sample image is detected, and the predicted position information about the salient region in the sample image is obtained; finally, the parameters of the saliency detection model are adjusted based on the marked position information and predicted position information of the sample image about the salient region. Finally, the control terminal 203 uploads the adjusted parameters to the network 202 , and sends the adjusted parameters to the sample image acquisition terminal 201 through the network 202 .

As an example, the sample image acquisition terminal 201 may include an image acquisition device, and the control terminal 203 may include a visual processing device or a remote server with visual information processing capability. Network 202 may employ wired or wireless connections. Wherein, when the control terminal 203 is a visual processing device, the sample image acquisition terminal 201 can be connected to the visual processing device through a wired connection, such as data communication through a bus; When the control terminal 203 is a remote server, the sample image acquisition terminal 201 can exchange data with the remote server through a wireless network.

Or, in some scenarios, the sample image acquisition terminal 201 may be a visual processing device with a video acquisition module, or a host with a camera. At this time, the training method of the image optimization model according to the embodiment of the present invention may be executed by the sample image acquisition terminal 201 , and the above-mentioned system architecture may not include the network 202 and the control terminal 203 .

In some disclosed embodiments, the at least one sample image includes multiple image types. For example, two, three or more, etc. are included. By using sample images of various image types to train the saliency detection model, the trained saliency detection model can perform image processing on various types of images, thereby improving the applicability of the saliency detection model. Optionally, the image type includes at least two of an image obtained by photographing the target, a hand-drawn drawing, and a cartoon image. The images captured by the target can be divided into visible light images and infrared images. A hand drawing can be a drawing drawn by hand on paper and photographed to obtain a hand drawing, or it can be a drawing drawn on a drawing software, for example, a simple Mickey Mouse drawn by an artist on a hand-painted board. In this embodiment of the present invention, the hand-drawn drawing is further limited to a drawing of a preset background color and a preset foreground color, and the foreground is composed of monochrome lines, for example, the background is white, and the foreground is the outline of Mickey Mouse composed of black lines. . A cartoon image can be a virtual image with multiple foreground colors.

In order to better understand the images, hand-drawn drawings and cartoon drawings obtained by shooting the target according to the embodiment of the present invention, please refer to FIGS. 3 to 5 at the same time, and FIG. 3 is an embodiment of the training method of the saliency detection model of the present invention. shown in A schematic diagram of an image obtained by shooting a target, FIG. 4 is a schematic diagram of a hand drawing shown in an embodiment of a training method for a saliency detection model of the present invention, and FIG. 5 is a training method for a saliency detection model according to an embodiment of the present invention. Schematic representation of the cartoon figure shown. As shown in Fig. 3, Fig. 3 is an image obtained by photographing a real apple, Fig. 4 is a sketch of an apple drawn on real paper, and Fig. 5 is a cartoon image of an apple. By using the sample images corresponding to common image types to train the saliency detection model, the trained saliency detection model is more suitable for daily life or work. In the embodiment of the present invention, 10,000 upper and lower images of the target, 20,000 upper and lower hand drawings, and 20,000 upper and lower cartoon images are selected for training.

In some disclosed embodiments, the preset image type is a hand drawing. Since the hand-drawn drawing is likely to have break points during the drawing process, the hand-drawn drawing is filtered according to the absence of contours, so that the quality of the contours of the salient regions in the left-hand drawing is better. Wherein, based on the missing contour of the salient region in the target sample image, the method of filtering the target sample image may be: filling the contour of the salient region in the target sample image to obtain the filled sample image. Then, the difference between the salient regions in the filled sample image and the target sample image is obtained. Wherein, if the outline of the salient region in the target sample image is not missing or the missing is small, the salient region in the filling sample image and the target sample image before filling is the same or the difference is within a preset range. If the contours of the salient regions in the target sample image are largely missing, the difference between the salient regions in the filled sample image and the target sample image before filling is large. In the case that the difference meets the preset requirements, Filter the target sample image. By obtaining the difference between the salient region in the filled sample image and the target sample image, the missing contour of the salient region can be quickly obtained. Wherein, since it is necessary to remove the target sample image with defects in the salient region in the sample image, in the embodiment of the present invention, the preset requirement is that the difference is greater than the preset difference value.

In order to better understand the difference between the hand drawing of the missing saliency region and the hand drawing after filling, please refer to Fig. 6 and Fig. 7, Fig. 6 shows the training method of the saliency detection model of the present invention in one embodiment. Fig. 7 is a schematic diagram showing the filled-in hand-drawing in an embodiment of a training method for a saliency detection model of the present invention.

As shown in Figures 6 and 7, the outline of the salient area in the hand-painted drawing before filling is an arc, the angle between the two endpoints and the center of the circle is 45°, and the area of the salient area can be connected by a line segment. The area smaller than the full circle is obtained, and the contour of the salient region after filling is the full circle. The area of the salient region is the area of the full circle. Obviously, the area of the saliency area after filling is quite different from the area of the saliency area before filling. At this time, the hand drawing before filling can be removed and not allowed to participate in the training of the model.

The method of filling in the contour of the salient region in the target sample image to obtain the filling sample image may be: performing a closing operation on the target sample image to obtain the filling sample image. The closing operation refers to performing an expansion operation on the target sample image first, and then performing an erosion operation or a scaling operation. Among them, the closing operation can close small lakes (ie, small holes) and bridge small cracks, while the overall position and shape remain unchanged. The dilation operation can make the salient region round The contour gap is bridged, and the thickness of the contour in the salient region can be reduced by the scaling operation. As mentioned above, the hand drawing may be in the form of black lines on a white background, wherein the salient area of the hand drawing is the area surrounded by the black lines, and the outline of the salient area is the black line. The closing operation on the target sample image may be, for example, the closing operation on the contour of the salient region. That is, the black lines are expanded first, and then the expanded black lines are scaled or eroded, so that the contour thickness of the salient region in the filled sample image is the same or different from that of the salient region in the target sample image before filling. within the preset range. In this way, in the process of acquiring the difference between the saliency region in the filled sample image and the target sample image, the contour difference between the two can be ignored.

Wherein, the manner of obtaining the difference between the saliency region in the fill-in sample image and the target sample image may be to obtain a first area of the fill-in sample image with respect to the saliency region, and a second area with respect to the saliency region in the target sample image. area. Generally, any method for obtaining the area of the region is acceptable, and the method for obtaining the area of the salient region is not specifically limited here. For example, the way to obtain the second area can be to use a line segment to connect the two ends of the contour gap to form a closed area, so as to calculate the area of the closed area. Of course, you can also use the two ends of the gap as the origin, and draw two horizontal and vertical lines respectively. Straight lines, four straight lines may have two intersections. Calculate the area of the enclosed area formed by the two straight lines connected by each intersection point and the saliency area, and take the smaller enclosed area as the second area. Take the difference between the first area and the second area as the difference. For example, the difference between the second area minus the first area is used as the difference with respect to the saliency area in the filled-in sample image and the target sample image. some males In an embodiment, the difference between the areas occupied by the contours of the salient regions before and after filling can be used as the difference. If there is a large gap in the contour of the salient region in the target sample image, there may be a large difference in the area of the salient region before and after filling, so the target sample can be determined according to the difference in the area of the salient region before and after filling. Whether the contours of salient regions in the image are missing.

In some disclosed embodiments, after the target sample image is filtered, the training method for the saliency detection model further includes the following steps: obtaining information about the saliency area of the target sample image based on the position information of the saliency area filled in the sample image. Label location information. For example, the contour of the salient region filling the sample image is obtained as the labeling position information about the contour of the salient region in the target sample image. And, take the contour and its surrounding area as the salient area. By filling in the position information of the salient region of the sample image, the marked position information of the salient region of the target sample image can be determined, which can ensure the integrity of the salient region.

In some disclosed embodiments, before using the saliency detection model to detect the filtered sample image to obtain the predicted position information about the saliency region in the sample image, the training method of the saliency detection model further includes the following steps: Data enhancement is performed on the filtered sample images. Among them, there are various ways of data enhancement, for example, including filling the background area except the salient area in the sample image. Among them, preset primitive values can be used for filling. For example, uniformly fill with 0 primitives, or uniformly fill with other primitive values. Of course, different primitive positions can also be filled with different primitive values. There are no specific rules for the filling method here. Certainly. In some disclosed embodiments, the data enhancement method may also be at least one of adding noise, Gaussian blurring, cropping, and rotation. Among them, Gaussian blur processing can also be called Gaussian smoothing. The main function is to reduce image noise and reduce the level of detail. The main method is to adjust the color value of the primitive according to the Gaussian curve, and selectively blur the image. Cropping refers to cropping the training sample images into images of different sizes, for example, cropping the training sample images into images with a size of 1024*2048 or 512*512. Of course, this size is only an example, in other In the embodiment, images that are cropped to other sizes may be adopted, and therefore, the cropped size is not specified here. The rotation can be to rotate the training sample image by 90°, 180° or 270°. Of course, in other embodiments, the data enhancement method may also be to adjust the resolution or the like. The applicability of the saliency detection model can be improved by data enhancement of the sample images.

In some disclosed embodiments, the saliency detection model is the network structure of MobileNetV3. The saliency detection model includes a feature extraction sub-network, a first detection sub-network and a second detection sub-network. Wherein, the first detection sub-network and the second detection sub-network adopt a cascade structure. That is, the output of the first detection sub-network is used as the input of the second detection sub-network. In some embodiments, the structures of the first detection sub-network and the second detection sub-network are the same. Wherein, using the saliency detection model to detect the filtered sample image, and obtaining the predicted position information about the saliency region in the sample image, the method may be: using the feature extraction sub-network to perform feature extraction on the sample image, Obtain the feature map corresponding to the sample image. Then, use the first detection sub-network to perform initial detection on the feature map, and obtain the relevant salient features in the sample image. The initial location information of the sex zone. The initial location information may be presented in the form of a saliency map. Then, the feature map and the initial position information are fused to obtain the fusion result. For example, the fusion method may be to multiply the feature map and the initial position information to obtain the fusion result. The fusion result is finally detected by the second detection sub-network to obtain the predicted position information of the sample image. The final predicted location information can also be presented in the form of a saliency map. For a better understanding of the saliency map, please refer to FIG. 8 and FIG. 9 , FIG. 8 is a schematic diagram showing a sample image according to an embodiment of the training method of the saliency detection model of the present invention, and FIG. 9 is the training method of the saliency detection model of the present invention. An embodiment shows a schematic diagram of a prominent figure. As shown in Figure 8 and Figure 9, the sample image includes a table and a toy duck on the table, the saliency detection model detects the sample image, and the output initial position information (saliency map) is shown in Figure 9 , the primitive value of the toy duck's position is 1, and the primitive value of the rest of the position is 0. From this, the position of the toy duck in the sample image can be clearly obtained. Due to the simple network structure of MobileNetV3, by using the network structure of MobileNetV3, the detection efficiency can be accelerated, and devices with smaller processing capabilities can also use the saliency detection model to achieve saliency detection; After the network performs initial detection on the feature map, the second detection sub-network is used to perform final detection on the initial detection result, which can improve the detection accuracy.

In some disclosed embodiments, a saliency detection model is used to process the sample image, respectively, to obtain predicted position information about the saliency region in the sample image, based on the marked position information and predicted position information of the saliency region in the sample image, Ways to tune the parameters of the saliency detection model include as follows.

Several sample images are selected from the plurality of sample images as the current sample image. Among them, several refer to 1 and above. That is, here, one of the sample images may be selected from the multiple sample images as the current sample image, or two or more sample images may be selected as the current sample image. In some embodiments, the image types to which the selected sample images belong include all image types of the plurality of sample images. For example, when the image types of the multiple sample images include the above three image types in total, several sample images selected from the multiple sample images also include the above three image types. The number of sample images of each image type may be the same or different. Then, the saliency detection model is used to process the current sample image to obtain the prediction result of the current sample image. For example, take the current sample image as a batch, and use the saliency detection model to process the sample images of this batch to obtain a batch of prediction results. Then, based on the annotation results and prediction results of the current sample image, the parameters of the saliency detection model are adjusted. Optionally, the parameters of the model can be adjusted by using the loss between each annotation result in a batch and its corresponding prediction result. In this way, the parameters need to be adjusted several times, or it can be combined with each annotation result and its corresponding prediction result. The loss between the prediction results adjusts the parameters of the model, which only needs to adjust the parameters of the model once. The selection of several sample images from multiple sample images as the current sample image and the subsequent steps are repeatedly performed until the saliency detection model meets the preset requirements. The preset requirement here may be the size of the error between the prediction result given by the model and the labeling result. The specific error size is determined according to the actual needs. Yes, there is no provision here. Optionally, several sample images selected from the plurality of sample images each time may be the same as the partial sample images selected last time. In other disclosed embodiments, several sample images selected from the plurality of sample images are different each time. Several sample images are selected from multiple sample images as the current sample image, and the saliency detection model is used to process the current sample image, which can improve the training speed.

In some disclosed embodiments, the annotation information of the sample image further includes the real image type of the sample image, and the prediction result of the sample image includes the predicted image type of the sample image. Wherein, when the saliency detection model is a target classification model, the prediction result of the saliency detection model includes the predicted category of the target and the predicted image type of the sample image. When the saliency detection model is a saliency detection model, the predicted position information is the predicted category of the target in the sample image and the predicted image type of the sample image. Adjusting the parameters of the saliency detection model by using annotated location information about the content of the sample image and predicted location information of its content, and/or the real image type of the sample image and the predicted image type of the sample image, This makes the adjusted saliency detection model more applicable.

In some disclosed embodiments, based on the labeled location information and predicted location information of the saliency region of the sample image, the parameters of the saliency detection model can be adjusted by: The first loss of primitives. The first loss of each primitive in the sample image is weighted to obtain the second loss of the sample image. Based on the second loss, the parameters of the saliency detection model are adjusted. The way to obtain the first loss can be to make a difference between the marked position information and the predicted position information to get first loss. By weighting the first loss of each primitive, it is more accurate to use the weighted second loss to adjust the parameters of the saliency detection model.

Among them, the weight of the first loss of the primitive is related to the boundary distance of the primitive. The boundary distance of the primitive is the distance between the primitive and the boundary of the real saliency area, and the real saliency area is the saliency area defined by the annotation location information in the sample image. Wherein, the distance between the primitive here and the boundary of the real saliency region may be the minimum distance from the boundary of the saliency region. For example, the position of the primitive in the upper left corner of the sample image is (0, 0), and the boundary of the real saliency area includes (0, 1), (0, 2), etc. The position of this primitive is the boundary of the real saliency area. The distance between them is 1. By determining the weight according to the boundary distance of the primitive, it is more accurate to use the weighted second loss to adjust the parameters of the saliency detection model.

In some embodiments, the smaller the boundary distance of the primitive point, the greater the weight of the primitive's first loss. That is, the weight of the first loss of the primitive point is negatively correlated with the boundary distance of the primitive point. The boundary distance of the primitive is negatively correlated with the weight of the first loss of the primitive, making the obtained second loss more accurate.

In some disclosed embodiments, based on the labeled position information and predicted position information of the saliency region of the sample image, the parameters of the saliency detection model may be adjusted by: obtaining the third loss based on the actual image type and the predicted image type. . Then, based on the second loss and the third loss, the parameters of the saliency detection model are adjusted. For example, a third loss is obtained based on the error between the real image type and the predicted image type. For example, a second loss is determined by combining the error between a batch of predicted location information and the corresponding annotation information, and combining a batch of predicted image types with ground truth The error between the image types determines a third loss. Combine the second loss and the third loss to adjust the parameters of the saliency detection model. By adjusting the parameters of the saliency detection model using a second loss between the annotated location information about the content of the sample image and the predicted location information of its content and a third loss based on the real image type and the predicted image type, it is possible to improve the Suitability of saliency detection models.

For example, the second loss optimizes the parameters of the model, so that the predicted location information obtained by the saliency detection model is closer to the labeled location information, that is, the error between the two becomes smaller. By using the third loss to adjust the parameters of the model, the distance between the feature vectors of images representing the same object but belonging to different image types is closer in the feature space, so that the feature vectors of images of different image types are all in the feature space with closer distances. For example, the trained saliency detection model has a closer distance in the feature space for feature vectors obtained by extracting features from hand-drawn drawings representing apples, cartoon drawings, and images obtained by photographing apples.

In some disclosed embodiments, based on the second loss and the third loss, the method of adjusting the parameters of the saliency detection model may be: obtaining a loss difference between the second loss and the third loss. The parameters of the saliency detection model are then adjusted using the loss difference and the third loss. For example, the loss difference is obtained by taking the difference between the second loss and the third loss. Using the second loss difference and the third loss difference, the parameters of the saliency detection model may be adjusted by using one of the losses to adjust the parameters of the model first, and then using the other loss to adjust the parameters of the model. By adjusting the parameters of the saliency detection model using the loss difference between the second loss and the third loss and the third loss, it is possible to improve the Applicability of high saliency detection models.

In some disclosed embodiments, the saliency detection model further includes an image type classification sub-network.

Among them, the image type classification sub-network connects the feature extraction sub-network. The image type classification network is used to classify the image type of the sample image, and the predicted image type of the sample image is obtained. In some embodiments, the feature map extracted by the feature extraction sub-network is input into the image type classification network to obtain the predicted image type of the sample image. The method of adjusting the parameters of the saliency detection model by using the difference of the loss and the third loss may be: adjusting the parameters of the image type classification sub-network by using the third loss. and using the loss difference to adjust the parameters of the feature extraction sub-network, the first detection sub-network and the second detection sub-network. Both the loss difference and the third loss are used to adjust the parameters in a positive way. By using the loss difference to adjust the feature extraction sub-network, the first detection sub-network and the second detection sub-network in the saliency detection model, the predicted location information about the content of the sample image obtained by the saliency detection model More accurate, and using the third loss to adjust the parameters of the image type classification network can improve the accuracy of the image type classification network.

In some disclosed embodiments, the trained saliency detection model can be deployed to the mobile phone, and the AR/VR end performs image processing. The saliency detection method can also be applied to software such as photography and video recording filters.

The above scheme filters the target sample images according to the missing contours of the salient regions of the acquired target sample images of the preset image type, so that the salient regions in the retained sample images are relatively small. In order to be complete, the saliency detection model is trained by using the retained sample images of higher quality, so that the result of subsequent image detection by the trained saliency detection model can be more accurate.

The execution body of the training method for the saliency detection model may be a training device for the saliency detection model. For example, the training method for the saliency detection model may be executed by a terminal device or a server or other processing device, where the terminal device may be User Equipment (UE), mobile device, user terminal, terminal, cellular phone, wireless phone, Personal Digital Assistant (PDA), handheld device, computing device, vehicle-mounted device, wearable device, etc. In some possible implementations, the training method of the saliency detection model may be implemented by the processor calling computer-readable instructions stored in the memory.

Please refer to FIG. 10. FIG. 10 is a schematic flowchart of an embodiment of a method for detecting significance of the present invention. As shown in FIG. 10 , the saliency detection method provided by the embodiment of the present invention includes the following steps.

Step S21: Acquire the image to be processed.

There are many ways to obtain the image to be processed, for example, the image to be processed can be obtained by photographing a photographing element in the execution device executing the saliency detection method, or the image to be processed can be obtained from other devices according to various communication methods. The image type of the image to be processed may be one of multiple image types. For example, the image type of the image to be processed may be one or more of an image obtained by photographing a target and a hand-drawn cartoon. In some disclosed embodiments, the image to be processed may also be obtained from the video. For example, put a The segment video is input to the saliency detection model, and the saliency detection model obtains each video frame in the video, and uses each video frame as an image to be processed.

Step S22: using the saliency detection model to process the image to be processed to obtain the predicted position information about the saliency area in the content of the image to be processed, wherein the saliency detection model is obtained by training the training method of the saliency detection model .

The saliency detection model in the embodiment of the present invention includes a feature extraction sub-network, a first detection sub-network, and a second detection sub-network. Among them, the saliency detection model is trained using sample images of various image types. For example, the image to be processed is input into the saliency detection model from the input end of the saliency detection model. The saliency detection model processes the to-be-processed image to obtain predicted location information about the saliency region in the content of the to-be-processed image.

In the above solution, by using the saliency detection model trained by the above-mentioned saliency detection model training method to process the image to be processed, the accuracy of image processing can be improved.

In some disclosed embodiments, the saliency detection method further includes at least one of the following steps after processing the to-be-processed image by using a saliency detection model to obtain predicted location information about the saliency region in the content of the to-be-processed image.

1. Display the predicted location information on the interface displaying the image to be processed. Among them, there are various ways of displaying, for example, the predicted position information is marked on the image to be processed, so that the image to be processed and the corresponding predicted position information can be displayed on the display interface together. Of course, it can also be displayed on different display interfaces The areas respectively display the image to be processed and the corresponding predicted location information. In some disclosed embodiments, if there are two or more images to be processed, the corresponding images to be processed and their predicted location information may be displayed in different areas of the display interface, or the images to be processed and their predicted location information may be displayed in the form of page turning. Its predicted location information. Wherein, when the image to be processed is obtained from the video, it is determined whether the predicted position information of the video frames of the consecutive preset number of frames is the same, and if so, the predicted position information is considered to be correct. If not, the predicted location information is considered incorrect. Among them, you can choose to output the correct predicted position information, and not output the wrong predicted position information, or you can choose to annotate and output the correct and wrong predicted position information correspondingly. The preset number of frames may be 5 frames, 10 frames, etc., which may be determined according to specific usage scenarios.

2. Use the predicted position information to extract bones from the salient regions to obtain the target bones. And, select a bone model for the target bone as the source bone. Among them, animation driving data is set on the source bone. Then, the first animation driving data related to the source bone is transferred to the target bone to obtain the second animation driving data of the target bone. The target skeleton is obtained by performing skeleton extraction based on the target in the image to be processed.

In some disclosed embodiments, using the predicted position information to perform bone extraction on the salient region to obtain the target skeleton may be: performing contour extraction on the salient region to obtain the contour of the target, and then using the contour to generate a 3D mesh for the target. lattice model. Finally, the target bones are extracted from the 3D mesh model.

The way to obtain the source skeleton can be: classify the image to be processed, get the category of the target object, and select the skeleton model that matches the category as the source bone. The target bone is the bone of the target object. For example, in this embodiment of the present invention, a predicted label mapping may be used, and a data set label mapping may also be used. The classification result of the target object by the predicted label mapping includes the predicted skeletal topology type of the target object, for example, the predicted skeletal topology type includes bipedal, quadrupedal and so on. That is, the process of predicting the label mapping is mainly to predict the skeletal topological characteristics of the target object. The classification result of the dataset label mapping needs to give the specific type of the target object in the input image, for example, the target object is a cat, a dog, a giant panda, a bear, and so on. This embodiment of the present invention chooses to use predicted label mapping. In the specific application process, if the target object is a giant panda, and the target object category given by the predicted label mapping is quadruped, a skeleton model matching the category is selected as the initial source bone. The initial source bone selected is a four-legged bear. Although pandas and bears are different, they actually have roughly the same skeletal topology, so transferring the bear's animation-driven data to pandas can also appear in a natural and reasonable form. That is, although the category of the target object cannot be obtained by predicting the label mapping, it does not affect the driving of the final target skeleton. At the same time, the computational cost is reduced because the predicted label mapping does not further know the specific category of the target object.

After determining the source bone that matches the target bone, perform bone node mapping between the source bone and the target bone to obtain the node mapping relationship between the two. In some disclosed embodiments, the manner of obtaining the node mapping relationship between the two may be: determining the number of bone branches where each node of the source bone and the target bone is located. The nodes in the source bone and the target bone are mapped in order according to the order of the number of bone branches from more to less. Among them, the bone The node with the largest number of skeleton branches is generally called the root node. Among them, the number of bone branches where the node is located is called the degree for the time being. That is, the mapping relationship between the nodes with the larger degree in the two bones is first constructed, and then the mapping relationship between the nodes with the smaller degree is constructed. Alternatively, the mapping can be performed using the principle of the minimum error value of the skeleton branch mapping. Among them, if the number of nodes between the source bone and the target bone is different, the smallest many-to-one mapping with the lowest cost is selected. For example, the mapping can be performed by performing a one-to-one joint match in the sequence in which the many-to-one or skip mapping occurs.

In some disclosed embodiments, the final target bone is consistent with the node topology of the source bone. Or, the nodes between the final target bone and the final source bone are mapped one-to-one. That is, the node topology of the final target bone and the final source bone may exist in two forms, one is that the node topology of the final target bone is completely consistent with the final source bone, and the other is the node in the final target bone. All the nodes of the final source bone correspond to it, but there are some nodes that have no mapping relationship in the final source bone. That is, it needs to be ensured that after the animation is migrated, the nodes of the final target bone have corresponding animation driving data.

After obtaining the node mapping relationship between the two, perform topology alignment and node alignment.

Wherein, the manner of performing topology structure alignment may include at least one of the following steps.

One is to update the node topology of one of the bones when there are multiple nodes mapped to the same node between the source bone and the target bone. Among them, the updated nodes between the two bones are mapped one by one. pass Updating the node topology of the bones can make the case where multiple nodes between two bones are mapped to the same node, and adjust the one-to-one mapping between the nodes between the two bones to reduce the unreasonableness in the process of subsequent animation driving the final target bone situation occurs.

Wherein, updating the node topology of one of the bones can be divided into multiple cases: the first case is to update the first bone where multiple nodes are located when multiple nodes are located in the same bone branch. Among them, one of the first bone and the second bone is the source bone, and the other is the target bone. By updating the first bone where multiple nodes are located, the situation where multiple nodes between two bones are mapped to the same node is adjusted to one-to-one mapping between nodes between two bones, thereby reducing the process of subsequent animation driving the final target bone Unreasonable situations occur. Optionally, the manner of updating the first bone where the multiple nodes are located may be to combine multiple nodes in the first bone into one first node. The first node retains the mapping relationship of the multiple nodes before merging. And, the position of the first node is the average of the positions of all merged nodes.

Referring to FIG. 11 at the same time, FIG. 11 is a first schematic diagram showing a mapping relationship according to an embodiment of the saliency detection method of the present invention. As shown in Figure 11, when the second node and the third node in the target bone are mapped to the second node in the source bone at the same time. In this case, the second and third nodes in the target bone are merged into one first node. Among them, the position of the first node is the average of the positions of the second node and the third node in the target bone. Among them, when the first bone is the source bone, because the nodes in the source bone carry animation driving data, after the nodes are merged, The animation driving data of the first node needs to be acquired. At this time, the animation driving data of all the nodes to be merged can be merged. For example, the animation driving data can generally be represented by a matrix, and the combination of the matrices can be represented by matrix multiplication, that is, the animation driving data of the first node can be obtained by multiplying the animation driving data. The second case is when multiple nodes are in different bone branches, updating the second bone that does not include multiple nodes. Among them, one of the first bone and the second bone is the source bone, and the other is the target bone. Optionally, a second node where the branches of the bones where the multiple nodes are located converge is found in the first bone. The specific method may be to traverse the parent nodes in turn to obtain the second node. And find the third node mapped to the second node in the second bone. Then, the node topology structure corresponding to the multiple nodes is found, and at least one bone branch is added at the third node. In this embodiment of the present invention, the parent node of a node refers to a node in a skeleton branch that is adjacent to the node and closer to the root node than the node. The multiple nodes are mapped one-to-one with the newly added skeleton branch at the third node and the nodes in the original skeleton branch. The newly added bone branch may be a copy of the original bone branch. The copied content includes animation data and the transformation relationship between the node and its parent node. For example, if the original bone branch includes three nodes, the newly added bone branch also includes three nodes, and the animation driving data of the three nodes in the newly added bone branch is obtained by copying the corresponding nodes in the original bone branch animation data obtained.

Referring to FIG. 12 at the same time, FIG. 12 is a second schematic diagram illustrating a mapping relationship according to an embodiment of the saliency detection method of the present invention. As shown in Figure 12, the node topology on the left is the node topology of the source bone, and the node on the right Topology is the node topology of the target bone. In Figure 12, the first node of the target bone is mapped to the first node of the source bone, the second node of the target bone is mapped to the second node of the source bone, and the second node of the target bone includes two branches , that is, the left branch and the right branch, where the first node in the left branch and the first node in the right branch are mapped to the third node of the source bone, the second node in the left branch and the right branch The second node maps to the fourth node of the source bone. This also means that two nodes in the target bone are mapped to the third node of the source bone, and these two nodes belong to different branches, and two nodes in the target bone are mapped to the fourth node of the source bone, and this The two nodes belong to different branches. Among them, the two branches meet at the second node of the target bone. Find the second node in the source bone that is mapped to the target bone as the second node. According to the node topology corresponding to the two nodes of the target bone, a new bone branch is added at the second node of the source bone. Among them, there are two nodes in the newly added bone branch. At this point, all the nodes in the target bone are in one-to-one correspondence with the nodes in the source bone. Therefore, in the case of implementing node-to-node mapping in this way, the node topology of the first bone can also be preserved to the greatest extent.

The second is to update the node topology of the bone where the node with no mapping relationship is located when there is no mapping relationship in the bone. The two bones include a source bone and a target bone, and the updated nodes between the two bones are mapped one-to-one. By updating the node topology of the bone where the node without a mapping relationship is located, the nodes without a mapping relationship are reduced, so that the nodes between the updated two bones are mapped one by one, thereby reducing the subsequent animation drive. An unreasonable situation occurs in the process of the final target bone. Optionally, a node without a mapping relationship is merged into an adjacent node with a mapping relationship. Among them, the adjacent node is the parent node or child node in the bone where the node without mapping relationship is located. In this embodiment of the present invention, nodes without a mapping relationship are merged with their parent nodes.

Please refer to FIG. 13 . FIG. 13 is a third schematic diagram illustrating a mapping relationship according to an embodiment of the method for detecting significance of the present invention. As shown in Figure 13, the first node of the target bone is mapped to the first node of the source bone, the second node of the target bone is mapped to the third node of the source bone, and the third node of the target bone is mapped to the source bone The fourth node of the bone. Among them, the second node of the source bone has no mapping relationship. You can merge the second node of the source bone to its parent node, that is, to the first node of the source bone. Of course, the merging of nodes in the source bones will be accompanied by the merging between animation-driven data, and the merging between animation-driven data will not be repeated here.

Among them, the node alignment is mainly to determine the first pose transformation relationship between the source bone and the target bone.

For example, in the order from the root node to the leaf source node, align each source node in the final source skeleton with the corresponding mapped target node in the final target skeleton, so as to obtain the first node between each source node and the mapped target node. A pose transformation relationship. As mentioned above, the root node is the node with the largest number of bone branches. The root node refers to the root node in the final source bone, and similarly, the root target node refers to the root node of the final target bone. The final source bone and final target bone refer to the source bone and target bone after topology alignment. Among them, a leaf node refers to a node with a parent but A node with no children. The leaf source node refers to the leaf node in the final source bone, and the leaf target node refers to the leaf node in the final target bone. That is, the root node and the root target node that has a mapping relationship with the root node are first aligned. Then align the leaf source node connected to the root node and the leaf target node with a mapping relationship with the leaf source node, and so on, until all the nodes in the final target bone are aligned with the nodes of the final source bone one by one. In some disclosed embodiments, the root target node of the final target bone may be directly used as the origin of the first coordinate system.

The pose transformation relationship is the transformation relationship between the source node and the mapped target node in the first coordinate system. By translating the root node of the final source bone and the root target node of the final target bone to the origin of the first coordinate system, the offset between the root node of the final source bone and the root target node of the final target bone can be obtained. For example, for each source node in the final source bone, get the offset needed to align the source node to the mapped target node. Among them, the offset includes translation and rotation components. In general, a scaling component is included in the translation component. Then, based on the offset corresponding to the source node, the first pose transformation relationship of the source node is obtained.

Among them, if the topology of the source bone changes, the animation data on the source bone also changes correspondingly. For example, if two source nodes in the source bone are merged, the animation data corresponding to the nodes will also be merged.

In this way, the animation data on the source bone can be transferred to the target bone to drive the movement of the target in the image to be processed.

By performing the above at least one step after obtaining the prediction information, the convenience in the use process is improved.

And the saliency area output by the saliency detection model obtained by using the training method of the saliency detection model above, and the saliency area is extracted from the saliency area to obtain the target skeleton, so that the obtained target skeleton is more accurate.

In the above solution, by using the saliency detection model trained by the above-mentioned saliency detection model training method to process the image to be processed, the accuracy of image processing can be improved.

Wherein, the execution body of the saliency detection method may be a saliency detection apparatus, for example, the salience detection method may be executed by a terminal device or a server or other processing device, wherein the terminal device may be a user equipment (User Equipment, UE) , mobile devices, user terminals, terminals, cellular phones, wireless phones, personal digital assistants (Personal Digital Assistant, PDA), handheld devices, computing devices, in-vehicle devices, wearable devices, etc. In some possible implementations, the significance detection method may be implemented by the processor calling computer-readable instructions stored in the memory.

Please refer to FIG. 14. FIG. 14 is a schematic structural diagram of an embodiment of a training apparatus for a saliency detection model of the present invention. The training device 30 for the saliency detection model includes a first acquisition module 31 , a screening module 32 , a first detection module 33 and an adjustment module 34 . The first acquisition module 31 is configured to acquire at least one sample image, wherein the at least one sample image includes a target sample image belonging to a preset image type; the screening module 32 is configured to be based on the target sample image If the contour of the medium saliency area is missing, filter the target sample image; the first detection module 33 is configured to use the saliency The detection model detects the filtered sample image, and obtains the predicted position information about the salient region in the sample image; the adjustment module 34 is configured to be based on the marked position information and predicted position information of the salient region based on the sample image , adjust the parameters of the saliency detection model.

The above scheme filters the target sample image according to the missing outline of the salient region of the obtained target sample image of the preset image type, so that the salient region in the retained sample image is relatively complete. , and then use the retained sample images with higher quality to train the saliency detection model, which can make the result of subsequent image detection by the trained saliency detection model more accurate.

In some embodiments, the screening module 32 is configured to filter the target sample image based on the absence of the contour of the salient region in the target sample image, including: filling in the contour of the salient region in the target sample image, Obtain the fill-in sample image; obtain the difference between the fill-in sample image and the target sample image with respect to the saliency area; filter the target sample image when the difference meets the preset requirements.

In the above solution, the sample image is filtered according to the absence of the contour, so that the quality of the contour of the salient region in the remaining sample image is better. In addition, the missing contour of the salient region can be quickly acquired by acquiring the difference between the saliency region in the fill-in sample image and the target sample image.

In some embodiments, the preset requirement is that the difference is greater than the preset difference value; the screening module 32 is configured to fill in the contour of the salient region in the target sample image to obtain the filled sample image, including: Perform the closing operation to obtain the filled sample image; obtain the filled sample image and the target The difference in the saliency area in the sample image includes: obtaining the first area of the filling sample image regarding the saliency area, and the second area regarding the saliency area in the target sample image; combining the first area and the second area The difference is the difference.

In the above scheme, if there is a large gap in the contour of the salient region in the target sample image, there may be a large difference in the area of the salient region before and after filling, so according to the difference in the area of the salient region before and after filling, Determines whether the contours of salient regions in the target sample image are missing.

In some embodiments, after filtering the target sample image based on the absence of the contour of the salient region in the target sample image, the filtering module 32 is further configured to: fill in the position information of the salient region of the sample image based on the location information , to obtain the annotation location information about the saliency region of the target sample image.

The above solution can ensure the integrity of the salient region by filling in the position information of the salient region of the sample image to determine the marked position information about the salient region of the target sample image.

In some embodiments, the at least one sample image includes multiple image types.

In the above solution, the saliency detection model is trained by using sample images of various image types, so that the saliency detection model obtained by training can perform image processing on various types of images, thereby improving the applicability of the saliency detection model. sex.

In some embodiments, the multiple image types include At least two of the captured images, hand-drawn drawings and cartoon drawings.

In the above solution, by using sample images corresponding to common image types for training the image processing model, the image processing model obtained by training is more suitable for daily life or work.

In some embodiments, the adjustment module 34 is configured to adjust the parameters of the saliency detection model based on the labeled location information and predicted location information of the saliency region of the sample image, including: obtaining samples based on the labeled location information and predicted location information The first loss of each image element in the image; the first loss of each image element in the sample image is weighted to obtain the second loss of the sample image; based on the second loss, the parameters of the saliency detection model are adjusted.

In the above solution, by weighting the first loss of each primitive, it is more accurate to use the weighted second loss to adjust the parameters of the saliency detection model.

In some embodiments, the weight of the first loss of the primitive is related to the boundary distance of the primitive, the boundary distance of the primitive is the distance between the primitive and the boundary of the real saliency area, and the real saliency area is the sample image The saliency area defined by the label position information in the .

In the above solution, by determining the weight according to the boundary distance of the primitives, it is more accurate to use the weighted second loss to adjust the parameters of the saliency detection model.

In some embodiments, the smaller the boundary distance of the primitive, the greater the weight of the primitive's first loss.

In the above scheme, the boundary distance of the primitive is negatively correlated with the weight of the first loss of the primitive, so that the obtained second loss is more accurate.

In some embodiments, the saliency detection model includes at least one of the following: the saliency detection model is the network structure of MobileNetV3, and the saliency detection model includes a feature extraction sub-network, a first detection sub-network and a second detection sub-network The first detection module 33 is configured to use the saliency detection model to detect the filtered sample image, and obtain the predicted position information about the saliency area in the sample image, including: using the feature extraction sub-network to Perform feature extraction on the image to obtain the feature map corresponding to the sample image; use the first detection sub-network to perform initial detection on the feature map to obtain the initial position information about the salient region in the sample image; combine the feature map and the initial position information Perform fusion to obtain a fusion result; use the second detection sub-network to perform final detection on the fusion result to obtain the predicted position information of the sample image.

The above scheme, due to the simple network structure of MobileNetV3, can speed up the detection efficiency by using the network structure of MobileNetV3, and can also enable devices with smaller processing capabilities to use the saliency detection model to achieve saliency detection; After the first detection sub-network performs initial detection on the feature map, the second detection sub-network is used to perform final detection on the initial detection result, which can improve the detection accuracy.

In some embodiments, the first detection module 33 is configured to use the saliency detection model to detect the filtered sample image and obtain the predicted location information about the saliency region in the sample image, the screening module 32 It is also configured to: perform data enhancement on the filtered sample image; wherein, the data enhancement method includes filling the background area except the salient area in the sample image.

The above solution can improve the applicability of the saliency detection model by enhancing the data of the sample image.

Please refer to FIG. 15 . FIG. 15 is a schematic structural diagram of an embodiment of a saliency detection apparatus according to the present invention. The significance detection device 40 includes a second acquisition module 41 and a second detection module 42 . The second acquisition module 41 is configured to acquire the to-be-processed image; the second detection module 42 is configured to process the to-be-processed image by using the saliency detection model to obtain the predicted position of the saliency region in the content of the to-be-processed image information, wherein the saliency detection model is obtained by training the above-mentioned saliency detection model training method.

In the above solution, by using the saliency detection model trained by the saliency detection model training method to detect the to-be-processed image, the accuracy of obtaining the predicted position information about the saliency region can be improved.

In some embodiments, after using the saliency detection model to process the to-be-processed image to obtain the predicted location information about the saliency region in the content of the to-be-processed image, the saliency detection device further includes a functional module (not shown in the figure) , the function module is configured as: using the predicted position information to extract the salient area to obtain the target bone; select a bone model for the target bone as the source bone; transfer the first animation driving data related to the source bone to the target On the bone, get the second animation driving data of the target bone.

The above solution can improve the accuracy of the target skeleton by using the predicted position information to perform skeleton extraction on the salient region.

Please refer to FIG. 16 , which is a schematic structural diagram of an embodiment of an electronic device of the present invention. The electronic device 50 includes a memory 51 and a processor 52, and the processor 52 is used for executing program instructions stored in the memory 51 to achieve The steps in any of the above-mentioned saliency detection model training method embodiments and/or the steps in the saliency detection method embodiments. In an implementation scenario, the electronic device 50 may include, but is not limited to, medical equipment, microcomputers, desktop computers, and servers. In addition, the electronic device 50 may also include mobile devices such as notebook computers and tablet computers, which are not described here. limited.

The processor 52 is used to control itself and the memory 51 to implement the steps in any of the above-mentioned embodiments of the training method for the saliency detection model. The processor 52 may also be referred to as a CPU (Central Processing Unit, central processing unit). The processor 52 may be an integrated circuit chip with signal processing capabilities. The processor 52 may also be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), or a field-programmable gate array (FPGA). Or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. Additionally, the processor 52 may be commonly implemented by an integrated circuit die.

The above scheme filters the target sample image according to the missing outline of the salient region of the obtained target sample image of the preset image type, so that the salient region in the retained sample image is relatively complete. , and then use the retained sample images with higher quality to train the saliency detection model, which can make the result of subsequent image detection by the trained saliency detection model more accurate.

Please refer to FIG. 17 , which is a schematic structural diagram of an embodiment of a computer-readable storage medium of the present invention. The computer-readable storage medium 60 stores program instructions 61 that can be executed by the processor, and the program instructions 61 are used to implement the steps in the training method embodiment of any of the above-mentioned saliency detection models and/or the saliency detection method embodiments. step.

The above scheme filters the target sample image according to the missing outline of the salient region of the obtained target sample image of the preset image type, so that the salient region in the retained sample image is relatively complete. , and then use the retained sample images with higher quality to train the saliency detection model, which can make the result of subsequent image detection by the trained saliency detection model more accurate.

In some embodiments, the functions or modules included in the apparatus provided in the embodiments of the present invention may be used to execute the methods described in the above method embodiments, and for implementation, reference may be made to the descriptions in the above method embodiments.

The above descriptions of the various embodiments tend to emphasize the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, details are not repeated herein.

In the several embodiments provided by the present invention, it should be understood that the disclosed method and apparatus may be implemented in other manners. For example, the device implementations described above are only illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other divisions. For example, units or elements may be combined or integrated. to another system, or some features can be ignored, or not implemented. Another point, shown or discussed mutual coupling or direct coupling or communication connection The connection may be indirect coupling or communication connection of devices or units through some interfaces, and may be in electrical, mechanical or other forms.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware, or can be implemented in the form of software functional units. The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, Several instructions are included to cause a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, removable hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or CD, etc. medium.

Industrial Applicability

The embodiment of the present invention discloses a saliency detection method and a training method, device and computer-readable storage medium of the model. The training method of the saliency detection model includes: acquiring at least one sample image, wherein at least one sample image The image includes the target sample image belonging to the preset image type; based on the missing contour of the salient region in the target sample image, the target sample image filter the image; use the saliency detection model to detect the filtered sample image to obtain the predicted position information about the salient region in the sample image; based on the marked position information and predicted position information of the salient region in the sample image , adjust the parameters of the saliency detection model. In the above solution, the accuracy of the output result of the saliency detection model can be improved by screening the sample images and then using the filtered sample images to train the saliency detection model.

S11~S14: Steps

Claims (14)

A training method for a saliency detection model, the method is executed by an electronic device, the method includes: acquiring at least one sample image, wherein the at least one sample image includes a target sample belonging to a preset image type image; filter the target sample image based on the missing outline of the saliency region in the target sample image; use a saliency detection model to detect the filtered sample image to obtain the The predicted position information about the saliency region in the sample image; based on the marked position information about the saliency region in the sample image and the predicted position information, the parameters of the saliency detection model are adjusted; wherein, the Filtering the target sample image based on the missing contour of the salient region in the target sample image includes: filling the contour of the salient region in the target sample image to obtain a filled sample image obtaining the difference with respect to the saliency region in the filling sample image and the target sample image; and filtering the target sample image when the difference meets a preset requirement. The method according to claim 1, wherein the preset requirement is that the difference is greater than a preset difference value; the contour of the salient region in the target sample image is filled to obtain a filled sample image Like, including: Perform a closing operation on the target sample image to obtain the fill-in sample image; the obtaining the difference between the fill-in sample image and the target sample image with respect to the salient region includes: obtaining the saliency region Fill in the first area of the sample image with respect to the saliency region, and the second area of the target sample image with respect to the saliency region; determine the difference between the first area and the second area for the difference. The method according to claim 2, wherein, after filtering the target sample image based on the absence of contours of salient regions in the target sample image, the method further comprises: based on the Filling in the position information of the salient region of the sample image, to obtain the marked position information of the target sample image with respect to the salient region. The method of claim 1 or 2, wherein the at least one sample image includes multiple image types. The method according to claim 4, wherein the multiple image types include at least two of images obtained by photographing real objects, hand-drawn drawings, and cartoon drawings. The method according to claim 1, wherein the adjusting the parameters of the saliency detection model based on the labeled location information and the predicted location information of the saliency region based on the sample image includes: the labeled position information and the predicted position information of the saliency region of the sample image, and obtain the first loss; weighting the first loss of each primitive in the sample image to obtain the second loss of the sample image; adjusting the parameters of the saliency detection model based on the second loss. The method according to claim 6, wherein the weight of the first loss of the primitive is related to the boundary distance of the primitive, and the boundary distance of the primitive is the boundary between the primitive and the real saliency area The real saliency area is the saliency area defined by the annotation location information in the sample image. The method according to claim 7, wherein the smaller the boundary distance of the primitive, the greater the weight of the first loss of the primitive. The method according to claim 1 or 2, wherein the saliency detection model includes at least one of the following: the saliency detection model is the network structure of MobileNetV3, and the saliency detection model includes a feature extraction sub-network and a first detection sub-network and a second detection sub-network; the saliency detection model is used to detect the filtered sample image to obtain the predicted position information about the saliency region in the sample image , including: using the feature extraction sub-network to perform feature extraction on the sample image to obtain a feature map corresponding to the sample image; using the first detection sub-network to perform initial detection on the feature map, Obtain the initial position information about the salient region in the sample image; fuse the feature map and the initial position information to obtain a fusion Result: using the second detection sub-network to perform final detection on the fusion result to obtain the predicted position information of the sample image. The method according to claim 1 or 2, wherein, before using the saliency detection model to detect the filtered sample image to obtain the predicted position information about the saliency region in the sample image , the method further includes: performing data enhancement on the filtered sample image; wherein, the data enhancement method includes filling the background area in the sample image other than the salient area. A saliency detection method, comprising: acquiring an image to be processed; using a saliency detection model to process the image to be processed to obtain predicted position information about a saliency area in the content of the image to be processed, wherein the The saliency detection model described above is trained by any method of request items 1 to 10. The method according to claim 11, wherein after the to-be-processed image is processed by using a saliency detection model to obtain predicted position information about the saliency region in the content of the to-be-processed image, the The method further includes: using the predicted position information, performing bone extraction on the salient region to obtain a target bone; selecting a bone model for the target bone as a source bone; driving a first animation related to the source bone Data transferred to the stated purpose On the target bone, the second animation driving data of the target bone is obtained. An electronic device includes a memory and a processor, the processor is configured to execute program instructions stored in the memory, so as to implement the method described in any one of claim 1 to 12. A computer-readable storage medium on which program instructions are stored, wherein the program instructions implement the method described in any one of claim 1 to 12 when the program instructions are executed by a processor.

Images