WO2024001675A1 - 虚拟特效挂载平面检测方法、装置、设备及存储介质 - Google Patents

虚拟特效挂载平面检测方法、装置、设备及存储介质 Download PDF

Info

Publication number
WO2024001675A1
WO2024001675A1 PCT/CN2023/098144 CN2023098144W WO2024001675A1 WO 2024001675 A1 WO2024001675 A1 WO 2024001675A1 CN 2023098144 W CN2023098144 W CN 2023098144W WO 2024001675 A1 WO2024001675 A1 WO 2024001675A1
Authority
WO
WIPO (PCT)
Prior art keywords
corner
frame
point
target
line segment
Prior art date
Application number
PCT/CN2023/098144
Other languages
English (en)
French (fr)
Inventor
徐勇志
王鹏
郭亨凯
Original Assignee
北京字跳网络技术有限公司
脸萌有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京字跳网络技术有限公司, 脸萌有限公司 filed Critical 北京字跳网络技术有限公司
Publication of WO2024001675A1 publication Critical patent/WO2024001675A1/zh

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/24—Aligning, centring, orientation detection or correction of the image
    • G06V10/245—Aligning, centring, orientation detection or correction of the image by locating a pattern; Special marks for positioning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/24—Aligning, centring, orientation detection or correction of the image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/762—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using clustering, e.g. of similar faces in social networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/762—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using clustering, e.g. of similar faces in social networks
    • G06V10/763—Non-hierarchical techniques, e.g. based on statistics of modelling distributions

Definitions

  • Embodiments of the present disclosure relate to the field of image processing technology, and in particular, to a virtual special effects mounting plane detection method, device, electronic equipment, computer-readable storage media, computer program products, and computer programs.
  • the mounting plane is usually based on the gradient of pixels in the image. The information is certain.
  • Embodiments of the present disclosure provide a virtual special effects mounting plane detection method, device, electronic equipment, computer-readable storage media, computer program products, and computer programs.
  • embodiments of the present disclosure provide a method for detecting a virtual special effects mounting plane, including:
  • the edge corners are used to generate at least two quadrilateral structural frames, which represent the outline of a plane of the object in the image to be detected; each of the structures is obtained through the target elimination point corresponding to the structural frame.
  • the matching degree of the frame and based on the matching degree, determine the target frame, wherein the object plane corresponding to the target frame is used to mount virtual special effects, and the matching degree indicates that the object plane corresponding to the structure frame is suitable for mounting The extent of virtual special effects.
  • an embodiment of the present disclosure provides a device for detecting a virtual special effects mounting plane, including:
  • An acquisition module is used to obtain a contour line segment, and obtain a corresponding edge corner point based on the contour line segment, wherein the contour line segment represents the outline of the object in the image to be detected, and the edge corner point is between the contour line segments. the intersection point;
  • a generation module configured to generate at least two quadrilateral structural frames through the edge corners, the structural frames representing the outline of a plane of the object in the image to be detected;
  • a detection module configured to obtain the matching degree of each structural frame through the target vanishing point corresponding to the structural frame, and determine the target frame based on the matching degree, wherein the object plane corresponding to the target frame is used for When mounting virtual special effects, the matching degree represents the degree to which the object plane corresponding to the structure frame is suitable for mounting virtual special effects.
  • an electronic device including:
  • a processor and a memory communicatively connected to the processor
  • the memory stores computer execution instructions
  • the processor executes the computer execution instructions stored in the memory to implement the virtual special effects mounting plane detection method described in the first aspect and various possible designs of the first aspect.
  • embodiments of the present disclosure provide a computer-readable storage medium.
  • Computer-executable instructions are stored in the computer-readable storage medium.
  • the processor executes the computer-executable instructions, the above first aspect and the first aspect are implemented.
  • the virtual special effects mounting plane detection method is described.
  • embodiments of the present disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the virtual special effects mounting plane detection described in the first aspect and various possible designs of the first aspect. method.
  • embodiments of the present disclosure provide a computer program that, when executed by a processor, implements the virtual special effects mounting plane detection method as described in the first aspect and various possible designs of the first aspect.
  • the virtual special effects mounting plane detection method, device, electronic equipment, computer-readable storage medium, computer program product and computer program obtained by this embodiment obtain the contour line segment and obtain the corresponding edge corner point based on the contour line segment.
  • the contour line segment represents the outline of the object in the image to be detected
  • the edge corner point is the intersection point between the contour line segments; through the edge corner point, at least two quadrilateral structural borders are generated, and the structural border Characterize the outline of a plane of the object in the image to be detected; obtain the matching degree of each structural frame through the target vanishing point corresponding to the structural frame, and determine the target frame based on the matching degree, where,
  • the object plane corresponding to the target frame is used to mount virtual special effects, and the matching degree represents the degree to which the object plane corresponding to the structure frame is suitable for mounting virtual special effects.
  • Figure 1 is an application scenario diagram of the virtual special effects mounting plane detection method provided by an embodiment of the present disclosure.
  • FIG. 2 is a schematic flowchart 1 of a method for detecting a virtual special effects mounting plane provided by an embodiment of the present disclosure.
  • FIG. 3 is a schematic diagram of a contour segment and an edge corner provided by an embodiment of the present disclosure.
  • Figure 4 is a schematic diagram of a clustering fusion process provided by an embodiment of the present disclosure.
  • Figure 5 is a schematic diagram of generating a structure frame based on the combination of corner structures provided by an embodiment of the present disclosure.
  • FIG. 6 is a schematic flowchart 2 of a method for detecting a virtual special effects mounting plane provided by an embodiment of the present disclosure.
  • Figure 7 is a schematic diagram of a target corner structure including a corner structure.
  • Figure 8 is a schematic diagram of a target corner structure including two corner structures.
  • Figure 9 is a schematic diagram of a visible angle provided by an embodiment of the present disclosure.
  • Figure 10 is a structural block diagram of a virtual special effects mounting plane detection device provided by an embodiment of the present disclosure.
  • FIG. 11 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure.
  • FIG. 12 is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present disclosure.
  • FIG 1 is an application scenario diagram of the virtual special effects mounting plane detection method provided by the embodiment of the present disclosure.
  • the image to be processed can be detected On the surface of each object (such as the wall in the figure), there are mounting locations (shown as area a and area b in the figure) that can be used to mount "advertising information”.
  • the virtual special effects of the corresponding "advertising information" can be mounted to the corresponding mounting locations respectively, thereby achieving the purpose of dynamically inserting advertising information into pictures and videos.
  • advertising information in other specific application scenarios, after determining the mounting location, you can also mount other virtual special effects, such as photos of people, virtual props, etc. The effects are similar and will not be described again here.
  • the mounting position corresponding to the virtual special effects for example, mounting "advertising information” on the "wall” or "house facade” in the picture, so as to achieve visual consistency with the image content and the quality of the mounted image.
  • the mounting plane is usually determined based on the gradient information of pixels in the image.
  • line segments are extracted based on the gradient information between pixel points in the image, and then based on the positional relationship between the line segments, the mounting plane is determined.
  • the structural border of the loading plane is purely based on the gradient information of pixel grayscale without any semantic information.
  • the length of the line segments is usually incomplete and the endpoints of the line segments are inaccurate.
  • the structural borders based on line segment detection in related technologies have poor robustness and rules. Complex and slow.
  • the mounting plane facing the camera shooting center point is rectangular, while the mounting plane diagonally opposite the camera shooting center point
  • the plane may be a trapezoid
  • the shape is complex, and it is difficult to perform a large number of annotations. Therefore, general object detection based on deep learning can only detect objects in the image and generate a two-dimensional rectangular box describing the image position, but it is difficult to detect the mounting plane.
  • the inspection of the mounting plane in the image in related technologies has problems such as low detection accuracy and poor robustness.
  • Embodiments of the present disclosure provide a method for detecting a virtual special effects mounting plane.
  • the structural features of the object in the image are extracted, and a structural frame representing the outline of the object plane is generated.
  • the vanishing points can then be used to express Based on the characteristics of the image shooting angle, the structure frame is screened through the elimination point to determine the object plane suitable for mounting virtual special effects, thereby solving the above problems.
  • FIG 2 is a schematic flowchart 1 of a method for detecting a virtual special effects mounting plane provided by an embodiment of the present disclosure.
  • the method of this embodiment can be applied to electronic equipment such as terminal equipment and servers.
  • the virtual special effects mounting plane detection method includes:
  • Step S101 Obtain the contour line segment, and obtain the corresponding edge corner point based on the contour line segment, where the contour line segment represents the outline of the object in the image to be detected, and the edge corner point is the intersection point between the contour line segments.
  • the contour line segments are all straight line segments obtained after detecting the target image and representing the outline of the object in the image to be processed. There are usually multiple contour line segments. The extension direction of each contour line segment depends on the specific object structure. Identical or different, some contour line segments can intersect, and the corner point where the contour line segments intersect is the edge corner point.
  • Figure 3 is a schematic diagram of a contour line segment and edge corners provided by an embodiment of the present disclosure. As shown in Figure 3, the image can be a video frame. After detecting the video frame, multiple images representing the outline of the house are obtained.
  • a line segment is a contour segment, where, based on specific recognition results, the contour segment can be all or part of the line segments that constitute the object (such as the house in the picture).
  • the intersection point between the contour line segments can be determined, that is, the edge corner point.
  • the edge corner point is determined by the intersection relationship of the contour line segments that characterize the outline of the object, so it can express the object's shape. Structure.
  • the process of detecting the image to be processed and obtaining the contour line segments can be obtained by identifying the line segments of the image to be processed using a pre-trained neural network model. The specific acquisition method will not be described again.
  • the endpoints of two contour line segments intersect and coincide at one point (i.e., the two contour line segments forming an "L" shape).
  • the edge corner point is the point where the endpoints of the two contour line segments coincide. More specifically, the fact that the endpoints of two contour line segments intersect and coincide with one point means that the coordinate distance corresponding to the endpoints of the two contour line segments is less than a preset threshold, which will not be described again here.
  • the endpoint of one contour segment intersects with the middle of another contour segment (that is, two contour segments forming a "T" shape). In this case, the endpoint that intersects with the middle of the contour segment is, that is, is the edge corner point.
  • the two contour segments may intersect with each other in the middle of the segment (that is, two contour segments forming an "X" shape). In this case, the two contour segments do not have edge corners. .
  • the step of obtaining the corresponding edge corner points based on the contour line segments in step S101 includes:
  • Step S1011 Obtain at least two intersection corner points based on the intersection relationship of each contour line segment.
  • Step S1012 Based on the density clustering algorithm, perform clustering and fusion on the intersecting corner points to generate edge corner points.
  • the intersection relationship of each contour line segment can be realized through an intersection detection algorithm between line segments.
  • the specific implementation method is related to the technology and will not be described again.
  • the obtained intersection corner points are merged based on the density clustering algorithm, so that the similar intersection corner points are merged into the same corner point, that is, the edge corner point.
  • density clustering is also called density-based clustering. This type of algorithm assumes that the clustering structure can be determined by the tightness of the sample distribution. Normally, density clustering algorithms examine the connectivity between samples from the perspective of sample density, and continuously expand clusters based on connectable samples to obtain the final clustering result. The specific implementation method of density clustering will not be described again.
  • FIG. 4 is a schematic diagram of a clustering fusion process provided by an embodiment of the present disclosure.
  • L1, L2, L3, L4, and L5 are five contour line segments in the image to be processed. According to the five contour line segments From the intersection relationship between P1, P2, P3, P4, P5, and P6, a total of 6 intersection corners can be obtained. Afterwards, after clustering and fusion, P1, P2, P3, and P4 among them are determined as edge angles.
  • Step S102 Generate at least two quadrilateral structural frames through edge corners.
  • the structural frames represent a plane outline of the object in the image to be detected.
  • each edge corner point corresponds to two contour line segments.
  • a closed quadrilateral that is, a structural border
  • the quadrilateral structural frame constructed based on the edge corner points can represent the outline of a plane of the object in the image to be detected.
  • each edge corner point can be traversed in turn to form an "L-shaped" structure composed of two contour line segments corresponding to the edge corner point, and an "L-shaped” structure composed of two contour line segments corresponding to other edge corner points. L-shaped" structures are combined, and at least two edge corner points that can form a closed quadrilateral are detected, thereby generating a structure frame.
  • step S102 the specific implementation steps of step S102 include:
  • Step S1021 Obtain the corner structure corresponding to each edge corner point.
  • the corner structure includes two contour line segments constituting the edge corner point.
  • Step S1022 According to the positional relationship between the corner structures corresponding to each edge corner point, obtain at least one set of corner structure combinations.
  • the corner structure combination includes at least one corner structure, and the corner structures in the corner structure combination belong to For the same quadrilateral, when the corner structure combination includes more than two corner structures, at least one outline segment in any corner structure in the corner structure combination partially overlaps with another corner structure.
  • Step S1023 Generate a structural frame based on the combination of corner structures.
  • the corner structure refers to two "L"-shaped contour line segments corresponding to the edge corner points.
  • the lengths and directions of the corner structures corresponding to each edge corner point may be different. Therefore, endpoint intersections and line segments may overlap between the corner structures corresponding to each edge corner point, and then combined to form a quadrilateral.
  • the validity of the corner structures is first tested before traversing possible combinations of each corner structure.
  • determine whether the corner structure is a valid corner structure through at least one of the following conditions:
  • Condition 3 The lengths of the two contour segments of the corner structure are within the preset range.
  • the validity detection of the corner structure can be achieved by at least one of the above three conditions. Among them, the three-dimensional angle between the two contour line segments can be normalized based on the two-dimensional angle and the preset camera focal length parameter. Obtained by unified calculation, the details will not be repeated. Through the above-mentioned process of detecting the validity of corner structures, invalid corner structures can be removed, thereby reducing the time-consuming process of traversing combinations of corner structures to generate combinations of corner structures and improving detection efficiency.
  • edge corner points can express the structural characteristics of the object, when the edge corner points are used as the vertex corners of the outline of a plane of the object in the image to be processed, its corresponding corner structure is the plane of the object in the image to be processed. Two sides of the silhouette. Furthermore, based on the corner structure combination composed of multiple corner structures, when the multiple corner structures in the corner structure combination belong to the same quadrilateral, and some line segments or endpoints of the corner structures overlap, then it can be based on Combining corner structures to try to generate structural borders. More specifically, whether the corner structure in the corner structure combination can be generated can be judged by whether the contour line segment of each corner structure belongs to two vanishing points (i.e., the horizontal vanishing point and the vertical vanishing point). Structural borders.
  • Figure 5 is a schematic diagram of generating a structural frame based on a combination of corner structures provided by an embodiment of the present disclosure.
  • the combination of corner structures includes corner structures C1, C2, and C3, where C1 includes an outline segment L1. and L2, C2 includes contour line segments L3 and L4, and C3 includes contour line segments L5 and L6.
  • This embodiment In, by traversing the positional relationship between the edge and corner structures corresponding to the edge corner points, a matching edge and corner structure combination is obtained, thereby generating a structural border, and using the structural characteristics of the object to achieve accurate positioning of the object plane in the image to be processed. Improved the detection accuracy of the finalized mounting plane.
  • Step S103 Obtain the matching degree of each structure frame through the target vanishing point corresponding to the structure frame, and determine the target frame based on the matching degree.
  • the object plane corresponding to the target frame is used to mount virtual special effects, and the matching degree represents the structure frame.
  • the corresponding object plane is suitable for mounting virtual special effects.
  • a corresponding object plane can be determined based on the position of the structure frame in the image to be processed.
  • not all object planes are suitable for mounting virtual special effects. For example, the area is too small, the display angle of the object plane in the image is too small, etc. Therefore, it is necessary to further screen the structure frame to determine the corresponding object plane suitable for mounting virtual special effects.
  • the matching degree of the structure frame is evaluated through the target extinction point corresponding to the structure frame, where the matching degree represents the degree to which the object plane corresponding to the structure frame is suitable for mounting virtual special effects.
  • the matching degree can be achieved by specifically normalizing the score. For example, 1 is the highest, that is, the most suitable for mounting virtual special effects; 0 is the lowest, that is, the least suitable for mounting virtual special effects.
  • the matching degree corresponding to the structure border is between (0, 1).
  • the structure border whose matching degree is greater than the matching degree threshold (for example, 0.6) is determined as the target border; or, the structural border with the preset number (for example, 3) of the maximum matching degree is determined as the target
  • the border can be set as needed.
  • the specific implementation steps for obtaining the matching degree of each structural frame through the target vanishing point corresponding to the structural frame include:
  • Step S1031 Obtain the first evaluation value of the structure frame.
  • the first evaluation value represents the number of relevant line segments corresponding to the target elimination point.
  • the relevant line segments are the contour line segments belonging to the target elimination point.
  • Step S1032 Obtain the matching degree of the structure frame according to the first evaluation value.
  • the vanishing point also known as the vanishing point, refers to the intersection point of a set of parallel lines in three-dimensional space after projection on a two-dimensional plane.
  • the elimination point detection of a single frame of two-dimensional image is to detect the intersection point of the projection line of N groups of three-dimensional space parallel lines contained in the two-dimensional image as the elimination point.
  • the structural border of the quadrilateral is composed of the contour segments corresponding to the edge corners, and each contour segment belongs to a unique elimination point. Therefore, through the structural border, at least two elimination points can be determined, that is, the target elimination point. point.
  • the relevant line segments are contour segments whose extension lines pass through the erasing point in the image to be processed.
  • the target erasure point in the steps of this embodiment, it corresponds to the relevant line segment.
  • the greater the number of relevant line segments corresponding to the target erasure point the more extension lines of the contour line segments pass through the target erasure point, thus indicating that the target erasure point is The higher the accuracy of the shadow point, the higher the first evaluation value.
  • the accuracy of the structure border is evaluated through the first evaluation value of the target extinction point.
  • the larger the first evaluation value the higher the matching degree.
  • the structural borders are evaluated from the perspective of accuracy, and one or more structural borders with the highest accuracy are obtained as the target borders, thereby improving the accuracy of checking the mounting plane.
  • the corresponding edge corner points are obtained, where the contour line segments represent the outline of the object in the image to be detected, and the edge corner points are the intersection points between the contour line segments; through the edge angle point, generate at least two quadrilateral structural borders, which represent the outline of a plane of the object in the image to be detected; by The target vanishing point corresponding to the structure frame is used to obtain the matching degree of each structure frame, and based on the matching degree, the target frame is determined.
  • the object plane corresponding to the target frame is used to mount virtual special effects, and the matching degree represents the object plane corresponding to the structure frame. Suitable for mounting virtual effects.
  • the structure frame By detecting the edge corner points, the structure frame is generated, and then the elimination points are used to filter the structure frame, thereby determining the target frame used to locate the mounting plane. Due to the use of the structural characteristics of the object in the image, combined with the elimination points, the The obtained target frame can accurately locate the object plane suitable for mounting virtual special effects in the image to be detected, thereby realizing the detection of the mounting plane, improving the detection accuracy of the mounting plane and the detection robustness in different image scenarios.
  • FIG. 6 is a schematic flowchart 2 of a method for detecting a virtual special effects mounting plane provided by an embodiment of the present disclosure. Based on the embodiment shown in Figure 2, this embodiment further refines steps S102 and S103, and adds the step of determining the mounting direction of virtual special effects.
  • the virtual special effects mounting plane detection method includes:
  • Step S201 Obtain the contour line segment, and obtain the corresponding edge corner point based on the contour line segment, where the contour line segment represents the outline of the object in the image to be detected, and the edge corner point is the intersection point between the contour line segments.
  • Step S202 Obtain the corner structure corresponding to each edge corner point.
  • the corner structure includes two contour line segments constituting the edge corner point.
  • Step S203 According to the positional relationship between the corner structures corresponding to each edge corner point, obtain at least one set of corner structure combinations.
  • the corner structure combination includes at least one corner structure, and the corner structures in the corner structure combination belong to For the same quadrilateral, when the corner structure combination includes more than two corner structures, at least one outline segment in any corner structure in the corner structure combination partially overlaps with another corner structure.
  • step S201 is the step of obtaining the contour line segments and edge corner points
  • step S202-S203 is the step of obtaining the corresponding corner structure based on each edge corner point, and then traversing each corner structure to obtain the combination of corner structures.
  • Step S204 Obtain the number of corner structures included in each corner structure combination.
  • Step S205 Determine a corner structure combination whose number of corner structures is greater than a preset quantity threshold as a target corner structure combination.
  • the group of corner structure combinations obtained in step S203 can be divided into different levels of corner structure combinations based on the number of corner structures contained therein.
  • the corner structure combination only includes an "L" "L-shaped" corner structure is a first-level corner structure combination; a corner structure combination including two "L-shaped” corner structures is a second-level corner structure combination; and so on, the higher the level.
  • the more side and corner structures included in the combination of side and corner structures the more accurate the quadrilateral that can be determined.
  • Filter by the level of the corner structure combination that is, the number of corner structures included in the corner structure combination
  • filter out the corner structure combinations that contain more corner structures For example, filter out the corner structure combinations that include more than two corner structures.
  • the corner structure combination is used as the target corner structure combination for subsequent processing, reducing the number of corner structure combinations to be detected, which can further improve the detection efficiency and shorten the detection time of the mounting plane.
  • the level of the corner structure combination (that is, the number of corner structures included in the corner structure combination) can be sorted , in order from high to low, the corner structure combination of the highest level (the largest number of corner structures) is prioritized to be determined as the target corner structure combination, and then the subsequent steps S206, S207 and other steps are performed in order to conduct subsequent matching degree evaluation. , after passing the re-evaluation, generate a corresponding target border frame (i.e.
  • step S212 and then return to step S205, and then sort according to the level, and determine the corner structure combination with a lower level as the target corner structure combination, and proceed to the next round Process until the number of target frames meets the preset number, thereby improving the overall detection efficiency and achieving the purpose of quickly locating the mounting plane that meets the requirements.
  • Step S206 Obtain the horizontal erasing points and vertical erasing points corresponding to each corner structure in the target corner structure combination.
  • Step S207 Generate a first relevant line segment corresponding to the horizontal erasing point and a second relevant line segment corresponding to the vertical erasing point, and form a structural frame based on the first relevant line segment, the second relevant line segment and the corner structure, wherein the first The first end of the relevant line segment intersects the first end of the second relevant line segment, the second end of the first relevant line segment intersects the end point of one contour segment, and the second end of the second relevant line segment intersects the end point of another contour segment.
  • Steps S206-S207 are the specific process of constructing the structure frame based on the target corner structure combination. The following is a detailed description based on the situation:
  • FIG. 7 is a schematic diagram of a target corner structure including a corner structure.
  • the target corner structure includes a corner structure S1.
  • S1 is composed of contour line segments L1 and L2, that is, the target corner structure combination provides a quadrilateral.
  • L3 i.e., the first relevant line segment
  • L4 i.e., the (i.e., the second relevant line segment)
  • FIG 8 is a schematic diagram of a target corner structure including two corner structures.
  • the target corner structure includes corner structures S1 and S2.
  • S1 is composed of contour line segments L1 and L2
  • S2 is composed of contour line segment L3. and L4, in which L2 and L3 partially overlap, that is, the target corner structure combination provides three sides of the quadrilateral.
  • the horizontal erasing point P1 corresponding to L1 and the vertical erasing point P2 corresponding to L2 are obtained.
  • L3 corresponds to the vertical erasing point P2
  • L4 corresponds to the horizontal erasing point P1.
  • L4 and L5 are on the same straight line and partially overlap (the end points overlap)
  • L2 and L3 are on the same straight line and partially overlap
  • L4+L5 is the union of L4 and L5
  • L2+L3 is the union of L2 and L3.
  • the number of corner structures in the target corner structure combination The less, the longer the length of the first and second correlation line segments that need to be supplemented based on the vertical vanishing point and the horizontal vanishing point, and the less the expression of the real object structural features in the corresponding image to be processed, so the detection
  • the mounting plane is less accurate. Therefore, in the step of determining the target corner structure combination, by selecting a corner structure combination including a larger number of corner structures as the target corner structure combination, the accuracy of the mounting plane detection can be improved.
  • Step S208 Obtain multiple candidate elimination points through each contour line segment and edge corner point, wherein the relevant line segments corresponding to the candidate elimination point pass through the edge corner point.
  • Step S209 Determine the corresponding target elimination point according to the edge corner point corresponding to the structural frame.
  • the candidate elimination points include horizontal and vertical elimination points, and the horizontal and vertical elimination points can be two contours corresponding to each edge corner point. Obtained by detecting line segments. If the obtained horizontal cancellation points and/or vertical cancellation points are real (that is, the number of relevant line segments of the cancellation points is greater than the quantity threshold), then the generated The horizontal cancellation point to be selected and/or the vertical cancellation point to be selected are, for example, passing through edge corners.
  • the corresponding target erasure point can be determined, more specifically, the target erasure point
  • the points include horizontal target erasing points and vertical target erasing points.
  • multiple candidate elimination points are obtained based on the contour line segments and edge corner points, and the object structure information contained in the edge corner points is used to exclude other elimination points that have nothing to do with the edge corner points. Therefore, in subsequent detection In the process, the detection time is reduced and the detection efficiency is improved.
  • Step S210 Obtain the first evaluation value and at least one second evaluation value of the structure frame.
  • the first evaluation value represents the number of relevant line segments corresponding to the target vanishing point
  • the second evaluation value represents the object plane corresponding to the structure frame in the image to be detected. visual characteristics.
  • the structure borders are evaluated through the first evaluation value and at least one second evaluation value of the structure borders, and an object plane suitable for mounting virtual special effects is screened out.
  • the specific implementation method of the first evaluation value has been introduced in the embodiment shown in FIG. 2 and will not be described again here.
  • the second evaluation value is based on the first evaluation value and further combined with the visual characteristics of the object plane corresponding to the structure frame in the image to be detected to evaluate whether the object plane corresponding to the structure frame is suitable for mounting virtual special effects.
  • the second evaluation value includes at least one of the following: visual angle, image area, number of internal associated line segments, and number of internal conflicting line segments.
  • the visible angle is the angle between the object plane corresponding to the structure frame and the camera centerline;
  • Figure 9 is a schematic diagram of a visible angle provided by an embodiment of the present disclosure, and the camera centerline is perpendicular to the image
  • the greater the angle between the object plane and the camera center line maximum 90 degrees
  • the more the object plane faces the user's viewing angle so the display effect is better, and the second evaluation value is higher.
  • the smaller the angle between the object plane and the camera centerline maximum is 0 degrees
  • the further the object plane deviates from the user's viewing angle so the display effect is worse and the second evaluation value is lower.
  • the image area is the area of the object plane corresponding to the structure frame in the image to be detected.
  • the larger the image area the better the display effect, and the higher the second evaluation value, which will not be described again.
  • internal associated line segments are contour line segments that are located within the structure border and belong to the target fade point corresponding to the structure border; internal conflict line segments are located within the structure border and do not belong to the target fade point corresponding to the structure border. Contour line segment.
  • both internal associated line segments and internal conflict line segments are line segments located within the structure border.
  • virtual special effects such as advertising information
  • it will Mount virtual effects on areas with no valid information, such as walls with repeated textures.
  • Step S211 Perform weighted calculation based on the first evaluation value and the second evaluation value to obtain the matching degree of the structure border.
  • Step S212 Determine the target border based on the matching degree.
  • the matching degree of the structure frame can be obtained by performing a weighted calculation on the above-mentioned first evaluation value and a plurality of second evaluation values.
  • the weighting coefficients of the first evaluation value and each second evaluation value can be set according to specific needs, and are not specifically limited here.
  • a weighted calculation is performed through the first evaluation value and a plurality of second evaluation values to obtain the matching degree corresponding to the structure frame, and then one or more targets corresponding to the object planes that are most suitable for mounting virtual special effects are selected.
  • the border enables accurate detection of the mounting plane and improves the visual performance of mounted virtual special effects.
  • Step S213 Obtain the preset camera focal length parameter, and obtain the three-dimensional direction of the target frame based on the camera focal length parameter and the target vanishing point corresponding to the target frame.
  • the three-dimensional direction is the normal direction of the plane corresponding to the target frame.
  • Step S214 Determine the mounting direction of the virtual special effects according to the three-dimensional direction corresponding to the target frame.
  • the target frame is a two-dimensional frame and only one two-dimensional plane can be determined.
  • the preset camera focal length parameter can be used to add corresponding information to the target frame.
  • the three-dimensional direction thereby determining the mounting direction when mounting virtual effects.
  • the camera focal length parameter is f
  • the target erasing point includes the target transverse erasing point and the target longitudinal erasing point
  • the three-dimensional angle corresponding to P1 is P1_3d
  • the three-dimensional angle corresponding to P2 is P2_3d, respectively:
  • P1_3d (p1x,p1y,f)/
  • P2_3d (p2x,p2y,f)/
  • the mounting direction when mounting virtual special effects on the mounting plane is further determined to avoid display errors caused by wrong mounting directions and improve the performance of virtual special effects. Visual performance.
  • FIG. 10 is a structural block diagram of a virtual special effects mounting plane detection device provided by an embodiment of the present disclosure. For convenience of explanation, only parts related to the embodiments of the present disclosure are shown.
  • the virtual special effects mounting plane detection device 3 includes:
  • the acquisition module 31 is used to obtain the contour line segments, and obtain the corresponding edge corner points based on the contour line segments, where the contour line segments represent the outline of the object in the image to be detected, and the edge corner points are the intersection points between the contour line segments;
  • the generation module 32 is used to generate at least two quadrilateral structural borders through edge corners, and the structural borders represent the outline of a plane of the object in the image to be detected;
  • the detection module 33 is used to obtain the matching degree of each structural frame through the target vanishing point corresponding to the structural frame, and determine the target frame based on the matching degree, wherein the object plane corresponding to the target frame is used to mount virtual special effects, and the matching degree Indicates the degree to which the object plane corresponding to the structure frame is suitable for mounting virtual special effects.
  • the generation module 32 is specifically used for:
  • the corner structure includes two contour line segments that constitute the edge corner point. According to the positional relationship between the corner structures corresponding to each edge corner point, at least one set of corner structure combinations is obtained.
  • the corner structure combination includes at least one corner structure, and the corner structures in the corner structure combination belong to the same quadrilateral. When the corner structure combination includes more than two corner structures, any corner structure combination At least one contour line segment in one corner structure partially overlaps with the other corner structure; a structure border is generated based on the combination of corner structures.
  • the generation module 32 when the generation module 32 generates a structural frame according to a combination of corner structures, it is specifically used to: obtain the horizontal and vertical elimination points corresponding to each corner structure in the combination of corner structures. ;Generate transverse vanishing point correspondence The first relevant line segment and the second relevant line segment corresponding to the vertical vanishing point, and a structural frame is formed based on the first relevant line segment, the second relevant line segment and the corner structure, wherein the first end of the first relevant line segment and the second relevant line segment The first end of the relevant line segment intersects, the second end of the first relevant line segment intersects the end point of one contour segment, and the second end of the second relevant line segment intersects the end point of another contour segment.
  • the generation module 32 before generating the structure frame according to the combination of corner structures, is also configured to: obtain the number of corner structures included in each combination of corner structures; set the number of corner structures to be greater than The corner structure combination with the preset quantity threshold is determined as the target corner structure combination; when the generation module 32 generates the structure frame according to the corner structure combination, it is specifically used to: pass each corner structure in the target corner structure combination , generate structure borders.
  • the detection module 33 is specifically configured to: obtain a first evaluation value of the structure frame.
  • the first evaluation value represents the number of relevant line segments corresponding to the target elimination point, and the relevant line segments belong to the target elimination point.
  • the contour line segment of the point; according to the first evaluation value, the matching degree of the structure border is obtained.
  • the detection module 33 is also used to: obtain a second evaluation value of the structure frame, where the second evaluation value represents the visual characteristics of the object plane corresponding to the structure frame in the image to be detected; the detection module 33 When obtaining the matching degree of the structure frame according to the first evaluation value, it is specifically used to: perform weighted calculation based on the first evaluation value and the second evaluation value to obtain the matching degree of the structure frame.
  • the second evaluation value includes at least one of the following: visible angle, image area, number of internal associated line segments, and number of internal conflicting line segments; wherein the visible angle is the angle corresponding to the structure frame.
  • the image area is the area of the object plane corresponding to the structure frame in the image to be detected;
  • the internal associated line segment is the contour line segment located within the structure frame and belonging to the target elimination point corresponding to the structure frame ;
  • Internal conflict line segments are contour segments that are located within the structure frame and do not belong to the target elimination point corresponding to the structure frame.
  • the generation module 32 is also configured to: obtain multiple candidate elimination points through each contour line segment and edge corner point, wherein the relevant line segments corresponding to the candidate elimination point pass through the edge corner points. ; According to the edge corners corresponding to the structure border, determine the corresponding target elimination point.
  • the acquisition module 31 is specifically configured to: obtain at least two intersecting corner points based on the intersection relationship of each contour line segment; perform clustering and fusion of the intersecting corner points based on a density clustering algorithm to generate an edge corner.
  • the detection module 33 is also used to: obtain the preset camera focal length parameter; obtain the three-dimensional direction of the target frame based on the camera focal length parameter and the target vanishing point corresponding to the target frame, and the three-dimensional direction is the target The normal direction of the plane corresponding to the border; determine the mounting direction of the virtual special effects based on the three-dimensional direction corresponding to the target border.
  • the virtual special effects mounting plane detection device 3 provided in this embodiment can execute the technical solution of the above method embodiment. Its implementation principles and technical effects are similar, and will not be described again in this embodiment.
  • FIG 11 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. As shown in Figure 11, the electronic device 4 includes:
  • Processor 41 and memory 42 communicatively connected to processor 41;
  • Memory 42 stores computer execution instructions
  • the processor 41 executes the computer execution instructions stored in the memory 42 to implement the virtual special effects mounting plane detection method in the embodiment shown in FIGS. 2 to 9 .
  • processor 41 and the memory 42 are connected through the bus 43 .
  • the electronic device 900 may be a terminal device or a server.
  • terminal devices may include but are not limited to mobile phones, laptops, digital broadcast receivers, personal digital assistants (Personal Digital Assistant, PDA for short), tablet computers (Portable Android Device, PAD for short), portable multimedia players (Portable Mobile terminals such as Media Player (PMP for short), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and fixed terminals such as digital TVs, desktop computers, etc.
  • PDA Personal Digital Assistant
  • PAD Personal Android Device
  • portable multimedia players Portable Mobile terminals such as Media Player (PMP for short
  • vehicle-mounted terminals such as vehicle-mounted navigation terminals
  • fixed terminals such as digital TVs, desktop computers, etc.
  • the electronic device shown in FIG. 12 is only an example and should not impose any limitations on the functions and scope of use of the embodiments of the present disclosure.
  • the electronic device 900 may include a processing device (such as a central processing unit, a graphics processor, etc.) 901, which may process data according to a program stored in a read-only memory (Read Only Memory, ROM for short) 902 or from a storage device.
  • a processing device such as a central processing unit, a graphics processor, etc.
  • 908 mounts the program in the random access memory (Random Access Memory, RAM for short) 903 to perform various appropriate actions and processing.
  • RAM 903 Random Access Memory
  • various programs and data required for the operation of the electronic device 900 are also stored.
  • the processing device 901, ROM 902 and RAM 903 are connected to each other via a bus 904.
  • An input/output (I/O) interface 905 is also connected to bus 904.
  • the following devices can be connected to the I/O interface 905: input devices 906 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; including, for example, a Liquid Crystal Display (LCD). ), an output device 907 such as a speaker, a vibrator, etc.; a storage device 908 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 909.
  • the communication device 909 may allow the electronic device 900 to communicate wirelessly or wiredly with other devices to exchange data.
  • FIG. 12 illustrates electronic device 900 with various means, it should be understood that implementation or availability of all illustrated means is not required. More or fewer means may alternatively be implemented or provided.
  • embodiments of the present disclosure include a computer program product including a computer program carried on a computer-readable medium, the computer program containing program code for performing the method illustrated in the flowchart.
  • the computer program may be downloaded and installed from the network via communication device 909, or from storage device 908, or from ROM 902.
  • the processing device 901 When the computer program is executed by the processing device 901, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.
  • Embodiments of the present disclosure also include a computer program that, when executed by a processor, implements the above functions defined in the method of the embodiment of the present disclosure.
  • the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two.
  • the computer-readable storage medium may be, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination thereof.
  • Computer readable storage media may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard drive, random access memory (RAM), read only memory (ROM), removable Programmable read-only memory (Electrical Programmable ROM, EPROM or flash memory), optical fiber, portable compact disk read-only memory (Compact Disc ROM, CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.
  • a computer-readable storage medium may be any tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device.
  • a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code therein. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above.
  • a machine-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device .
  • Program code contained on a computer-readable medium can be transmitted using any appropriate medium, including but not limited to: wires, optical cables, radio frequency (Radio Frequency, RF), etc., or any suitable combination of the above.
  • the above-mentioned computer-readable medium may be included in the above-mentioned electronic device; it may also exist independently without being assembled into the electronic device.
  • the computer-readable medium carries one or more programs.
  • the electronic device When the one or more programs are executed by the electronic device, the electronic device performs the method shown in the above embodiment.
  • Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional Procedural programming language—such as "C" or a similar programming language.
  • the program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server.
  • the remote computer can be connected to the user's computer through any kind of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or it can be connected to an external computer Computer (e.g. connected via the Internet using an Internet service provider).
  • LAN Local Area Network
  • WAN Wide Area Network
  • each block in the flowchart or block diagram may represent a module, segment, or portion of code that contains one or more logic functions that implement the specified executable instructions.
  • the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown one after another may actually execute substantially in parallel, or they may sometimes execute in the reverse order, depending on the functionality involved.
  • each block of the block diagram and/or flowchart illustration, and combinations of blocks in the block diagram and/or flowchart illustration can be implemented by special purpose hardware-based systems that perform the specified functions or operations. , or can be implemented using a combination of specialized hardware and computer instructions.
  • the units involved in the embodiments of the present disclosure can be implemented in software or hardware.
  • the name of the unit does not constitute a limitation on the unit itself under certain circumstances.
  • the first acquisition unit can also be described as "the unit that acquires at least two Internet Protocol addresses.”
  • exemplary types of hardware logic components include: field programmable gate array (Field Programmable Gate Array, FPGA), application specific integrated circuit (Application Specific Integrated Circuit, ASIC), application specific standard product (Application Specific Standard Product (ASSP), System on Chip (SOC), Complex Programmable Logic Device (CPLD), etc.
  • a machine-readable medium may be a tangible medium that may contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.
  • the machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium.
  • Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any suitable combination of the foregoing.
  • machine-readable storage media would include one or more wire-based electrical connections, laptop disks, hard drives, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.
  • RAM random access memory
  • ROM read only memory
  • EPROM or flash memory erasable programmable read only memory
  • CD-ROM portable compact disk read-only memory
  • magnetic storage device or any suitable combination of the above.
  • a method for detecting a virtual special effects mounting plane including:
  • the edge corners are used to generate at least two quadrilateral structural frames, which represent the outline of a plane of the object in the image to be detected; each of the structures is obtained through the target elimination point corresponding to the structural frame.
  • the matching degree of the frame and based on the matching degree, determine the target frame, wherein the object plane corresponding to the target frame is used to mount virtual special effects, and the matching degree indicates that the object plane corresponding to the structure frame is suitable for mounting Extent of virtual special effects.
  • generating at least two quadrilateral structural frames through the edge corner points includes: obtaining a corner structure corresponding to each of the edge corner points, and the corner structure includes Two contour line segments constituting the edge corner points; according to the positional relationship between the corner structures corresponding to the edge corner points, at least one set of corner structure combinations is obtained, and the corner structure combination includes at least one edge Corner structure, the corner structures in the corner structure combination belong to the same quadrilateral.
  • the corner structure combination includes more than two corner structures, any corner structure in the corner structure combination At least one contour line segment partially overlaps with another corner structure; the structural frame is generated according to the combination of the corner structures.
  • generating the structural frame according to the corner structure combination includes: obtaining the horizontal vanishing point and vertical vanishing point corresponding to each corner structure in the corner structure combination. Elimination point; generate a first relevant line segment corresponding to the horizontal erasure point and a second relevant line segment corresponding to the vertical erasure point, and based on the first relevant line segment, the second relevant line segment and the The corner structure forms a structural frame, wherein the first end of the first related line segment intersects the first end of the second related line segment, and the second end of the first related line segment intersects the end point of an outline line segment, The second end of the second relevant line segment intersects the end point of the other contour line segment.
  • the method before generating the structural frame according to the corner structure combination, further includes: obtaining the number of corner structures included in each of the corner structure combinations; The corner structure combination whose number of corner structures is greater than the preset quantity threshold is determined as the target corner structure combination; generating the structure frame according to the corner structure combination includes: using the target corner structure Combining each corner structure to generate the structure border.
  • obtaining the matching degree of each structural frame through a target vanishing point corresponding to the structural frame includes: obtaining a first evaluation value of the structural frame,
  • the first evaluation value represents the number of relevant line segments corresponding to the target elimination point, and the relevant line segments are contour segments belonging to the target elimination point; according to the first evaluation value, the matching of the structure frame is obtained Spend.
  • the method further includes: obtaining a second evaluation value of the structural frame, the second evaluation value representing the position of the object plane corresponding to the structural frame in the image to be detected.
  • the visual characteristics; according to the first evaluation value, obtaining the matching degree of the structural frame includes: performing a weighted calculation based on the first evaluation value and the second evaluation value to obtain the matching degree of the structural frame.
  • the second evaluation value includes at least one of the following: visual angle, image area, number of internal associated line segments, number of internal conflicting line segments; wherein the visual clip The angle is the angle between the object plane corresponding to the structural frame and the camera centerline; the image area is the area of the object plane corresponding to the structural frame in the image to be detected; the internal associated line segment is located in the Within the structure frame and belonging to the corresponding structure frame The contour line segment of the target fade point; the internal conflict line segment is a contour line segment located within the structure frame and not belonging to the target fade point corresponding to the structure frame.
  • the method further includes: obtaining a plurality of candidate elimination points through each of the contour segments and the edge corner points, wherein the candidate elimination points correspond to The relevant line segment passes through the edge corner point; according to the edge corner point corresponding to the structure frame, the corresponding target elimination point is determined.
  • obtaining the corresponding edge corner points based on the contour line segments includes: obtaining at least two intersecting corner points based on the intersection relationship of each contour line segment; based on a density clustering algorithm, Clustering and fusion is performed on the intersecting corner points to generate the edge corner points.
  • the method further includes: obtaining a preset camera focal length parameter; and obtaining a three-dimensional image of the target frame based on the camera focal length parameter and the target vanishing point corresponding to the target frame.
  • the three-dimensional direction is the normal direction of the plane corresponding to the target frame; the mounting direction of the virtual special effect is determined according to the three-dimensional direction corresponding to the target frame.
  • the structure frame By detecting the edge corner points, the structure frame is generated, and then the elimination points are used to filter the structure frame, thereby determining the target frame used to locate the mounting plane. Due to the use of the structural characteristics of the object in the image, combined with the elimination points, the The obtained target frame can accurately locate the object plane suitable for mounting virtual special effects in the image to be detected, thereby realizing the detection of the mounting plane, improving the detection accuracy of the mounting plane and the detection robustness in different image scenarios.
  • a virtual special effects mounting plane detection device including:
  • An acquisition module is used to obtain a contour line segment, and obtain a corresponding edge corner point based on the contour line segment, wherein the contour line segment represents the outline of the object in the image to be detected, and the edge corner point is between the contour line segments. the intersection point;
  • a generation module configured to generate at least two quadrilateral structural frames through the edge corners, the structural frames representing the outline of a plane of the object in the image to be detected;
  • a detection module configured to obtain the matching degree of each structural frame through the target vanishing point corresponding to the structural frame, and determine the target frame based on the matching degree, wherein the object plane corresponding to the target frame is used for To mount virtual special effects, the matching degree represents the degree to which the object plane corresponding to the structure frame is suitable for mounting virtual special effects.
  • the generation module is specifically configured to: obtain a corner structure corresponding to each edge corner point, where the corner structure includes two contour line segments that constitute the edge corner point. ; According to the positional relationship between the corner structures corresponding to each of the edge corner points, at least one set of corner structure combinations is obtained.
  • the corner structure combination includes at least one corner structure, and the corner structure combinations include The corner structures belong to the same quadrilateral.
  • the corner structure combination includes more than two corner structures, at least one contour line segment in any corner structure in the corner structure combination is consistent with the other corner structure. Partial overlap; the structural border is generated according to the combination of the corner structures.
  • the generation module when the generation module generates the structural frame according to the corner structure combination, it is specifically used to: obtain the lateral direction corresponding to each corner structure in the corner structure combination.
  • the erasing point and the vertical erasing point ; generate the first relevant line segment corresponding to the horizontal erasing point and the second relevant line segment corresponding to the vertical erasing point, and based on the first relevant line segment, the third relevant line segment
  • Two related line segments and the corner structure form a structural frame, wherein the first end of the first related line segment intersects the first end of the second related line segment, and the second end of the first related line segment intersects with a The endpoints of the contour line segments intersect, and the second end of the second related line segment intersects the endpoint of another contour line segment.
  • the generation module before generating the structural frame according to the corner structure combination, is further configured to: obtain the information of the corner structures included in each of the corner structure combinations. Quantity; place the corner structure The corner structure combination whose quantity is greater than the preset quantity threshold is determined as the target corner structure combination; when the generation module generates the structure frame according to the corner structure combination, it is specifically used to: pass the target Each corner structure in the combination of corner structures generates the structure frame.
  • the detection module is specifically configured to: obtain a first evaluation value of the structure border, where the first evaluation value represents the number of relevant line segments corresponding to the target vanishing point. , the relevant line segment is a contour line segment belonging to the target vanishing point; according to the first evaluation value, the matching degree of the structure frame is obtained.
  • the detection module is further configured to obtain a second evaluation value of the structural frame, where the second evaluation value represents the position of the object plane corresponding to the structural frame in the to-be- Detect visual features in the image; when obtaining the matching degree of the structural frame according to the first evaluation value, the detection module is specifically configured to perform weighting based on the first evaluation value and the second evaluation value. Calculate and obtain the matching degree of the structure border.
  • the second evaluation value includes at least one of the following: visual angle, image area, number of internal associated line segments, number of internal conflicting line segments; wherein the visual clip The angle is the angle between the object plane corresponding to the structural frame and the camera centerline; the image area is the area of the object plane corresponding to the structural frame in the image to be detected; the internal associated line segment is located in the Contour line segments within the structural frame and belonging to the target vanishing point corresponding to the structural frame; the internal conflict line segments are contours located within the structural frame and not belonging to the target vanishing point corresponding to the structural frame line segment.
  • the generation module is further configured to: obtain a plurality of candidate shadow elimination points through each of the contour line segments and the edge corner points, wherein the candidate shadow elimination points The relevant line segment corresponding to the point passes through the edge corner point; according to the edge corner point corresponding to the structure frame, the corresponding target elimination point is determined.
  • the acquisition module is specifically configured to: obtain at least two intersecting corner points based on the intersection relationship of each contour line segment; and cluster the intersecting corner points based on a density clustering algorithm. Class fusion, generates the edge corner points.
  • the detection module is further configured to: obtain a preset camera focal length parameter; and obtain the target based on the camera focal length parameter and the target vanishing point corresponding to the target frame.
  • the three-dimensional direction of the frame which is the normal direction of the plane corresponding to the target frame; determine the mounting direction of the virtual special effect according to the three-dimensional direction corresponding to the target frame.
  • an electronic device including: a processor, and a memory communicatively connected to the processor;
  • the memory stores computer execution instructions
  • the processor executes the computer execution instructions stored in the memory to implement the virtual special effects mounting plane detection method described in the first aspect and various possible designs of the first aspect.
  • a computer-readable storage medium is provided.
  • Computer-executable instructions are stored in the computer-readable storage medium.
  • a processor executes the computer-executed instructions, Implement the virtual special effects mounting plane detection method described in the first aspect and various possible designs of the first aspect.
  • embodiments of the present disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the virtual special effects mounting plane detection described in the first aspect and various possible designs of the first aspect. method.
  • embodiments of the present disclosure provide a computer program that, when executed by a processor, implements the virtual special effects mounting plane detection method as described in the first aspect and various possible designs of the first aspect.
  • the above description is only Preferred embodiments of the present disclosure and description of the technical principles applied.
  • Those skilled in the art should understand that the disclosure scope involved in the present disclosure is not limited to technical solutions composed of specific combinations of the above technical features, but should also cover solutions composed of the above technical features or without departing from the above disclosed concept.
  • a technical solution is formed by replacing the above features with technical features with similar functions disclosed in this disclosure (but not limited to).

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Artificial Intelligence (AREA)
  • Databases & Information Systems (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Health & Medical Sciences (AREA)
  • Probability & Statistics with Applications (AREA)
  • Image Analysis (AREA)

Abstract

本公开实施例提供一种虚拟特效挂载平面检测方法、装置、电子设备、计算机可读存储介质、计算机程序产品及计算机程序，通过获取轮廓线段，并基于轮廓线段，得到对应的边缘角点，其中，轮廓线段表征待检测图像中物体的轮廓，边缘角点为轮廓线段之间的交点；通过边缘角点，生成至少两个四边形的结构边框，结构边框表征待检测图像中的物体的一个平面的轮廓；通过结构边框对应的目标消影点，获得各结构边框的匹配度，并基于匹配度，确定目标边框，其中，目标边框对应的物体平面用于挂载虚拟特效，匹配度表征结构边框对应的物体平面适合于挂载虚拟特效的程度。

Description

虚拟特效挂载平面检测方法、装置、设备及存储介质
相关申请的交叉引用
本公开要求于2022年6月29日提交的申请号为202210761378.8、发明名称为“虚拟特效挂载平面检测方法、装置、设备及存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本公开中。
技术领域
本公开实施例涉及图像处理技术领域，尤其涉及一种虚拟特效挂载平面检测方法、装置、电子设备、计算机可读存储介质、计算机程序产品及计算机程序。
背景技术
当前，在图片和视频中添加虚拟特效时，需要首先定位一个挂载平面，之后将虚拟特效素材添加在该挂载平面上进行显示，相关技术中，挂载平面通常是基于图像中像素的梯度信息确定的。
然而，相关技术中检测挂载平面的方法，存在检测准确性低，不同图像场景下鲁棒性差等问题。
发明内容
本公开实施例提供一种虚拟特效挂载平面检测方法、装置、电子设备、计算机可读存储介质、计算机程序产品及计算机程序。
第一方面，本公开实施例提供一种虚拟特效挂载平面检测方法，包括：
获取轮廓线段，并基于所述轮廓线段，得到对应的边缘角点，其中，所述轮廓线段表征待检测图像中物体的轮廓，所述边缘角点为所述轮廓线段之间的交点；通过所述边缘角点，生成至少两个四边形的结构边框，所述结构边框表征所述待检测图像中的物体的一个平面的轮廓；通过所述结构边框对应的目标消影点，获得各所述结构边框的匹配度，并基于所述匹配度，确定目标边框，其中，所述目标边框对应的物体平面用于挂载虚拟特效，所述匹配度表征所述结构边框对应的物体平面适合于挂载虚拟特效的程度。
第二方面，本公开实施例提供一种虚拟特效挂载平面检测装置，包括：
获取模块，用于获取轮廓线段，并基于所述轮廓线段，得到对应的边缘角点，其中，所述轮廓线段表征待检测图像中物体的轮廓，所述边缘角点为所述轮廓线段之间的交点；
生成模块，用于通过所述边缘角点，生成至少两个四边形的结构边框，所述结构边框表征所述待检测图像中的物体的一个平面的轮廓；
检测模块，用于通过所述结构边框对应的目标消影点，获得各所述结构边框的匹配度，并基于所述匹配度，确定目标边框，其中，所述目标边框对应的物体平面用于挂载虚拟特效，所述匹配度表征所述结构边框对应的物体平面适合于挂载虚拟特效的程度。
第三方面，本公开实施例提供一种电子设备，包括：
处理器，以及与所述处理器通信连接的存储器；
所述存储器存储计算机执行指令；
所述处理器执行所述存储器存储的计算机执行指令，以实现如上第一方面以及第一方面各种可能的设计所述的虚拟特效挂载平面检测方法。
第四方面，本公开实施例提供一种计算机可读存储介质，所述计算机可读存储介质中存储有计算机执行指令，当处理器执行所述计算机执行指令时，实现如上第一方面以及第一方面各种可能的设计所述的虚拟特效挂载平面检测方法。
第五方面，本公开实施例提供一种计算机程序产品，包括计算机程序，该计算机程序被处理器执行时实现如上第一方面以及第一方面各种可能的设计所述的虚拟特效挂载平面检测方法。
第六方面，本公开实施例提供了一种计算机程序，所述计算机程序被处理器执行时实现如上第一方面以及第一方面各种可能的设计所述虚拟特效挂载平面检测方法。
本实施例提供的虚拟特效挂载平面检测方法、装置、电子设备、计算机可读存储介质、计算机程序产品及计算机程序，通过获取轮廓线段，并基于所述轮廓线段，得到对应的边缘角点，其中，所述轮廓线段表征待检测图像中物体的轮廓，所述边缘角点为所述轮廓线段之间的交点；通过所述边缘角点，生成至少两个四边形的结构边框，所述结构边框表征所述待检测图像中的物体的一个平面的轮廓；通过所述结构边框对应的目标消影点，获得各所述结构边框的匹配度，并基于所述匹配度，确定目标边框，其中，所述目标边框对应的物体平面用于挂载虚拟特效，所述匹配度表征所述结构边框对应的物体平面适合于挂载虚拟特效的程度。
附图说明
为了更清楚地说明本公开实施例或相关技术中的技术方案，下面将对实施例或相关技术描述中所需要使用的附图作一简单地介绍，显而易见地，下面描述中的附图是本公开的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。
图1为本公开实施例提供的虚拟特效挂载平面检测方法的一种应用场景图。
图2为本公开实施例提供的虚拟特效挂载平面检测方法的流程示意图一。
图3为本公开实施例提供的一种轮廓线段和边缘角点的示意图。
图4为本公开实施例提供的一种聚类融合的过程示意图。
图5为本公开实施例提供的一种基于边角结构组合生成结构边框的示意图。
图6为本公开实施例提供的虚拟特效挂载平面检测方法的流程示意图二。
图7为包括一个边角结构的目标边角结构的示意图。
图8为包括两个边角结构的目标边角结构的示意图。
图9为本公开实施例提供的一种可视夹角的示意图。
图10为本公开实施例提供的虚拟特效挂载平面检测装置的结构框图。
图11为本公开实施例提供的一种电子设备的结构示意图。
图12为本公开实施例提供的电子设备的硬件结构示意图。
具体实施方式
为使本公开实施例的目的、技术方案和优点更加清楚，下面将结合本公开实施例中的附图，对本公开实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本公开一部分实施例，而不是全部的实施例。基于本公开中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本公开保护的范围。
下面对本公开实施例的应用场景进行解释：
本公开实施例提供的虚拟特效挂载平面检测方法，可以应用于各类应用中，为视频或图片增加虚拟特效的应用场景。更具体地，例如在视频(视频中的各帧)中的动态插入广告信息、虚拟道具等。图1为本公开实施例提供的虚拟特效挂载平面检测方法的一种应用场景图，如图1所示，通过本公开实施例提供的虚拟特效挂载平面检测方法，可以检测出待处理图像中的各物体(例如图中的墙壁)的表面上，可以用于挂载“广告信息”的挂载位置(图中示为区域a和区域b)。之后，可以将对应的“广告信息”的虚拟特效，分别挂载至对应的挂载位置，从而实现在图片、视频中动态插入广告信息的目的。当然，除了广告信息外，在其他具体的应用场景下，在确定挂载位置后，还可以挂载其他虚拟特效，例如人物照片、虚拟道具等，实现效果类似，此处不再赘述。
在上述应用场景下，由于在向图像中插入“广告信息”等虚拟特效时，需要避免插入的虚拟特效影响图像的正常展示，避免造成虚拟特效与图像内容的违和感，因此，需要先确定虚拟特效对应的挂载位置，例如，将“广告信息”挂载在图片中的“墙体”或“房屋外立面”上，从而达到和图像内容的视觉一致性，以及挂载后图像的真实性。相关技术中，挂载平面通常是基于图像中像素的梯度信息确定的，具体地，即基于图像中像素点之间的梯度信息，进行线段提取，再基于线段之间的位置关系，确定表征挂载平面的结构边框。然而，基于这类线段的提取纯粹基于像素灰度的梯度信息，没有任何语义信息，导致线段通常长度不完整，且线段端点不准确，基于相关技术中的线段检测的结构边框鲁棒性差，规则复杂且速度慢。此外，由于挂载平面在图像中的视觉形状，受到相机拍摄角度和拍摄内容的影响(参考图1，正对相机拍摄中心点的挂载平面为矩形，而斜对相机拍摄中心点的挂载平面可能是梯形)，形状复杂，难以进行大量标注，因此基于深度学习的通用物体检测，只能够检测图像中的物体，生成描述图像位置的二维矩形框，但难以实现挂载平面的检测。
因此，综上，由于挂载平面的特殊性，相关技术中对图像中挂载平面的检查，存在检测准确性低、鲁棒性差等问题。
本公开实施例提供一种虚拟特效挂载平面检测方法，通过检查图像中的边缘角点，来提取图像中物体的结构特征，生成表征物体平面的轮廓的结构边框，再利用消影点能够表达图像拍摄角度的特点，通过消影点对结构边框进行筛选，确定适用于挂载虚拟特效的物体平面，从而以解决上述问题。
参考图2，图2为本公开实施例提供的虚拟特效挂载平面检测方法的流程示意图一。本实施例的方法可以应用在终端设备、服务器等电子设备中，该虚拟特效挂载平面检测方法包括：
步骤S101：获取轮廓线段，并基于轮廓线段，得到对应的边缘角点，其中，轮廓线段表征待检测图像中物体的轮廓，边缘角点为轮廓线段之间的交点。
示例性地，轮廓线段是对目标图像进行检测后，得到的所有表征待处理图像中物体的轮廓的直线线段，其中，轮廓线段通常有多条，根据具体的物体结构，各轮廓线段的延伸方向相同或不同，其中，部分轮廓线段能够相交，轮廓线段相交的角点，即为边缘角点。图3为本公开实施例提供的一种轮廓线段和边缘角点的示意图，如图3所示，该图像可以为一个视频帧，对该视频帧进行检测后，得到多个表征房屋的轮廓的线段，即轮廓线段，其中，基于具体的识别结果，轮廓线段可以是构成物体(例如图中的房屋)的全部线段，也可以部分线段。进一步地，根据各轮廓线段之间的位置关系，可以确定轮廓线段之间的交点，即边缘角点，边缘角点是通过表征物体轮廓的轮廓线段的相交关系确定的，因此能够表现出物体的结构特征。其中，对待处理图像进行检测，得到轮廓线段的过程，可以通过预训练的神经网络模型对待处理图像进行线段识别而获得，具体获取方式不再赘述。
在一些可能的情况下，两条轮廓线段的端点相交并重合于一点(即构成“L”型的两条轮廓线段)，此种情况下，边缘角点即两条轮廓线段的端点重合点。更具体地，两条轮廓线段的端点相交并重合于一点，是指两条轮廓线段的端点对应的坐标距离小于一个预设阈值，此处不再赘述。在另一些可能的情况下，一条轮廓线段的端点与另一条轮廓线段的中部相交(即构成“T”型的两条轮廓线段)，此种情况下，与轮廓线段的中部相交的端点，即为边缘角点。在再一些可能的情况下，两条轮廓线段可以是线段中部与线段中部相交(即构成“X”型的两条轮廓线段)，则此种情况下的两条轮廓线段，不具有边缘角点。
示例性地，在获得边缘角点后，对于一些相互靠近的轮廓线段，各自形成的对应的边缘角点之间的距离也很近，并不会提高更多有效的物体的结构信息，同时，在后续基于边缘角点生成表征物体平面的轮廓的结构边框时，若该结构边框的面积过小(由于边缘角点过密导致)，则也无法用于指示挂载平面挂载虚拟特效。因此，将可以将对边缘角点进行融合，以降低边缘角点的数量，减少无效的检测过程，提高整体的检测速度。具体地，步骤S101中基于轮廓线段，得到对应的边缘角点的步骤，包括：
步骤S1011：根据各轮廓线段的相交关系，得到至少两个相交角点。
步骤S1012：基于密度聚类算法，对相交角点进行聚类融合，生成边缘角点。
示例性地，各轮廓线段的相交关系，可以通过线段之间的相交检测算法实现，具体地实现方法为相关技术，不再赘述。之后，将获得的相交角点基于密度聚类算法进行合并，使相近的相交角点合并为同一个角点，即边缘角点。其中，密度聚类亦称基于密度的聚类(density-based clustering)，此类算法假设聚类结构能通过样本分布的紧密程度确定。通常情形下，密度聚类算法从样本密度的角度来考察样本之间的可连接性，并基于可连接样本不断扩展聚类簇以获得最终的聚类结果。密度聚类的具体实现方法不再赘述。
通过密度聚类算法，对相交角点进行聚类融合，仅保留两条轮廓线段的端点相交并重合于一点的边缘角点(即“L”型的两条轮廓线段对应的边缘角点)。图4为本公开实施例提供的一种聚类融合的过程示意图，如图4所示，L1、L2、L3、L4、L5为待处理图像中的5条轮廓线段，根据该5条轮廓线段之间的相交关系，可以得到P1、P2、P3、P4、P5、P6，共6个相交角点，之后，进行聚类融合后，将其中的P1、P2、P3、P4，确定为边缘角点，而位于中部的相交角点P5、P6，则不作为边缘角点，从而在后续通过边缘角点确定结构边框时，避免生成面积过小的无效结构边框，从而提高挂载平面的检测效率。
步骤S102：通过边缘角点，生成至少两个四边形的结构边框，结构边框表征待检测图像中的物体的一个平面的轮廓。
示例性地，在获得边缘角点后，由于边缘角点是基于相交的两条轮廓线段生成的，因此，每一边缘角点均对应两条轮廓线段。而基于一个或多个边缘角点对应的轮廓线段之间的位置关系，可以构造封闭的四边形，即结构边框。由于边缘角点能够表现物体的结构特征，因此，基于边缘角点构造的四边形的结构边框，能够表征待检测图像中的物体的一个平面的轮廓。其中，具体地，例如，可以通过依次遍历每一边缘角点，将该边缘角点对应的两条轮廓线段构成的“L型”结构，与其他边缘角点对应的两条轮廓线段构成的“L型”结构进行组合，检测出其中能够构成封闭四边形的至少两个边缘角点，从而生成结构边框。
在一种可能的实现方式中，步骤S102的具体实现步骤包括：
步骤S1021：获取各边缘角点对应的边角结构，边角结构包括两条构成边缘角点的轮廓线段。
步骤S1022：根据各边缘角点对应的边角结构之间的位置关系，得到至少一组边角结构组合，边角结构组合中包括至少一个边角结构，边角结构组合中的边角结构归属于同一个四边形，当边角结构组合中包括两个以上的边角结构时，边角结构组合中的任一边角结构中的至少一条轮廓线段与另一边角结构部分重叠。
步骤S1023：根据边角结构组合，生成结构边框。
示例性地，边角结构是指边缘角点对应的“L”型的两条轮廓线段。各边缘角点对应的边角结构的长度、方向可能不同，因此，各边缘角点对应的边角结构之间可能会发生端点相交和线段重合，进而组合形成四边形。在一种可能的实现方式中，在遍历各边角结构可能的组合之前，先对边角结构的有效性进行检测。
具体地，例如，通过以下至少一个条件，判断边角结构是否为有效边角结构：
条件1：边角结构的两条轮廓线段都归属于有效的消影点，且两个消影点是不同的。
条件2：边角结构的两条轮廓线段之间的二维角度和三维角度均大于对应的预设阈值。
条件3：边角结构的两条轮廓线段的长度在预设范围内。
边角结构的有效性检测，可以通过上述三个条件中的至少一个实现，其中，两条轮廓线段之间的三维角度，可以在二维角度的基础上，基于预设的相机焦距参数进行归一化计算而获得，具体不再赘述。通过上述对边角结构的有效性检测的过程，可以将去除无效的边角结构，从而降低遍历各边角结构的组合而生成边角结构组合的耗时，提高检测效率。
由于边缘角点能够表现物体的结构特征，边缘角点作为待处理图像中的物体的一个平面的轮廓的顶角时，其对应的边角结构，即为待处理图像中的物体的一个平面的轮廓的两条边。进一步地，基于多个边角结构构成的边角结构组合，当边角结构组合中的多个边角结构归属于同一个四边形，且边角结构的部分线段重合或端点重合时，则可基于边角结构组合，来尝试生成结构边框。更具体地，可通过各边角结构的轮廓线段是否归属于两个消影点(即横向消影点和竖向消影点)，来判断该边角结构组合中的边角结构是否能够生成结构边框。
图5为本公开实施例提供的一种基于边角结构组合生成结构边框的示意图，如图5所示，边角结构组合中包括边角结构C1、C2、C3，其中，C1包括轮廓线段L1和L2，C2包括轮廓线段L3和L4，C3包括轮廓线段L5和L6。通过遍历所有的边角结构，当找到由C1、C2、C3构成的边角结构组合后，即可基于C1、C2、C3，确定一个四边形的结构边框。本实施例 中，通过边缘角点对应的边角结构之间的位置关系进行遍历，得到匹配的边角结构组合，从而生成结构边框，利用物体的结构特征，实现对待处理图像中的物体平面的准确定位，提高了最终确定的挂载平面的检测精度。
步骤S103：通过结构边框对应的目标消影点，获得各结构边框的匹配度，并基于匹配度，确定目标边框，其中，目标边框对应的物体平面用于挂载虚拟特效，匹配度表征结构边框对应的物体平面适合于挂载虚拟特效的程度。
示例性地，在确定结构边框后，根据结构边框在待处理图像中的位置，即可确定一个对应的物体平面。通过上述步骤得到的结构边框通常有多个，因此也对应待处理图像中的多个物体平面。然而，对于挂载虚拟特效的具体场景而言，并非所有的物体平面均适合用于挂载虚拟特效。例如面积过小、物体平面在图像中的展示角度过小等，因此，需要进一步的对结构边框进行筛选，确定出其中适合于挂载虚拟特效的物体平面对应。
具体地，通过结构边框对应的目标消影点，来对结构边框的匹配度进行评估，其中，匹配度表征结构边框对应的物体平面适合于挂载虚拟特效的程度。具体地，匹配度可以通过具体地归一化分数来实现，例如，1为最高，即最合适于挂载虚拟特效；0为最低，即最不合适于挂载虚拟特效。结构边框对应的匹配度位于(0，1)之间。通过结构边框对应的匹配度，将匹配度大于匹配度阈值(例如0.6)的结构边框，确定为目标边框；或者，将匹配度最大的预设数量(例如3个)的结构边框，确定为目标边框，具体可根据需要设置。
进一步地，在一种可能的实现方式中，通过结构边框对应的目标消影点，获得各结构边框的匹配度的具体实现步骤，包括：
步骤S1031：获取结构边框的第一评估值，第一评估值表征目标消影点对应的相关线段的数量，相关线段为归属于目标消影点的轮廓线段。
步骤S1032：根据第一评估值，获得结构边框的匹配度。
示例性地，消影点又称灭点，是指三维空间中一组平行线在二维平面投影后的交点。单帧二维图像的消影点检测，即将二维图像中所包含的N组三维空间平行线的投影线的交点检测出来作为消影点。四边形的结构边框是由边缘角点对应的轮廓线段所构成的，而每一轮廓线段，归属于唯一的消影点，因此，通过结构边框，可以确定至少两个消影点，即目标消影点。进一步地，对于消影点对应有多个相关线段，相关线段为待处理图像中，延长线经过消影点的轮廓线段。对于本实施例步骤中的目标消影点，对应的为相关线段，目标消影点对应的相关线段的数量越多，说明越多的轮廓线段的延长线经过目标消影点，因此说明目标消影点的准确性越高，即第一评估值越高。
进一步地，结构边框的准确性，与对应的目标消影点的第一评估值的大小，响应的具体对应关系，具体来说，目标消影点的第一评估值越大，目标消影点的准确性越高，则归属于该目标消影点的轮廓线段，越能准确的表现物体结构，因此对应的结构边框准确性越高。因此，通过目标消影点的第一评估值来对结构边框的匹配度进行评估，第一评估值越大，则越高匹配度。从而从准确性的角度对结构边框进行评估，得到准确性最高的一个或多个结构边框作为目标边框，从而提高检查挂载平面的准确性。
在本实施例中，通过获取轮廓线段，并基于轮廓线段，得到对应的边缘角点，其中，轮廓线段表征待检测图像中物体的轮廓，边缘角点为轮廓线段之间的交点；通过边缘角点，生成至少两个四边形的结构边框，结构边框表征待检测图像中的物体的一个平面的轮廓；通过 结构边框对应的目标消影点，获得各结构边框的匹配度，并基于匹配度，确定目标边框，其中，目标边框对应的物体平面用于挂载虚拟特效，匹配度表征结构边框对应的物体平面适合于挂载虚拟特效的程度。通过检测边缘角点，生成结构边框，再利用消影点对结构边框进行筛选，从而确定出用于定位挂载平面的目标边框，由于利用了图像中物体的结构特征，结合消影点，使得到的目标边框能够准确的定位出待检测图像中适合挂载虚拟特效的物体平面，从而实现挂载平面的检测，提高挂载平面的检测精度和不同图像场景下的检测鲁棒性。
参考图6，图6为本公开实施例提供的虚拟特效挂载平面检测方法的流程示意图二。本实施例在图2所示实施例的基础上，进一步对步骤S102和步骤S103进行细化，并增加了确定虚拟特效的挂载方向的步骤，该虚拟特效挂载平面检测方法包括：
步骤S201：获取轮廓线段，并基于轮廓线段，得到对应的边缘角点，其中，轮廓线段表征待检测图像中物体的轮廓，边缘角点为轮廓线段之间的交点。
步骤S202：获取各边缘角点对应的边角结构，边角结构包括两条构成边缘角点的轮廓线段。
步骤S203：根据各边缘角点对应的边角结构之间的位置关系，得到至少一组边角结构组合，边角结构组合中包括至少一个边角结构，边角结构组合中的边角结构归属于同一个四边形，当边角结构组合中包括两个以上的边角结构时，边角结构组合中的任一边角结构中的至少一条轮廓线段与另一边角结构部分重叠。
其中，步骤S201为获得轮廓线段和边缘角点的步骤，S202-S203为基于各边缘角点获得对应的边角结构，再遍历各边角结构，得到边角结构组合的步骤，上述步骤在图2所示实施例中已进行介绍，可参见之前实施例中的详细介绍，不再赘述。
步骤S204：获取各边角结构组合中包括的边角结构的数量。
步骤S205：将边角结构的数量大于预设的数量阈值的边角结构组合，确定为目标边角结构组合。
示例性地，在步骤S203中得到的组边角结构组合中，基于其中包含的边角结构的数量，可以分为不同级别的边角结构组合，例如，边角结构组合仅中包括一个“L型”的边角结构，则为一级边角结构组合；边角结构组合中包括两个“L型”的边角结构，则为二级边角结构组合；以此类推，级别越高的边角结构组合，包括的边角结构越多，相应的，所能确定的四边形越准确。通过边角结构组合的级别(即边角结构组合中包括的边角结构的数量)进行筛选，将包含较多边角结构的边角结构组合筛选出，例如，将包括两个以上边角结构的边角结构组合作为目标边角结构组合进行后续处理，减少待检测的边角结构组合的数量，可以进一步的提高检测效率，缩短挂载平面的检测时间。
在一种可能的实现方式中，参考图6中虚线所示，步骤S205的具体实现中，可以通过对边角结构组合的级别(即边角结构组合中包括的边角结构的数量)进行排序，从高至低的顺序，优先将最高级别(边角结构的数量最多)的边角结构组合确定为目标边角结构组合，之后依次执行后续的S206、S207等步骤，进行后续的匹配度评估，再评估合格后，生成一个对应的目标边框以(即步骤S212)，之后返回步骤S205，再按照级别排序，将低一个级别的边角结构组合确定为目标边角结构组合，进行下一轮处理，直至目标边框的数量满足预设数量，从而提高整体检测效率，实现快速定位满足要求的挂载平面的目的。
步骤S206：获取目标边角结构组合中各边角结构对应的横向消影点和竖向消影点。
步骤S207：生成横向消影点对应的第一相关线段和竖向消影点对应的第二相关线段，并基于第一相关线段、第二相关线段和边角结构形成结构边框，其中，第一相关线段的第一端与第二相关线段的第一端相交，第一相关线段的第二端与一条轮廓线段的端点相交，第二相关线段的第二端与另一条轮廓线段的端点相交。
步骤S206-S207是基于目标边角结构组合构造结构边框的具体过程，下面分情况进行详细说明：
图7为包括一个边角结构的目标边角结构的示意图，参考图7所示，目标边角结构中包括边角结构S1，S1由轮廓线段L1和L2构成，即目标边角结构组合提供四边形的两条边；在此基础上，获取L1对应的竖向消影点P1和L2对应的横向消影点P2，之后，分别以P1和P2为起点，向L2和L1的开放端点引射线L12和L21，形成交点P3、P4、P5，其中P3即为L1的开放端点，P4即为L2的开放端点，P5是L12和L21的交点，之后，以P5为端点，分别向P3和P4引线段，生成新的线段L3(即第一相关线段)和L4(即(即第二相关线段)，从而生成由L1、L2、L3、L4围成的四边形，即结构边框。
图8为包括两个边角结构的目标边角结构的示意图，参考图8所示，目标边角结构中包括边角结构S1和S2，S1由轮廓线段L1和L2构成，S2由轮廓线段L3和L4构成，其中，L2和L3部分重合，即目标边角结构组合提供四边形的三条边。在此基础上，获取L1对应的横向消影点P1和L2对应的竖向消影点P2，相应的，L3对应竖向消影点P2，L4对应横向消影点P1。之后，分别以P1和P2为起点，向L4和L1的开放端点引射线L14和L21，形成交点P3、P4、P5，其中P3即为L4的开放端点，P4即为L1的开放端点，P5是L14和L21的交点，之后，以P5为端点，分别向P3和P4引线段，生成新的线段L5(第一相关线段)和L6(第二相关线段)，从而生成由L4+L5、L2+L3、L1、L6围成的四边形，即结构边框。其中，L4和L5位于同一直线且部分重合(端点重合)、L2和L3位于同一直线且部分重合，L4+L5为L4和L5的并集，L2+L3为L2和L3的并集。
由上述实施例可知，通过目标边角结构组合构造结构边框时，目标边角结构组合中的边角结构的数量越多，则需要基于竖向消影点和横向消影点补充的第一相关线段、第二相关线段的长度越短，也即待处理图像中真实的物体结构特征的表达越多，因此检测的挂载平面越准确，反之，目标边角结构组合中的边角结构的数量越少，则需要基于竖向消影点和横向消影点补充的第一相关线段、第二相关线段的长度越长，相对的待处理图像中真实的物体结构特征的表达越少，因此检测的挂载平面越不准确。因此，在确定目标边角结构组合的步骤中，通过选择包括较多数量的边角结构的边角结构组合作为目标边角结构组合，可以提高挂载平面检测的准确性。
步骤S208：通过各轮廓线段和边缘角点，获得多个待选消影点，其中，待选消影点对应的相关线段经过边缘角点。
步骤S209：根据结构边框对应的边缘角点，确定对应的目标消影点。
示例性地，在经过步骤S201，得到待处理图像中的轮廓线段和边缘角点之后，基于轮廓线段和对应的边缘角点，可以检测出待处理图像中的多个消影点，即本实施例步骤中的待选消影点。示例性地，待选消影点包括横向待选消影点和竖向待选消影点，横向待选消影点和竖向待选消影点可以每一边缘角点对应的两条轮廓线段进行检测而获得，若得到的横向待选消影点和/或竖向待选消影点是真实的(即消影点的相关线段的数量大于数量阈值)，则生成的 横向待选消影点和/或竖向待选消影点比如经过边缘角点。之后，在根据上述步骤中得到的结构边框对应的边缘角点(一个结构边框所对应的边缘角点对应的消影点相同)，可确定对应的目标消影点，更具体地，目标消影点包括横向目标消影点和竖向目标消影点。
本实施例中，通过基于轮廓线段和边缘角点获得多个待选消影点，利用了边缘角点所包含的物体结构信息，排除了边缘角点无关的其他消影点，因此在后续检测过程中，降低了检测耗时，提高了检测效率。
步骤S210：获取结构边框的第一评估值和至少一个第二评估值，第一评估值表征目标消影点对应的相关线段的数量，第二评估值表征结构边框对应的物体平面在待检测图像中的视觉特征。
示例性地，在获得多个结构边框后，进一步地，通过结构边框的第一评估值和至少一个第二评估值对结构边框进行评估，筛选出适合于挂载虚拟特效的物体平面。其中，第一评估值的具体实现方式在图2所示实施例中已进行介绍，此处不再赘述。而第二评估值是在第一评估值的基础上，进一步地结合结构边框对应的物体平面在待检测图像中的视觉特征，来对结构边框对应的物体平面是否适合于挂载虚拟特效进行评估。具体地，第二评估值包括以下至少一种：可视夹角、图像面积、内部关联线段的数量、内部冲突线段的数量。
示例性地，其中，可视夹角为结构边框对应的物体平面与相机中心线的夹角；图9为本公开实施例提供的一种可视夹角的示意图，相机中心线为垂直于图像平面的法线方向，物体平面与相机中心线的夹角越大(最大为90度)，则物体平面越正对用户的观看视角，因此展示效果越好，第二评估值越高，反之，物体平面与相机中心线的夹角越小(最大为0度)，则物体平面越偏离用户的观看视角，因此展示效果越差，第二评估值越低。参考图9所示，以水平方向为例，物体平面A的可视夹角为R＝90度，物体平面正对相机，展示效果最好；当可视夹角为R＝30度时，物体平面侧对相机，展示效果较差。
示例性地，图像面积为结构边框对应的物体平面在待检测图像中的面积，图像面积越大，展示效果越好，第二评估值越高，不再赘述。
示例性地，内部关联线段为位于结构边框内，且归属于结构边框对应的目标消影点的轮廓线段；内部冲突线段为位于结构边框内，且不归属于结构边框对应的目标消影点的轮廓线段。
具体地，内部关联线段和内部冲突线段均为位于结构边框内的线段，在图像内挂载广告信息等虚拟特效的应该场景下，为了避免挂载的虚拟特效对图像中本身内容的影响，会将虚拟特效挂载在一些无有效信息的区域，例如纹理重复的墙体上。通过检测内部关联线段和内部冲突线段，若内部关联线段较多，即均为归属于相同的目标消影点的轮廓线段较多时，则说明该结构边框对应的物体平面内，包含的大多为向同一方向延伸的重复线条，所携带的有效信息较少；而当内部冲突线段较多时，即归属于不同的目标消影点的轮廓线段较多时，则说明该结构边框对应的物体平面内，包含的大多为向不同方向延伸的线条，可能携带较多的有效信息。因此，内部关联线段的数量越多，第二评估值越高，内部冲突线段的数量越多，第二评估值越低。
步骤S211：基于第一评估值和第二评估值进行加权计算，得到结构边框的匹配度。
步骤S212：基于匹配度，确定目标边框。
进一步地，示例性地，可以通过对上述的第一评估值和多个第二评估值进行加权计算，即可得到结构边框的匹配度。其中，第一评估值和各第二评估值的加权系数，可以通过具体的需要设置，此处不进行具体限定。
本实施例中，通过第一评估值和多个第二评估值进行加权计算，得到结构边框对应的匹配度，进而选择其中的一个或多个最为适合于挂载虚拟特效的物体平面对应的目标边框，实现了对挂载平面的精确检测，提高挂载的虚拟特效的视觉表现。
步骤S213：获取预设的相机焦距参数，并基于相机焦距参数和目标边框对应的目标消影点，得到目标边框的三维方向，三维方向为目标边框对应的平面的法线方向。
步骤S214：根据目标边框对应的三维方向，确定虚拟特效的挂载方向。
示例性地，进一步地，在确定目标边框后，目标边框作为一个二维的边框，仅可以确定一个二维平面，在此基础上，可以通过预设的相机焦距参数，为该目标边框增加对应的三维方向，从而使确定挂载虚拟特效时的挂载方向。
具体地，例如，相机焦距参数为f，目标消影点包括目标横向消影点和目标纵向消影点，目标横向消影点为P1＝(p1x,p1y)，目标横向消影点为P2＝(p2x,p2y)，P1对应的三维角度为P1_3d，P2对应的三维角度为P2_3d，分别为：
P1_3d＝(p1x,p1y,f)/|(p1x，p1y，f)|
P2_3d＝(p2x,p2y,f)/|(p2x，p2y，f)|
之后，利用上述已得到的两个三维方向，通过正交约束，即可得到对应的三维方向，具过程为相关技术，此处不再赘述。
本实施例中，通过在目标边框的基础上，增加三维方向，进一步的确定挂载平面的上挂载虚拟特效时的挂载方向，避免错误的挂载方向导致的显示错误，提高虚拟特效的视觉表现。
对应于上文实施例的虚拟特效挂载平面检测方法，图10为本公开实施例提供的虚拟特效挂载平面检测装置的结构框图。为了便于说明，仅示出了与本公开实施例相关的部分。参照图10，虚拟特效挂载平面检测装置3，包括：
获取模块31，用于获取轮廓线段，并基于轮廓线段，得到对应的边缘角点，其中，轮廓线段表征待检测图像中物体的轮廓，边缘角点为轮廓线段之间的交点；
生成模块32，用于通过边缘角点，生成至少两个四边形的结构边框，结构边框表征待检测图像中的物体的一个平面的轮廓；
检测模块33，用于通过结构边框对应的目标消影点，获得各结构边框的匹配度，并基于匹配度，确定目标边框，其中，目标边框对应的物体平面用于挂载虚拟特效，匹配度表征结构边框对应的物体平面适合于挂载虚拟特效的程度。
在本公开的一个实施例中，生成模块32，具体用于：
获取各边缘角点对应的边角结构，边角结构包括两条构成边缘角点的轮廓线段；根据各边缘角点对应的边角结构之间的位置关系，得到至少一组边角结构组合，边角结构组合中包括至少一个边角结构，边角结构组合中的边角结构归属于同一个四边形，当边角结构组合中包括两个以上的边角结构时，边角结构组合中的任一边角结构中的至少一条轮廓线段与另一边角结构部分重叠；根据边角结构组合，生成结构边框。
在本公开的一个实施例中，生成模块32在根据边角结构组合，生成结构边框时，具体用于：获取边角结构组合中各边角结构对应的横向消影点和竖向消影点；生成横向消影点对应 的第一相关线段和竖向消影点对应的第二相关线段，并基于第一相关线段、第二相关线段和边角结构形成结构边框，其中，第一相关线段的第一端与第二相关线段的第一端相交，第一相关线段的第二端与一条轮廓线段的端点相交，第二相关线段的第二端与另一条轮廓线段的端点相交。
在本公开的一个实施例中，生成模块32在根据边角结构组合，生成结构边框之前，还用于：获取各边角结构组合中包括的边角结构的数量；将边角结构的数量大于预设的数量阈值的边角结构组合，确定为目标边角结构组合；生成模块32在根据边角结构组合，生成结构边框时，具体用于：通过目标边角结构组合中的各边角结构，生成结构边框。
在本公开的一个实施例中，检测模块33，具体用于：获取结构边框的第一评估值，第一评估值表征目标消影点对应的相关线段的数量，相关线段为归属于目标消影点的轮廓线段；根据第一评估值，获得结构边框的匹配度。
在本公开的一个实施例中，检测模块33，还用于：获取结构边框的第二评估值，第二评估值表征结构边框对应的物体平面在待检测图像中的视觉特征；检测模块33在根据第一评估值，获得结构边框的匹配度时，具体用于：基于第一评估值和第二评估值进行加权计算，得到结构边框的匹配度。
在本公开的一个实施例中，第二评估值包括以下至少一种：可视夹角、图像面积、内部关联线段的数量、内部冲突线段的数量；其中，可视夹角为结构边框对应的物体平面与相机中心线的夹角；图像面积为结构边框对应的物体平面在待检测图像中的面积；内部关联线段为位于结构边框内，且归属于结构边框对应的目标消影点的轮廓线段；内部冲突线段为位于结构边框内，且不归属于结构边框对应的目标消影点的轮廓线段。
在本公开的一个实施例中，生成模块32，还用于：通过各轮廓线段和边缘角点，获得多个待选消影点，其中，待选消影点对应的相关线段经过边缘角点；根据结构边框对应的边缘角点，确定对应的目标消影点。
在本公开的一个实施例中，获取模块31，具体用于：根据各轮廓线段的相交关系，得到至少两个相交角点；基于密度聚类算法，对相交角点进行聚类融合，生成边缘角点。
在本公开的一个实施例中，检测模块33，还用于：获取预设的相机焦距参数；基于相机焦距参数和目标边框对应的目标消影点，得到目标边框的三维方向，三维方向为目标边框对应的平面的法线方向；根据目标边框对应的三维方向，确定虚拟特效的挂载方向。
其中，获取模块31、生成模块32、检测模块33依次连接。本实施例提供的虚拟特效挂载平面检测装置3可以执行上述方法实施例的技术方案，其实现原理和技术效果类似，本实施例此处不再赘述。
图11为本公开实施例提供的一种电子设备的结构示意图，如图11所示，该电子设备4包括：
处理器41，以及与处理器41通信连接的存储器42；
存储器42存储计算机执行指令；
处理器41执行存储器42存储的计算机执行指令，以实现如图2-图9所示实施例中的虚拟特效挂载平面检测方法。
其中，可选地，处理器41和存储器42通过总线43连接。
相关说明可以对应参见图2-图9所对应的实施例中的步骤所对应的相关描述和效果进行理解，此处不做过多赘述。
参考图12，其示出了适于用来实现本公开实施例的电子设备900的结构示意图，该电子设备900可以为终端设备或服务器。其中，终端设备可以包括但不限于诸如移动电话、笔记本电脑、数字广播接收器、个人数字助理(Personal Digital Assistant，简称PDA)、平板电脑(Portable Android Device，简称PAD)、便携式多媒体播放器(Portable Media Player，简称PMP)、车载终端(例如车载导航终端)等等的移动终端以及诸如数字TV、台式计算机等等的固定终端。图12示出的电子设备仅仅是一个示例，不应对本公开实施例的功能和使用范围带来任何限制。
如图12所示，电子设备900可以包括处理装置(例如中央处理器、图形处理器等)901，其可以根据存储在只读存储器(Read Only Memory，简称ROM)902中的程序或者从存储装置908挂载到随机访问存储器(Random Access Memory，简称RAM)903中的程序而执行各种适当的动作和处理。在RAM 903中，还存储有电子设备900操作所需的各种程序和数据。处理装置901、ROM 902以及RAM 903通过总线904彼此相连。输入/输出(Input/Output，I/O)接口905也连接至总线904。
通常，以下装置可以连接至I/O接口905：包括例如触摸屏、触摸板、键盘、鼠标、摄像头、麦克风、加速度计、陀螺仪等的输入装置906；包括例如液晶显示器(Liquid Crystal Display，简称LCD)、扬声器、振动器等的输出装置907；包括例如磁带、硬盘等的存储装置908；以及通信装置909。通信装置909可以允许电子设备900与其他设备进行无线或有线通信以交换数据。虽然图12示出了具有各种装置的电子设备900，但是应理解的是，并不要求实施或具备所有示出的装置。可以替代地实施或具备更多或更少的装置。
特别地，根据本公开的实施例，上文参考流程图描述的过程可以被实现为计算机软件程序。例如，本公开的实施例包括一种计算机程序产品，其包括承载在计算机可读介质上的计算机程序，该计算机程序包含用于执行流程图所示的方法的程序代码。在这样的实施例中，该计算机程序可以通过通信装置909从网络上被下载和安装，或者从存储装置908被安装，或者从ROM 902被安装。在该计算机程序被处理装置901执行时，执行本公开实施例的方法中限定的上述功能。本公开的实施例还包括一种计算机程序，该计算机程序在被处理器执行时实现本公开实施例的方法中限定的上述功能。
需要说明的是，本公开上述的计算机可读介质可以是计算机可读信号介质或者计算机可读存储介质或者是上述两者的任意组合。计算机可读存储介质例如可以是——但不限于——电、磁、光、电磁、红外线、或半导体的系统、装置或器件，或者任意以上的组合。计算机可读存储介质的更具体的例子可以包括但不限于：具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、随机访问存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(Electrical Programmable ROM，EPROM或闪存)、光纤、便携式紧凑磁盘只读存储器(Compact Disc ROM，CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。在本公开中，计算机可读存储介质可以是任何包含或存储程序的有形介质，该程序可以被指令执行系统、装置或者器件使用或者与其结合使用。而在本公开中，计算机可读信号介质可以包括在基带中或者作为载波一部分传播的数据信号，其中承载了计算机可读的程序代码。这种传播的数据信号可以采用多种形式，包括但不限于电磁信号、光信号或上述的任意合适的组合。计算 机可读信号介质还可以是计算机可读存储介质以外的任何计算机可读介质，该计算机可读信号介质可以发送、传播或者传输用于由指令执行系统、装置或者器件使用或者与其结合使用的程序。计算机可读介质上包含的程序代码可以用任何适当的介质传输，包括但不限于：电线、光缆、射频(Radio Frequency，RF)等等，或者上述的任意合适的组合。
上述计算机可读介质可以是上述电子设备中所包含的；也可以是单独存在，而未装配入该电子设备中。
上述计算机可读介质承载有一个或者多个程序，当上述一个或者多个程序被该电子设备执行时，使得该电子设备执行上述实施例所示的方法。
可以以一种或多种程序设计语言或其组合来编写用于执行本公开的操作的计算机程序代码，上述程序设计语言包括面向对象的程序设计语言—诸如Java、Smalltalk、C++，还包括常规的过程式程序设计语言—诸如“C”语言或类似的程序设计语言。程序代码可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络——包括局域网(Local Area Network，简称LAN)或广域网(Wide Area Network，简称WAN)—连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。
附图中的流程图和框图，图示了按照本公开各种实施例的系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段、或代码的一部分，该模块、程序段、或代码的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。也应当注意，在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个接连地表示的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合，可以用执行规定的功能或操作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。
描述于本公开实施例中所涉及到的单元可以通过软件的方式实现，也可以通过硬件的方式来实现。其中，单元的名称在某种情况下并不构成对该单元本身的限定，例如，第一获取单元还可以被描述为“获取至少两个网际协议地址的单元”。
本文中以上描述的功能可以至少部分地由一个或多个硬件逻辑部件来执行。例如，非限制性地，可以使用的示范类型的硬件逻辑部件包括：现场可编程门阵列(Field Programmable Gate Array，FPGA)、专用集成电路(Application Specific Integrated Circuit，ASIC)、专用标准产品(Application Specific Standard Product，ASSP)、片上系统(System on Chip，SOC)、复杂可编程逻辑设备(Complex Programmable Logic Device，CPLD)等等。
在本公开的上下文中，机器可读介质可以是有形的介质，其可以包含或存储以供指令执行系统、装置或设备使用或与指令执行系统、装置或设备结合地使用的程序。机器可读介质可以是机器可读信号介质或机器可读储存介质。机器可读介质可以包括但不限于电子的、磁性的、光学的、电磁的、红外的、或半导体系统、装置或设备，或者上述内容的任何合适组合。机器可读存储介质的更具体示例会包括基于一个或多个线的电气连接、便携式计算机盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦除可编程只读存储器(EPROM 或快闪存储器)、光纤、便捷式紧凑盘只读存储器(CD-ROM)、光学储存设备、磁储存设备、或上述内容的任何合适组合。
第一方面，根据本公开的一个或多个实施例，提供了一种虚拟特效挂载平面检测方法，包括：
获取轮廓线段，并基于所述轮廓线段，得到对应的边缘角点，其中，所述轮廓线段表征待检测图像中物体的轮廓，所述边缘角点为所述轮廓线段之间的交点；通过所述边缘角点，生成至少两个四边形的结构边框，所述结构边框表征所述待检测图像中的物体的一个平面的轮廓；通过所述结构边框对应的目标消影点，获得各所述结构边框的匹配度，并基于所述匹配度，确定目标边框，其中，所述目标边框对应的物体平面用于挂载虚拟特效，所述匹配度表征所述结构边框对应的物体平面适合于挂载虚拟特效的程度。
根据本公开的一个或多个实施例，所述通过所述边缘角点，生成至少两个四边形的结构边框，包括：获取各所述边缘角点对应的边角结构，所述边角结构包括两条构成所述边缘角点的轮廓线段；根据各所述边缘角点对应的边角结构之间的位置关系，得到至少一组边角结构组合，所述边角结构组合中包括至少一个边角结构，所述边角结构组合中的边角结构归属于同一个四边形，当所述边角结构组合中包括两个以上的边角结构时，所述边角结构组合中的任一边角结构中的至少一条轮廓线段与另一边角结构部分重叠；根据所述边角结构组合，生成所述结构边框。
根据本公开的一个或多个实施例，所述根据所述边角结构组合，生成所述结构边框，包括：获取所述边角结构组合中各边角结构对应的横向消影点和竖向消影点；生成所述横向消影点对应的第一相关线段和所述竖向消影点对应的第二相关线段，并基于所述第一相关线段、所述第二相关线段和所述边角结构形成结构边框，其中，所述第一相关线段的第一端与所述第二相关线段的第一端相交，所述第一相关线段的第二端与一条轮廓线段的端点相交，所述第二相关线段的第二端与另一条轮廓线段的端点相交。
根据本公开的一个或多个实施例，在根据所述边角结构组合，生成所述结构边框之前，还包括：获取各所述边角结构组合中包括的边角结构的数量；将所述边角结构的数量大于预设的数量阈值的边角结构组合，确定为目标边角结构组合；所述根据所述边角结构组合，生成所述结构边框，包括：通过所述目标边角结构组合中的各边角结构，生成所述结构边框。
根据本公开的一个或多个实施例，所述通过所述结构边框对应的目标消影点，获得各所述结构边框的匹配度，包括：获取所述结构边框的第一评估值，所述第一评估值表征所述目标消影点对应的相关线段的数量，所述相关线段为归属于所述目标消影点的轮廓线段；根据所述第一评估值，获得所述结构边框的匹配度。
根据本公开的一个或多个实施例，所述方法还包括：获取所述结构边框的第二评估值，所述第二评估值表征所述结构边框对应的物体平面在所述待检测图像中的视觉特征；根据所述第一评估值，获得所述结构边框的匹配度，包括：基于所述第一评估值和所述第二评估值进行加权计算，得到所述结构边框的匹配度。
根据本公开的一个或多个实施例，所述第二评估值包括以下至少一种：可视夹角、图像面积、内部关联线段的数量、内部冲突线段的数量；其中，所述可视夹角为所述结构边框对应的物体平面与相机中心线的夹角；所述图像面积为所述结构边框对应的物体平面在所述待检测图像中的面积；所述内部关联线段为位于所述结构边框内，且归属于所述结构边框对应 的目标消影点的轮廓线段；所述内部冲突线段为位于所述结构边框内，且不归属于所述结构边框对应的目标消影点的轮廓线段。
根据本公开的一个或多个实施例，所述方法还包括：通过各所述轮廓线段和所述边缘角点，获得多个待选消影点，其中，所述待选消影点对应的相关线段经过所述边缘角点；根据所述结构边框对应的边缘角点，确定对应的目标消影点。
根据本公开的一个或多个实施例，所述基于所述轮廓线段，得到对应的边缘角点，包括：根据各轮廓线段的相交关系，得到至少两个相交角点；基于密度聚类算法，对所述相交角点进行聚类融合，生成所述边缘角点。
根据本公开的一个或多个实施例，所述方法还包括：获取预设的相机焦距参数；基于所述相机焦距参数和所述目标边框对应的目标消影点，得到所述目标边框的三维方向，所述三维方向为所述目标边框对应的平面的法线方向；根据所述目标边框对应的三维方向，确定所述虚拟特效的挂载方向。
通过检测边缘角点，生成结构边框，再利用消影点对结构边框进行筛选，从而确定出用于定位挂载平面的目标边框，由于利用了图像中物体的结构特征，结合消影点，使得到的目标边框能够准确的定位出待检测图像中适合挂载虚拟特效的物体平面，从而实现挂载平面的检测，提高挂载平面的检测精度和不同图像场景下的检测鲁棒性。
第二方面，根据本公开的一个或多个实施例，提供了一种虚拟特效挂载平面检测装置，包括：
获取模块，用于获取轮廓线段，并基于所述轮廓线段，得到对应的边缘角点，其中，所述轮廓线段表征待检测图像中物体的轮廓，所述边缘角点为所述轮廓线段之间的交点；
生成模块，用于通过所述边缘角点，生成至少两个四边形的结构边框，所述结构边框表征所述待检测图像中的物体的一个平面的轮廓；
检测模块，用于通过所述结构边框对应的目标消影点，获得各所述结构边框的匹配度，并基于所述匹配度，确定目标边框，其中，所述目标边框对应的物体平面用于挂载虚拟特效，所述匹配度表征所述结构边框对应的物体平面适合于挂载虚拟特效的程度
根据本公开的一个或多个实施例，所述生成模块，具体用于：获取各所述边缘角点对应的边角结构，所述边角结构包括两条构成所述边缘角点的轮廓线段；根据各所述边缘角点对应的边角结构之间的位置关系，得到至少一组边角结构组合，所述边角结构组合中包括至少一个边角结构，所述边角结构组合中的边角结构归属于同一个四边形，当所述边角结构组合中包括两个以上的边角结构时，所述边角结构组合中的任一边角结构中的至少一条轮廓线段与另一边角结构部分重叠；根据所述边角结构组合，生成所述结构边框。
根据本公开的一个或多个实施例，所述生成模块在根据所述边角结构组合，生成所述结构边框时，具体用于：获取所述边角结构组合中各边角结构对应的横向消影点和竖向消影点；生成所述横向消影点对应的第一相关线段和所述竖向消影点对应的第二相关线段，并基于所述第一相关线段、所述第二相关线段和所述边角结构形成结构边框，其中，所述第一相关线段的第一端与所述第二相关线段的第一端相交，所述第一相关线段的第二端与一条轮廓线段的端点相交，所述第二相关线段的第二端与另一条轮廓线段的端点相交。
根据本公开的一个或多个实施例，所述生成模块在根据所述边角结构组合，生成所述结构边框之前，还用于：获取各所述边角结构组合中包括的边角结构的数量；将所述边角结构 的数量大于预设的数量阈值的边角结构组合，确定为目标边角结构组合；所述生成模块在根据所述边角结构组合，生成所述结构边框时，具体用于：通过所述目标边角结构组合中的各边角结构，生成所述结构边框。
根据本公开的一个或多个实施例，所述检测模块，具体用于：获取所述结构边框的第一评估值，所述第一评估值表征所述目标消影点对应的相关线段的数量，所述相关线段为归属于所述目标消影点的轮廓线段；根据所述第一评估值，获得所述结构边框的匹配度。
根据本公开的一个或多个实施例，所述检测模块，还用于：获取所述结构边框的第二评估值，所述第二评估值表征所述结构边框对应的物体平面在所述待检测图像中的视觉特征；所述检测模块在根据所述第一评估值，获得所述结构边框的匹配度时，具体用于：基于所述第一评估值和所述第二评估值进行加权计算，得到所述结构边框的匹配度。
根据本公开的一个或多个实施例，所述第二评估值包括以下至少一种：可视夹角、图像面积、内部关联线段的数量、内部冲突线段的数量；其中，所述可视夹角为所述结构边框对应的物体平面与相机中心线的夹角；所述图像面积为所述结构边框对应的物体平面在所述待检测图像中的面积；所述内部关联线段为位于所述结构边框内，且归属于所述结构边框对应的目标消影点的轮廓线段；所述内部冲突线段为位于所述结构边框内，且不归属于所述结构边框对应的目标消影点的轮廓线段。
根据本公开的一个或多个实施例，所述生成模块，还用于：通过各所述轮廓线段和所述边缘角点，获得多个待选消影点，其中，所述待选消影点对应的相关线段经过所述边缘角点；根据所述结构边框对应的边缘角点，确定对应的目标消影点。
根据本公开的一个或多个实施例，所述获取模块，具体用于：根据各轮廓线段的相交关系，得到至少两个相交角点；基于密度聚类算法，对所述相交角点进行聚类融合，生成所述边缘角点。
根据本公开的一个或多个实施例，所述检测模块，还用于：获取预设的相机焦距参数；基于所述相机焦距参数和所述目标边框对应的目标消影点，得到所述目标边框的三维方向，所述三维方向为所述目标边框对应的平面的法线方向；根据所述目标边框对应的三维方向，确定所述虚拟特效的挂载方向。
第三方面，根据本公开的一个或多个实施例，提供了一种电子设备，包括：处理器，以及与所述处理器通信连接的存储器；
所述存储器存储计算机执行指令；
所述处理器执行所述存储器存储的计算机执行指令，以实现如上第一方面以及第一方面各种可能的设计所述的虚拟特效挂载平面检测方法。
第四方面，根据本公开的一个或多个实施例，提供了一种计算机可读存储介质，所述计算机可读存储介质中存储有计算机执行指令，当处理器执行所述计算机执行指令时，实现如上第一方面以及第一方面各种可能的设计所述的虚拟特效挂载平面检测方法。
第五方面，本公开实施例提供一种计算机程序产品，包括计算机程序，该计算机程序被处理器执行时实现如上第一方面以及第一方面各种可能的设计所述的虚拟特效挂载平面检测方法。
第六方面，本公开实施例提供了一种计算机程序，所述计算机程序被处理器执行时实现如上第一方面以及第一方面各种可能的设计所述虚拟特效挂载平面检测方法。以上描述仅为 本公开的较佳实施例以及对所运用技术原理的说明。本领域技术人员应当理解，本公开中所涉及的公开范围，并不限于上述技术特征的特定组合而成的技术方案，同时也应涵盖在不脱离上述公开构思的情况下，由上述技术特征或其等同特征进行任意组合而形成的其它技术方案。例如上述特征与本公开中公开的(但不限于)具有类似功能的技术特征进行互相替换而形成的技术方案。
此外，虽然采用特定次序描绘了各操作，但是这不应当理解为要求这些操作以所示出的特定次序或以顺序次序执行来执行。在一定环境下，多任务和并行处理可能是有利的。同样地，虽然在上面论述中包含了若干具体实现细节，但是这些不应当被解释为对本公开的范围的限制。在单独的实施例的上下文中描述的某些特征还可以组合地实现在单个实施例中。相反地，在单个实施例的上下文中描述的各种特征也可以单独地或以任何合适的子组合的方式实现在多个实施例中。
尽管已经采用特定于结构特征和/或方法逻辑动作的语言描述了本主题，但是应当理解所附权利要求书中所限定的主题未必局限于上面描述的特定特征或动作。相反，上面所描述的特定特征和动作仅仅是实现权利要求书的示例形式。

Claims (15)

  1. 一种虚拟特效挂载平面检测方法，包括：
    获取轮廓线段，并基于所述轮廓线段，得到对应的边缘角点，其中，所述轮廓线段表征待检测图像中物体的轮廓，所述边缘角点为所述轮廓线段之间的交点；
    通过所述边缘角点，生成至少两个四边形的结构边框，所述结构边框表征所述待检测图像中的物体的一个平面的轮廓；
    通过所述结构边框对应的目标消影点，获得各所述结构边框的匹配度，并基于所述匹配度，确定目标边框，其中，所述目标边框对应的物体平面用于挂载虚拟特效，所述匹配度表征所述结构边框对应的物体平面适合于挂载虚拟特效的程度。
  2. 根据权利要求1所述的方法，其中，所述通过所述边缘角点，生成至少两个四边形的结构边框，包括：
    获取各所述边缘角点对应的边角结构，所述边角结构包括两条构成所述边缘角点的轮廓线段；
    根据各所述边缘角点对应的边角结构之间的位置关系，得到至少一组边角结构组合，所述边角结构组合中包括至少一个边角结构，所述边角结构组合中的边角结构归属于同一个四边形，当所述边角结构组合中包括两个以上的边角结构时，所述边角结构组合中的任一边角结构中的至少一条轮廓线段与另一边角结构部分重叠；
    根据所述边角结构组合，生成所述结构边框。
  3. 根据权利要求2所述的方法，其中，所述根据所述边角结构组合，生成所述结构边框，包括：
    获取所述边角结构组合中各边角结构对应的横向消影点和竖向消影点；
    生成所述横向消影点对应的第一相关线段和所述竖向消影点对应的第二相关线段，并基于所述第一相关线段、所述第二相关线段和所述边角结构形成结构边框，其中，所述第一相关线段的第一端与所述第二相关线段的第一端相交，所述第一相关线段的第二端与一条轮廓线段的端点相交，所述第二相关线段的第二端与另一条轮廓线段的端点相交。
  4. 根据权利要求2或3所述的方法，其中，在根据所述边角结构组合，生成所述结构边框之前，还包括：
    获取各所述边角结构组合中包括的边角结构的数量；
    将所述边角结构的数量大于预设的数量阈值的边角结构组合，确定为目标边角结构组合；
    所述根据所述边角结构组合，生成所述结构边框，包括：
    通过所述目标边角结构组合中的各边角结构，生成所述结构边框。
  5. 根据权利要求1至4中任一项所述的方法，其中，所述通过所述结构边框对应的目标消影点，获得各所述结构边框的匹配度，包括：
    获取所述结构边框的第一评估值，所述第一评估值表征所述目标消影点对应的相关线段的数量，所述相关线段为归属于所述目标消影点的轮廓线段；
    根据所述第一评估值，获得所述结构边框的匹配度。
  6. 根据权利要求5所述的方法，其中，所述方法还包括：
    获取所述结构边框的第二评估值，所述第二评估值表征所述结构边框对应的物体平面在所述待检测图像中的视觉特征；
    根据所述第一评估值，获得所述结构边框的匹配度，包括：
    基于所述第一评估值和所述第二评估值进行加权计算，得到所述结构边框的匹配度。
  7. 根据权利要求6所述的方法，其中，所述第二评估值包括以下至少一种：
    可视夹角、图像面积、内部关联线段的数量、内部冲突线段的数量；
    其中，所述可视夹角为所述结构边框对应的物体平面与相机中心线的夹角；
    所述图像面积为所述结构边框对应的物体平面在所述待检测图像中的面积；
    所述内部关联线段为位于所述结构边框内，且归属于所述结构边框对应的目标消影点的轮廓线段；
    所述内部冲突线段为位于所述结构边框内，且不归属于所述结构边框对应的目标消影点的轮廓线段。
  8. 根据权利要求1至7中任一项所述的方法，其中，所述方法还包括：
    通过各所述轮廓线段和所述边缘角点，获得多个待选消影点，其中，所述待选消影点对应的相关线段经过所述边缘角点；
    根据所述结构边框对应的边缘角点，确定对应的目标消影点。
  9. 根据权利要求1至8中任一项所述的方法，其中，所述基于所述轮廓线段，得到对应的边缘角点，包括：
    根据各轮廓线段的相交关系，得到至少两个相交角点；
    基于密度聚类算法，对所述相交角点进行聚类融合，生成所述边缘角点。
  10. 根据权利要求1至9中任一项所述的方法，其中，所述方法还包括：
    获取预设的相机焦距参数；
    基于所述相机焦距参数和所述目标边框对应的目标消影点，得到所述目标边框的三维方向，所述三维方向为所述目标边框对应的平面的法线方向；
    根据所述目标边框对应的三维方向，确定所述虚拟特效的挂载方向。
  11. 一种虚拟特效挂载平面检测装置，包括：
    获取模块，用于获取轮廓线段，并基于所述轮廓线段，得到对应的边缘角点，其中，所述轮廓线段表征待检测图像中物体的轮廓，所述边缘角点为所述轮廓线段之间的交点；
    生成模块，用于通过所述边缘角点，生成至少两个四边形的结构边框，所述结构边框表征所述待检测图像中的物体的一个平面的轮廓；
    检测模块，用于通过所述结构边框对应的目标消影点，获得各所述结构边框的匹配度，并基于所述匹配度，确定目标边框，其中，所述目标边框对应的物体平面用于挂载虚拟特效，所述匹配度表征所述结构边框对应的物体平面适合于挂载虚拟特效的程度。
  12. 一种电子设备，包括：处理器，以及与所述处理器通信连接的存储器；
    所述存储器存储计算机执行指令；
    所述处理器执行所述存储器存储的计算机执行指令，以实现如权利要求1至10中任一项所述的方法。
  13. 一种计算机可读存储介质，其中，所述计算机可读存储介质中存储有计算机执行指令，当处理器执行所述计算机执行指令时，实现如权利要求1至10中任一项所述的虚拟特效挂载平面检测方法。
  14. 一种计算机程序产品，包括计算机程序，该计算机程序被处理器执行时实现权利要 求1至10中任一项所述的虚拟特效挂载平面检测方法。
  15. 一种计算机程序，所述计算机程序被处理器执行时实现如权利要求1至10中任一项所述的虚拟特效挂载平面检测方法。
PCT/CN2023/098144 2022-06-29 2023-06-02 虚拟特效挂载平面检测方法、装置、设备及存储介质 WO2024001675A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210761378.8 2022-06-29
CN202210761378.8A CN117351185A (zh) 2022-06-29 2022-06-29 虚拟特效挂载平面检测方法、装置、设备及存储介质

Publications (1)

Publication Number Publication Date
WO2024001675A1 true WO2024001675A1 (zh) 2024-01-04

Family

ID=89358150

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/098144 WO2024001675A1 (zh) 2022-06-29 2023-06-02 虚拟特效挂载平面检测方法、装置、设备及存储介质

Country Status (2)

Country Link
CN (1) CN117351185A (zh)
WO (1) WO2024001675A1 (zh)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108921897A (zh) * 2013-06-03 2018-11-30 支付宝（中国）网络技术有限公司 定位卡片区域的方法和设备
US20190197693A1 (en) * 2017-12-22 2019-06-27 Abbyy Development Llc Automated detection and trimming of an ambiguous contour of a document in an image
US20200226805A1 (en) * 2019-01-15 2020-07-16 Facebook, Inc. Identifying Planes in Artificial Reality Systems
CN112308769A (zh) * 2020-10-30 2021-02-02 脸萌有限公司 图像合成方法、设备及存储介质

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108921897A (zh) * 2013-06-03 2018-11-30 支付宝（中国）网络技术有限公司 定位卡片区域的方法和设备
US20190197693A1 (en) * 2017-12-22 2019-06-27 Abbyy Development Llc Automated detection and trimming of an ambiguous contour of a document in an image
US20200226805A1 (en) * 2019-01-15 2020-07-16 Facebook, Inc. Identifying Planes in Artificial Reality Systems
CN112308769A (zh) * 2020-10-30 2021-02-02 脸萌有限公司 图像合成方法、设备及存储介质

Also Published As

Publication number Publication date
CN117351185A (zh) 2024-01-05

Similar Documents

Publication Publication Date Title
CN112233216B (zh) 游戏图像处理方法、装置及电子设备
CN110781823B (zh) 录屏检测方法、装置、可读介质及电子设备
CN108509621B (zh) 景区全景图的景点识别方法、装置、服务器和存储介质
CN111222509B (zh) 目标检测方法、装置及电子设备
CN108960012B (zh) 特征点检测方法、装置及电子设备
CN110084797A (zh) 平面检测方法、装置、电子设备和存储介质
CN114627244A (zh) 三维重建方法及装置、电子设备、计算机可读介质
CN112634235A (zh) 产品图像的边界检测方法和电子设备
CN113344957B (zh) 图像处理方法、图像处理装置和非瞬时性存储介质
US20240203030A1 (en) 3d model rendering method and apparatus, electronic device, and storage medium
CN114863071A (zh) 目标对象标注方法、装置、存储介质及电子设备
CN112257598B (zh) 图像中四边形的识别方法、装置、可读介质和电子设备
CN111862342B (zh) 增强现实的纹理处理方法、装置、电子设备及存储介质
WO2024001675A1 (zh) 虚拟特效挂载平面检测方法、装置、设备及存储介质
CN109816791B (zh) 用于生成信息的方法和装置
WO2023138467A1 (zh) 虚拟物体的生成方法、装置、设备及存储介质
WO2023138468A1 (zh) 虚拟物体的生成方法、装置、设备及存储介质
US11893677B1 (en) Bounding volume hierarchy (BVH) widening based on node compressibility
CN112465692A (zh) 图像处理方法、装置、设备及存储介质
CN111870954B (zh) 一种高度图生成方法、装置、设备及存储介质
CN114494379A (zh) 基于fpfh的凸包辅助三维点云配准方法、设备及介质
CN114387241A (zh) 动画效果检测方法、装置、电子设备和存储介质
CN111354070A (zh) 一种立体图形生成方法、装置、电子设备及存储介质
KR100738097B1 (ko) 3차원 디스플레이를 위한 쿼리의 효율적 처리 방법 및 장치
CN112818748B (zh) 确定视频中的平面的方法和装置、存储介质和电子设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23829859

Country of ref document: EP

Kind code of ref document: A1