CN114677350A

CN114677350A - Connection point extraction method and device, computer equipment and storage medium

Info

Publication number: CN114677350A
Application number: CN202210306611.3A
Authority: CN
Inventors: 罗益承; 者雪飞; 暴林超; 陈长海; 薛唐立
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2022-03-25
Filing date: 2022-03-25
Publication date: 2022-06-28

Abstract

The embodiment of the application discloses a connection point extraction method and device, computer equipment and a storage medium, and belongs to the technical field of computers. The method comprises the following steps: identifying two-dimensional line segments from a plurality of images containing a target object respectively; generating line cloud data based on two-dimensional line segments identified from the plurality of images; determining a plurality of contour points and three-dimensional contour line segments corresponding to the contour points in the line cloud data; and extracting the connecting points of the target object from the three-dimensional contour line segments corresponding to the contour points. In the scheme provided by the embodiment of the application, a mode of extracting the connection point of the target object by using line cloud data is provided, the line cloud data generated based on the identified two-dimensional line segment is used for extracting the connection point of the target object based on the three-dimensional contour line segment contained in the line cloud data, so that the determined connection point is a point on the three-dimensional contour of the target object and an intersection point of at least three surfaces of the target object, and the accuracy of the connection point is ensured.

Description

Connection point extraction method and device, computer equipment and storage medium

Technical Field

The embodiment of the application relates to the technical field of computers, in particular to a connection point extraction method and device, computer equipment and a storage medium.

Background

In practical applications, it is usually necessary to construct a three-dimensional outline of some object, for example, for a building in a real environment, constructing a three-dimensional outline of the building in a certain three-dimensional space, and the like. The object in the real environment includes a plurality of faces, and an intersection of at least three of the plurality of faces included in the object is referred to as a connection point, for example, an intersection of an east wall surface, a south wall surface, and a roof surface of a building in the real environment is a connection point. When the three-dimensional contour of the object is constructed, the connection points of the object need to be determined in the three-dimensional space, and then the connection points in the three-dimensional space are connected, so that the three-dimensional contour of the object can be obtained. How to accurately determine the connection point of the object is a problem that needs to be solved urgently at present.

Disclosure of Invention

The embodiment of the application provides a connection point extraction method and device, computer equipment and a storage medium, which can ensure the accuracy of connection points. The technical scheme is as follows:

in one aspect, a connection point extraction method is provided, and the method includes:

identifying two-dimensional line segments from a plurality of images containing a target object respectively, wherein the postures of the target object in the images are different;

generating line cloud data based on two-dimensional line segments identified from the plurality of images, the line cloud data comprising a plurality of three-dimensional line segments of the target object;

determining a plurality of contour points and three-dimensional contour line segments corresponding to the contour points in the line cloud data, wherein the contour points are end points of the three-dimensional line segments in the line cloud data, and the distance between the contour points and the corresponding three-dimensional contour line segments is smaller than the distance between the contour points and other three-dimensional line segments;

and extracting the connecting points of the target object from the three-dimensional contour line segments corresponding to the contour points.

In one possible implementation manner, before extracting the connection point of the target object from the three-dimensional contour line segment corresponding to the plurality of contour points, the method further includes:

for each endpoint of each three-dimensional contour line segment, determining a difference value between the coordinate of the endpoint and the coordinate of a contour point corresponding to the three-dimensional contour line segment as an updated coordinate of the endpoint;

and respectively determining the three-dimensional contour line segment after updating of each three-dimensional contour line segment based on the coordinate after updating of the endpoint of each three-dimensional contour line segment.

In another possible implementation manner, the determining two-dimensional line segments in the plurality of images that match each other based on the two-dimensional line segments in the plurality of images and the internal references and the external references corresponding to the plurality of images includes:

for a first image and a second image in the plurality of images, mapping a first line segment in the first image to the second image based on internal reference and external reference corresponding to the first image and internal reference and external reference corresponding to the second image to obtain a second line segment, wherein the first line segment is any two-dimensional line segment in the first image;

and under the condition that a third line segment in the second image is similar to the second line segment, determining that the first line segment is matched with the third line segment, wherein the third line segment is any two-dimensional line segment in the first image.

In another aspect, there is provided a connection point extracting apparatus, including:

the identification module is used for respectively identifying two-dimensional line segments from a plurality of images containing target objects, wherein the postures of the target objects in the images are different;

a generating module for generating line cloud data based on two-dimensional line segments identified from the plurality of images, the line cloud data comprising a plurality of three-dimensional line segments of the target object;

the determining module is used for determining a plurality of contour points and three-dimensional contour line segments corresponding to the contour points in the line cloud data, wherein the contour points are end points of the three-dimensional line segments in the line cloud data, and the distance between the contour points and the corresponding three-dimensional contour line segments is smaller than the distance between the contour points and other three-dimensional line segments;

and the extraction module is used for extracting the connecting points of the target object from the three-dimensional contour line segments corresponding to the contour points.

In one possible implementation, the extraction module includes:

the extraction unit is used for extracting the characteristics of the three-dimensional contour line segments corresponding to the contour points to obtain three-dimensional contour characteristics;

a first determination unit, configured to determine, based on the three-dimensional contour feature, a plurality of candidate points and a probability corresponding to each candidate point, where the probability indicates a likelihood that the candidate point is a connection point on the contour of the target object;

a first obtaining unit, configured to obtain a connection point of the target object from the multiple candidate points based on probabilities corresponding to the multiple candidate points, where the probability corresponding to the connection point is greater than probabilities corresponding to other candidate points.

In another possible implementation manner, the obtaining unit is configured to determine, as a 1 st connection point of the target object, an alternative point with a highest probability among the multiple alternative points; deleting the alternative points of which the distance from the 1 st connecting point is less than a first distance threshold from the current remaining alternative points; determining the alternative point with the highest probability in the residual alternative points after deletion as the 2 nd connecting point of the target object; and deleting the alternative points of which the distance between the alternative points and the 2 nd connecting point is smaller than the first distance threshold value from the currently remaining alternative points until the alternative points do not remain any more or until the nth connecting point of the target object is determined, wherein the distance between the currently remaining alternative points and the nth connecting point is smaller than the first distance threshold value, and n is an integer larger than 1.

In another possible implementation manner, the extracting unit is configured to generate a first feature based on coordinates of end points of three-dimensional contour line segments corresponding to the plurality of contour points, where the first feature includes a local feature corresponding to each contour point; respectively reducing the dimension of the local features corresponding to each contour point in the first features, and forming a second feature by using a plurality of local features after dimension reduction; performing global dimension reduction on the first feature to obtain a third feature; and splicing the second characteristic and the third characteristic to obtain the three-dimensional contour characteristic.

In another possible implementation manner, the apparatus further includes:

the determining module is further configured to determine, based on coordinates of end points of the plurality of three-dimensional line segments, an average coordinate value and a distance between each end point in each three-dimensional line segment and an origin of a coordinate system, where the origin of the coordinate system is an origin of a three-dimensional coordinate system in which the line cloud data is located;

a selecting module for selecting a maximum distance from the plurality of currently determined distances;

the determining module is further configured to determine, for each endpoint in the line cloud data, a difference between the coordinate of the endpoint and the coordinate average, and determine a ratio between the difference and the maximum distance as the updated coordinate of the endpoint;

and the updating module is further used for determining the three-dimensional line segments after the three-dimensional line segments are updated based on the coordinates after the end points of the three-dimensional line segments are updated for each three-dimensional line segment in the line cloud data, and forming the three-dimensional line segments after the plurality of three-dimensional line segments are updated into the updated line cloud data.

In another possible implementation manner, the determining module is further configured to determine, for each connection point, a product between the coordinates of the connection point and the maximum distance, and determine a sum of the product and the average as the updated coordinates of the connection point.

In another possible implementation manner, the determining module is configured to select any endpoint from the endpoints of the multiple three-dimensional line segments, and determine the endpoint as a 1 st contour point; determining a first distance between each currently remaining end point and the 1 st contour point, and determining an end point with the largest first distance in the currently remaining end points as a 2 nd contour point; for each currently remaining end point, determining a second distance between the end point and each currently determined contour point, and determining a minimum distance in a plurality of second distances corresponding to the end point as a third distance corresponding to the end point; and determining the endpoint with the maximum third distance in the current remaining endpoints as the 3 rd contour point until the mth contour point is obtained, wherein m is an integer larger than 2.

In another possible implementation manner, the determining module is configured to determine, for each contour point in the plurality of contour points, a distance between the contour point and each three-dimensional line segment in the line cloud data; determining the number of three-dimensional line segments with the distance between the three-dimensional line segments and the contour point smaller than a second distance threshold value from the line cloud data; and under the condition that the number is not less than a number threshold, determining the three-dimensional contour line segment corresponding to the contour point by using the number threshold three-dimensional line segment with the minimum distance between the contour point and the line cloud data, wherein the distance between the contour point and the corresponding three-dimensional contour line segment is less than the distance between the contour point and other three-dimensional line segments.

In another possible implementation manner, the determining module is further configured to determine, as the three-dimensional contour line segment corresponding to the contour point, a three-dimensional line segment in the line cloud data, where a distance between the line cloud data and the contour point is smaller than the second distance threshold, if the number is smaller than the number threshold.

In another possible implementation manner, the determining module is further configured to determine, for each endpoint of each three-dimensional contour line segment, a difference between coordinates of the endpoint and coordinates of a contour point corresponding to the three-dimensional contour line segment as updated coordinates of the endpoint; and respectively determining the three-dimensional contour line segment after updating of each three-dimensional contour line segment based on the coordinate after updating of the endpoint of each three-dimensional contour line segment.

In another possible implementation manner, the apparatus further includes:

and the forming module is used for connecting the plurality of connecting points to form the three-dimensional outline of the target object based on the distribution positions of the plurality of connecting points of the target object.

In another possible implementation manner, the generating module includes:

a second obtaining unit, configured to obtain internal parameters and external parameters corresponding to each image based on the pose of the target object in the plurality of images;

a second determining unit, configured to determine two-dimensional line segments in the plurality of images, which match each other, based on the two-dimensional line segments in the plurality of images and the internal and external references corresponding to the plurality of images;

the reconstruction unit is used for reconstructing two-dimensional line segments matched with each other in the plurality of images based on the internal parameters and the external parameters of the plurality of images to obtain three-dimensional line segments corresponding to the two-dimensional line segments matched with each other;

and the composition unit is used for composing the obtained three-dimensional line segments into the line cloud data.

In another possible implementation manner, the second determining unit is configured to, for a first image and a second image in the multiple images, map a first line segment in the first image into the second image based on an internal reference and an external reference corresponding to the first image and an internal reference and an external reference corresponding to the second image, so as to obtain a second line segment, where the first line segment is any two-dimensional line segment in the first image; and under the condition that a third line segment in the second image is similar to the second line segment, determining that the first line segment is matched with the third line segment, wherein the third line segment is any two-dimensional line segment in the first image.

In another aspect, a computer device is provided, which includes a processor and a memory, where at least one computer program is stored, and the at least one computer program is loaded by the processor and executed to implement the operations performed by the connection point extraction method according to the above aspect.

In another aspect, a computer-readable storage medium is provided, in which at least one computer program is stored, the at least one computer program being loaded and executed by a processor to implement the operations performed by the connection point extraction method according to the above aspect.

In a further aspect, a computer program product is provided, comprising a computer program which, when executed by a processor, performs the operations performed by the connection point extraction method according to the above aspect.

In the solution provided in the embodiment of the present application, considering that the connection point is an intersection point of at least three surfaces of a plurality of surfaces included in the target object and is located on a three-dimensional contour of the target object, a manner of extracting the connection point of the target object by using line cloud data is proposed, since poses of the target object in a plurality of images are different, the multiple images are multi-view images of the target object, the two-dimensional line segments identified from the multi-view images comprise two-dimensional contour line segments of the target object, the three-dimensional contour line segment of the target object should be included in the line cloud data generated based on the recognized two-dimensional line segment, and then the connection point of the target object is extracted based on the three-dimensional contour line segment included in the line cloud data, the determined connection point is a point on the three-dimensional contour of the target object and is the intersection point of at least three surfaces of the target object, so that the accuracy of the connection point is ensured.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

FIG. 1 is a schematic diagram of an implementation environment provided by an embodiment of the present application;

fig. 2 is a flowchart of a connection point extraction method provided in an embodiment of the present application;

fig. 3 is a flowchart of another connection point extraction method provided in an embodiment of the present application;

fig. 4 is a schematic diagram of line cloud data provided in an embodiment of the present application;

FIG. 5 is a flowchart for obtaining three-dimensional contour features according to an embodiment of the present application;

fig. 6 is a flowchart of acquiring line cloud data according to an embodiment of the present disclosure;

fig. 7 is a flowchart of another connection point extraction method provided in an embodiment of the present application;

FIG. 8 is a flowchart of acquiring a connection point of a target object according to an embodiment of the present disclosure;

FIG. 9 is a flowchart of extracting connection points based on a connection point extraction model according to an embodiment of the present application;

FIG. 10 is a schematic illustration of a three-dimensional profile provided by an embodiment of the present application;

fig. 11 is a schematic structural diagram of a connection point extraction apparatus provided in an embodiment of the present application;

fig. 12 is a schematic structural diagram of a connection point extraction apparatus provided in an embodiment of the present application;

fig. 13 is a schematic structural diagram of a terminal according to an embodiment of the present application;

fig. 14 is a schematic structural diagram of a server according to an embodiment of the present application.

Detailed Description

To make the objects, technical solutions and advantages of the embodiments of the present application more clear, the embodiments of the present application will be further described in detail with reference to the accompanying drawings.

The terms "first," "second," "third," and the like as used herein may be used herein to describe various concepts that are not limited by these terms unless otherwise specified. These terms are only used to distinguish one concept from another. For example, a first feature may be termed a second feature, and, similarly, a second feature may be termed a first feature, without departing from the scope of the present application.

As used herein, the terms "at least one," "a plurality," "each," and "any," at least one of which includes one, two, or more than two, and a plurality of which includes two or more than two, each of which refers to each of the corresponding plurality, and any of which refers to any of the plurality. For example, the plurality of images includes 3 images, each of the 3 images is referred to as each of the 3 images, and any one of the 3 images is referred to as any one of the 3 images, which can be a first image, a second image, or a third image.

In the specific implementation manner of the present application, related data such as images, line cloud data, etc. when the above embodiments of the present application are applied to specific products or technologies, user permission or consent needs to be obtained, and the collection, use and processing of the related data need to comply with related laws and regulations and standards of related countries and regions.

Artificial Intelligence (AI) is a theory, method, technique and application system that uses a digital computer or a machine controlled by a digital computer to simulate, extend and expand human Intelligence, perceive the environment, acquire knowledge and use the knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technique of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a manner similar to human intelligence. Artificial intelligence is the research of the design principle and the realization method of various intelligent machines, so that the machines have the functions of perception, reasoning and decision making.

The artificial intelligence technology is a comprehensive subject and relates to the field of extensive technology, namely the technology of a hardware level and the technology of a software level. The artificial intelligence infrastructure generally includes technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and the like.

Computer Vision technology (CV) is a science for researching how to make a machine "look", and more specifically, it refers to that a camera and a Computer are used to replace human eyes to perform machine Vision such as identification and measurement on a target, and further image processing is performed, so that the Computer processing becomes an image more suitable for human eyes to observe or is transmitted to an instrument to detect. As a scientific discipline, computer vision research-related theories and techniques attempt to build artificial intelligence systems that can capture information from images or multidimensional data. The computer vision technology generally includes image processing, image Recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content/behavior Recognition, three-dimensional object reconstruction, 3D (3-Dimension) technology, virtual reality, augmented reality, synchronous positioning and map construction, automatic driving, smart transportation and other technologies, and also includes common human face Recognition, fingerprint Recognition and other biological feature Recognition technologies.

Machine Learning (ML) is a multi-domain cross discipline, and relates to a plurality of disciplines such as probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and the like. The special research on how a computer simulates or realizes the learning behavior of human beings so as to acquire new knowledge or skills and reorganize the existing knowledge structure to continuously improve the performance of the computer. Machine learning is the core of artificial intelligence, is the fundamental approach for computers to have intelligence, and is applied to all fields of artificial intelligence. Machine learning and deep learning generally include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

According to the scheme, the machine learning technology based on artificial intelligence trains the connection point extraction model, the trained connection point extraction model is utilized, and the connection point of the target object can be extracted from the line cloud data corresponding to the target object.

The connection point extraction method provided by the embodiment of the application is executed by computer equipment. Optionally, the computer device is a terminal or a server. Optionally, the server is an independent physical server, or a server cluster or distributed system formed by a plurality of physical servers, or a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, web service, cloud communication, middleware service, domain name service, security service, CDN (Content Delivery Network), big data and artificial intelligence platform. Optionally, the terminal is a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a smart voice interaction device, a smart home appliance, a vehicle-mounted terminal, an aircraft, and the like, but is not limited thereto.

In some embodiments, the computer program according to the embodiments of the present application may be deployed to be executed on one computer device or on multiple computer devices located at one site, or may be executed on multiple computer devices distributed at multiple sites and interconnected by a communication network, and the multiple computer devices distributed at the multiple sites and interconnected by the communication network can form a block chain system.

In one possible implementation, the computer device is provided as a server. Fig. 1 is a schematic diagram of an implementation environment provided by an embodiment of the present application. Referring to fig. 1, the implementation environment includes a terminal 101 and a server 102. The terminal 101 and the server 102 are connected through a wireless or wired network, which is not limited in this application.

The terminal 101 is configured to capture a plurality of images including any object having different poses in the plurality of images. The terminal 101 transmits the plurality of images to the server 102 through a network connection with the server 102. The server 102 is configured to provide a connection point extraction service for the terminal 101, and process a plurality of images sent by the terminal 101 so as to obtain a connection point of the target object.

Alternatively, the terminal 101 has installed thereon a target application served by the server 102, by which the terminal 101 can implement functions such as three-dimensional contour construction, data transmission, and the like. Optionally, the target application is a target application in an operating system of the terminal 101, or a target application provided by a third party. For example, the target application is a contour construction application having a function of three-dimensional contour construction, but of course, the contour construction application can also have other functions, such as a comment function, a shopping function, a navigation function, and the like.

The terminal 101 is configured to capture a plurality of images including any object through the target application, and send the plurality of images to the server through the target application. The server 102 is configured to receive a plurality of images sent by the terminal 101, process the images, extract connection points included in the object, construct a three-dimensional contour of the object in a three-dimensional space based on the extracted connection points, send the three-dimensional contour to the terminal 101, and the terminal 101 receives the three-dimensional contour and displays the three-dimensional contour in the target application.

Fig. 2 is a flowchart of a connection point extraction method provided in an embodiment of the present application, where the method is executed by a computer device, and as shown in fig. 2, the method includes:

201. a computer device identifies two-dimensional line segments from a plurality of images containing a target object, respectively, the poses of the target object being different in the plurality of images.

The target object is any type of object, for example, a building, a vehicle, or a cartoon character. The image is obtained by photographing a target object in a real environment, and may be a virtual object generated in some way. For example, for any object in the real environment, the object is photographed from multiple angles, and multiple images including the object are obtained, and the posture of the object in the multiple images is different. For another example, the user draws the same object on multiple drawings, and the postures of the object on different drawings are different, and multiple images containing the object can be obtained by shooting the multiple drawings.

The plurality of images are a plurality of perspective images of the target object, and the posture of the target object in the plurality of images is different, which indicates that the plurality of images present different angles of the target object, for example, in the plurality of images, a first image shows a front side of the target object, a second image shows a side of the target object, a third object shows a back side of the target object, and the like.

In this embodiment, the plurality of images include a target object, the target object has a contour, each image may include other information in addition to contour information of the target object, and the two-dimensional line segments identified from the plurality of images include contour line segments of the target object in the images and may also include other line segments. For example, if the target object is a building, any image shows an outline of the building at a certain angle and includes windows on the wall surface of the building, and if the image includes a car in addition to the building, the two-dimensional line segment recognized from the image may include an outline segment of the building, an outline segment of the car, an outline segment of a window on the building, and the like. Therefore, processing based on the two-dimensional line segments recognized from the plurality of images is required to recognize the contour of the target object.

202. The computer device generates line cloud data based on the two-dimensional line segments identified from the plurality of images, the line cloud data including a plurality of three-dimensional line segments of the target object.

The line cloud data is data in any three-dimensional space, and a plurality of three-dimensional line segments included in the line cloud data are all located in the three-dimensional space. In this embodiment, since the plurality of images all include the target object and the postures of the target object in the plurality of images are different, the two-dimensional line segments of the target object in the plurality of images at different postures can be used to generate three-dimensional line segments corresponding to the two-dimensional line segments, the generated three-dimensional line segments form the line cloud data, and then the line cloud data is used to extract the connection points on the three-dimensional contour of the target object.

And if the two-dimensional line segments recognized from the plurality of images include a two-dimensional contour line segment of the target object, the three-dimensional contour line segment of the target object should be included in the line cloud data generated based on the recognized two-dimensional line segments, and then the three-dimensional contour line segment of the target object may be recognized from the line cloud data to extract the connection point of the target object therefrom.

203. The computer equipment determines a plurality of contour points and three-dimensional contour line segments corresponding to the contour points in the line cloud data, wherein the contour points are end points of the three-dimensional line segments in the line cloud data, and the distance between the contour points and the corresponding three-dimensional contour line segments is smaller than the distance between the contour points and other three-dimensional line segments.

The contour points are points on a three-dimensional contour of the target object, and the three-dimensional contour line segments are line segments on the three-dimensional contour of the target object. In the embodiment of the present application, the line cloud data includes a plurality of three-dimensional line segments, each of the three-dimensional line segments has two end points, and in any one of the three-dimensional line segments, the probability that an end point of the three-dimensional line segment is on the three-dimensional contour of the target object is greater than the probability that other points in the three-dimensional line segment are on the three-dimensional contour of the target object, so that a contour point is selected from the end points of the three-dimensional line segments included in the line cloud data. Further, since the contour point is a point on the contour of the target object, a three-dimensional line segment closer to the contour point is considered as a three-dimensional contour line segment of the target object. And after a plurality of contour points are determined, determining a three-dimensional line segment which is closer to the contour points in the line cloud data as a three-dimensional contour line segment corresponding to the contour points.

204. And the computer equipment extracts the connecting points of the target object from the three-dimensional contour line segments corresponding to the contour points.

In the embodiment of the present application, the object in the real environment includes a plurality of faces, and an intersection point of at least three faces of the plurality of faces included in the object is referred to as a connection point. Since the three-dimensional contour line segment is a line segment on the three-dimensional contour of the target object in the line cloud data, the three-dimensional contour line segment includes a connection point on the three-dimensional contour of the target object. The connecting point of the target object can be determined through the three-dimensional contour line segments corresponding to the contour points, so that the determined connecting point is a point on the three-dimensional contour of the target object and is an intersection point of at least three surfaces of the target object, and the accuracy of the connecting point is guaranteed.

In the solution provided in the embodiment of the present application, considering that the connection point is an intersection point of at least three surfaces in the plurality of surfaces included in the target object and is located on the three-dimensional contour of the target object, a manner of extracting the connection point of the target object by using line cloud data is proposed, since the postures of the target object in the plurality of images are different, the multiple images are multi-view images of the target object, the two-dimensional line segments identified from the multi-view images comprise two-dimensional contour line segments of the target object, the three-dimensional contour line segment of the target object should be included in the line cloud data generated based on the recognized two-dimensional line segment, and then the connection point of the target object is extracted based on the three-dimensional contour line segment included in the line cloud data, the determined connection point is a point on the three-dimensional contour of the target object and is the intersection point of at least three surfaces of the target object, so that the accuracy of the connection point is ensured.

On the basis of the embodiment shown in fig. 2, the connection point of the target object can be extracted by using the three-dimensional contour features corresponding to the three-dimensional contour line segments, and the specific process is described in the following embodiments.

Fig. 3 is a flowchart of a connection point extraction method provided in an embodiment of the present application, where the method is executed by a computer device, and as shown in fig. 3, the method includes:

301. the computer device identifies two-dimensional line segments from a plurality of images containing the target object, respectively.

In the embodiment of the present application, each two-dimensional line segment has two end points, and the coordinates of each end point are coordinates in an image coordinate system. The plurality of images are the same size, and the origin of the image coordinate systems of the plurality of images is at the same position in the plurality of images. For example, each image coordinate system takes the center point of the image as the origin of the image coordinate system, or each image coordinate system takes the upper left corner point of the image as the origin of the image coordinate system.

In one possible implementation, the two-dimensional line segment identified from each image is represented in the form of two-dimensional coordinates of two end points that the two-dimensional line segment has and a connection relationship between the two end points. In any image, the coordinates of two end points and the connection relationship between the two end points are determined, namely a two-dimensional line segment formed by the two end points is determined.

In one possible implementation, a Line Segment Detector (LSD) is adopted to identify two-dimensional Line segments from each image respectively; or, based on the line segment identification model, identifying a two-dimensional line segment from each image, which is not limited in the present application.

The Line Segment identification model is L-CNN (Convolutional Neural Network Based on search) or TP-LSD (Tri-Points Based Line Segment Detector) and the like.

302. The computer device generates line cloud data based on two-dimensional line segments identified from the plurality of images.

In this embodiment, the line cloud data is data in any three-dimensional space, and a plurality of three-dimensional line segments included in the line cloud data are located in the three-dimensional space. As shown in fig. 4, the line cloud data includes a plurality of three-dimensional line segments in a three-dimensional space, where any two three-dimensional line segments may or may not intersect.

303. The computer device determines a plurality of contour points and three-dimensional contour line segments corresponding to the plurality of contour points in the line cloud data.

The contour points are points on a three-dimensional contour of the target object, and the three-dimensional contour line segments are line segments on the three-dimensional contour of the target object. The contour points determined from the line cloud data are end points of three-dimensional line segments in the line cloud data, and for a three-dimensional contour line segment corresponding to any contour point, the three-dimensional contour line segment corresponding to the contour point is a three-dimensional line segment which is closer to the contour point, that is, in the line cloud data, the distance between the contour point and the corresponding three-dimensional contour line segment is smaller than the distance between the contour point and other three-dimensional line segments.

In one possible implementation, the process of determining a plurality of contour points includes the following steps 3031-3033.

3031. And the computer equipment selects any endpoint from the endpoints of the three-dimensional line segments and determines the endpoint as the 1 st contour point.

In the embodiment of the application, in the line cloud data, each three-dimensional line segment has two end points, that is, a plurality of three-dimensional line segments have a plurality of end points, any end point is selected from the line cloud data in a random selection mode, and the selected end point is determined as the 1 st contour point.

3032. The computer device determines a first distance between each of the remaining end points and the 1 st contour point, and determines the end point with the largest first distance in the remaining end points as the 2 nd contour point.

After the 1 st contour point is determined, the remaining end points are the end points of the plurality of three-dimensional line segments except the 1 st contour point. For example, the line cloud data includes 10 three-dimensional line segments, the 10 three-dimensional line segments have 20 end points, the currently determined 1 st end point is removed from the 20 end points, and the remaining 19 end points are currently remaining end points.

For any end point which is remained at present, the greater the first distance between the end point and the 1 st contour point, the greater the possibility that the end point is located in the edge region of the target object, the greater the possibility that the end point is located on the three-dimensional contour of the target object, therefore, after determining the first distance between each end point which is remained at present and the 1 st contour point, the end point with the largest first distance in the end points which are remained at present is determined as the 2 nd contour point.

In one possible implementation, the line cloud data includes coordinates of end points of each three-dimensional line segment, and a first distance between each end point currently remaining and a 1 st contour point is determined based on the coordinates of each end point currently remaining and the coordinates of the 1 st contour point.

3033. The computer equipment determines a second distance between each endpoint which is remained at present and each contour point which is determined at present, and determines the minimum distance in a plurality of second distances corresponding to the endpoint as a third distance corresponding to the endpoint; and determining the endpoint with the maximum third distance in the current remaining endpoints as the 3 rd contour point until the m-th contour point is obtained.

Where m is a predetermined integer greater than 2, for example, 100 or 80. After the 2 nd contour point is determined, the remaining end points are the end points of the plurality of three-dimensional line segments except the 1 st contour point and the 2 nd contour point.

In the case where 2 or more contour points are currently determined, since the distances between any remaining end point and the plurality of contour points currently determined are different due to different distribution positions of the currently determined contour points, it is necessary to determine the distance between the end point and each of the determined contour points, determine the minimum distance among the distances between the end point and the plurality of contour points as a third distance corresponding to the end point, where the third distance corresponding to the end point reflects the possibility that the end point is located in the edge region of the target object with reference to the currently determined contour point, that is, reflects the possibility that the end point is a point on the three-dimensional contour of the target object, and thus, after determining the third distance corresponding to each of the currently remaining end points, determine the end point corresponding to the maximum third distance, which is the most likely contour point among the remaining end points currently, therefore, the contour points are determined from the line cloud data in this way to ensure that the selected contour points are points in the line cloud data close to the edge region of the target object, thereby ensuring the accuracy of the selected contour points.

In one possible implementation, the process of determining three-dimensional contour line segments corresponding to a plurality of contour points includes steps 3034-3037.

3034. The computer device determines, for each contour point of the plurality of contour points, a distance between the contour point and each three-dimensional line segment in the line cloud data, respectively.

In the case of determining a plurality of contour points, for each contour point in the plurality of contour points, based on the position of the contour point and the position of each three-dimensional line segment in the line cloud data, the distance between the contour point and each three-dimensional line segment can be determined.

In one possible implementation, the distance between any contour point and any three-dimensional line segment satisfies the following relationship:

wherein P represents any contour point of a plurality of contour points, A and B represent two end points of a three-dimensional line segment respectively,

a vector representing the end point a to the end point B of the three-dimensional line segment,

a vector representing the end point a to the contour point P of the three-dimensional line segment,

for representing vectors

And vector

The vector product of, |, is used to represent the modulus.

3035. The computer device determines, from the line cloud data, a number of three-dimensional line segments having a distance to the contour point that is less than a second distance threshold.

Wherein the second distance threshold is an arbitrary value. After the distance between the contour point and each three-dimensional line segment in the line cloud data is determined, the determined distances are respectively compared with a second distance threshold value to determine the number of distances smaller than the second distance threshold value, that is, the number of three-dimensional line segments of which the distance between the contour point and the three-dimensional line segments is smaller than the second distance threshold value.

3036. And under the condition that the number is not less than the number threshold, the computer equipment determines the three-dimensional contour line segment corresponding to the contour point by using the number threshold three-dimensional line segment with the minimum distance between the three-dimensional line segment and the contour point in the line cloud data.

And the distance between the contour point and the corresponding three-dimensional contour line segment is smaller than the distance between the contour point and other three-dimensional line segments. The number threshold is any number set in advance, for example, the number threshold is 10 or 12, and the like. And selecting a number threshold of three-dimensional line segments which are closer to the contour point as the three-dimensional contour line segments corresponding to the contour point under the condition that the number of the three-dimensional line segments of which the distance from the contour point is smaller than a second distance threshold is not smaller than a number threshold after the distance between the contour point and each three-dimensional line segment is based on the distance between the contour point and each three-dimensional line segment, so as to ensure that the distance between the determined three-dimensional contour line segment and the contour point is smaller than the second distance threshold, determine as many three-dimensional contour line segments as possible for the contour point, and ensure that the number of the determined three-dimensional contour line segments is not more than the number threshold. In the embodiment of the present application, the number of three-dimensional contour line segments corresponding to each contour point may be equal to or less than the number threshold.

In the embodiment of the present application, after the contour point is determined, a three-dimensional line segment closer to the contour point needs to be selected from the line cloud data to enrich the three-dimensional contour line segment of the target object. In order to ensure that the selected three-dimensional contour line segment is accurate enough, a second distance threshold value is set, the three-dimensional contour line segment which is close to the contour point is determined, and the distance between the contour point and the corresponding three-dimensional line segment is smaller than the second distance threshold value, so that the accuracy of the determined three-dimensional contour line segment is ensured. And selecting a number of threshold three-dimensional line segments with the minimum distance from the contour point from the line cloud data to ensure that the three-dimensional contour line segments as many as possible are determined for the contour point, and the determined three-dimensional contour line segments do not exceed a set number threshold to avoid data redundancy caused by selecting too many three-dimensional contour line segments, so that the simplification of the line cloud data can be realized, and the reduction of the data amount required to be subsequently processed is ensured.

3037. And under the condition that the number is smaller than the number threshold, determining the three-dimensional line segment, of which the distance between the line cloud data and the contour point is smaller than the second distance threshold, as the three-dimensional contour line segment corresponding to the contour point.

And under the condition that the number of the three-dimensional line segments with the distance to the contour point smaller than the second distance threshold is smaller than the number threshold, determining the three-dimensional line segments with the distance to the contour point smaller than the second distance threshold as the three-dimensional contour line segments corresponding to the contour point so as to ensure that as many three-dimensional contour line segments as possible are determined for the contour point, the determined three-dimensional contour line segments do not exceed the set number threshold, and the distance between the determined three-dimensional contour line segments and the contour point is smaller than the second distance threshold.

For example, the number threshold is 10, and for any contour point, the number of three-dimensional line segments with the distance from the contour point being smaller than the second distance threshold is 8, then 8 three-dimensional contour line segments with the smallest distance are determined for the contour point. For another example, if the number threshold is 10, and for any contour point, the number of three-dimensional line segments whose distance from the contour point is smaller than the second distance threshold is 15, then 10 three-dimensional contour line segments with the smallest distance are determined for the contour point.

304. And the computer equipment performs feature extraction on the three-dimensional contour line segments corresponding to the contour points to obtain three-dimensional contour features.

The three-dimensional contour feature is used for representing three-dimensional contour line segments corresponding to the contour points. The three-dimensional profile feature can be represented in any form, for example, the three-dimensional profile feature can be represented in the form of a multi-dimensional feature matrix.

In the embodiment of the application, the three-dimensional contour line segments corresponding to the determined contour line segments are the line segments on the three-dimensional contour of the target object, so that the obtained three-dimensional contour line segments are subjected to feature extraction, and the features capable of representing the three-dimensional contour of the target object can be obtained.

305. The computer device determines a plurality of candidate points and a probability of each candidate point corresponding based on the three-dimensional profile feature.

Wherein the alternative points are extracted from the three-dimensional contour features and may be connection points on a three-dimensional contour on the target object. The probability indicates a likelihood that the candidate point is a connection point on the contour of the target object.

In the embodiment of the application, the three-dimensional contour feature can represent the three-dimensional contour of the target object, and the alternative points and the corresponding probabilities can be selected from the three-dimensional contour feature through the three-dimensional contour feature.

306. And the computer equipment acquires the connecting point of the target object from the plurality of alternative points based on the probabilities corresponding to the plurality of alternative points.

Wherein, the probability corresponding to the connecting point is greater than the probability corresponding to other alternative points. For any alternative point, the greater the probability that the alternative point corresponds to is, the greater the probability that the alternative point is a connection point on the contour of the target object is, and therefore, the alternative point with the higher probability is selected from the multiple alternative points as the connection point of the target object.

In one possible implementation, a target number of connection points is obtained from the plurality of candidate points based on probabilities corresponding to the plurality of candidate points.

Wherein the target number is an arbitrary number. Optionally, the number of targets is the same as the number of determined contour points.

For example, the number of the targets is 30, 100 candidate points and the probability corresponding to each candidate point are determined based on the three-dimensional contour features, the 100 candidate points are sorted according to the corresponding probability from high to low, 30 candidate points which are sorted in the front are selected from the 100 sorted candidate points, and the candidate points are determined as the connection points of the target object.

In one possible implementation, the process of selecting a connection point of the target object from the candidate points includes the following steps 3061-3062:

3061. the computer equipment determines the alternative point with the highest probability in the multiple alternative points as the 1 st connecting point of the target object; and deleting the alternative points with the distance between the alternative points and the 1 st connecting point smaller than a first distance threshold from the currently remaining alternative points.

Wherein the more probable the alternative points are more probable on the contour of the target object. Therefore, the alternative point with the highest probability is selected as the 1 st connection point of the target object. After the 1 st connection point is determined, the currently remaining alternative points are alternative points except for the 1 st connection point in the plurality of alternative points.

In the embodiment of the application, it is considered that the distance between the connection points on the contour of the target object is not too short, therefore, a first distance threshold is set, and after a connection point is determined, the candidate point which is closer to the connection point is deleted, so as to ensure that the distance between the subsequently determined connection point and the currently determined connection point is large enough, and avoid that the determined connection points are too dense.

In one possible implementation, step 3061 includes: under the condition that the maximum probability in the probabilities corresponding to the multiple candidate points is greater than a probability threshold value, the computer equipment determines the candidate point with the maximum probability in the multiple candidate points as the 1 st connecting point of the target object; and deleting the alternative points with the distance between the alternative points and the 1 st connecting point smaller than a first distance threshold from the currently remaining alternative points. The probability threshold is an arbitrary value.

3062. The computer equipment determines the alternative point with the highest probability in the residual alternative points after deletion as the 2 nd connecting point of the target object; and deleting the alternative points of which the distance between the alternative points and the 2 nd connecting point is less than the first distance threshold value from the currently remaining alternative points until the alternative points do not remain any more or until the nth connecting point of the target object is determined, wherein the distance between the currently remaining alternative points and the nth connecting point is less than the first distance threshold value, and n is an integer greater than 1.

After the 2 nd connecting point is determined, the remaining alternative points are alternative points except the 1 st connecting point and the 2 nd connecting point in the plurality of alternative points.

In the embodiment of the application, according to the corresponding probabilities of the multiple candidate points, the candidate point with the highest probability is selected from the multiple candidate points at each time as the connection point of the target object, the candidate point with the current selected connection point closer to the current selected connection point is deleted, and then the steps are repeated to select the next connection point, so that the multiple connection points of the target object can be obtained. In the process of selecting the connection point of the target object, if one alternative point currently remains after deleting the alternative point with the distance from the currently determined connection point smaller than the first distance threshold, the remaining alternative point is taken as the last connection point of the target object, and then the alternative point is not remained currently. And if the nth connecting point of the target object is determined currently, and the distances between the currently remaining alternative points and the nth connecting point are all smaller than a first distance threshold, deleting the currently remaining alternative points. The method for restraining the non-maximum value is adopted, the connecting point of the target object is determined from the multiple alternative points, the probability that the connecting point of the target object corresponds to is large enough, the distance between the multiple determined connecting points is enough, and the situation that the determined connecting points are too dense is avoided.

In one possible implementation, step 3062 includes: under the condition that the maximum probability in the probabilities corresponding to the remaining alternative points after deletion is greater than the probability threshold value, the computer equipment determines the alternative point with the maximum probability in the remaining alternative points after deletion as the 2 nd connection point of the target object; and deleting the alternative points of which the distance between the alternative points and the 2 nd connecting point is less than the first distance threshold value from the current remaining alternative points until the alternative points do not remain any more, or until the probabilities of the remaining alternative points are all less than the probability threshold value.

In one possible implementation, forming a first point set by a plurality of candidate points, and selecting a connection point of the target object from the candidate points includes: determining the alternative point with the highest probability in the first point set as the 1 st connection point of the target object, adding the 1 st connection point into the second point set, and deleting the 1 st connection point and the alternative point of which the distance between the 1 st connection point and the alternative point is smaller than a first distance threshold value from the first point set; determining the alternative point with the highest probability in the current first point set as the 2 nd connecting point of the target object, and adding the 2 nd connecting point into the second point set; deleting a 2 nd connecting point and an alternative point of which the distance between the 2 nd connecting point and the alternative point is less than a first distance threshold from the first point set; until the first set of points is empty.

In this embodiment, when the first point set is empty, the connection points in the current second point set are the connection points of the target object.

It should be noted that, in the embodiment of the present application, after the three-dimensional contour feature is extracted by using the three-dimensional contour line segment, the connection point of the target object is determined by using the three-dimensional contour feature, but in another embodiment, the step 304 and the step 306 need not be executed, and instead, the connection point of the target object is extracted from the three-dimensional contour line segments corresponding to the plurality of contour points in other manners, so that the probability of the connection point corresponding to the connection point is greater than the probability of the other points in the three-dimensional contour line segment.

In one possible implementation, after the step 306, the method further includes: and connecting the plurality of connecting points to form the three-dimensional contour of the target object based on the distribution positions of the plurality of connecting points of the target object.

The three-dimensional contour of the target object is a line structure formed by a plurality of line segments, and each line segment in the three-dimensional contour is formed by connecting two connecting points. For example, a three-dimensional wire frame structure formed by connecting a plurality of connection points is a three-dimensional contour of the target object. In this embodiment, the line cloud data is data in a three-dimensional space, the determined multiple connection points are also points in the three-dimensional space, that is, coordinates of each connection point in the three-dimensional space are determined, and according to positions indicated by the coordinates of the multiple connection points, the multiple connection points can be connected to form a three-dimensional contour of the target object.

It should be noted that, in the embodiment of the present application, the three-dimensional contour features extracted from the three-dimensional contour line segments corresponding to the contour points are used to determine the connection points of the target object, and in another embodiment, before step 305, the method further includes: acquiring image characteristics of a plurality of images, fusing the image characteristics of the plurality of images with the three-dimensional profile characteristics to obtain updated three-dimensional profile characteristics, and then determining the connection point of the target object by using the updated three-dimensional profile characteristics according to the step 305 and the step 306. Wherein the image features are used to characterize the corresponding image. The image features of the images are fused with the three-dimensional contour feature, so that the information contained in the updated three-dimensional contour feature is enriched, and the accuracy of the subsequently determined contour point is ensured.

In the solution provided in the embodiment of the present application, considering that the connection point is an intersection point of at least three surfaces of a plurality of surfaces included in the target object and is located on a three-dimensional contour of the target object, a manner of extracting the connection point of the target object by using line cloud data is proposed, since poses of the target object in a plurality of images are different, the multiple images are multi-view images of the target object, the two-dimensional line segments identified from the multi-view images comprise two-dimensional contour line segments of the target object, the three-dimensional contour line segment of the target object should be included in the line cloud data generated based on the recognized two-dimensional line segment, and then the connection point of the target object is extracted based on the three-dimensional contour line segment included in the line cloud data, the determined connection point is a point on the three-dimensional contour of the target object and is the intersection point of at least three surfaces of the target object, so that the accuracy of the connection point is ensured. And moreover, the line cloud data is used for extracting the connecting points of the target object, the connecting points of the target object do not need to be extracted by the point cloud data, the data volume contained by the line cloud data is smaller than that contained by the point cloud data, the calculation amount for extracting the connecting points is reduced, and the cost for extracting the point cloud data or reconstructing the point cloud is avoided.

And in the process of determining the contour points from the line cloud data, randomly selecting one end point from the line cloud data as a 1 st contour point, then selecting the end point with the largest distance from the 1 st contour point as a 2 nd contour point, when 2 or more than 2 contour points are currently determined, because the distribution positions of the currently determined contour points are different, the distances between any end point currently remaining and the currently determined contour points are different, then determining the minimum distance in the distances between the end point and the contour points as a third distance corresponding to the end point, wherein the third distance corresponding to the end point can reflect the possibility that the end point is positioned in the edge area of the target object by taking the currently determined contour point as reference, namely the possibility that the end point can reflect the point on the three-dimensional contour of the target object, and then taking the end point with the largest third distance as a next contour point, the selected contour points are guaranteed to be points close to the edge area of the target object in the line cloud data, and therefore accuracy of the selected contour points is guaranteed.

And in order to ensure that the selected three-dimensional contour line segment is accurate enough, a second distance threshold value is set, the three-dimensional contour line segment which is close to the contour point is determined, and the distance between the contour point and the corresponding three-dimensional line segment is smaller than the second distance threshold value, so that the accuracy of the determined three-dimensional contour line segment is ensured. And selecting a number of threshold three-dimensional line segments with the minimum distance from the contour point from the line cloud data to ensure that the three-dimensional contour line segments as many as possible are determined for the contour point, and the determined three-dimensional contour line segments do not exceed a set number threshold to avoid data redundancy caused by selecting too many three-dimensional contour line segments, so that the simplification of the line cloud data can be realized, and the reduction of the data amount required to be subsequently processed is ensured.

And according to the probability corresponding to the multiple candidate points, selecting the candidate point with the maximum probability from the multiple candidate points as the connection point of the target object each time, deleting the candidate point with the current selected connection point closer to the current selected connection point, and then repeating the steps to select the next connection point, so that the multiple connection points of the target object can be obtained. According to the mode, the probability that the connection points of the target object correspond to each other can be guaranteed to be large enough, the distances among the plurality of determined connection points are guaranteed to be sufficient, and the determined connection points are prevented from being too dense.

It should be noted that, on the basis of the embodiment shown in fig. 3, the three-dimensional contour feature can be obtained by using coordinates of end points of the three-dimensional contour line segment and by adopting a local feature and global feature stitching manner, as shown in fig. 5, the process of obtaining the three-dimensional contour feature includes:

501. the computer device generates a first feature based on coordinates of endpoints of the three-dimensional contour line segments corresponding to the plurality of contour points.

Wherein the first features comprise local features corresponding to each contour point. In the embodiment of the present application, each contour point corresponds to a three-dimensional contour line segment, and each three-dimensional contour line segment is represented by two end points of the three-dimensional contour line segment and a connection relationship between the two end points, that is, coordinates of the end points of each three-dimensional contour line segment can represent the corresponding three-dimensional contour line segment, so that, based on the coordinates of the end points of the three-dimensional contour line segment corresponding to each contour point, a local feature corresponding to each end point can be determined, and the local features corresponding to a plurality of end points constitute the first feature.

In one possible implementation, this step 501 includes: respectively extracting the characteristics of the three-dimensional contour line segments corresponding to each contour point to obtain the local characteristics corresponding to each contour point; and splicing the local features corresponding to the contour points to obtain a first feature.

The three-dimensional contour line segment corresponding to each contour point is a three-dimensional line segment which is close to the contour point, and the three-dimensional contour line segment corresponding to the contour point can represent the contour near the contour point on the three-dimensional contour of the target object, so that the three-dimensional contour line segment corresponding to the contour point is subjected to feature extraction to obtain a feature, the feature is a local feature corresponding to the contour point, and the local feature can represent the contour near the contour point on the three-dimensional contour of the target object.

Optionally, the process of obtaining the local feature corresponding to each contour point includes: and for each contour point in the plurality of contour points, forming a fourth feature corresponding to the contour point by using the coordinates of the end points of the three-dimensional contour line segment corresponding to the contour point, and performing feature extraction on the fourth feature to obtain a local feature corresponding to the contour point.

For example, the number of the contour points is Z, the number of the three-dimensional contour line segments corresponding to each contour point is L, each three-dimensional contour line segment has 2 end points, and the coordinates of each end point are three-dimensional coordinates, then the dimension of the fourth feature corresponding to each contour point is lx 2 × 3, where L represents the number of the three-dimensional contour line segments corresponding to the contour point, 2 represents the number of end points of each three-dimensional contour line segment, and 3 represents that the coordinates of each end point are 3-dimensional coordinates. The dimension of the fifth feature formed by the fourth features corresponding to the Z contour points is Z × L × 2 × 3, that is, the fifth feature is a tensor formed by coordinates of end points of the three-dimensional contour line segments corresponding to the plurality of contour points. After feature extraction is performed on the fourth features corresponding to each contour point, the dimension of the local features corresponding to each contour point is L multiplied by 2 multiplied by C, namely the dimension of the feature of the coordinate dimension of each endpoint is changed from 3 to C, and the dimension of the first feature formed by splicing the local features corresponding to the Z contour points is Z multiplied by L multiplied by 2 multiplied by C.

Optionally, the process of obtaining the local feature includes: for any contour point in the plurality of contour points, under the condition that the number of the three-dimensional line segments corresponding to the contour point is equal to a number threshold, forming local characteristics corresponding to the contour point by using the coordinates of the end points of the three-dimensional contour line segments corresponding to the contour point; and under the condition that the number of the three-dimensional contour line segments corresponding to the contour point is smaller than the number threshold, determining the difference value between the number and the number threshold as a first number, determining the repeated contour line segments of the first number from the three-dimensional contour line segments corresponding to the contour point, and forming the local characteristics corresponding to the contour point by using the coordinates of the end points of the three-dimensional contour line segments corresponding to the contour point and the coordinates of the end points of the repeated contour line segments.

The repeated contour line segments corresponding to any contour point are the first number of three-dimensional contour line segments with the minimum distance from the contour point in the three-dimensional contour line segments corresponding to the contour point. For example, if the number threshold is 10 and the number of the three-dimensional contour line segments corresponding to the contour point is 7, then 3 repeated contour line segments are determined from the three-dimensional contour line segments corresponding to the contour point, and the distance between the 3 repeated contour line segments and the contour point is smaller than the distance between the remaining 4 three-dimensional contour line segments and the contour point.

In the embodiment of the application, under the condition that the number of the three-dimensional contour line segments corresponding to any contour point is equal to the number threshold, the coordinates of the end points of the three-dimensional contour line segments corresponding to the contour point form local features corresponding to the contour point; when the number of the three-dimensional contour line segments corresponding to the contour point is smaller than the number threshold, the repeated contour line segments corresponding to the contour point need to be determined from the three-dimensional contour line segments corresponding to the contour point, so as to ensure that the three-dimensional contour line segments corresponding to the contour point and the number corresponding to the repeated contour line segments reach the number threshold, and ensure that the dimensions of the local features determined for each contour point are the same.

For example, the number of the plurality of contour points is Z, the number threshold is L, each three-dimensional contour line segment has 2 end points, the coordinates of each end point are three-dimensional coordinates, and the number of the three-dimensional contour line segments corresponding to each contour point may only be equal to or less than the number threshold L. For each contour point, under the condition that the number of the three-dimensional contour line segments corresponding to the contour point is equal to the number threshold value L, setting the dimension of a local feature formed by the coordinates of the end points of the three-dimensional contour line segments corresponding to the contour point to be L multiplied by 2 multiplied by 3; when the number X of the three-dimensional contour line segments corresponding to the contour point is smaller than the number threshold L, if the dimension of the local feature formed by the coordinates of the end points of the three-dimensional contour line segments corresponding to the contour point is X × 2 × 3, and the dimension of the local feature is different from L × 2 × 3, the repeated contour line segment corresponding to the contour point is determined, the dimension of the local feature formed by the end point coordinates of the three-dimensional contour line segment corresponding to the contour point and the end point coordinates of the repeated contour line segment is L × 2 × 3, according to this way, the dimension of the local feature corresponding to each contour point is ensured to be L × 2 × 3, and the dimension of the first feature formed by the local features corresponding to a plurality of contour points is Z × L × 2 × 3.

502. And the computer equipment reduces the dimension of the local feature corresponding to each contour point in the first feature respectively, and forms a second feature by the plurality of local features after dimension reduction.

The dimension reduction refers to processing the feature values contained in the features, reducing the dimension of the features to reduce the number of the feature values contained in the features, and the dimension reduction of the features can be realized by adopting a mode of maximum pooling or average pooling and the like. For example, the local feature is a three-dimensional feature matrix, and the dimension of the three-dimensional feature matrix is L × 2 × C, that is, the local feature includes the number of feature values of L × 2 × C. Dimension reduction is performed on the local feature, that is, the number of feature values included in the local feature is L × 2 × C, so as to reduce the dimension of the local feature, thereby reducing the number of feature values included in the local feature, where, for example, the dimension of the local feature after dimension reduction is C, that is, the number of feature values included in the local feature after dimension reduction is C, that is, the dimension of the local feature after dimension reduction is smaller and the number of feature values included in the local feature is smaller than the dimension of the local feature before dimension reduction.

In the embodiment of the present application, the local feature corresponding to each contour point is usually subjected to dimension reduction, and the plurality of local features after the dimension reduction constitute the second feature, so that the second feature includes the local feature of the contour of the target object.

For example, the dimension of the first feature is Z × L × 2 × C, the number of the plurality of contour points is Z, the dimension of the local feature corresponding to each contour point is L × 2 × C, the dimension of each local feature is reduced, the dimension of each local feature after the dimension reduction is C, the plurality of local features after the dimension reduction form a second feature, and the dimension of the second feature is Z × C.

In one possible implementation manner, the local feature corresponding to each contour point includes a plurality of first-dimension features and second-dimension features, and the process of performing dimension reduction on the local feature includes: and for the local features corresponding to any contour point, reducing the dimensions of the features with the first dimension in the local features to obtain the reduced-dimension local features corresponding to the local features.

For example, the dimension of the local feature corresponding to any contour point is lx2 × C, that is, the local feature includes 3 dimensions, the first dimensions include dimension L and dimension 2, the second dimension is dimension C, the feature of the 2 nd dimension (dimension 2) is first pooled maximally, the dimension of the local feature after maximal pooling is lxc, and then the feature of the 1 st dimension (dimension L) in the pooled local features is pooled averagely to obtain the reduced-dimension local feature, where the dimension of the reduced-dimension local feature is C.

503. And the computer equipment performs global dimension reduction on the first characteristic to obtain a third characteristic.

Wherein the third feature is a global feature of the contour of the target object.

In one possible implementation, the first feature includes a plurality of third-dimension features and fourth-dimension features, and the step 503 includes: and reducing the dimension of each third dimension feature in the first features to obtain the third feature.

For example, the dimension of the first feature is Z × L × 2 × C, the number of the plurality of contour points is Z, the local feature dimension corresponding to each contour point is L × 2 × C, the plurality of third dimensions include dimension Z, dimension L, and dimension 2, the fourth dimension is dimension C, the feature of each third dimension is reduced, and the obtained dimension of the third feature is C. For another example, the first features include 3 third-dimension features and 1 fourth-dimension feature, the 3 rd third-dimension (dimension 2) features are subjected to maximum pooling, and the first third-dimension (dimension Z) and the second third-dimension (dimension L) features are subjected to average pooling, so that the third features are obtained.

504. And the computer equipment splices the second characteristic and the third characteristic to obtain the three-dimensional contour characteristic.

The second feature can represent the local three-dimensional contour of the target object, the third feature can represent the global contour of the target object, and the second feature and the third feature are spliced, so that the spliced three-dimensional contour feature not only contains the local feature of the contour of the target object, but also contains the global feature of the contour of the target object, information contained in the three-dimensional contour feature is enriched, and the accuracy of the three-dimensional contour feature is ensured.

In one possible implementation, the dimension of the second feature is different from that of the third feature, and then the step 504 includes: and carrying out dimension transformation on the third features, and splicing the transformed third features with the first features to obtain the three-dimensional profile features.

And the dimension of the transformed third feature is the same as that of the first feature.

For example, the dimension of the second feature is Z × C, the dimension of the third feature is C, the third feature is subjected to dimension transformation, and the dimension of the changed third feature is Z × C, the second feature and the transformed third feature are spliced to obtain the dimension of the three-dimensional contour feature, which is Z × 2C.

In the scheme provided by the embodiment of the application, the first feature is generated by using the coordinates of the end points of the three-dimensional contour line segments corresponding to the contour points, and the first feature is subjected to local and global processing to obtain the second feature and the third feature of the target object.

Based on the embodiment shown in fig. 3, the line cloud data can be generated by using the internal parameters and the external parameters corresponding to the multiple images, and as shown in fig. 6, the process of acquiring the line cloud data includes:

601. the computer equipment acquires internal parameters and external parameters corresponding to each image based on the postures of the target objects in the plurality of images.

In the embodiment of the present application, the pose of the target object in each image is different, the internal reference corresponding to the image indicates the conversion relationship between an image coordinate system and a camera coordinate system, the external reference corresponding to the image indicates the conversion relationship between a camera coordinate system and a world coordinate system, the camera coordinate system is a coordinate system of a device capturing the image, for example, the camera coordinate system is a three-dimensional rectangular coordinate system established by taking a focus center of the device as an origin and taking an optical axis as a Z axis, and an X axis and a Y axis of the three-dimensional rectangular coordinate system are parallel to an X axis and a Y axis of the image coordinate system. Optionally, the internal and external parameters are represented in the form of a matrix.

In a possible implementation manner, based on the matching relationship between the pixel points in the plurality of images and the positions of the pixel points in the images, the internal parameters and the external parameters corresponding to each image can be determined.

In the embodiment of the present application, any point on the target object corresponds to a pixel point in the plurality of images, the pixel points of the point on the target object in different images may be different, and the corresponding pixel points in the plurality of images are mutually matched. Determining the matching relationship among the pixel points in the images and the positions of the pixel points which are matched with each other in each image, triangulating the pixel points which are matched with each other, mapping the pixel points which are matched with each other into a three-dimensional space to obtain a plurality of three-dimensional characteristic points, determining the internal parameters corresponding to each image based on the matching relationship among the pixel points and the three-dimensional characteristic points in the image, the positions of the pixel points in the image and the positions of the three-dimensional characteristic points in the three-dimensional space for each image, and obtaining the internal parameters corresponding to each image according to the mode. Taking any image in the plurality of images as a reference image, setting the external parameter corresponding to the reference image as a fixed external parameter, determining a reference pixel point matched with any pixel point in the reference image and a three-dimensional characteristic point matched with the pixel point in a three-dimensional space for any image except the reference image in the plurality of images, namely the pixel point, the reference pixel point and the three-dimensional characteristic point correspond to the same point on a target object, determining the external parameter corresponding to the image based on the coordinates of the pixel point, the coordinates of the reference pixel point and the three-dimensional characteristic point, the internal parameter corresponding to the image and the fixed external parameter, and obtaining the external parameter corresponding to each image according to the steps.

In a possible implementation manner, an SFM (Structure From Motion) framework such as a COLMAP (a general Motion Structure and a multi-view stereo pipeline) or an application such as Reality Capture is adopted, and the internal reference and the external reference corresponding to each image are determined based on a plurality of images, which is not limited in this application.

602. The computer device determines two-dimensional line segments in the plurality of images, which are matched with each other, based on the two-dimensional line segments in the plurality of images and the internal and external parameters corresponding to the plurality of images.

Under the condition that the internal parameters and the external parameters corresponding to the multiple images are determined, based on the internal parameters and the external parameters of the multiple images, the matching relation among pixel points in the multiple images can be determined, and therefore two-dimensional line segments matched with each other in the multiple images are determined. In the embodiment of the present application, a plurality of two-dimensional line segments that match each other indicate the same three-dimensional line segment on the three-dimensional contour of the target object.

In one possible implementation, this step 602 includes: for a first image and a second image in the plurality of images, mapping a first line segment in the first image to the second image based on the internal reference and the external reference corresponding to the first image and the internal reference and the external reference corresponding to the second image to obtain a second line segment; in the case that a third line segment in the second image is similar to the second line segment, it is determined that the first line segment matches the third line segment.

The method comprises the steps that a first image and a second image are any two images in a plurality of images, under the condition that internal reference and external reference corresponding to the first image and internal reference and external reference corresponding to the second image are determined, the matching relation between pixel points in the first image and pixel points in the second image can be determined, the first line segment in the first image can be mapped to the second line segment, and therefore a new line segment is obtained, the new line segment can be matched with a two-dimensional line segment in the second image, whether the two-dimensional line segment is matched with the new line segment in the second image or not is determined, and whether the second image is matched with the first line segment or not is determined. In an embodiment of the present application, the manner of determining the two-dimensional line segments matching with each other in the images is determined by using a limit constraint between the multi-view images, and in another embodiment, the two-dimensional line segments matching with each other in different images can also be determined in other manners, which is not limited in this application.

Optionally, the process of mapping the first line segment into the second image comprises: and respectively mapping the first end point and the second end point of the first line segment to the second image to obtain a third end point corresponding to the first end point and a fourth end point corresponding to the second end point, and connecting the third end point and the fourth end point to form a second line segment.

Optionally, the process of determining whether the second line segment is similar to the line segment in the second image includes: determining whether the position of the second line segment is the same as the position of the third image, and determining that the second line segment is similar to the third line segment when the position of the second line segment is the same as the position of the third line segment; alternatively, a difference between coordinates of an endpoint in the second line segment and coordinates of an endpoint in the third line segment is determined, and in the case where the difference is less than a threshold, the second line segment is determined to be similar to the third line segment.

603. And the computer equipment reconstructs the two-dimensional line segments matched with each other in the plurality of images based on the internal reference and the external reference of the plurality of images to obtain the three-dimensional line segments corresponding to the two-dimensional line segments matched with each other.

In the embodiment of the present application, for any contour line segment on the contour of the object, the contour line segment corresponds to a two-dimensional line segment in the plurality of images, and the two-dimensional line segments corresponding to the contour line segment in the plurality of images are matched with each other. For a group of two-dimensional line segments matched with each other, the group of two-dimensional line segments are mapped into a three-dimensional space based on internal parameters and external parameters of a plurality of images to obtain the same three-dimensional line segment, and a plurality of groups of two-dimensional line segments matched with each other in the plurality of images are mapped into the three-dimensional space to obtain a plurality of three-dimensional line segments.

604. And the computer equipment forms the obtained three-dimensional line segment into the line cloud data.

In the embodiment of the application, the plurality of images comprise a plurality of groups of mutually matched two-dimensional line segments, and for each group of mutually matched two-dimensional line segments, a three-dimensional line segment can be reconstructed, so that a plurality of three-dimensional line segments are obtained, and the plurality of three-dimensional line segments form line cloud data.

In the scheme provided by the embodiment of the application, a mode for generating line cloud data by using a two-dimensional image is provided, a multi-view image containing a target object is obtained, internal parameters and external parameters corresponding to different images are estimated by using different postures of the target object in the multi-view image, and two-dimensional line segments matched with each other are mapped into a three-dimensional space by using the matching relation between the two-dimensional line segments in the images and the internal parameters and the external parameters corresponding to the images, so that the line cloud data is formed, and the accuracy of the generated line cloud data is ensured.

On the basis of the embodiment shown in fig. 2, in the process of obtaining the connection point of the target object, the line cloud data and the three-dimensional contour line segment need to be updated, as shown in fig. 7, the connection point extraction method includes:

701. a computer device identifies two-dimensional line segments from a plurality of images containing a target object, respectively, the poses of the target object being different in the plurality of images.

702. The computer device generates line cloud data based on the two-dimensional line segments identified from the plurality of images, the line cloud data including a plurality of three-dimensional line segments of the target object.

It should be noted that the steps 701-702 are the same as the steps 201-202, and are not described herein again.

703. The computer device determines a coordinate average value and a distance between each end point in each three-dimensional line segment and an origin of a coordinate system based on coordinates of end points of the plurality of three-dimensional line segments.

And the origin of the coordinate system is the origin of the three-dimensional coordinate system in which the line cloud data is positioned. In the embodiment of the application, each three-dimensional line segment has two end points, and the coordinate average value is determined based on the coordinates of the two end points of the three-dimensional line segments. And determining the distance between each end point and the origin of the coordinate system according to the coordinates of each end point in each three-dimensional line segment and the coordinates of the origin of the coordinate system.

In one possible implementation, the distance between each end point and the origin of the coordinate system is determined, and L2norm (L2 norm) or other means is adopted, which is not limited in this application.

704. The computer device selects a maximum distance from the plurality of distances that have been currently determined.

After determining the distance between each end point of the line segments and the origin of the coordinate system, a plurality of distances are determined, and the maximum distance is determined from the plurality of distances.

705. The computer device determines, for each endpoint in the line cloud data, a difference between the coordinate of the endpoint and the average value of the coordinates, and determines a ratio between the difference and the maximum distance as the updated coordinate of the endpoint.

706. And for each three-dimensional line segment in the line cloud data, the computer equipment determines the three-dimensional line segment after the three-dimensional line segment is updated based on the updated coordinates of the end points of the three-dimensional line segment, and the three-dimensional line segments after the three-dimensional line segment is updated form the updated line cloud data.

The coordinates of each endpoint can be updated according to the above step 705, each three-dimensional line segment has two endpoints, and for each three-dimensional line segment, the updated endpoint of the three-dimensional line segment forms the updated three-dimensional line segment of the three-dimensional line segment.

In this embodiment of the application, since the plurality of three-dimensional line segments included in the line cloud data are generated by using two-dimensional line segments in different images, so that the dimensions of the plurality of generated three-dimensional line segments may be different, after the line cloud data is generated, normalization processing needs to be performed on the three-dimensional line segments in the line cloud data, coordinates of end points of each three-dimensional line segment are updated, and the updated end points form an updated three-dimensional line segment, so that updated line cloud data is obtained, so that the dimensions of the three-dimensional line segments in the updated line cloud data are at the same dimension level, and a connection point is extracted subsequently.

707. The computer device determines a plurality of contour points and three-dimensional contour line segments corresponding to the contour points in the updated line cloud data.

The contour points are end points of three-dimensional line segments in the line cloud data, and the distance between each contour point and the corresponding three-dimensional contour line segment is smaller than the distance between the contour point and other three-dimensional line segments. Step 707 is the same as step 203, and will not be described again.

708. And for each endpoint of each three-dimensional contour line segment, the computer equipment determines the difference value between the coordinate of the endpoint and the coordinate of the contour point corresponding to the three-dimensional contour line segment as the updated coordinate of the endpoint.

After a plurality of contour points and three-dimensional contour line segments corresponding to each contour point are determined, namely the coordinates of the contour points and the coordinates of the end points of each three-dimensional contour line segment are determined. And for any three-dimensional contour line segment, updating the coordinates of the end points of the three-dimensional contour line segment by using the coordinates of the contour points corresponding to the three-dimensional contour line segment to obtain the updated end points of the three-dimensional contour line segment.

709. And the computer equipment respectively determines the updated three-dimensional contour line segment of each three-dimensional contour line segment based on the updated coordinates of the end points of each three-dimensional contour line segment.

And after the updated coordinates of the end points of each three-dimensional contour line segment are obtained, the updated three-dimensional contour line segment of the three-dimensional contour line segment is formed based on the updated coordinates of the two end points belonging to the same three-dimensional contour line segment.

In the embodiment of the present application, when determining the three-dimensional contour line segment corresponding to the contour point, the three-dimensional contour line segment corresponding to the contour point is selected with the contour point as a center, that is, the three-dimensional contour line segments corresponding to the contour point are distributed with the contour point as a center, and then each three-dimensional contour line segment corresponding to the contour point is updated by using the coordinates of the contour point according to the above manner, so as to realize the decentralized processing of the three-dimensional contour line segment.

710. And the computer equipment extracts the connecting points of the target object from the updated three-dimensional contour line segments corresponding to the contour points.

Step 710 is similar to step 204, and will not be described herein again.

711. The computer device determines, for each connection point, a product between the coordinates of the connection point and the maximum distance, and determines a sum of the product and the average as updated coordinates of the connection point.

Wherein the maximum distance is the maximum distance determined in step 704 and the average value is the average value determined in step 703. In the embodiment of the application, after the line cloud data is obtained, the line cloud data is updated, and coordinates of end points in the updated line cloud data may change, so that after the connection point of the target object is obtained, the coordinates of the connection point need to be restored to obtain real coordinates of the connection point. When the three-dimensional line segments in the line cloud data are updated, the line cloud data are updated by using the average value and the maximum distance determined in the

steps

703 and 704, so that after the connection points of the target object are obtained, the coordinates of the connection points are restored according to the average value and the maximum value, so that the updated coordinates of the connection points are matched with the original line cloud data, and the accuracy of the updated coordinates of the connection points is ensured.

In one possible implementation, after the step 711, the method further includes: and connecting the plurality of connecting points to form the three-dimensional contour of the target object based on the distribution positions of the plurality of updated connecting points of the target object.

It should be noted that, in the embodiment of the present application, the cloud data, the three-dimensional contour line segment, and the coordinates of the connection points are updated as an example to obtain the final connection point of the target object, and in another embodiment, the three-dimensional contour line segment corresponding to the contour point by the contour point is determined directly by using the cloud data without performing the

steps

703, 708, 709, and 711, and then the connection point of the target object is extracted from the three-dimensional contour line segment.

In the scheme provided by the embodiment of the application, considering that the scales of two-dimensional line segments in different images are possibly different, after line cloud data are obtained, normalization processing is performed on the line cloud data first to ensure that the scales of three-dimensional line segments in the line cloud data after normalization processing are consistent, so that connection points can be obtained by using the line cloud data after normalization processing later, and then coordinates of the connection points are restored to enable the coordinates of the updated connection points to be matched with original line cloud data, and the accuracy of the coordinates of the updated connection points is ensured. And when the three-dimensional contour line segment corresponding to the contour point is determined, the three-dimensional contour line segment corresponding to the contour point is selected by taking the contour point as the center, and each three-dimensional contour line segment corresponding to the contour point is updated by utilizing the coordinates of the contour point so as to realize the decentralized processing of the three-dimensional contour line segment, thereby ensuring the accuracy of the determined connection point.

In addition to the embodiments shown in fig. 2 to 7, the connection point extraction model can be used to extract the connection point of the target object, and as shown in fig. 8, the process of acquiring the connection point of the target object includes:

801. a computer device acquires a multi-view image containing a target object.

802. And the computer equipment extracts two-dimensional line segments from the multi-view images, carries out pose estimation on the multi-view images and determines internal parameters and external parameters corresponding to the multi-view images.

803. And the computer equipment generates a three-dimensional line segment based on the extracted two-dimensional line segment and the internal reference and the external reference corresponding to the multi-view image to obtain line cloud data.

804. The computer device determines the connection point of the target object based on the connection point extraction model according to the above step 303 and 306, that is, the connection point estimation result is obtained.

It should be noted that, on the basis of the embodiment shown in fig. 8, the connection point extraction model includes a local feature sub-model, a global feature sub-model, a connection sub-model, a classification sub-model and a regression sub-model, as shown in fig. 9, that is, a process of extracting connection points based on the connection point extraction model includes:

901. a computer device acquires a multi-view image containing a target object.

902. And the computer equipment extracts two-dimensional line segments from the multi-view images, carries out pose estimation on the multi-view images and determines internal parameters and external parameters corresponding to the multi-view images.

903. And the computer equipment generates a three-dimensional line segment based on the extracted two-dimensional line segment and the internal reference and the external reference corresponding to the multi-view image to obtain line cloud data.

904. The computer device determines a plurality of contour points and three-dimensional contour line segments corresponding to the plurality of contour points in the line cloud data based on the connection point extraction model.

905. And the computer equipment forms the coordinates of the end points of the three-dimensional contour line segments corresponding to the contour points into first features based on the local feature submodels, reduces the dimension of the local features corresponding to each contour point in the first features respectively, and forms a second feature from the plurality of local features after dimension reduction.

906. And the computer equipment forms the coordinates of the end points of the three-dimensional contour line segments corresponding to the contour points into a first feature based on the global feature submodel, and performs global dimension reduction on the first feature to obtain a third feature.

In one possible implementation manner, the local feature submodel and the global feature submodel are similar in structure and each of the local feature submodel and the global feature submodel is composed of a plurality of fully-connected layers, a BN (Batch Normalization) layer, and a nonlinear layer.

Optionally, the local feature submodel and the global feature submodel each include a multilayer perceptron, the multilayer perceptron in the local feature submodel is configured to form the first feature based on coordinates of endpoints of the three-dimensional contour line segments corresponding to the multiple contour points, and the multilayer perceptron in the global feature submodel is configured to form the first feature based on coordinates of endpoints of the three-dimensional contour line segments corresponding to the multiple contour points. In the embodiment of the present application, the number of input channels of the multi-layered sensor is 3, and the number of output channels is C. For example, the dimension of the fifth feature input by the multi-layer perceptron is Z × L × 2 × 3, and the dimension of the first feature obtained by the multi-layer perceptron is Z × L × 2 × C.

907. And the computer equipment splices the second characteristic and the third characteristic based on the connection sub-model to obtain the three-dimensional profile characteristic.

908. The computer equipment processes the three-dimensional profile characteristics based on the regression sub-model to obtain a plurality of alternative points; and processing the three-dimensional profile characteristics based on the classification submodels to obtain the probability of each alternative point.

In the embodiment of the application, the regression sub-model and the classification sub-model are sequentially stacked by a fully connected layer, a BN layer, a non-linear layer and a Dropout (neuron for preventing overfitting) layer.

In one possible implementation, the output channel of the regression submodel is 3 and the output channel of the classification submodel is 2. The regression submodel outputs the three-dimensional coordinates of each candidate point, the classification submodel outputs a set of probabilities corresponding to each candidate point, the set of probabilities including 2 probabilities, the 1 st probability indicating the likelihood that the candidate point is a connection point on the three-dimensional contour of the target object, the 2 nd probability indicating the likelihood that the candidate point is not a connection point on the three-dimensional contour of the target object.

909. And the computer equipment acquires the connecting points of the target object from the multiple candidate points based on the connecting point extraction model and the corresponding probabilities of the multiple candidate points.

In addition to the embodiments shown in fig. 8 and 9, after the connection points extracted by the connection point extraction model are used, the obtained connection points are connected to form the three-dimensional contour of the target object, and as shown in fig. 10, the three-dimensional contour obtained based on the connection point extraction model is similar to the real three-dimensional contour of the target object, which indicates that the connection point extraction model is accurate enough.

It should be noted that, on the basis of the embodiment shown in fig. 8 or fig. 9, before extracting the connection point based on the connection point extraction model, the connection point extraction model needs to be trained, that is, the training process of the connection point extraction model includes: obtaining a plurality of sample images containing sample objects, wherein the sample objects in the plurality of sample images have different postures, obtaining sample line cloud data according to the step 901-903, and determining a positive sample connection point from the sample line cloud data, wherein the positive sample connection point is a real connection point on a three-dimensional contour of the sample object in the sample line cloud data, and the negative sample connection point is other points in the sample line cloud data, which are not the real connection point on the three-dimensional contour of the sample object. Based on the join point extraction model, the predicted join point of the target object and the corresponding probability indicating the possibility that the predicted join point is a join point on the three-dimensional contour of the sample object are determined, as per the above step 904-. Based on the positive sample connection point, the negative sample connection point, the predicted connection point and the probability corresponding to the predicted connection point, a connection point extraction model is trained.

In a possible implementation mode, the connection point extraction model is subjected to iterative training, and the training of the connection point extraction model is stopped under the condition that the iteration times reach a time threshold; or stopping training the connection point extraction model in other ways, which is not limited in the present application.

After the connection point extraction model is trained, the performance of the connection point extraction model is tested on a test data set by taking a probability threshold of 0.5 and a second distance threshold of 10 as examples, and the test results are shown in table 1. As can be seen from Table 1, the accuracy, recall rate and precision of the connection point extraction model are sufficiently high. A true positive indicates the likelihood of predicting a positive sample as a positive sample, and a false negative indicates the likelihood of predicting a positive sample as a negative sample. Through the L2 distance between the positive sample and the negative sample in the test result, the accuracy rate of true positive in the true negative sample is 12.97, the accuracy rate of true positive + false negative is 15.19, and the recall rate of the connecting point is 96.69%, namely the accuracy of the connecting point extraction model is high enough.

TABLE 1

Fig. 11 is a schematic structural diagram of a connection point extraction apparatus provided in an embodiment of the present application, and as shown in fig. 11, the apparatus includes:

a recognition module 1101, configured to recognize two-dimensional line segments from a plurality of images including a target object, respectively, where poses of the target object in the plurality of images are different;

a generating module 1102 for generating line cloud data based on two-dimensional line segments identified from the plurality of images, the line cloud data including a plurality of three-dimensional line segments of the target object;

a determining module 1103, configured to determine, in line cloud data, a plurality of contour points and a plurality of three-dimensional contour line segments corresponding to the contour points, where a contour point is an end point of a three-dimensional line segment in the line cloud data, and a distance between a contour point and a corresponding three-dimensional contour line segment is smaller than distances between a contour point and other three-dimensional line segments;

and an extracting module 1104, configured to extract a connection point of the target object from the three-dimensional contour line segment corresponding to the plurality of contour points.

In one possible implementation, the extraction module 1104 includes:

an extracting unit 1141, configured to perform feature extraction on three-dimensional contour line segments corresponding to the plurality of contour points to obtain three-dimensional contour features;

a first determining unit 1142, configured to determine, based on the three-dimensional contour feature, a plurality of candidate points and a probability corresponding to each candidate point, where the probability indicates a possibility that the candidate point is a connection point on the contour of the target object;

a first obtaining unit 1143, configured to obtain a connection point of the target object from the multiple candidate points based on probabilities corresponding to the multiple candidate points, where the probability corresponding to the connection point is greater than probabilities corresponding to other candidate points.

In another possible implementation manner, the first obtaining unit 1143 is configured to determine, as a 1 st connection point of the target object, an alternative point with a highest probability in the multiple alternative points; deleting the alternative points of which the distance from the 1 st connecting point is less than a first distance threshold from the current remaining alternative points; determining the alternative point with the maximum probability in the residual alternative points after deletion as the 2 nd connection point of the target object; and deleting the alternative points of which the distance from the 2 nd alternative point to the nth connection point is smaller than the first distance threshold value from the currently remaining alternative points until the alternative points do not remain any more or until the nth connection point of the target object is determined, wherein the distance between the currently remaining alternative points and the nth connection point is smaller than the first distance threshold value, and n is an integer larger than 1.

In another possible implementation manner, the extracting unit 1141 is configured to generate a first feature based on coordinates of end points of three-dimensional contour line segments corresponding to a plurality of contour points, where the first feature includes a local feature corresponding to each contour point; respectively reducing the dimension of the local feature corresponding to each contour point in the first feature, and forming a second feature by using a plurality of local features after dimension reduction; carrying out global dimension reduction on the first features to obtain third features; and splicing the second characteristic and the third characteristic to obtain the three-dimensional profile characteristic.

In another possible implementation, as shown in fig. 12, the apparatus further includes:

the determining module 1103 is further configured to determine, based on coordinates of end points of the multiple three-dimensional line segments, an average coordinate value and a distance between each end point in each three-dimensional line segment and an origin of a coordinate system, where the origin of the coordinate system is an origin of a three-dimensional coordinate system in which the line cloud data is located;

a selecting module 1105, configured to select a maximum distance from the currently determined multiple distances;

the determining module 1103 is further configured to determine, for each endpoint in the line cloud data, a difference between a coordinate of the endpoint and an average value of the coordinates, and determine a ratio between the difference and the maximum distance as an updated coordinate of the endpoint;

the updating module 1106 is further configured to, for each three-dimensional line segment in the line cloud data, determine a three-dimensional line segment after updating the three-dimensional line segment based on the coordinates of the updated end points of the three-dimensional line segment, and configure the three-dimensional line segment after updating the plurality of three-dimensional line segments into updated line cloud data.

In another possible implementation, the determining module 1103 is further configured to determine, for each connection point, a product between the coordinate of the connection point and the maximum distance, and determine a sum of the product and the average as the updated coordinate of the connection point.

In another possible implementation manner, the determining module 1103 is configured to select any endpoint from endpoints of multiple three-dimensional line segments, and determine the endpoint as a 1 st contour point; determining a first distance between each of the remaining current end points and the 1 st contour point, and determining the end point with the largest first distance in the remaining current end points as the 2 nd contour point; for each currently remaining endpoint, determining a second distance between the endpoint and each currently determined contour point, and determining the minimum distance in a plurality of second distances corresponding to the endpoint as a third distance corresponding to the endpoint; and determining the endpoint with the maximum third distance in the current remaining endpoints as the 3 rd contour point until the mth contour point is obtained, wherein m is an integer larger than 2.

In another possible implementation manner, the determining module 1103 is configured to determine, for each contour point in the plurality of contour points, a distance between the contour point and each three-dimensional line segment in the line cloud data; determining the number of three-dimensional line segments with the distance from the contour point smaller than a second distance threshold value from the line cloud data; and under the condition that the number is not less than the number threshold, determining the three-dimensional contour line segment corresponding to the contour point by using the number threshold with the minimum distance between the contour point and the line cloud data, wherein the distance between the contour point and the corresponding three-dimensional contour line segment is less than the distance between the contour point and other three-dimensional line segments.

In another possible implementation manner, the determining module 1103 is further configured to, in a case that the number is smaller than the number threshold, determine, as the three-dimensional contour line segment corresponding to the contour point, a three-dimensional line segment in the line cloud data, where the distance between the three-dimensional line segment and the contour point is smaller than a second distance threshold.

In another possible implementation manner, the determining module 1103 is further configured to, for each endpoint of each three-dimensional contour line segment, determine a difference between the coordinate of the endpoint and the coordinate of the contour point corresponding to the three-dimensional contour line segment as an updated coordinate of the endpoint; and respectively determining the updated three-dimensional contour line segment of each three-dimensional contour line segment based on the updated coordinates of the end points of each three-dimensional contour line segment.

a forming module 1107, configured to connect the plurality of connection points to form a three-dimensional contour of the target object based on the distribution positions of the plurality of connection points of the target object.

In another possible implementation, as shown in fig. 12, the generating module 1102 includes:

a second obtaining unit 1121 configured to obtain internal parameters and external parameters corresponding to each image based on the posture of the target object in the plurality of images;

a second determining unit 1122, configured to determine two-dimensional line segments matching each other in the plurality of images based on the two-dimensional line segments in the plurality of images and the internal and external references corresponding to the plurality of images;

the reconstruction unit 1123 is configured to reconstruct two-dimensional line segments, which are matched with each other, in the multiple images based on the internal reference and the external reference of the multiple images, so as to obtain three-dimensional line segments corresponding to the two-dimensional line segments, which are matched with each other;

a constructing unit 1124 configured to construct the obtained three-dimensional line segments into line cloud data.

In another possible implementation manner, the second determining unit 1122 is configured to map, for a first image and a second image in the multiple images, a first line segment in the first image into the second image based on an internal reference and an external reference corresponding to the first image and an internal reference and an external reference corresponding to the second image, so as to obtain a second line segment, where the first line segment is any two-dimensional line segment in the first image; and under the condition that a third line segment in the second image is similar to the second line segment, determining that the first line segment is matched with the third line segment, wherein the third line segment is any two-dimensional line segment in the first image.

It should be noted that: the connection point extracting apparatus provided in the above embodiment is only illustrated by dividing the functional modules, and in practical applications, the functions may be distributed by different functional modules according to needs, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the functions described above. In addition, the connection point extraction device and the connection point extraction method provided by the above embodiments belong to the same concept, and specific implementation processes thereof are detailed in the method embodiments and are not described herein again.

The embodiment of the present application further provides a computer device, where the computer device includes a processor and a memory, where the memory stores at least one computer program, and the at least one computer program is loaded by the processor and executed to implement the operations performed by the connection point extraction method of the foregoing embodiment.

Optionally, the computer device is provided as a terminal. Fig. 13 shows a block diagram of a terminal 1300 according to an exemplary embodiment of the present application. Terminal 1300 includes: a processor 1301 and a memory 1302.

Processor 1301 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 1301 may be implemented in at least one hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), and a PLA (Programmable Logic Array). The processor 1301 may also include a main processor and a coprocessor, where the main processor is a processor for Processing data in an awake state, and is also referred to as a Central Processing Unit (CPU); a coprocessor is a low power processor for processing data in a standby state. In some embodiments, the processor 1301 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing content that the display screen needs to display. In some embodiments, processor 1301 may further include an AI (Artificial Intelligence) processor for processing computational operations related to machine learning.

Memory 1302 may include one or more computer-readable storage media, which may be non-transitory. Memory 1302 may also include high speed random access memory, as well as non-volatile memory, such as one or more magnetic disk storage devices, flash memory storage devices. In some embodiments, a non-transitory computer readable storage medium in the memory 1302 is used to store at least one computer program for execution by the processor 1301 to implement the connection point extraction method provided by the method embodiments in the present application.

In some embodiments, terminal 1300 may further optionally include: a peripheral interface 1303 and at least one peripheral. Processor 1301, memory 1302, and peripheral interface 1303 may be connected by a bus or signal line. Each peripheral device may be connected to the peripheral device interface 1303 via a bus, signal line, or circuit board. Specifically, the peripheral device includes: at least one of radio frequency circuitry 1304, display screen 1305, camera assembly 1306, audio circuitry 1307, and power supply 1308.

Peripheral interface 1303 may be used to connect at least one peripheral associated with I/O (Input/Output) to processor 1301 and memory 1302. In some embodiments, processor 1301, memory 1302, and peripheral interface 1303 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1301, the memory 1302, and the peripheral device interface 1303 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

The Radio Frequency circuit 1304 is used to receive and transmit RF (Radio Frequency) signals, also called electromagnetic signals. The radio frequency circuitry 1304 communicates with communication networks and other communication devices via electromagnetic signals. The radio frequency circuit 1304 converts an electrical signal into an electromagnetic signal to transmit, or converts a received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 1304 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and so forth. Radio frequency circuit 1304 may communicate with other terminals via at least one wireless communication protocol. The wireless communication protocols include, but are not limited to: the world wide web, metropolitan area networks, intranets, generations of mobile communication networks (2G, 3G, 4G, and 5G), Wireless local area networks, and/or WiFi (Wireless Fidelity) networks. In some embodiments, the radio frequency circuit 1304 may also include NFC (Near Field Communication) related circuits, which are not limited in this application.

The display screen 1305 is used to display a UI (User Interface). The UI may include graphics, text, icons, video, and any combination thereof. When the display screen 1305 is a touch display screen, the display screen 1305 also has the ability to capture touch signals on or over the surface of the display screen 1305. The touch signal may be input to the processor 1301 as a control signal for processing. At this point, the display 1305 may also be used to provide virtual buttons and/or a virtual keyboard, also referred to as soft buttons and/or a soft keyboard. In some embodiments, display 1305 may be one, disposed on the front panel of terminal 1300; in other embodiments, the display 1305 can be at least two, respectively disposed on different surfaces of the terminal 1300 or in a folded design; in other embodiments, display 1305 may be a flexible display disposed on a curved surface or on a folded surface of terminal 1300. Even further, the display 1305 may be arranged in a non-rectangular irregular figure, i.e., a shaped screen. The Display panel 1305 may be made of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), or other materials.

The camera assembly 1306 is used to capture images or video. Optionally, camera head assembly 1306 includes a front camera and a rear camera. The front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, the number of the rear cameras is at least two, and each rear camera is any one of a main camera, a depth-of-field camera, a wide-angle camera and a telephoto camera, so that the main camera and the depth-of-field camera are fused to realize a background blurring function, and the main camera and the wide-angle camera are fused to realize panoramic shooting and VR (Virtual Reality) shooting functions or other fusion shooting functions. In some embodiments, camera assembly 1306 may also include a flash. The flash lamp can be a monochrome temperature flash lamp or a bicolor temperature flash lamp. The double-color-temperature flash lamp is a combination of a warm-light flash lamp and a cold-light flash lamp, and can be used for light compensation at different color temperatures.

The audio circuit 1307 may include a microphone and a speaker. The microphone is used for collecting sound waves of a user and the environment, converting the sound waves into electric signals, and inputting the electric signals to the processor 1301 for processing, or inputting the electric signals to the radio frequency circuit 1304 for realizing voice communication. For stereo capture or noise reduction purposes, multiple microphones may be provided, each at a different location of terminal 1300. The microphone may also be an array microphone or an omni-directional pick-up microphone. The speaker is used to convert electrical signals from the processor 1301 or the radio frequency circuitry 1304 into sound waves. The loudspeaker can be a traditional film loudspeaker or a piezoelectric ceramic loudspeaker. When the speaker is a piezoelectric ceramic speaker, the speaker can be used for purposes such as converting an electric signal into a sound wave audible to a human being, or converting an electric signal into a sound wave inaudible to a human being to measure a distance. In some embodiments, audio circuitry 1307 may also include a headphone jack.

Power supply 1308 is used to provide power to various components within terminal 1300. The power source 1308 may be alternating current, direct current, disposable batteries, or rechargeable batteries. When the power source 1308 includes a rechargeable battery, the rechargeable battery may be a wired rechargeable battery or a wireless rechargeable battery. The wired rechargeable battery is a battery charged through a wired line, and the wireless rechargeable battery is a battery charged through a wireless coil. The rechargeable battery may also be used to support fast charge technology.

Those skilled in the art will appreciate that the configuration shown in fig. 13 is not intended to be limiting with respect to terminal 1300 and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components may be employed.

Optionally, the computer device is provided as a server. Fig. 14 is a schematic structural diagram of a server according to an embodiment of the present application, where the server 1400 may generate relatively large differences due to different configurations or performances, and may include one or more processors (CPUs) 1401 and one or more memories 1402, where the memory 1402 stores at least one computer program, and the at least one computer program is loaded and executed by the processors 1401 to implement the methods provided by the foregoing method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input/output interface, so as to perform input/output, and the server may also include other components for implementing the functions of the device, which are not described herein again.

The embodiment of the present application further provides a computer-readable storage medium, where at least one computer program is stored in the computer-readable storage medium, and the at least one computer program is loaded and executed by a processor to implement the operations performed by the connection point extraction method of the foregoing embodiment.

The embodiment of the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the computer program implements the operations performed by the connection point extraction method according to the above embodiment.

It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.

The above description is only an alternative embodiment of the present application, and is not intended to limit the present application, and any modifications, equivalents, improvements and the like made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method of connection point extraction, the method comprising:

2. The method according to claim 1, wherein the extracting connection points of the target object from three-dimensional contour line segments corresponding to the plurality of contour points comprises:

performing feature extraction on the three-dimensional contour line segments corresponding to the contour points to obtain three-dimensional contour features;

determining a plurality of alternative points and a probability corresponding to each alternative point based on the three-dimensional profile feature, the probability indicating a likelihood that the alternative point is a connection point on the profile of the target object;

and acquiring the connection point of the target object from the multiple candidate points based on the probabilities corresponding to the multiple candidate points, wherein the probability corresponding to the connection point is greater than the probabilities corresponding to other candidate points.

3. The method of claim 2, wherein the obtaining the connection point of the target object from the plurality of candidate points based on the probabilities of the plurality of candidate points comprises:

determining the alternative point with the highest probability in the multiple alternative points as the 1 st connecting point of the target object; deleting the alternative points of which the distance from the 1 st connecting point is less than a first distance threshold value from the current remaining alternative points;

determining the alternative point with the highest probability in the residual alternative points after deletion as the 2 nd connecting point of the target object; and deleting the alternative points of which the distance between the alternative points and the 2 nd connecting point is smaller than the first distance threshold value from the currently remaining alternative points until the alternative points are not left any more, or until the nth connecting point of the target object is determined, wherein the distance between the currently remaining alternative points and the nth connecting point is smaller than the first distance threshold value, and n is an integer larger than 1.

4. The method according to claim 2, wherein said performing feature extraction on the three-dimensional contour line segments corresponding to the plurality of contour points to obtain three-dimensional contour features comprises:

generating a first feature based on coordinates of end points of the three-dimensional contour line segments corresponding to the plurality of contour points, wherein the first feature comprises a local feature corresponding to each contour point;

respectively reducing the dimension of the local features corresponding to each contour point in the first features, and forming a second feature by using a plurality of local features after dimension reduction;

performing global dimension reduction on the first feature to obtain a third feature;

and splicing the second characteristic and the third characteristic to obtain the three-dimensional contour characteristic.

5. The method of claim 1, wherein prior to determining a plurality of contour points and three-dimensional contour line segments corresponding to the plurality of contour points in the line cloud data, the method further comprises:

determining a coordinate average value and a distance between each end point in each three-dimensional line segment and a coordinate system origin point based on coordinates of end points of the plurality of three-dimensional line segments, wherein the coordinate system origin point is an origin point in a three-dimensional coordinate system where the line cloud data is located;

selecting a maximum distance from a plurality of currently determined distances;

for each endpoint in the line cloud data, determining a difference value between the coordinate of the endpoint and the coordinate average value, and determining a ratio between the difference value and the maximum distance as the updated coordinate of the endpoint;

and for each three-dimensional line segment in the line cloud data, determining the three-dimensional line segment after the three-dimensional line segment is updated based on the updated coordinates of the end points of the three-dimensional line segment, and forming the three-dimensional line segments after the three-dimensional line segment is updated into the updated line cloud data.

6. The method according to claim 5, wherein after extracting the connection points of the target object from the three-dimensional contour line segments corresponding to the plurality of contour points, the method further comprises:

for each connection point, determining a product between the coordinates of the connection point and the maximum distance, and determining a sum of the product and the average as updated coordinates of the connection point.

7. The method of claim 1, wherein determining a plurality of contour points in the line cloud data comprises:

selecting any end point from the end points of the three-dimensional line segments, and determining the end point as a 1 st contour point;

determining a first distance between each currently remaining end point and the 1 st contour point, and determining an end point with the largest first distance in the currently remaining end points as a 2 nd contour point;

for each currently remaining end point, determining a second distance between the end point and each currently determined contour point, and determining a minimum distance in a plurality of second distances corresponding to the end point as a third distance corresponding to the end point; and determining the endpoint with the maximum third distance in the current remaining endpoints as the 3 rd contour point until the mth contour point is obtained, wherein m is an integer larger than 2.

8. The method of claim 1, wherein determining three-dimensional contour line segments in the line cloud data corresponding to the plurality of contour points comprises:

for each contour point in the plurality of contour points, respectively determining a distance between the contour point and each three-dimensional line segment in the line cloud data;

determining the number of three-dimensional line segments with the distance between the three-dimensional line segments and the contour point smaller than a second distance threshold value from the line cloud data;

and under the condition that the number is not less than a number threshold, determining the three-dimensional contour line segment corresponding to the contour point by using the number threshold three-dimensional line segment with the minimum distance between the contour point and the line cloud data, wherein the distance between the contour point and the corresponding three-dimensional contour line segment is less than the distance between the contour point and other three-dimensional line segments.

9. The method of claim 8, wherein after determining, from the line cloud data, a number of three-dimensional line segments having a distance to the contour point less than a second distance threshold, the method further comprises:

and determining the three-dimensional line segments, of which the distances between the line cloud data and the contour points are smaller than the second distance threshold, as the three-dimensional contour line segments corresponding to the contour points when the number is smaller than the number threshold.

10. The method according to any one of claims 1-9, further comprising:

and connecting the plurality of connecting points to form the three-dimensional contour of the target object based on the distribution positions of the plurality of connecting points of the target object.

11. The method of any one of claims 1-9, wherein generating line cloud data based on two-dimensional line segments identified from the plurality of images comprises:

acquiring internal parameters and external parameters corresponding to each image based on the postures of the target objects in the plurality of images;

determining two-dimensional line segments matched with each other in the plurality of images based on the two-dimensional line segments in the plurality of images and the internal parameters and the external parameters corresponding to the plurality of images;

reconstructing two-dimensional line segments matched with each other in the plurality of images based on the internal parameters and the external parameters of the plurality of images to obtain three-dimensional line segments corresponding to the two-dimensional line segments matched with each other;

and forming the obtained three-dimensional line segments into the line cloud data.

12. A connection point extraction apparatus, characterized in that the apparatus comprises:

a generating module for generating line cloud data based on two-dimensional line segments identified from the plurality of images, the line cloud data including a plurality of three-dimensional line segments of the target object;

13. A computer device, characterized in that the computer device comprises a processor and a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor to implement the operations performed by the connection point extraction method according to any one of claims 1 to 11.

14. A computer-readable storage medium, wherein at least one computer program is stored in the computer-readable storage medium, and the at least one computer program is loaded into and executed by a processor to perform the operations performed by the connection point extraction method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program, characterized in that the computer program, when being executed by a processor, carries out the operations being performed by the connection point extraction method according to any one of claims 1 to 11.