Disclosure of Invention
The embodiment of the application provides a three-dimensional reconstruction method, a three-dimensional reconstruction device and a three-dimensional reconstruction system based on a graphics processor, so as to solve the technical problems at least partially.
The embodiment of the application adopts the following technical scheme:
in a first aspect, an embodiment of the present application provides a three-dimensional reconstruction method based on a graphics processor, where the method includes:
determining an object in the environment;
determining each actual marking point of the object;
determining the maximum distance between the actual marking point and other actual marking points as the specified distance of the actual marking point for each actual marking point;
dividing the designated distance according to the designated step length to obtain a designated number of distance intervals;
dividing each thread of the graphics processor into a plurality of thread groups, such that each distance interval corresponds to one thread group;
for each distance interval, distributing the data corresponding to the distance interval acquired by the auxiliary module to a thread group corresponding to the distance interval, so that the thread group processes the distributed data;
and executing three-dimensional reconstruction according to the processing result obtained by each thread group.
In an optional embodiment of the present description, the auxiliary module includes an auxiliary camera for image capturing of the object.
In an alternative embodiment of the present description, the graphics processor includes at least one of: GPU and multi-core CPU; and/or the presence of a gas in the gas,
the object is a rigid body.
In an optional embodiment of this specification, allocating data corresponding to the distance interval, acquired by the secondary module, to a thread group corresponding to the distance interval includes:
identifying each reference mark point of the object in each frame of image collected by the dynamic auxiliary module;
determining, for each reference marker point, the respective available distances between the reference marker point and the other reference marker points;
and allocating the available distance corresponding to the distance interval in the available distances to the thread group corresponding to the distance interval.
In an optional embodiment of the present specification, allocating an available distance corresponding to the distance interval in the available distances to a thread group corresponding to the distance interval includes:
determining an available distance smaller than the specified distance of the actual marking point in each available distance as an available distance corresponding to the actual marking point for each actual marking point;
determining an available distance matched with the distance interval from available distances corresponding to the actual marking points in the available distances corresponding to the actual marking points as an available distance corresponding to the distance interval for each distance interval of the actual marking points;
and distributing the available distance corresponding to the distance interval to the thread group corresponding to the distance interval.
In an optional embodiment of this specification, after performing three-dimensional reconstruction according to a processing result obtained by each thread group, the method further includes:
and displaying the 3D image obtained by three-dimensional reconstruction to a user.
In a second aspect, an embodiment of the present application further provides a three-dimensional reconstruction system based on a graphics processor, where the system includes a reconstruction processor and an active-auxiliary module;
the reconstruction processor is configured to: determining an object in the environment; determining each actual marking point of the object; determining the maximum distance between the actual marking point and other actual marking points as the specified distance of the actual marking point for each actual marking point; dividing the designated distance according to the designated step length to obtain a designated number of distance intervals; dividing each thread of the graphics processor into a plurality of groups, such that each distance interval corresponds to a thread group; for each distance interval, distributing the data corresponding to the distance interval acquired by the auxiliary module to a thread group corresponding to the distance interval, so that the thread group processes the distributed data; executing three-dimensional reconstruction according to the processing result obtained by each thread group;
the active-auxiliary module is configured to: the dynamic auxiliary module comprises a dynamic auxiliary camera which is used for acquiring images of the object.
In a third aspect, an embodiment of the present application further provides a three-dimensional reconstruction apparatus based on a graphics processor, configured to implement any one of the foregoing methods in the first aspect.
In a fourth aspect, an embodiment of the present application further provides an electronic device, including:
a processor; and
a memory arranged to store computer executable instructions that, when executed, cause the processor to perform any one of the methods of the first aspect.
In a fifth aspect, embodiments of the present application further provide a computer-readable storage medium storing one or more programs which, when executed by an electronic device including a plurality of application programs, cause the electronic device to perform any of the methods of the first aspect.
The embodiment of the application adopts at least one technical scheme which can achieve the following beneficial effects: in the three-dimensional reconstruction process in this specification, in the reconstruction process, the data processing capability provided by the threads of the graphics processor is divided according to the distance between the actual mark points on the object (i.e., the side length between the actual mark points), so as to obtain a plurality of thread groups, and different thread groups process data corresponding to different distance intervals. The data processing capacity provided by the graphic processor is effectively utilized, and the data processing efficiency is improved. In addition, the multithreading processing architecture of the graphics processor is also beneficial to increasing the data processing speed.
Detailed Description
The present invention will be described in further detail with reference to the following detailed description and accompanying drawings. Wherein like elements in different embodiments are numbered with like associated elements. In the following description, numerous details are set forth in order to provide a better understanding of the present application. However, those skilled in the art will readily recognize that some of the features may be omitted or replaced with other elements, materials, methods in different instances. In some instances, certain operations related to the present application have not been shown or described in detail in order to avoid obscuring the core of the present application from excessive description, and it is not necessary for those skilled in the art to describe these operations in detail, so that they may be fully understood from the description in the specification and the general knowledge in the art.
Furthermore, the features, operations, or characteristics described in the specification may be combined in any suitable manner to form various embodiments. Also, the various steps or actions in the method descriptions may be transposed or transposed in order, as will be apparent to one of ordinary skill in the art. Thus, the various sequences in the specification and drawings are for the purpose of describing certain embodiments only and are not intended to imply a required sequence unless otherwise indicated where such sequence must be followed.
The numbering of the components as such, e.g., "second", etc., is used herein only to distinguish the objects as described, and does not have any sequential or technical meaning. The term "connected" and "coupled" when used in this application, unless otherwise indicated, includes both direct and indirect connections (couplings).
The technical solutions provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings.
The graphics processor-based three-dimensional reconstruction method in the present specification may include the steps of:
s100: objects in the environment are determined.
In an alternative embodiment of the present description, the processes in the present description may be performed by a three-dimensional reconstruction system of a graphics processor. In particular, a three-dimensional reconstruction system of a graphics processor may include a reconstruction processor and an active auxiliary module, and the processes in this specification may be performed by the reconstruction processor.
The dynamic auxiliary module in the specification is used for acquiring images of a specified environment, different dynamic auxiliary modules are different in pose when acquiring the images of the specified environment, and the images acquired by the dynamic auxiliary module comprise a plurality of two-dimensional points.
The three-dimensional reconstruction process in this specification does not specifically limit the objects in the environment involved. Alternatively, the objects in the environment may be two or more.
The dynamic assistant module in this specification may be a physical camera having a physical structure, or may be a virtual camera corresponding to a viewpoint of a user.
The specific circumstances in this specification may be determined according to actual needs. Illustratively, the specified environment may be a certain room, or a certain area in a certain room.
When the three-dimensional reconstruction process based on the graphic processor in this specification is applied to the field of the holographic display technology, the specified environment may be an environment in which a holographic sand table (sand table display device) is located, as shown in fig. 2 for example. In this example, the execution subject of the process in this specification may be a control module of a holographic presentation system to which the holographic sand table belongs.
S102: determining each actual marking point of the object.
The actual marking point mode can be obtained through a manual calibration mode, and then the calibrated actual marking point is input into a reconstruction processor.
Since the designated environment is a certain space, the actual marked points in the designated environment can be represented by three-dimensional coordinates. Optionally, in the holographic display scene, an origin of a three-dimensional coordinate system for determining three-dimensional coordinates is at the center of the holographic sand table.
S104: for each actual marking point, determining the maximum distance between the actual marking point and other actual marking points as the designated distance (L) of the actual marking pointMax)。
Suppose there is N on an object
RAn actual mark point corresponds to an object
And (4) arranging edges. The edges of these rigid bodies are matched to the N identified by the system
sThe edges of each reference mark point are provided with
And (4) respectively. The edge of each actual mark point is longer than the edge of the reference mark point, if the lengths of the two reference mark points are the same, the two current reference mark points correspond to the two current actual mark points, and then the two actual mark points are displayed on the screenA match may be made. Thereafter, a three-dimensional reconstruction may be performed based on the matched reference marker points.
Since a plurality of actual marking points are determined for each object, the designated distance of each actual marking point needs to be determined.
S106: and dividing the specified distance according to the specified step length to obtain a specified number of distance intervals.
For example: l isMaxThe value of (d) is 1.0 meter and the specified step size is 0.1 meter. 10 individual distance intervals are obtained, the specified number being 10.
S108: the threads of the graphics processor are divided into thread groups such that each distance interval corresponds to a thread group.
For example, each thread group corresponds to a bucket, divided into 10 buckets, bucket 1 contains 0 to 0.1 meter long edges, bucket 2 contains 0.1 to 0.2 meter long edges, and so on. Alternatively, the threads may be divided equally into thread groups. The resulting number of thread groups is greater than or equal to the specified number.
Side length (i.e. distance) less than LMaxThe edges of (a) can be allocated to different buckets by specific lengths, and the operation can also realize multithread acceleration.
S110: and for each distance interval, distributing the data corresponding to the distance interval acquired by the auxiliary module to the thread group corresponding to the distance interval, so that the thread group processes the distributed data.
When performing the processing, if the distance between the reference mark points is 0.15 m long, only the sides of the reference mark points in the bucket 2 need to be matched, greatly reducing the amount of calculation.
Optionally, the graphics processor is a GPU. The graphics processor processes all points in the data simultaneously to increase the computation speed. The GPU has the advantage over a Central Processing Unit (CPU) that a large number of threads can be generated, which reduces the total computation time when computing in large numbers in parallel.
S112: and executing three-dimensional reconstruction according to the processing result obtained by each thread group.
And then, displaying the 3D image obtained by three-dimensional reconstruction to a user.
Therefore, through the three-dimensional reconstruction process based on the graphics processor in the specification, a corresponding relationship can be established between the thread group and each distance interval, and the reasonable distribution of the processing resources of the graphics processing among the distance intervals is realized.
Since the rigid body is mobile, the process is applied to each acquired frame image. Generation of RSThe side lengths of all 3D points are calculated by each thread. If the side length is less than LMaxAnd reserving the reference mark points corresponding to the edges. Less than LMaxNumber of sides (by E'STo represent) is much smaller than ES. Using multiple threads to make a side length less than LMaxIs collected to another variable, and occupies only E 'of memory'sLength of (E) compared to occupation of E at subsequent accessesSThe long memory size reduces the computation time by reducing the memory bandwidth required to access the data.
In an optional embodiment of the present description, the auxiliary module includes an auxiliary camera for image capturing of the object. Further, the graphics processor may be a multi-core CPU and the object may be a rigid body. Each reference marker point corresponds to an actual marker point on the object.
When the collected image packet comprises a plurality of frames which are continued according to time, each reference mark point of the object in each frame of image collected by the auxiliary module can be identified; determining, for each reference marker point, the respective available distances between the reference marker point and the other reference marker points; determining an available distance smaller than the specified distance of the actual marking point in each available distance as an available distance corresponding to the actual marking point for each actual marking point; determining an available distance matched with the distance interval from available distances corresponding to the actual marking points in the available distances corresponding to the actual marking points as an available distance corresponding to the distance interval for each distance interval of the actual marking points; and distributing the available distance corresponding to the distance interval to the thread group corresponding to the distance interval.
Based on the same idea, the embodiments of the present specification further provide a graphics processor-based three-dimensional reconstruction apparatus corresponding to the partial process shown in fig. 1. As shown in fig. 3, the graphics processor-based three-dimensional reconstruction apparatus may include one or more of the following modules:
an object determination module 300 configured to: objects in the environment are determined.
An actual marker determination module 302 configured to: determining each actual marking point of the object.
A specified distance determination module 304 configured to: and determining the maximum distance between the actual marking point and other actual marking points as the specified distance of the actual marking point for each actual marking point.
A distance interval determination module 306 configured to: and dividing the specified distance according to the specified step length to obtain a specified number of distance intervals.
A thread group determination module 308 configured to: the threads of the graphics processor are divided into thread groups such that each distance interval corresponds to a thread group.
An assignment module 310 configured to: and for each distance interval, distributing the data corresponding to the distance interval acquired by the auxiliary module to the thread group corresponding to the distance interval, so that the thread group processes the distributed data.
A three-dimensional reconstruction module 312 configured to: and executing three-dimensional reconstruction according to the processing result obtained by each thread group.
In an optional embodiment of the present description, the auxiliary module includes an auxiliary camera for image capturing of the object.
In an alternative embodiment of the present description, the graphics processor includes at least one of: GPU and multi-core CPU; and/or, the object is a rigid body.
In an optional embodiment of the present description, the allocation module is specifically configured to: identifying each reference mark point of the object in each frame of image collected by the dynamic auxiliary module;
determining, for each reference marker point, the respective available distances between the reference marker point and the other reference marker points;
and allocating the available distance corresponding to the distance interval in the available distances to the thread group corresponding to the distance interval.
In an optional embodiment of the present description, the allocation module is specifically configured to: determining an available distance smaller than the specified distance of the actual marking point in each available distance as an available distance corresponding to the actual marking point for each actual marking point;
determining an available distance matched with the distance interval from available distances corresponding to the actual marking points in the available distances corresponding to the actual marking points as an available distance corresponding to the distance interval for each distance interval of the actual marking points;
and distributing the available distance corresponding to the distance interval to the thread group corresponding to the distance interval.
In an alternative embodiment of the present description, the apparatus further comprises a display module. And the display module is configured to display the three-dimensional reconstructed 3D image to a user.
It can be understood that, the three-dimensional reconstruction apparatus based on a graphics processor can implement the steps of the three-dimensional reconstruction process based on a graphics processor performed by the three-dimensional reconstruction apparatus based on a graphics processor provided in the foregoing embodiments, and the related explanations regarding the three-dimensional reconstruction method based on a graphics processor are applicable to the three-dimensional reconstruction apparatus based on a graphics processor, and are not repeated herein.
Fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present application. Referring to fig. 4, at a hardware level, the electronic device includes a processor, and optionally further includes an internal bus, a network interface, and a memory. The Memory may include a Memory, such as a Random-Access Memory (RAM), and may further include a non-volatile Memory, such as at least 1 disk Memory. Of course, the electronic device may also include hardware required for other services.
The processor, the network interface, and the memory may be connected to each other via an internal bus, which may be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, an EISA (Extended Industry Standard Architecture) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one double-headed arrow is shown in FIG. 4, but that does not indicate only one bus or one type of bus.
And the memory is used for storing programs. In particular, the program may include program code comprising computer operating instructions. The memory may include both memory and non-volatile storage and provides instructions and data to the processor.
The processor reads a corresponding computer program from the nonvolatile memory into the memory and then runs the computer program to form a three-dimensional reconstruction device based on the graphics processor on a logic level. And the processor is used for executing the program stored in the memory and is particularly used for executing any one of the three-dimensional reconstruction processes based on the graphics processor.
The method performed by the three-dimensional reconstruction device based on the graphics processor according to the embodiment shown in fig. 1 of the present application can be applied to or implemented by a processor. The processor may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in a processor or instructions in the form of software. The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components. The various methods, steps, and logic blocks disclosed in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in a memory, and a processor reads information in the memory and completes the steps of the method in combination with hardware of the processor.
The electronic device may further perform at least a part of the method steps performed by the three-dimensional reconstruction apparatus based on the graphics processor in fig. 1, and implement a three-dimensional reconstruction apparatus based on the graphics processor, which is not described herein again in this embodiment of the present application.
The present application also provides a computer-readable storage medium storing one or more programs, where the one or more programs include instructions, which when executed by an electronic device including a plurality of application programs, enable the electronic device to perform the method performed by the graphics processor-based three-dimensional reconstruction apparatus in the embodiment shown in fig. 1, and in particular to perform any one of the aforementioned graphics processor-based three-dimensional reconstruction methods.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The above description is only an example of the present application and is not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.