CN112254677B

CN112254677B - Multi-position combined 3D acquisition system and method based on handheld device

Info

Publication number: CN112254677B
Application number: CN202011105994.5A
Authority: CN
Inventors: 左忠斌; 左达宇
Original assignee: Tianmu Aishi Beijing Technology Co Ltd
Current assignee: Tianmu Aishi Beijing Technology Co Ltd
Priority date: 2020-10-15
Filing date: 2020-10-15
Publication date: 2022-10-21
Anticipated expiration: 2040-10-15
Also published as: CN112254677A

Abstract

The embodiment of the invention provides a multi-position combined 3D acquisition system and a method based on handheld equipment, wherein the multi-position combined 3D acquisition system comprises the following steps: the system comprises a plurality of handheld 3D acquisition devices, wherein in the plurality of handheld 3D acquisition devices, the acquisition range of each handheld 3D acquisition device on a target object is at least overlapped with the acquisition ranges of other two handheld 3D acquisition devices on the target object; the handheld 3D acquisition equipment comprises an image acquisition device, a rotating device and a handheld part; wherein the collection direction of the image collection device is the direction deviating from the rotation center. The handheld self-rotating 3D acquisition equipment is arranged at a plurality of positions, so that a complete multi-position combined 3D acquisition system is formed. The acquisition of the inner space of a complex surface or a large-range target object is realized.

Description

Multi-position combined 3D acquisition system and method based on handheld device

Technical Field

The invention relates to the technical field of topography measurement, in particular to the technical field of 3D topography measurement.

Background

In making a 3D measurement, it is necessary to first acquire 3D information. Currently common methods include the use of machine vision and structured light, laser ranging, lidar.

Structured light, laser ranging and laser radar all need an active light source to emit to a target object, and the target object is affected under certain conditions, and the cost of the light source is high. And the light source structure is more accurate, easily damages.

The machine vision mode is to collect the pictures of the object at different angles and match and splice the pictures to form a 3D model, so that the cost is low and the use is easy. When the device collects pictures at different angles, a plurality of cameras can be arranged at different angles of an object to be detected, and the pictures can be collected from different angles through rotation of a single camera or a plurality of cameras. However, in either of these two methods, the capturing position of the camera needs to be set around the target (referred to as a wraparound method), but this method needs a large space for setting the capturing position for the image capturing device.

Moreover, besides the 3D construction of a single object, there are also requirements for 3D model construction of the internal space of the object and 3D model construction of the peripheral large field of view, which are difficult to achieve by the conventional surrounding type 3D acquisition device. Particularly, the surface of the target object is complex (the surface is uneven and the unevenness is deep) in the internal space or the large field range, and at the time, each part of the surface pit or the surface bump is difficult to cover by collecting at a single position, so that a complete 3D model is difficult to obtain during final synthesis, even synthesis fails, or synthesis time is prolonged. And in some situations, 3D acquisition needs to be performed quickly on site, which takes a lot of time if installation configuration is performed, and many environments do not allow for fixed installation. For example, when temporary part inspection is performed, there is no fixed inspection equipment on the production line, and a handheld device is needed to be flexibly used.

In the prior art, it has also been proposed to define the camera position using empirical formulas including rotation angle, object size, object distance, to take into account the speed of the synthesis and the effect. However in practice this has been found to be feasible in wrap-around 3D acquisitions, where the target size may be measured in advance. However, it is difficult to measure the target object in advance in an open space, and it is necessary to acquire 3D information of streets, traffic intersections, building groups, tunnels, traffic flows, and the like (not limited thereto). Which makes this approach difficult to work. Even if the dimensions of fixed, small objects, such as furniture, human body parts, etc., can be measured beforehand, this method is still subject to major limitations: the size of the target is difficult to accurately determine, and particularly, the target needs to be frequently replaced in certain application occasions, each measurement brings a large amount of extra workload, and professional equipment is needed to accurately measure irregular targets. The measured error causes the camera position setting error, thereby influencing the acquisition and synthesis speed and effect; accuracy and speed need to be further improved.

Although there are methods for optimizing the surround-type acquisition device in the prior art, there is no better optimization method in the prior art when the acquisition direction of the camera of the 3D acquisition and synthesis device and the direction of its rotation axis deviate from each other.

Therefore, there is an urgent need for a device capable of accurately, efficiently and conveniently collecting 3D information with complicated peripheral or internal space.

Disclosure of Invention

In view of the above, the present invention is proposed to provide a multi-position combined 3D acquisition system and method based on handheld device that overcomes or at least partially solves the above mentioned problems.

The embodiment of the invention provides a multi-position combined 3D acquisition system and a method based on handheld equipment, wherein the multi-position combined 3D acquisition system comprises the following steps: comprising a plurality of handheld 3D acquisition devices,

in the plurality of handheld 3D acquisition devices, the acquisition range of each handheld 3D acquisition device on the target object is at least overlapped with the acquisition ranges of the other two handheld 3D acquisition devices on the target object respectively;

the handheld 3D acquisition equipment comprises an image acquisition device, a rotating device and a handheld part; wherein the collection direction of the image collection device is the direction deviating from the rotation center.

In an alternative embodiment, the plurality of handheld 3D acquisition devices comprises a first type of handheld 3D acquisition device and a second type of handheld 3D acquisition device.

In an alternative embodiment, the sum of the acquisition ranges of the first type of handheld 3D acquisition device can cover the target object, and the sum of the acquisition ranges of the second type of handheld 3D acquisition device can cover a specific area of the target object.

In an optional embodiment, the plurality of handheld 3D acquisition devices include a first type of handheld 3D acquisition device and a second type of handheld 3D acquisition device, and a sum of acquisition ranges of the first type of handheld 3D acquisition device is greater than a sum of acquisition ranges of the second type of handheld 3D acquisition device.

In an alternative embodiment, for a specific region of the target object, a first type of handheld 3D acquisition device and a second type of handheld 3D acquisition device are used for scanning and acquiring together.

In an alternative embodiment, the specific area is a user-specified area.

In an alternative embodiment, the specific area is a previous synthesis failure area.

In an alternative embodiment, the specific area is an area where the degree of change of the contour unevenness is greater than a preset threshold.

The included angle alpha of the optical axes of the image acquisition device at two adjacent acquisition positions meets the following condition:

wherein, R is the distance from the rotation center to the surface of the target object, T is the sum of the object distance and the image distance during acquisition, d is the length or the width of a photosensitive element of the image acquisition device, F is the focal length of a lens of the image acquisition device, and u is an empirical coefficient.

In an alternative embodiment u <0.498, for better synthesis u <0.411 is preferred, in particular u <0.359, in some applications u <0.281, or u <0.169, or u <0.041, or u <0.028.

In another aspect, the embodiment of the present invention further provides a 3D synthesis/identification apparatus and method, including any one of the above systems and methods.

In another aspect, the present invention provides an apparatus and a method for manufacturing/displaying an object, including any of the above systems and methods.

Invention and technical effects

1. The 3D information of the inner space of the target object is collected by the autorotation type intelligent vision 3D collecting equipment, and the autorotation type intelligent vision 3D collecting equipment is suitable for wider space and thinner space.

2. The method has the advantages that the acquisition position of the camera is optimized by measuring the distance between the rotation center and the target object and the distance between the image sensing element and the target object, so that the speed and the effect of 3D construction are considered.

3. The handheld self-rotating 3D acquisition equipment is arranged at a plurality of positions, so that a complete multi-position combined 3D acquisition system is formed. The acquisition of the inner space of a complex surface or a large-range target object is realized, and the implementation is more flexible.

4. The method is provided for the first time, and specific scanning acquisition is carried out on a specific area through the arrangement of two types of acquisition equipment, so that accurate and efficient acquisition of complex objects is realized.

Drawings

Various additional advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:

fig. 1 shows a schematic structural diagram of a 3D information acquisition device provided in an embodiment of the present invention;

fig. 2 is a schematic structural diagram illustrating another implementation manner of a 3D information acquisition device according to an embodiment of the present invention;

FIG. 3 shows a block diagram of a system provided by an embodiment of the present inventionOf multi-position combined 3D acquisition systemsSchematic representation.

The correspondence of reference numbers to the various components in the drawings is as follows:

1. an image acquisition device;

2. a rotating device;

3. and a bearing device.

Detailed Description

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

Hand-held type 3D information acquisition equipment structure

To solve the above technical problem, an embodiment of the present invention provides a 3D information collecting device, referred to as a 3D collecting device for short, referring to fig. 1, including an image collecting device 1, a rotating device 2 and a bearing device 3.

The image acquisition device 1 is connected with the rotating device 2, so that the rotating device 2 can be driven to stably rotate and scan, and 3D acquisition of peripheral objects is realized (the specific acquisition process is described in detail below). The rotating means 2 are mounted on a carrier means 3, the carrier means 3 being intended to carry the entire apparatus. The carrier means 3 may be a handle, so that the entire apparatus can be used for hand-held acquisition. The carrying device 3 may also be a base type carrying device for being installed on other devices, so that the whole intelligent 3D acquisition device is installed on other devices for common use. For example, the smart 3D acquisition device is mounted on a vehicle, and 3D acquisition is performed as the vehicle travels.

The bearing device 3 is used for bearing the weight of the whole equipment, and the rotating device 2 is connected with the bearing device 3. The carrying means may be a handle, a tripod, a base with a support means, etc. Typically, the rotating means is located in the central part of the carrying means to ensure balance. But in special cases it can be located anywhere on the carrier. And the carrier is not necessary. The rotating device may also be mounted directly in the application, for example on the roof of a vehicle. The inner space of the bearing device is used for accommodating a battery and supplying power to the 3D rotary acquisition stabilizing device. Simultaneously, for convenient to use, set up the button on bearing the device shell for control 3D rotation gathers stabilising arrangement. Including on/off stabilization, on/off 3D rotational acquisition.

As shown in fig. 2, the image capturing device 1 is connected to the rotating shaft of the rotating device 2, and is driven by the rotating device to rotate. Of course, the rotation axis of the rotation device may also be connected to the image capture device via a transmission device, such as a gear set. The rotating device 2 can be arranged in the handle, and part or all of the transmission device is also arranged in the handle, so that the volume of the equipment can be further reduced.

When the image pickup device performs 360 ° rotation in the horizontal plane, it picks up an image of a corresponding target object at a specific position (the specific pickup position will be described later in detail). The shooting can be performed synchronously with the rotation action, or shooting can be performed after the rotation of the shooting position is stopped, and the rotation is continued after the shooting is finished, and the like. The rotating device can be a motor, a stepping motor, a servo motor, a micro motor and the like. The rotating device (e.g., various motors) can rotate at a prescribed speed under the control of the controller and can rotate at a prescribed angle, thereby achieving optimization of the acquisition position, which will be described in detail below. Of course, the image acquisition device can be mounted on the rotating device in the existing equipment.

The above device may further include a distance measuring device, the distance measuring device is fixedly connected to the image collecting device, and a direction of the distance measuring device is the same as an optical axis direction of the image collecting device. Of course, the distance measuring device can also be fixedly connected to the rotating device, as long as the distance measuring device can synchronously rotate along with the image acquisition device. Preferably, an installation platform can be arranged, the image acquisition device and the distance measurement device are both positioned on the platform, and the platform is installed on a rotating shaft of the rotating device and driven to rotate by the rotating device. The distance measuring device can use various modes such as a laser distance measuring instrument, an ultrasonic distance measuring instrument, an electromagnetic wave distance measuring instrument and the like, and can also use a traditional mechanical measuring tool distance measuring device. Of course, in some applications, the 3D acquisition device is located at a specific location, and its distance from the target object is calibrated, without additional measurements.

The device also comprises a light source which can be arranged on the periphery of the image acquisition device, the rotating device and the mounting platform. Of course, the light source may be separately provided, for example, a separate light source may be used to illuminate the target. Even when the lighting conditions are good, no light source is used. The light source can be an LED light source or an intelligent light source, namely, the light source parameters are automatically adjusted according to the conditions of the target object and the ambient light. Typically, the light sources are distributed around the lens of the image capturing device, for example, the light sources are ring-shaped LED lamps around the lens. Since in some applications it is desirable to control the intensity of the light source. In particular, a light softening means, such as a light softening envelope, may be arranged in the light path of the light source. Or an LED surface light source is directly adopted, so that the light is soft, and the light is emitted more uniformly. Preferably, an OLED light source can be adopted, the size is smaller, the light is softer, and the flexible OLED light source has the flexible characteristic and can be attached to a curved surface.

In order to facilitate the actual size measurement of the target object, a plurality of marking points can be arranged at the position of the target object. And the coordinates of these marked points are known. The absolute size of the 3D synthetic model is obtained by collecting the mark points and combining the coordinates thereof. These marking points may be previously set points or may be laser light spots. The method of determining the coordinates of the points may comprise: (1) using laser for ranging: and emitting laser towards the target object by using the calibration device to form a plurality of calibration point light spots, and obtaining the coordinates of the calibration points through the known position relation of the laser ranging units in the calibration device. And emitting laser towards the target by using the calibration device, so that the light beam emitted by the laser ranging unit in the calibration device falls on the target to form a light spot. Since the laser beams emitted from the laser ranging units are parallel to each other, the positional relationship between the respective units is known. The two-dimensional coordinates in the emission plane of the plurality of light spots formed on the target object can be obtained. The distance between each laser ranging unit and the corresponding light spot can be obtained by measuring the laser beam emitted by the laser ranging unit, namely the depth information equivalent to a plurality of light spots formed on the target object can be obtained. I.e. the depth coordinate perpendicular to the emission plane, can be obtained. Thereby, three-dimensional coordinates of each spot can be obtained. (2) Using range finding in combination with angle measurement: and respectively measuring the distances of the plurality of mark points and the included angles between the mark points, thereby calculating respective coordinates. (3) Using other coordinate measurement tools: such as RTK, global coordinate positioning systems, satellite-sensitive positioning systems, position and pose sensors, etc.

Multi-position combined 3D acquisition system

As shown in fig. 3, the acquisition system includes a plurality of the above-described hand-held 3D information acquisition devices a, b, c \ 8230, which are respectively located at different spatial positions. The acquisition range of the acquisition equipment a comprises an area A, the acquisition range of the acquisition equipment B comprises an area B, the acquisition range of the acquisition equipment C comprises an area C (8230), and the like. Their acquisition regions at least satisfy that the intersection between two acquisition regions is not empty. In particular, the intersection that is not empty should be located on the target. That is, each acquisition device overlaps with the acquisition ranges of the other two acquisition devices at least, and particularly, the acquisition range of each acquisition device on the target object overlaps with the acquisition ranges of the other two acquisition devices on the target object at least.

Whether they are objects in the interior space or in a wide field of view, they may have areas of more complex surface, referred to as specific areas. These areas either have inwardly recessed deep holes/pits or outwardly projecting higher protrusions or both, thus constituting a larger degree of surface relief. This presents challenges to acquisition devices that acquire in one direction. Due to the concave and convex parts, no matter where the device is arranged, the specific area of the target object can be acquired only from a single direction through rotary scanning, and therefore information of the specific area is greatly lost.

Therefore, the acquisition devices at multiple positions can be set to scan and acquire the specific area, so that the information of the area is obtained from different angles. For example, the intersection of the a region and the B region includes the specific region; the common intersection of the area A, the area B and the area C comprises the specific area; the intersection of the a region and the B region and the intersection of the C region and the D region each include the specific region, and so on. That is, the specific area is repeatedly scanned, which may be referred to as a repeated scanning area, that is, the specific area is scanned and acquired by a plurality of acquisition devices. The above cases include that the intersection of the acquisition regions of two or more acquisition devices includes the specific region; the intersection of the acquisition regions of two or more acquisition devices and the intersection of the acquisition regions of other two or more acquisition devices both include the specific region.

The specific area can be obtained by analyzing the previous 3D synthesis condition, for example, the previous 3D synthesis fails or the area with higher failure rate; the regions may be defined in advance according to the experience of the operator, for example, regions having a large fluctuation in the unevenness, regions having a large degree of unevenness, or the like.

In another embodiment, multiple handheld 3D information acquisition devices are not required, but only one handheld 3D information acquisition device is used. The user holds the equipment, and the equipment is respectively positioned at different positions to carry out rotary scanning for multiple times to obtain the target object picture. At this time, it should be ensured that the scanning ranges of the handheld 3D information acquisition devices are overlapped to cover the whole target area each time the handheld 3D information acquisition devices are located.

3D information acquisition process

1. And selecting the number of the first type of 3D information acquisition equipment according to the size and the position of the target object, and arranging the position of each 3D information acquisition equipment.

(1) According to the acquisition requirement of the target object, the position where the 3D information acquisition equipment can be placed is set, and the distance between the 3D information acquisition equipment and the target object is determined.

(2) According to the size of the target object, the distance and the collection ranges A, B and C of the multiple 3D information collection devices a, B and C \8230, the number of the 3D information collection devices is selected, so that the sum of the collection ranges of the 3D information collection devices can cover the target object. But in general, the sum of the acquisition ranges of the 3D information acquisition devices is required to cover the size of the target, and in the case that the acquisition ranges of adjacent 3D information acquisition devices overlap, the sum of their acquisition ranges still covers the size of the target. For example, the overlap range accounts for more than 10% of the acquisition range.

(3) The selected plurality of 3D information acquisition devices a, b and c \8230arearranged at the positions with the distances from the target object relatively uniformly, so that the acquisition areas of the plurality of 3D information acquisition devices a, b and c \8230areensured to cover the target object.

2. And setting the number of the second type of 3D information acquisition equipment according to the size, the number and the position of the specific area of the target object, and arranging the position for each 3D information acquisition equipment.

(1) And determining the number and the position of the specific areas of the target object. The determination may be based on prior knowledge, or on visual results, or on the distribution of regions not synthesized in the previous acquisition.

(2) One or more second type 3D information acquisition devices are arranged for each specific area according to the size of the specific area of the object so that their acquisition range can cover the specific area.

(3) And determining the number of the second type of 3D information acquisition equipment according to the number and the position of the specific areas of the target object and the number of the second type of 3D information acquisition equipment required by each specific area, and arranging the position for each 3D information acquisition equipment. In general, one or more second-type 3D information acquisition devices are inserted between the first-type 3D information acquisition devices, so as to repeatedly acquire a region with a weak acquisition range of the first-type 3D information acquisition devices, that is, repeatedly acquire a specific region, and form a repeated scanning region. The second type of 3D information acquisition device may also be located at other positions (e.g., closer or further away from the target object) to ensure that the rescanning area can obtain sufficiently different angle pictures. The position of the first type of acquisition equipment is called a first type of acquisition position, and the position of the second type of acquisition equipment is called a second type of acquisition position.

3. After the first type and the second type of 3D information acquisition equipment are arranged, each 3D information acquisition equipment is controlled to rotate to scan the target object, and the rotation meets the optimization condition of an image acquisition device of the 3D information acquisition equipment. That is, the image pickup device of each 3D information pickup apparatus may be controlled to rotate by the controller according to the above conditions.

4. And sending pictures acquired by scanning of a plurality of 3D information acquisition devices to a processor, and performing synthetic modeling on the 3D model of the target object by using the pictures by the processor. Similarly, the plurality of pictures can be sent to a remote platform, a cloud platform, a server, an upper computer and/or a mobile terminal through a communication device, and 3D synthesis of the target object can be carried out by using a 3D model synthesis method.

In another embodiment, instead of setting a plurality of acquisition devices, one acquisition device is replaced with a plurality of acquisition devices in the above flow, and the acquisition position of each acquisition device is a position where the user holds the device to perform acquisition in sequence. Namely, the user holds the device and sequentially locates the first type and the second type of 3D information acquisition devices at the acquisition positions. And controlling the handheld acquisition equipment to rotate for acquisition each time when the handheld acquisition equipment is in an acquisition position (a first type acquisition position or a second type acquisition position). And finally, transmitting each acquired picture to a processor for 3D modeling and formation.

In another embodiment, in addition to the above-described combined acquisition using a plurality of 3D acquisition devices, it is understood that one 3D acquisition device or a limited number of 3D acquisition devices may be used to respectively perform the acquisition at the set positions in time division sequentially. That is, the images are not acquired simultaneously, but acquired at different positions in time division, and the images acquired at different times are collected to perform 3D synthesis. The different positions described here are the same as the positions described above for the different acquisition devices.

Optimization of camera position

In order to ensure that the device can give consideration to the effect and efficiency of 3D synthesis, the method can be used for optimizing the acquisition position of the camera besides the conventional method for optimizing the synthesis algorithm. Especially in the case of 3D acquisition synthesis devices in which the acquisition direction of the camera and the direction of its axis of rotation deviate from each other, the prior art does not mention how to perform a better optimization of the camera position for such devices. Even if some optimization methods exist, they are different empirical conditions obtained under different experiments. In particular, some existing position optimization methods need to obtain the size of the target, which is feasible in the wrap-around 3D acquisition, and can be measured in advance. However, it is difficult to measure in advance in an open space. It is therefore desirable to propose a method that can be adapted to the optimization of the camera position when the acquisition direction of the camera of the 3D acquisition synthesis device and the direction of its rotation axis deviate from each other. It is to the problem, and a technical contribution, that the present invention solves.

For this reason, the present invention has performed a large number of experiments, and it is concluded that empirical conditions preferably satisfied by the intervals of camera acquisition when acquisition is performed are as follows.

When 3D acquisition is carried out, the included angle alpha of the optical axis of the image acquisition device at two adjacent positions meets the following condition:

wherein the content of the first and second substances,

r is the distance from the center of rotation to the surface of the target,

t is the sum of the object distance and the image distance during acquisition, namely the distance between the photosensitive unit of the image acquisition device and the target object.

d is the length or width of a photosensitive element (CCD) of the image acquisition device, and when the two positions are along the length direction of the photosensitive element, the length of the rectangle is taken as d; when the two positions are along the width direction of the photosensitive element, d takes a rectangular width.

And F is the focal length of the lens of the image acquisition device.

u is an empirical coefficient.

Usually, a distance measuring device, for example a laser distance meter, is arranged on the acquisition device. The optical axis of the distance measuring device is parallel to the optical axis of the image acquisition device, so that the distance from the acquisition device to the surface of the target object can be measured, and R and T can be obtained by using the measured distance and according to the known position relation between the distance measuring device and each part of the acquisition device.

When the image acquisition device is at any one of the two positions, the distance from the photosensitive element to the surface of the target object along the optical axis is taken as T. In addition to this method, multiple averaging or other methods can be used, the principle being that the value of T should not deviate from the sum of the image distances from the object at the time of acquisition.

Similarly, when the image pickup device is at any one of the two positions, the distance from the rotation center to the surface of the object along the optical axis is defined as R. In addition to this method, multiple averaging or other methods may be used, with the principle that the value of R should not deviate from the radius of rotation at the time of acquisition.

In general, the size of an object is adopted as a method for estimating the position of a camera in the prior art. Since the object size will vary with the measured object. For example, when a large object is acquired by 3D information and then a small object is acquired, the size needs to be measured again and reckoning needs to be performed again. The inconvenient measurement and the repeated measurement bring errors in measurement, thereby causing errors in camera position estimation. According to the scheme, the experience conditions required to be met by the position of the camera are given according to a large amount of experimental data, and the size of an object does not need to be directly measured. In the empirical condition, d and F are both fixed parameters of the camera, and when the camera and the lens are purchased, a manufacturer can give corresponding parameters without measurement. R and T are only one straight-line distance, and can be conveniently measured by using a traditional measuring method, such as a ruler and a laser range finder. Meanwhile, in the apparatus of the present invention, the capturing direction of the image capturing device (e.g., camera) and the direction of the rotation axis thereof are away from each other, that is, the lens is oriented substantially opposite to the rotation center. At the moment, the included angle alpha of the optical axis for controlling the image acquisition device at the two positions is easier, and only the rotation angle of the rotary driving motor needs to be controlled. Therefore, it is more reasonable to use α to define the optimal position. Therefore, the empirical formula of the invention enables the preparation process to be convenient and fast, and simultaneously improves the arrangement accuracy of the camera position, so that the camera can be arranged in an optimized position, thereby simultaneously considering the 3D synthesis precision and speed.

According to a number of experiments, u should be less than 0.498 in order to ensure the speed and effect of the synthesis, and for better synthesis effect, u <0.411 is preferred, and particularly preferred u <0.359, and in some applications u <0.281, or u <0.169, or u <0.041, or u <0.028.

Experiments were carried out using the apparatus of the invention, and some experimental data are shown below, in mm. (the following data are given by way of example only)

The above data are obtained by experiments for verifying the conditions of the formula, and do not limit the invention. Without these data, the objectivity of the formula is not affected. Those skilled in the art can adjust the parameters of the equipment and the details of the steps as required to perform experiments, and obtain other data which also meet the conditions of the formula.

3D model synthesis method

A plurality of images acquired by the image acquisition device are sent into the processing unit, and a 3D model is constructed by using the following algorithm. The processing unit can be located in the acquisition equipment or remotely, such as a cloud platform, a server, an upper computer and the like.

The specific algorithm mainly comprises the following steps:

step 1: and performing image enhancement processing on all input photos. The following filters are used to enhance the contrast of the original picture and simultaneously suppress noise.

In the formula: g (x, y) is the gray value of the original image at (x, y), f (x, y) is the gray value of the original image at the position after being enhanced by the Wallis filter, and m _g Is the local gray average value, s, of the original image _g Is the local standard deviation of gray scale of the original image, m _f For the transformed image local gray scale target value, s _f The target value of the standard deviation of the local gray scale of the image after transformation. c belongs to (0, 1) as the expansion constant of the image variance, and b belongs to (0, 1) as the image brightness coefficient constant.

The filter can greatly enhance image texture modes of different scales in an image, so that the quantity and the precision of feature points can be improved when the point features of the image are extracted, and the reliability and the precision of a matching result are improved in photo feature matching.

Step 2: and extracting feature points of all input photos, and matching the feature points to obtain sparse feature points. And extracting and matching feature points of the photos by adopting a SURF operator. The SURF feature matching method mainly comprises three processes of feature point detection, feature point description and feature point matching. The method uses a Hessian matrix to detect characteristic points, a Box filter (Box Filters) is used for replacing second-order Gaussian filtering, an integral image is used for accelerating convolution to improve the calculation speed, and the dimension of a local image characteristic descriptor is reduced to accelerate the matching speed. The method mainly comprises the steps of (1) constructing a Hessian matrix, generating all interest points for feature extraction, and aiming at generating stable edge points (catastrophe points) of an image; (2) constructing scale space characteristic point positioning, comparing each pixel point processed by a Hessian matrix with 26 points in a two-dimensional image space and a scale space neighborhood, preliminarily positioning a key point, filtering the key point with weak energy and the key point with wrong positioning, and screening out a final stable characteristic point; (3) and determining the main direction of the characteristic point by adopting the harr wavelet characteristics in the circular neighborhood of the statistical characteristic point. In a circular neighborhood of the feature points, counting the sum of horizontal and vertical harr wavelet features of all points in a sector of 60 degrees, rotating the sector at intervals of 0.2 radian, counting the harr wavelet feature values in the region again, and taking the direction of the sector with the largest value as the main direction of the feature points; (4) and generating a 64-dimensional feature point description vector, and taking a 4-by-4 rectangular region block around the feature point, wherein the direction of the taken rectangular region is along the main direction of the feature point. Each subregion counts haar wavelet features of 25 pixels in both the horizontal and vertical directions, where both the horizontal and vertical directions are relative to the principal direction. The haar wavelet features are in 4 directions of the sum of the horizontal direction value, the vertical direction value, the horizontal direction absolute value and the vertical direction absolute value, and the 4 values are used as feature vectors of each subblock region, so that 4 x 4= 64-dimensional vectors are used as descriptors of Surf features in total; (5) and (3) matching the characteristic points, namely determining the matching degree by calculating the Euclidean distance between the two characteristic points, wherein the shorter the Euclidean distance is, the better the matching degree of the two characteristic points is.

And 3, step 3: inputting matched feature point coordinates, resolving the sparse three-dimensional point cloud of the target object and the position and posture data of the photographing camera by using a light beam method adjustment, namely obtaining model coordinate values of the sparse three-dimensional point cloud of the target object model and the position; and taking the sparse feature points as initial values, performing multi-view photo dense matching, and obtaining dense point cloud data. The process mainly comprises four steps: stereo pair selection, depth map calculation, depth map optimization and depth map fusion. For each image in the input data set, we select a reference image to form a stereo pair for use in computing the depth map. Therefore, we can get rough depth maps of all images, which may contain noise and errors, and we use its neighborhood depth map to perform consistency check to optimize the depth map of each image. And finally, carrying out depth map fusion to obtain the three-dimensional point cloud of the whole scene.

And 4, step 4: and reconstructing the curved surface of the target object by using the dense point cloud. The method comprises the steps of defining an octree, setting a function space, creating a vector field, solving a Poisson equation and extracting an isosurface. And obtaining an integral relation between the sampling point and the indicating function according to the gradient relation, obtaining a vector field of the point cloud according to the integral relation, and calculating approximation of the gradient field of the indicating function to form a Poisson equation. And (3) solving an approximate solution by using matrix iteration according to the Poisson equation, extracting an isosurface by adopting a moving square algorithm, and reconstructing a model of the measured point cloud.

And 5: fully automatic texture mapping of object models. And after the surface model is constructed, texture mapping is carried out. The main process comprises the following steps: (1) acquiring texture data to reconstruct a surface triangular surface grid of the target through an image; (2) and (5) reconstructing visibility analysis of the triangular surface of the model. Calculating a visible image set and an optimal reference image of each triangular surface by using the calibration information of the image; (3) and (5) performing triangular face clustering to generate texture patches. Clustering the triangular surfaces into a plurality of reference image texture patches according to the visible image set of the triangular surfaces, the optimal reference image and the neighborhood topological relation of the triangular surfaces; (4) the texture patches are automatically ordered to generate a texture image. And sequencing the generated texture patches according to the size relationship of the texture patches to generate a texture image with the minimum surrounding area, and obtaining the texture mapping coordinate of each triangular surface.

It should be noted that the above algorithm is an algorithm used by the present invention, and the algorithm is matched with the image acquisition condition, and the time and quality of the synthesis are considered by using the algorithm. It will be appreciated that conventional 3D synthesis algorithms known in the art may equally be used in conjunction with the inventive arrangements.

Examples of applications

In order to construct the inside 3D model of a certain rectangular shape exhibition hall, can hand 3D collection equipment and walk in the room, stop in different positions to through many images of rotatory collection building, remove collection equipment to a plurality of indoor positions and rotate the collection many times again, carry out the synthesis of 3D model according to the synthesis algorithm, thereby construct the 3D model in the room, be convenient for follow-up fitment, show.

The target object, and the object all represent objects for which three-dimensional information is to be acquired. The object may be a solid object or a plurality of object components. The three-dimensional information of the target object comprises a three-dimensional image, a three-dimensional point cloud, a three-dimensional grid, a local three-dimensional feature, a three-dimensional size and all parameters with the three-dimensional feature of the target object. Three-dimensional in the present invention means having XYZ three-direction information, particularly depth information, and is essentially different from only two-dimensional plane information. It is also fundamentally different from some definitions, which are called three-dimensional, panoramic, holographic, three-dimensional, but actually comprise only two-dimensional information, in particular not depth information.

The capture area in the present invention refers to a range in which an image capture device (e.g., a camera) can capture an image. The image acquisition device can be a CCD, a CMOS, a camera, a video camera, an industrial camera, a monitor, a camera, a mobile phone, a tablet, a notebook, a mobile terminal, a wearable device, intelligent glasses, an intelligent watch, an intelligent bracelet and all devices with image acquisition functions.

In the description provided herein, numerous specific details are set forth. It is understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be interpreted as reflecting an intention that: that is, the claimed invention requires more features than are expressly recited in each claim. Rather, as the invention reflects, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.

Those skilled in the art will appreciate that the modules in the device in an embodiment may be adaptively changed and disposed in one or more devices different from the embodiment. The modules or units or components of the embodiments may be combined into one module or unit or component, and furthermore they may be divided into a plurality of sub-modules or sub-units or sub-components. All of the features disclosed in this specification (including any accompanying abstract and drawings), and all of the processes or elements of any method or apparatus so disclosed, may be combined in any combination, except combinations where at least some of such features and/or processes or elements are mutually exclusive. Each feature disclosed in this specification (including the accompanying abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Furthermore, those skilled in the art will appreciate that while some embodiments herein include some features included in other embodiments, rather than other features, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, any of the claimed embodiments may be used in any combination in the present invention.

The various component embodiments of the invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It will be appreciated by those skilled in the art that a microprocessor or Digital Signal Processor (DSP) may be used in practice to implement some or all of the functionality of some or all of the components in an apparatus consistent with embodiments of the invention. The present invention may also be embodied as apparatus or device programs (e.g., computer programs and computer program products) for performing a portion or all of the methods described herein. Such programs implementing the present invention may be stored on computer-readable media or may be in the form of one or more signals. Such a signal may be downloaded from an internet website or provided on a carrier signal or in any other form.

It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the invention. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The usage of the words first, second and third, etcetera do not indicate any ordering. These words may be interpreted as names.

Thus, it should be appreciated by those skilled in the art that while various exemplary embodiments of the invention have been shown and described in detail herein, many other variations or modifications which are consistent with the principles of this invention may be determined or derived directly from the disclosure of the present invention without departing from the spirit and scope of the invention. Accordingly, the scope of the invention should be understood and interpreted to cover all such other variations or modifications.

Claims

1. A multi-position combined 3D acquisition system based on a handheld device, characterized by: comprising a plurality of handheld 3D acquisition devices,

the handheld 3D acquisition equipment comprises an image acquisition device, a rotating device and a handheld part; the acquisition direction of the image acquisition device is a direction departing from the rotation center;

2. The system of claim 1, wherein: the plurality of handheld 3D acquisition devices includes a first type of handheld 3D acquisition device and a second type of handheld 3D acquisition device.

3. The system of claim 2, wherein: the sum of the acquisition ranges of the first type of handheld 3D acquisition equipment can cover the target object, and the sum of the acquisition ranges of the second type of handheld 3D acquisition equipment can cover a specific area of the target object.

4. The system of claim 2, wherein: the plurality of handheld 3D acquisition devices comprise a first type of handheld 3D acquisition device and a second type of handheld 3D acquisition device, and the sum of the acquisition ranges of the first type of handheld 3D acquisition device is larger than that of the second type of handheld 3D acquisition device.

5. The system of claim 3, wherein: and for a specific area of the target object, scanning and acquiring by adopting a first handheld 3D acquisition device and a second handheld 3D acquisition device together.

6. The system of claim 3 or 5, wherein: the specific area is a user-specified area.

7. The system of claim 3 or 5, wherein: the specific region is a previous synthesis failure region.

8. The system of claim 3 or 5, wherein: the specific area is an area with the contour concave-convex variation degree larger than a preset threshold value.

9. The system of claim 1, wherein: u <0.498.

10. The system of claim 1, wherein: u <0.411.

11. The system of claim 1, wherein: u <0.359.

12. The system of claim 1, wherein: u <0.281.

13. The system of claim 1, wherein: u <0.169.

14. The system of claim 1, wherein: u <0.041.

15. The system of claim 1, wherein: u <0.028.

16. A 3D synthesis or recognition apparatus, characterized by: comprising a system according to any of the preceding claims 1-15.

17. An object manufacturing or display apparatus, characterized by: comprising a system according to any of the preceding claims 1-15.

18. A multi-position combined 3D acquisition method based on a handheld device is characterized in that: comprising a plurality of handheld 3D acquisition devices,

the included angle alpha of the optical axes of the image acquisition devices at two adjacent acquisition positions meets the following condition:

19. The method of claim 18, wherein: the plurality of handheld 3D acquisition devices include a first type of handheld 3D acquisition device and a second type of handheld 3D acquisition device.

20. The method of claim 19, wherein: the sum of the acquisition ranges of the first type of handheld 3D acquisition equipment can cover the target object, and the sum of the acquisition ranges of the second type of handheld 3D acquisition equipment can cover a specific area of the target object.

21. The method of claim 18, wherein: the plurality of handheld 3D acquisition devices comprise a first type of handheld 3D acquisition device and a second type of handheld 3D acquisition device, and the sum of the acquisition ranges of the first type of handheld 3D acquisition devices is larger than that of the second type of handheld 3D acquisition devices.

22. The method of claim 20, wherein: and for a specific area of the target object, scanning and acquiring by adopting a first handheld 3D acquisition device and a second handheld 3D acquisition device together.

23. The method of claim 20 or 22, wherein: the specific area is a user-designated area.

24. The method of claim 20 or 22, wherein: the specific region is a previous synthesis failure region.

25. The method of claim 20 or 22, wherein: the specific area is an area with the concave-convex change degree of the outline larger than a preset threshold value.

26. The method of claim 18, wherein: u <0.498.

27. The method of claim 18, wherein: u <0.411.

28. The method of claim 18, wherein: u <0.359.

29. The method of claim 18, wherein: u <0.281.

30. The method of claim 18, wherein: u <0.169.

31. The method of claim 18, wherein: u <0.041.

32. The method of claim 18, wherein: u <0.028.

33. A 3D synthesis or identification method, characterized by: comprising the method of any of the preceding claims 18-32.

34. A method of manufacturing or displaying an object, comprising: comprising the method of any of the preceding claims 18-32.