WO2022078433A1

WO2022078433A1 - Multi-location combined 3d image acquisition system and method

Info

Publication number: WO2022078433A1
Application number: PCT/CN2021/123762
Authority: WO
Inventors: 左忠斌; 左达宇
Original assignee: 左忠斌
Priority date: 2020-10-15
Filing date: 2021-10-14
Publication date: 2022-04-21

Abstract

An embodiment of the present invention provides a multi-location combined 3D image acquisition system and method. The system comprises a 3D image acquisition apparatus. The 3D image acquisition apparatus performs acquisition at multiple locations with respect to an acquisition target. The acquisition range of each acquisition location at least respectively overlaps with the acquisition ranges of the other acquisition locations. The 3D image acquisition apparatus comprises an image acquisition device and a rotation device. The image acquisition device performs acquisition in a direction facing away from the center of rotation. The invention is the first to propose a configuration in which a single autorotating 3D image acquisition apparatus is arranged at multiple locations so as to form a complete multi-location combined 3D image acquisition system. The invention enables image acquisition of a complex surface of an inner space of a target or of a target over a large field of view.

Description

A multi-point combined 3D acquisition system and method

technical field

The invention relates to the technical field of topography measurement, in particular to the technical field of 3D topography measurement.

Background technique

When performing 3D measurement, 3D information needs to be collected first. Commonly used methods include the use of machine vision and structured light, laser ranging, and lidar.

Structured light, laser ranging, and lidar all require an active light source to be emitted to the target, which will affect the target in some cases, and the cost of the light source is high. Moreover, the structure of the light source is relatively precise and easy to be damaged.

The machine vision method is to collect pictures of objects from different angles, and match and stitch these pictures to form a 3D model, which is low-cost and easy to use. When collecting pictures from different angles, multiple cameras can be set at different angles of the object to be tested, or pictures can be collected from different angles by rotating a single or multiple cameras. But no matter which of the two methods, the acquisition position of the camera needs to be set around the target (referred to as the surround type), but this method requires a large space to set the acquisition position for the image acquisition device.

Moreover, in addition to the 3D construction of a single target, there are usually requirements for the construction of 3D models of the internal space of the target and the construction of 3D models within a large surrounding field of view, which is difficult for traditional surround-type 3D acquisition equipment to achieve. . Especially in the inner space or in the large field of view, the surface of the target is more complicated (reflected in the unevenness and deep concavity and convexity of the surface). At this time, it is difficult to cover every part of the surface pits or protrusions by collecting at a single position, resulting in It is difficult to obtain a complete 3D model in the final synthesis, and even the synthesis fails, or the synthesis time is prolonged.

In the prior art, it has also been proposed to use an empirical formula including rotation angle, target size, and object distance to define the camera position, so as to take into account the synthesis speed and effect. In practice, however, this is found to be feasible in wrap-around 3D acquisition, where the target size can be measured in advance. However, in an open space, it is difficult to measure the target in advance. For example, it is necessary to collect and obtain 3D information of streets, traffic intersections, buildings, tunnels, traffic flows, etc. (not limited to this). This makes this approach ineffective. Even small fixed targets, such as furniture, human body parts, etc., can be measured in advance, but this method is still limited: the size of the target is difficult to accurately determine, especially in some applications. Frequent replacements bring a lot of extra work per measurement and require specialized equipment to accurately measure irregular targets. The measurement error leads to the camera position setting error, which will affect the acquisition and synthesis speed and effect; the accuracy and speed need to be further improved.

Although the prior art also has methods for optimizing the wrap-around capturing device, when the capturing direction of the camera of the 3D capturing and synthesizing device deviates from the direction of its rotation axis, there is no better optimization method in the prior art.

Therefore, there is an urgent need for a device that can accurately, efficiently and conveniently collect complex 3D information in the surrounding or internal space.

SUMMARY OF THE INVENTION

In view of the above problems, it is proposed that the present invention provides a multi-position combined 3D acquisition system and method that overcomes the above problems or at least partially solves the above problems.

Embodiments of the present invention provide a multi-point combined 3D acquisition system and method, including a 3D acquisition device,

The 3D collection device performs multi-point collection on the collection target, and the collection range of each collection point at least overlaps with the collection range of other collection points;

The 3D acquisition device includes an image acquisition device and a rotation device; wherein the acquisition direction of the image acquisition device is a direction away from the rotation center.

Optionally, the multi-point acquisition is that a plurality of 3D acquisition devices are respectively set to acquire at a plurality of points.

Optionally, the plurality of 3D acquisition devices include a first type of 3D acquisition device and a second type of 3D acquisition device.

Optionally, the sum of the collection ranges of the first type of 3D collection devices can cover the target, and the sum of the collection ranges of the second type of 3D collection devices can cover a specific area of the target.

Optionally, the multiple 3D acquisition devices include a first type of 3D acquisition device and a second type of 3D acquisition device, and the sum of the acquisition ranges of the first type of 3D acquisition devices is greater than the sum of the acquisition ranges of the second type of 3D acquisition devices.

Optionally, for a specific area of the target object, the first type of 3D acquisition device and the second type of 3D acquisition device are used to jointly scan and acquire.

Optionally, the specific area is a user-specified area.

Optionally, the specific area is an area where the previous synthesis failed.

Optionally, the specific area is an area with a large variation in contour concavo-convex.

Optionally, the 3D acquisition device includes a handheld 3D acquisition device.

Optionally, for the multi-point acquisition, a single 3D acquisition device is sequentially set to acquire at multiple points to complete the acquisition of multiple 3D acquisition locations.

Optionally, in the sequential collection by the 3D collection device, the collection range of the 3D collection device at one collection position overlaps with the collection range at another collection position, and the overlapped area is at least partially located on the target.

Optionally, the multiple 3D collection positions include a first type of 3D collection position and a second type of 3D collection position; the sum of the collection ranges can cover the target at the first type of 3D collection position, and the collection at the second type of 3D collection position The sum of the ranges can cover a specific area of the target.

Optionally, the above-mentioned specific area is a user-specified area.

Optionally, the above-mentioned specific area is the area where the previous synthesis failed.

Optionally, the above-mentioned specific area is an area where the contour unevenness changes greatly.

In an optional embodiment, the included angle α of the optical axes of the image acquisition devices at two adjacent acquisition positions satisfies the following conditions:

Among them, R is the distance from the rotation center to the surface of the target object, T is the sum of the object distance and the image distance during acquisition, d is the length or width of the photosensitive element of the image acquisition device, F is the lens focal length of the image acquisition device, and u is the experience coefficient.

In alternative embodiments, u<0.498, or u<0.41, or u<0.359, or u<0.281, or u<0.169, or u<0.041, or u<0.028.

Another aspect of the embodiments of the present invention further provides a 3D synthesis identification device and method, including the system described in any one of the preceding claims.

Another aspect of the embodiments of the present invention further provides an object manufacturing and display device and method, including the system described in any of the preceding claims.

Inventions and technical effects

1. For the first time, it is proposed to use the self-rotating intelligent visual 3D acquisition device to collect the 3D information of the internal space of the target, which is suitable for wider space and smaller space.

2. For the first time, it is proposed to optimize the camera acquisition position by measuring the distance between the rotation center and the target, and the distance between the image sensing element and the target, so as to take into account the speed and effect of 3D construction.

3. For the first time, it is proposed to set up a single self-rotating 3D acquisition device in multiple positions, so as to form a complete set of multi-position combined 3D acquisition system. Realize the acquisition of the inner space of complex surfaces or a wide range of objects.

4. For the first time, it is proposed to perform multi-position repeated scanning for areas with large changes in concave and convex to ensure the synthesis rate. That is, through the setting of two types of acquisition devices, specific scanning acquisition is performed for a specific area, so as to achieve accurate and efficient acquisition of complex objects.

Description of drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are for the purpose of illustrating preferred embodiments only and are not to be considered limiting of the invention. Also, the same components are denoted by the same reference numerals throughout the drawings. In the attached image:

FIG. 1 shows a schematic structural diagram of a 3D information collection device provided by an embodiment of the present invention;

FIG. 2 shows a schematic structural diagram of a handheld 3D information collection device provided by an embodiment of the present invention;

3 shows another schematic structural diagram of a handheld 3D information collection device provided by an embodiment of the present invention;

4 shows a schematic diagram of a multi-position combined 3D acquisition system provided by an embodiment of the present invention;

5 shows a schematic diagram of a multi-position combined handheld 3D acquisition system provided by an embodiment of the present invention;

6 shows a schematic diagram of the collection of a specific area by a multi-position combined 3D collection system provided by an embodiment of the present invention;

FIG. 7 shows a schematic diagram of a multiple combined 3D acquisition system provided by an embodiment of the present invention;

FIG. 8 shows another schematic diagram of a multiple combined 3D acquisition system provided by an embodiment of the present invention.

The corresponding relationship between the reference numerals in the accompanying drawings and the components is as follows:

1 image acquisition device;

2 rotating device;

3 carrying device.

Detailed ways

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be more thoroughly understood, and will fully convey the scope of the present disclosure to those skilled in the art.

3D information acquisition equipment structure

To solve the above technical problems, an embodiment of the present invention provides a multi-position combined 3D acquisition system, including 3D information acquisition equipment, as shown in FIG.

The image acquisition device 1 is connected with the rotating shaft of the rotating device 2 , and the rotating device 2 drives it to rotate. The acquisition direction of the image acquisition device 1 is the direction away from the rotation center. That is, the acquisition direction is directed outward relative to the center of rotation. The optical axis of the image acquisition device 1 may be parallel to the rotation plane, or may form a certain angle with the rotation plane, for example, within the range of -90°-90° based on the rotation plane. Usually, the rotation axis or its extension line (ie, the rotation center line) passes through the image acquisition device, that is, the image acquisition device still rotates in an autorotation manner. This is essentially different from the acquisition method (surround type) in which the traditional image acquisition device rotates around a certain object, that is, it is completely different from the surround type in which the image acquisition device rotates around the target object. The optical collection ports (eg lenses) of the image collection device are all facing away from the direction of the rotation axis, that is to say, the collection area of the image collection device has no intersection with the rotation center line. At the same time, because the optical axis of the image acquisition device has an included angle with the horizontal plane, this method is also quite different from the general self-rotation method, especially the target object whose surface is not perpendicular to the horizontal plane can be collected.

Of course, the rotating shaft of the rotating device can also be connected to the image capturing device through a deceleration device, for example, through a gear set or the like. When the image capturing device rotates 360° on the horizontal plane, it captures an image corresponding to the target at a specific position (the specific shooting position will be described in detail later). This shooting can be performed in synchronization with the rotation action, or after the shooting position stops rotating, and then continues to rotate after shooting, and so on. The above-mentioned rotating device may be a motor, a motor, a stepping motor, a servo motor, a micro motor, or the like. The rotating device (for example, various types of motors) can rotate at a specified speed under the control of the controller, and can rotate at a specified angle, so as to realize the optimization of the collection position. The specific collection position will be described in detail below. Of course, the rotating device in the existing equipment can also be used, and the image capturing device can be installed thereon.

The carrying device 3 is used to carry the weight of the entire equipment, and the rotating device 2 is connected with the carrying device 3 . The carrying device may be a tripod, a base with a supporting device, or the like. Typically, the rotating device is located in the center part of the carrier to ensure balance. However, in some special occasions, it can also be located at any position of the carrying device. Furthermore, the carrying device is not necessary. The swivel device can be installed directly in the application, eg on the roof of a vehicle.

Structure of handheld 3D information acquisition equipment

The embodiment also provides a handheld 3D information acquisition device, referred to as a 3D acquisition device, please refer to FIG.

The image acquisition device 1 is connected to the rotation device 2, so as to stably rotate and scan under the drive of the rotation device 2, and realize 3D acquisition of surrounding objects (the specific acquisition process will be described in detail below). The rotating device 2 is mounted on the carrying device 3, and the carrying device 3 is used to carry the entire equipment. The carrier 3 can be a handle, so that the entire device can be used for hand-held acquisition. The bearing device 3 can also be a base-type bearing device, which is used to be installed on other devices, so that the entire intelligent 3D acquisition device can be installed on other devices for common use. For example, an intelligent 3D acquisition device is installed on the vehicle and performs 3D acquisition as the vehicle travels.

The carrying device 3 is used to carry the weight of the entire equipment, and the rotating device 2 is connected with the carrying device 3 . The carrying device may be a handle, a tripod, a base with a supporting device, or the like. Typically, the rotating device is located in the center part of the carrier to ensure balance. However, in some special occasions, it can also be located at any position of the carrying device. Furthermore, the carrying device is not necessary. The rotating device can also be installed directly in the application, eg on the roof of a vehicle. The inner space of the carrying device is used to accommodate the battery, which is used to supply power to the 3D rotation acquisition stabilization device. At the same time, for the convenience of use, buttons are arranged on the casing of the carrying device to control the 3D rotation acquisition stabilization device. Including turning on/off the stabilization function and turning on/off the 3D rotation capture function.

As shown in FIG. 3 , the image acquisition device 1 is connected with the rotating shaft of the rotating device 2, and the rotating device drives the rotating device to rotate. Of course, the rotating shaft of the rotating device can also be connected with the image capturing device through a transmission device, for example, through a gear set or the like. At this time, the rotating device 2 can be arranged inside the handle, and part or all of the transmission device is also arranged inside the handle, which can further reduce the volume of the device.

When the image capturing device rotates 360° on the horizontal plane, it captures an image corresponding to the target at a specific position (the specific shooting position will be described in detail later). This shooting can be performed in synchronization with the rotation action, or after the shooting position stops rotating, and then continues to rotate after shooting, and so on. The above-mentioned rotating device may be a motor, a motor, a stepping motor, a servo motor, a micro motor, or the like. The rotating device (for example, various types of motors) can rotate at a specified speed under the control of the controller, and can rotate at a specified angle, so as to realize the optimization of the collection position. The specific collection position will be described in detail below. Of course, the rotating device in the existing equipment can also be used, and the image capturing device can be installed thereon.

A distance measuring device is also included. The distance measuring device is fixedly connected with the image acquisition device, and the pointing direction of the distance measuring device is the same as the optical axis direction of the image acquisition device. Of course, the distance measuring device can also be fixedly connected to the rotating device, as long as it can rotate synchronously with the image capturing device. Preferably, an installation platform may be provided, the image acquisition device and the distance measuring device are both located on the platform, the platform is installed on the rotating shaft of the rotating device, and is driven and rotated by the rotating device. The distance measuring device can use a variety of methods such as a laser distance meter, an ultrasonic distance meter, an electromagnetic wave distance meter, etc., or a traditional mechanical measuring tool distance measuring device. Of course, in some applications, the 3D acquisition device is located at a specific location, and its distance from the target has been calibrated, and no additional measurement is required.

It can also include a light source, and the light source can be arranged on the periphery of the image acquisition device, on the rotating device and on the installation platform. Of course, the light source can also be set independently, for example, an independent light source is used to illuminate the target. Even when lighting conditions are good, no light source is used. The light source can be an LED light source or an intelligent light source, that is, the parameters of the light source are automatically adjusted according to the conditions of the target object and the ambient light. Usually, the light sources are distributed around the lens of the image capture device, for example, the light sources are ring-shaped LED lights around the lens. Because in some applications it is necessary to control the intensity of the light source. In particular, a diffuser device, such as a diffuser housing, can be arranged on the light path of the light source. Or directly use the LED surface light source, not only the light is softer, but also the light is more uniform. More preferably, an OLED light source can be used, which has a smaller volume, softer light, and has flexible properties, which can be attached to a curved surface.

In order to facilitate the measurement of the actual size of the target, multiple marking points can be set at the position of the target. And the coordinates of these marker points are known. By collecting marker points and combining their coordinates, the absolute size of the 3D composite model is obtained. These marking points can be pre-set points or laser light spots. The method for determining the coordinates of these points may include: ①Using laser ranging: using a calibration device to emit laser light toward the target to form a plurality of calibration point spots, and obtain the calibration point coordinates through the known positional relationship of the laser ranging unit in the calibration device. Use the calibration device to emit laser light toward the target, so that the light beam emitted by the laser ranging unit in the calibration device falls on the target to form a light spot. Since the laser beams emitted by the laser ranging units are parallel to each other, and the positional relationship between the units is known. Then the two-dimensional coordinates on the emission plane of the multiple light spots formed on the target can be obtained. By measuring the laser beam emitted by the laser ranging unit, the distance between each laser ranging unit and the corresponding light spot can be obtained, that is, depth information equivalent to multiple light spots formed on the target can be obtained. That is, the depth coordinates perpendicular to the emission plane can be obtained. Thereby, the three-dimensional coordinates of each spot can be obtained. ②Using the combination of distance measurement and angle measurement: respectively measure the distance of multiple markers and the angle between each other, so as to calculate the respective coordinates. ③ Use other coordinate measurement tools: such as RTK, global coordinate positioning system, star-sensing positioning system, position and pose sensors, etc.

Multi-position combined 3D acquisition system

As shown in Figures 4-5, the acquisition system includes a plurality of the above-mentioned 3D information acquisition devices a, b, c..., which are located in different spatial positions. The collection range of the collection device a includes the area A, the collection range of the collection device b includes the area B, the collection range of the collection device c includes the area C... and so on. Their collection areas at least satisfy that the intersection between the two collection areas is not empty. In particular, the non-empty intersection should be located on the target. That is, each acquisition device overlaps at least the acquisition range of the other two acquisition devices, especially the acquisition range of each acquisition device on the target is at least the same as the acquisition range of the other two acquisition devices on the target. overlapping.

Whether it is an interior space or a target in a wide field of view, they may have areas with more complex surfaces, called specific areas. These areas either have inwardly recessed deep holes/pits, or outwardly projecting higher protrusions, or both, resulting in a greater degree of surface unevenness. This presents challenges for acquisition devices that acquire in one direction. Due to the concave and convex reasons, no matter where the device is set, the specific area of the target object can only be collected from a single direction by rotating and scanning, resulting in a large amount of information loss in the specific area.

Therefore, it is possible to set the collecting devices in multiple locations to scan and collect the specific area, so that the information of the area can be obtained from different angles. For example, the intersection of area A and area B includes this specific area; the common intersection of area A, area B, and area C includes this specific area; the intersection of area A and area B and the intersection of area C and area D include this specific area, etc. Wait. That is to say, the specific area is scanned repeatedly, which may also be called a repeated scanning area, that is, the specific area is scanned and collected by multiple collection devices. The above situation includes the intersection of the collection areas of two or more collection devices including the specific area; the intersection of the collection areas of two or more collection devices, and the other two or more collection devices The intersection of the collection areas, including the specific area.

The above-mentioned specific areas can be analyzed and obtained according to the previous 3D synthesis situation, such as the areas where the previous 3D synthesis failed or with a high failure rate; it can also be delineated in advance according to the operator's experience, such as the areas with large fluctuations in bumps, or the degree of bumps. A larger area, etc., that is, an area with a variation of bumps and ridges, or a region with a degree of variation of the bumps and volts greater than a preset threshold.

In another embodiment, multiple handheld 3D information collection devices are not required, but only one handheld 3D information collection device is used. The user holds the device and performs multiple rotation scans at different positions to obtain a picture of the target object. At this time, it should be ensured that the scanning range of the handheld 3D information acquisition device can cover the entire target area when it is located at each location.

3D information collection process

1. Select the number of the first type of 3D information collection equipment according to the size and position of the target, and arrange the position for each 3D information collection equipment.

(1) According to the target collection requirements, set the position where the 3D information collection equipment can be placed, and determine the distance between the 3D information collection equipment and the target.

(2) Select the number of 3D information collection devices according to the size of the target, the above distance and the collection ranges A, B, C... of multiple 3D information collection devices a, b, c..., so that the sum of the collection ranges of the 3D information collection devices can be Cover the target. However, under normal circumstances, not only is the sum of the collection ranges of the 3D information collection devices required to cover the size of the target, but also when the collection ranges of adjacent 3D information collection devices overlap, the sum of their collection ranges can still cover the target. object size. For example, the overlapping range is more than 10% of the acquisition range.

(3) Arrange the selected multiple 3D information collection devices a, b, c... relatively evenly at the above-mentioned distance from the target, so as to ensure that the collection areas of the multiple 3D information collection devices a, b, c... can cover the target thing.

2. Set the number of the second type of 3D information collection equipment according to the size, quantity and position of the specific area of the target, and arrange the position for each 3D information collection equipment.

(1) Determine the number and location of the specific area of the target. The way of determination includes based on pre-existing data, or based on visual results, or based on the distribution of areas that were not synthesized in the previous acquisition.

(2) According to the size of the specific area of the target, one or more 3D information acquisition devices of the second type are arranged for each specific area, so that their acquisition range can cover the specific area.

(3) Determine the number of the second type of 3D information acquisition equipment according to the number and location of the specific area of the target and the number of the second type of 3D information acquisition equipment required for each specific area, and arrange the location for each 3D information acquisition equipment . As shown in Figure 6, under normal circumstances, one or more second-type 3D information collection devices are inserted between the above-mentioned first-type 3D information collection devices, so as to form an area where the collection range of the first-type 3D information collection equipment is weak. Repeated acquisition, that is, repeated acquisition of a specific area to form a repeated scanning area. The second type of 3D information acquisition device can also be set at other positions (for example, closer to or farther from the target) to ensure that enough different angle pictures can be obtained in the repeated scanning area.

3. After the first and second types of 3D information acquisition devices are all arranged, start to control each 3D information acquisition device to rotate and scan the target object, and the rotation satisfies the optimization conditions for the image acquisition device of the 3D information acquisition device. That is, the controller can control the rotation of the image acquisition device of each 3D information acquisition device according to the above conditions.

4. Send the pictures scanned and collected by the multiple 3D information collecting devices to the processor, and the processor uses the multiple pictures to synthesize the 3D model of the target object. Similarly, the above-mentioned multiple pictures can also be sent to a remote platform, cloud platform, server, host computer and/or mobile terminal through a communication device, and a 3D model synthesis method is used to perform 3D synthesis of the target object.

In another embodiment, in addition to the above-described combined acquisition using multiple 3D acquisition devices, it can be understood that one 3D acquisition device or a limited number of 3D acquisition devices can be used to perform acquisition at the above set positions in sequence and time-sharing. . That is to say, the acquisition is not performed at the same time, but is acquired at different locations in a time-sharing manner, and images acquired at different times are collected for 3D synthesis. The different locations described here are the same as the locations described above for the different acquisition devices.

In another embodiment, it is not necessary to set up multiple collection devices, but replace the multiple collection devices in the above process with one, and the collection positions of each collection device are the positions where the user holds the device for collection in sequence. That is, the user holds the device and is located at the collection positions of the above-mentioned first and second types of 3D information collection equipment in sequence. Every time it is in the collection position (the first type of collection position or the second type of collection position), the handheld collection device is controlled to rotate for collection. Finally, the images collected each time are transmitted to the processor for 3D modeling and synthesis.

Multiple combined 3D acquisition system

As shown in Figures 7-8, the collection system includes one or more of the above-mentioned 3D information collection devices, and the collection devices are located at positions a, b, c... in sequence during the collection process. The collection range when the collection device is at position a includes area A, the collection range when the collection device is at position b includes area B, and the collection range when the collection device is at position c includes area C... and so on. Their collection areas at least satisfy that the intersection between the two collection areas is not empty. In particular, the non-empty intersection should be located on the target. That is, each acquisition device overlaps at least the acquisition range of the other two acquisition devices, especially the acquisition range of each acquisition device on the target is at least the same as the acquisition range of the other two acquisition devices on the target. overlapping.

Whether it is an interior space or an object in a wide field of view, they may have areas with more complex surfaces, called specific areas. These areas either have inwardly recessed deep holes/pits, or outwardly projecting higher protrusions, or both, resulting in a greater degree of surface unevenness. This presents challenges for acquisition devices that acquire in one direction. Due to the concave and convex reasons, no matter where the device is set, the specific area of the target object can only be collected from a single direction by rotating and scanning, resulting in a large amount of information loss in the specific area.

The above-mentioned specific areas can be analyzed and obtained according to the previous 3D synthesis situation, such as the areas where the previous 3D synthesis failed or with a high failure rate; it can also be delineated in advance according to the operator's experience, such as the areas with large fluctuations in bumps, or the degree of bumps. larger areas, etc.

3D information collection process

1. Select the first type of 3D information collection location and quantity according to the size and location of the target.

(2) According to the size of the target, the above distance and the collection range of multiple 3D information collection devices, select different collection positions a, b, c... The sum of C… can cover the target. But in general, not only is the sum of the collection ranges of the 3D information collection devices required to cover the size of the target, but also when the collection ranges of the 3D information collection devices overlap in adjacent positions, the sum of their collection ranges can still be Override target size. For example, the overlapping range accounts for more than 10% of the acquisition range.

(3) The selected 3D information collection equipment is placed in the collection positions a, b, c... in order to ensure that the 3D information collection equipment can cover the target in the collection area of the positions a, b, c....

(4) According to the above principles, place the 3D information acquisition device on each type of acquisition position in turn, and rotate and scan the target for acquisition. The rotation satisfies the optimization conditions for the image acquisition device of the 3D information acquisition device. That is to say, the controller can control the rotation of the image acquisition device of the 3D information acquisition device each time according to the above conditions.

2. Set the second type of 3D information collection position and quantity according to the size, quantity and position of the specific area of the target. As shown in Figure 4:

(2) According to the size of the specific area of the target, arrange one or more second-type 3D information collection positions for each specific area, and place the acquisition equipment in turn for re-rotation acquisition, so that their acquisition range can cover the specific area.

(3) According to the number and position of the specific area of the target and the number of the second type of 3D information collection positions required by each specific area, determine the number of times of the second type of 3D information collection, and arrange the position for each 3D information collection device. Usually, one or more second-type 3D information collection locations are inserted between the above-mentioned first-type 3D information collection locations, so as to form repeated collection of areas where the collection range of the first-type 3D information collection locations is weak. The area is repeatedly collected to form a repeated scanning area. The second type of 3D information collection position can also be set at other positions (for example, closer to or farther from the target) to ensure that enough different angle pictures can be obtained in the repeated scanning area.

(4) According to the above principles, place the 3D information collection device in each second-class collection position in turn, and rotate and scan the target for collection. The rotation satisfies the optimization conditions for the image acquisition device of the 3D information acquisition device. That is to say, the controller can control the rotation of the image acquisition device of the 3D information acquisition device each time according to the above conditions.

3. Send the pictures scanned and collected by multiple 3D information collection devices to the processor, and the processor uses the above multiple pictures to synthesize and model the 3D model of the target object. Similarly, the above-mentioned multiple pictures can also be sent to a remote platform, cloud platform, server, host computer and/or mobile terminal through a communication device, and a 3D model synthesis method is used to perform 3D synthesis of the target object.

The above acquisition can be completed by using a handheld device, so that the user can walk to different positions with the handheld device to perform multiple acquisitions, thereby constructing a 3D model of a larger space. For example, for a corridor, it is also possible to set up multiple devices, but the process is more complicated. However, if the user holds the device and moves to different positions of the corridor, different areas of the corridor can be collected separately, so as to finally synthesize the 3D model of the whole corridor. Of course, in the process of walking, for areas with complex corridor structures, you can stay at several more positions to ensure that these areas are repeatedly collected, that is, multiple second-type collection positions are formed.

Optimization of camera position

In order to ensure that the device can take into account the effect and efficiency of 3D synthesis, in addition to the conventional method of optimizing the synthesis algorithm, the method of optimizing the camera acquisition position can also be adopted. Especially when the acquisition direction of the camera of the 3D acquisition and synthesis device deviates from the direction of its rotation axis, the prior art for such a device does not mention how to better optimize the camera position. Even if some optimization methods exist, they are obtained under different empirical conditions under different experiments. In particular, some existing position optimization methods need to obtain the size of the target object, which is feasible in surround 3D acquisition and can be measured in advance. However, it is difficult to measure in advance in an open space. Therefore, it is necessary to propose a method for optimizing the camera position when the acquisition direction of the camera of the 3D acquisition and synthesis device and the direction of its rotation axis deviate from each other. This is exactly the problem to be solved by the present invention and the technical contribution made.

To this end, the present invention conducts a large number of experiments, and summarizes the following empirical conditions that the interval of camera acquisition is preferably satisfied during acquisition.

When performing 3D acquisition, the included angle α of the optical axis of the image acquisition device at two adjacent positions satisfies the following conditions:

in,

R is the distance from the center of rotation to the surface of the target,

T is the sum of the object distance and the image distance during acquisition, that is, the distance between the photosensitive unit of the image acquisition device and the target object.

d is the length or width of the photosensitive element (CCD) of the image acquisition device. When the above two positions are along the length direction of the photosensitive element, d is the length of the rectangle; when the above two positions are along the width direction of the photosensitive element, d is the rectangle. width.

F is the focal length of the lens of the image acquisition device.

u is the empirical coefficient.

Usually, a distance measuring device, such as a laser distance meter, is configured on the acquisition device. Adjust its optical axis to be parallel to the optical axis of the image acquisition device, then it can measure the distance from the acquisition device to the surface of the target object. Using the measured distance, according to the known positional relationship between the distance measuring device and the various components of the acquisition device, you can Get R and T.

When the image acquisition device is in any one of the two positions, the distance from the photosensitive element to the surface of the target object along the optical axis is taken as T. In addition to this method, multiple averaging methods or other methods can also be used. The principle is that the value of T should not deviate from the distance between the image and the object during acquisition.

In the same way, when the image acquisition device is in any one of the two positions, the distance from the center of rotation to the surface of the target object along the optical axis is taken as R. In addition to this method, multiple averaging methods or other methods can also be used, the principle of which is that the value of R should not deviate from the radius of rotation at the time of acquisition.

Usually, the size of the object is used as a method for estimating the position of the camera in the prior art. Because the size of the object will change with the change of the measured object. For example, after collecting 3D information of a large object, when collecting small objects, it is necessary to re-measure the size and re-calculate. The above-mentioned inconvenient measurements and multiple re-measurements will bring about measurement errors, resulting in incorrect camera position estimation. In this scheme, based on a large number of experimental data, the empirical conditions that the camera position needs to meet are given, and there is no need to directly measure the size of the object. In the empirical conditions, d and F are fixed parameters of the camera. When purchasing a camera and lens, the manufacturer will give the corresponding parameters without measurement. However, R and T are only a straight line distance, which can be easily measured by traditional measurement methods, such as straightedge and laser rangefinder. At the same time, in the apparatus of the present invention, the acquisition direction of the image acquisition device (eg, camera) is away from the direction of its rotation axis, that is, the orientation of the lens is substantially opposite to the rotation center. At this time, it is easier to control the included angle α of the optical axis between the two positions of the image acquisition device, and it is only necessary to control the rotation angle of the rotary drive motor. Therefore, it is more reasonable to use α to define the optimal position. Therefore, the empirical formula of the present invention makes the preparation process convenient and quick, and at the same time improves the arrangement accuracy of the camera position, so that the camera can be set in an optimized position, thereby taking into account the 3D synthesis accuracy and speed at the same time.

According to a large number of experiments, in order to ensure the speed and effect of synthesis, u should be less than 0.498. For better synthesis effect, u<0.411 is preferred, especially u<0.359. In some applications, u<0.281, or u<0.169, or u<0.041, or u<0.028.

Using the device of the present invention, experiments are carried out, and some experimental data are as follows, in mm. (The following data are only limited examples)

The above data are only obtained from experiments to verify the conditions of the formula, and do not limit the invention. Even the absence of these data does not affect the objectivity of the formula. Those skilled in the art can adjust the parameters of the equipment and the details of the steps to conduct experiments, and other data obtained are also in line with the conditions of the formula.

3D model synthesis method

The multiple images acquired by the image acquisition device are sent to the processing unit, and the following algorithm is used to construct a 3D model. The processing unit may be located in the acquisition device, or may be located remotely, such as a cloud platform, a server, a host computer, and the like.

The specific algorithm mainly includes the following steps:

Step 1: Perform image enhancement processing on all input photos. The following filters are used to enhance the contrast of the original photo and suppress noise at the same time.

In the formula: g(x, y) is the gray value of the original image at (x, y), f(x, y) is the gray value of the original image after enhancement by Wallis filter, and m _g is the local gray value of the original image. sg is the local grayscale standard deviation of the original image, _mf is the local _grayscale target value of the transformed image, and _sf is the localized grayscale standard deviation target value of the transformed image. c∈(0,1) is the expansion constant of the image variance, and b∈(0,1) is the image luminance coefficient constant.

The filter can greatly enhance the image texture patterns of different scales in the image, so it can improve the number and accuracy of feature points when extracting image point features, and improve the reliability and accuracy of matching results in photo feature matching.

Step 2: Extract feature points from all the input photos, and perform feature point matching to obtain sparse feature points. The SURF operator is used to extract and match the feature points of the photo. The SURF feature matching method mainly includes three processes, feature point detection, feature point description and feature point matching. This method uses the Hessian matrix to detect feature points, uses Box Filters to replace second-order Gaussian filtering, uses integral images to accelerate convolution to improve computational speed, and reduces the dimension of local image feature descriptors. to speed up matching. The main steps include ① constructing a Hessian matrix to generate all interest points for feature extraction. The purpose of constructing a Hessian matrix is to generate image stable edge points (mutation points); ② constructing the scale space feature point positioning, which will be processed by the Hessian matrix Each pixel point is compared with 26 points in the two-dimensional image space and scale space neighborhood, and the key points are initially located. (3) The main direction of the feature point is determined by using the harr wavelet feature in the circular neighborhood of the statistical feature point. That is, in the circular neighborhood of the feature points, the sum of the horizontal and vertical harr wavelet features of all points in the 60-degree sector is counted, and then the sector is rotated at intervals of 0.2 radians and the harr wavelet eigenvalues in the region are counted again. The direction of the sector with the largest value is used as the main direction of the feature point; (4) a 64-dimensional feature point description vector is generated, and a 4*4 rectangular area block is taken around the feature point, but the direction of the obtained rectangular area is along the main direction of the feature point. direction. Each sub-region counts the haar wavelet features of 25 pixels in the horizontal and vertical directions, where the horizontal and vertical directions are relative to the main direction. The haar wavelet features are 4 directions after the horizontal value, after the vertical value, after the absolute value of the horizontal direction and the sum of the absolute value of the vertical direction. These 4 values are used as the feature vector of each sub-block area, so there are 4* 4*4=64-dimensional vector as the descriptor of the Surf feature; ⑤ Feature point matching, by calculating the Euclidean distance between the two feature points to determine the matching degree, the shorter the Euclidean distance, the better the matching degree of the two feature points. .

Step 3: Input the coordinates of the matched feature points, and use the beam method to adjust the position and attitude data of the sparse target object 3D point cloud and the camera to obtain the sparse target object model 3D point cloud and position model coordinates. ; Take sparse feature points as the initial value, perform dense matching of multi-view photos, and obtain dense point cloud data. There are four main steps in this process: stereo pair selection, depth map calculation, depth map optimization, and depth map fusion. For each image in the input dataset, we select a reference image to form a stereo pair for computing the depth map. So we can get a rough depth map for all images, these depth maps may contain noise and errors, and we use its neighborhood depth map to perform a consistency check to optimize the depth map for each image. Finally, depth map fusion is performed to obtain a 3D point cloud of the entire scene.

Step 4: Use dense point cloud to reconstruct the target surface. Including several processes of defining octrees, setting function spaces, creating vector fields, solving Poisson equations, and extracting isosurfaces. The integral relationship between the sampling point and the indicator function is obtained from the gradient relationship, the vector field of the point cloud is obtained according to the integral relationship, and the approximation of the gradient field of the indicator function is calculated to form the Poisson equation. According to the Poisson equation, the approximate solution is obtained by matrix iteration, the isosurface is extracted by the moving cube algorithm, and the model of the measured object is reconstructed from the measured point cloud.

Step 5: Fully automatic texture mapping of the target model. After the surface model is constructed, texture mapping is performed. The main process includes: ① texture data acquisition through image reconstruction of the target surface triangle mesh; ② visibility analysis of the reconstructed model triangle. Use the calibration information of the image to calculate the visible image set of each triangular face and the optimal reference image; 3. The triangular face is clustered to generate texture patches. According to the visible image set of the triangular surface, the optimal reference image and the neighborhood topology relationship of the triangular surface, the triangular surface is clustered into several reference image texture patches; ④The texture patches are automatically sorted to generate texture images. Sort the generated texture patches according to their size relationship, generate a texture image with the smallest enclosing area, and obtain the texture mapping coordinates of each triangular surface.

It should be noted that the above algorithm is the algorithm used in the present invention, the algorithm cooperates with the image acquisition conditions, and the time and quality of synthesis are taken into account when using this algorithm. However, it can be understood that conventional 3D synthesis algorithms in the prior art can also be used in conjunction with the solution of the present invention.

Applications

In order to construct a 3D model inside an exhibition hall, a 3D acquisition device can be placed on the floor of the house, multiple images of the building can be acquired by rotating, and then the acquisition device can be moved to multiple indoor positions for multiple rotation acquisitions, and the 3D model can be synthesized according to the synthesis algorithm. , so as to build a 3D model of the house, which is convenient for subsequent decoration and display.

The above-mentioned target object, target object, and object all represent objects for which three-dimensional information is pre-acquired. It can be a solid object, or it can be composed of multiple objects. . The three-dimensional information of the target includes a three-dimensional image, a three-dimensional point cloud, a three-dimensional grid, a local three-dimensional feature, a three-dimensional size, and all parameters with the three-dimensional feature of the target. The so-called three-dimensional in the present invention refers to having three directional information of XYZ, especially having depth information, which is essentially different from having only two-dimensional plane information. It is also fundamentally different from some definitions that are called three-dimensional, panoramic, holographic, and three-dimensional, but actually only include two-dimensional information, especially not depth information.

The acquisition area mentioned in the present invention refers to the range that the image acquisition device 1 (eg, camera) can capture. The image acquisition device 1 in the present invention can be CCD, CMOS, camera, video camera, industrial camera, monitor, camera, mobile phone, tablet, notebook, mobile terminal, wearable device, smart glasses, smart watch, smart bracelet and belt All devices with image acquisition function.

In the description provided herein, numerous specific details are set forth. It will be understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

Similarly, it is to be understood that in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together into a single embodiment, figure, or its description. This disclosure, however, should not be construed as reflecting an intention that the invention as claimed requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the Detailed Description are hereby expressly incorporated into this Detailed Description, with each claim standing on its own as a separate embodiment of this invention.

Those skilled in the art will understand that the modules in the device in the embodiment can be adaptively changed and arranged in one or more devices different from the embodiment. The modules or units or components in the embodiments may be combined into one module or unit or component, and further they may be divided into multiple sub-modules or sub-units or sub-assemblies. All features disclosed in this specification (including accompanying claims, abstract and drawings) and any method so disclosed may be employed in any combination unless at least some of such features and/or procedures or elements are mutually exclusive. All processes or units of equipment are combined. Each feature disclosed in this specification (including accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Furthermore, it will be understood by those skilled in the art that although some of the embodiments herein include certain features, but not others, included in other embodiments, that combinations of features of the different embodiments are intended to be within the scope of the present invention And form different embodiments. For example, in the claims, any of the claimed embodiments may be used in any combination.

Various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art should understand that a microprocessor or a digital signal processor (DSP) may be used in practice to implement some or all functions of some or all of the components in the device according to the present invention according to the embodiments of the present invention. The present invention can also be implemented as apparatus or apparatus programs (eg, computer programs and computer program products) for performing part or all of the methods described herein. Such a program implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such signals may be downloaded from Internet sites, or provided on carrier signals, or in any other form.

It should be noted that the above-described embodiments illustrate rather than limit the invention, and that alternative embodiments may be devised by those skilled in the art without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the words first, second, and third, etc. do not denote any order. These words can be interpreted as names.

By now, those skilled in the art will recognize that, although various exemplary embodiments of the present invention have been illustrated and described in detail herein, the present invention may still be implemented in accordance with the present disclosure without departing from the spirit and scope of the present invention. The content directly determines or derives many other variations or modifications consistent with the principles of the invention. Accordingly, the scope of the present invention should be understood and deemed to cover all such other variations or modifications.

Claims

A multi-point combined 3D acquisition system and method, characterized in that it comprises a 3D acquisition device,

The 3D collection device performs multi-point collection on the collection target, and the collection range of each collection point at least overlaps with the collection range of other collection points;

The 3D acquisition device includes an image acquisition device and a rotation device; wherein the acquisition direction of the image acquisition device is a direction away from the rotation center.
The system according to claim 1, wherein the multi-point acquisition is that a plurality of 3D acquisition devices are respectively set at a plurality of points for acquisition.
The system of claim 2, wherein the plurality of 3D acquisition devices include a first type of 3D acquisition device and a second type of 3D acquisition device.
The system according to claim 3, wherein the sum of the collection ranges of the first type of 3D collection devices can cover the target, and the sum of the collection ranges of the second type of 3D collection devices can cover a specific area of the target.
The system of claim 3, wherein the plurality of 3D acquisition devices include a first type of 3D acquisition device and a second type of 3D acquisition device, and the sum of the acquisition ranges of the first type of 3D acquisition device is greater than the second type of 3D acquisition device The sum of the collection range of the collection device.
The system according to claim 3, characterized in that: for a specific area of the target object, the first type of 3D acquisition device and the second type of 3D acquisition device are used to scan and acquire together.
The system of claim 6, wherein the specific area is a user-designated area.
The system of claim 6, wherein the specific area is an area where the previous synthesis failed.
The system according to claim 6, wherein the specific area is an area with a large variation in contour concavo-convex.
The system of claim 1, wherein the 3D acquisition device comprises a handheld 3D acquisition device.
The system according to claim 1, wherein the multi-point acquisition is that a single 3D acquisition device is sequentially arranged at multiple points for acquisition, and the acquisition of multiple 3D acquisition positions is completed.
11. The system of claim 11, wherein: in the sequential acquisition by the 3D acquisition device, the acquisition range of the 3D acquisition device at one acquisition location overlaps with the acquisition range of another acquisition location, and the overlapped area is at least partially located at the target on things.
The system according to claim 11, wherein: the plurality of 3D collection positions include a first type of 3D collection position and a second type of 3D collection position; when the first type of 3D collection position is collected, the sum of the collection ranges can cover the target object, In the second type of 3D acquisition location, the sum of the acquisition ranges can cover a specific area of the target.
The system of claim 13, wherein the specific area is a user-specified area.
The system of claim 13, wherein the specific area is an area where the previous synthesis failed.
The system according to claim 13, wherein the specific area is an area with a large variation in contour concavo-convex.
The system according to claim 1, wherein the included angle α of the optical axes of the image acquisition devices at two adjacent acquisition positions satisfies the following conditions:

Among them, R is the distance from the rotation center to the surface of the target object, T is the sum of the object distance and the image distance during acquisition, d is the length or width of the photosensitive element of the image acquisition device, F is the lens focal length of the image acquisition device, and u is the experience coefficient.
The system of claim 9, wherein: u<0.498, or u<0.41, or u<0.359, or u<0.281, or u<0.169, or u<0.041, or u<0.028.
A 3D synthesis or identification device, comprising the system of any one of claims 1-18.
A 3D synthesis or identification method, comprising the system of any one of claims 1-18.
An object manufacturing or display apparatus comprising the system of any of claims 1-18.
A method of making or displaying an object, comprising the system of any one of claims 1-18.