CN112304250B

CN112304250B - Three-dimensional matching equipment and method between moving objects

Info

Publication number: CN112304250B
Application number: CN202011105318.8A
Authority: CN
Inventors: 左忠斌; 左达宇
Original assignee: Tianmu Aishi Beijing Technology Co Ltd
Current assignee: Tianmu Aishi Beijing Technology Co Ltd
Priority date: 2020-10-15
Filing date: 2020-10-15
Publication date: 2022-09-16
Anticipated expiration: 2040-10-15
Also published as: CN112304250A

Abstract

The embodiment of the invention provides a device and a method for realizing 3D matching between moving objects, which comprises a target object 3D information acquisition device and a first processor; the target object 3D information acquisition device is used for continuously scanning an object to be matched when the object to be matched moves, and acquiring a plurality of images capable of synthesizing a three-dimensional model at a plurality of moments; and the first processor is used for matching the three-dimensional models of at least two objects to be matched at the same time. The visual three-dimensional acquisition device with a simple structure is provided for the first time, the three-dimensional model construction of the object is realized, so that the mutual matching between the objects can be realized, and the acquisition speed and the accuracy are high.

Description

Three-dimensional matching equipment and method between moving objects

Technical Field

The invention relates to the technical field of topography measurement, in particular to the technical field of 3D topography measurement.

Background

At present, matching of objects is needed in many scenes, for example, rescue at sea by using a helicopter requires mutual alignment between the helicopter and a small ship on the water surface; or two trucks need to relay to carry goods and the like. I.e. two objects which are moving differently at the same time, are moving relative to each other, so that a clear image of both is needed for further manipulation if alignment, docking, insertion, etc. is to be performed. In the prior art, the two-dimensional images are adopted, which brings large errors. Of course, in the prior art, a combination of a two-dimensional image and a range finder is used, but the problems of complex equipment, high cost and difficult operation are caused. Therefore, there is a need for a device and a method for performing three-dimensional acquisition with a simple structure so as to achieve fast and accurate matching between objects.

In addition, in 3D information acquisition, methods commonly used at present include a machine vision method and a structured light, laser ranging, and laser radar method. Structured light, laser ranging and laser radar all need an active light source to emit to a target object, and can affect the target object under certain conditions, and the light source cost is high. And the light source structure is more accurate, easily damages. The machine vision mode is to collect the pictures of the object at different angles and match and splice the pictures to form a 3D model, so that the cost is low and the use is easy. When the device collects pictures at different angles, a plurality of cameras can be arranged at different angles of an object to be detected, and the pictures can be collected from different angles through rotation of a single camera or a plurality of cameras. However, in either of these two methods, the capturing position of the camera needs to be set around the target (referred to as a wraparound method), but this method needs a large space for setting the capturing position for the image capturing device. The applicable scenarios are greatly limited.

In the prior art, it has also been proposed to use empirical formulas including rotation angle, object size, object distance to define camera position, thereby taking into account the speed and effect of the synthesis. However, in practice this has been found to be feasible in wrap-around 3D acquisition, where the target size can be measured in advance. However, it is difficult to measure the target object in advance in an open space, and it is necessary to acquire 3D information of streets, traffic intersections, building groups, tunnels, traffic flows, and the like (not limited to this). Which makes this approach difficult to work. Even if the dimensions of fixed, small objects, such as furniture, human body parts, etc., can be measured beforehand, this method is still subject to major limitations: the size of the target is difficult to accurately determine, and particularly, the target needs to be frequently replaced in certain application occasions, each measurement brings a large amount of extra workload, and professional equipment is needed to accurately measure irregular targets. The measured error causes the camera position setting error, thereby influencing the acquisition and synthesis speed and effect; accuracy and speed need to be further improved.

Although there are methods for optimizing the surround-type acquisition device in the prior art, there is no better optimization method in the prior art when the acquisition direction of the camera of the 3D acquisition and synthesis device and the direction of its rotation axis deviate from each other.

Therefore, a technology for collecting space 3D information accurately, efficiently and widely applicable to a scene, and performing matching optimization on the space 3D information and the object 3D information so as to guide a matching action between the object and the space is urgently needed.

Disclosure of Invention

In view of the above, the present invention has been made to provide an apparatus and method for achieving 3D matching between moving objects that overcome or at least partially solve the above-mentioned problems.

The embodiment of the invention provides a device and a method for realizing 3D matching between moving objects, which comprises a target object 3D information acquisition device and a first processor;

the target object 3D information acquisition device is used for continuously scanning an object to be matched when the object to be matched moves, and acquiring a plurality of images capable of synthesizing a three-dimensional model at a plurality of moments;

and the first processor is used for matching the three-dimensional models of at least two objects to be matched at the same time.

In an alternative embodiment, the first processor is further configured to synthesize a three-dimensional model of the object to be matched.

In an optional embodiment, the system further comprises a second processor, located in the target object 3D information acquisition device, for synthesizing a three-dimensional model of the object to be matched opposite thereto;

in an alternative embodiment, the 3D information acquisition device of the target object timestamps the images when acquiring the above-mentioned images of the object to be matched at any time.

In an alternative embodiment, the spatial 3D information acquisition device comprises an image acquisition device, a rotation device;

the image acquisition device is connected with the rotating device and is driven to rotate by the rotating device;

the included angle alpha of the optical axes of the image acquisition devices at two adjacent acquisition positions meets the following condition:

wherein, R is the distance from the rotation center to the surface of the target object, T is the sum of the object distance and the image distance during acquisition, d is the length or the width of a photosensitive element of the image acquisition device, F is the focal length of a lens of the image acquisition device, and u is an empirical coefficient.

In an alternative embodiment u <0.498, for better synthesis u <0.411 is preferred, especially u <0.359, in some applications u <0.281, or u <0.169, or u <0.041, or u < 0.028.

In an alternative embodiment, the optical collection ports of the image collection device face away from the rotation axis.

In an optional embodiment, the target object 3D information acquisition device is disposed on the object to be matched.

In an optional embodiment, the target 3D information acquisition device is arranged separately from the object to be matched.

In an alternative embodiment, after the matching is completed, the processor outputs the matching result to the display device, the printing device, and/or the action execution device.

Invention and technical effects

1. The visual three-dimensional acquisition device with a simple structure is provided for the first time, the three-dimensional model construction of the object is realized, so that the mutual matching between the objects can be realized, and the acquisition speed and the accuracy are high.

2. The method has the advantages that the acquisition position of the camera is optimized by measuring the distance between the rotation center and the target object and the distance between the image sensing element and the target object, so that the speed and the effect of 3D construction are considered.

3. Through rotation formula collection equipment for can carry out quick scanning to the object, and not restricted by the occasion, make the matching between the object realize more easily.

4. The position of the acquisition equipment (arranged outside or on the target) is flexibly set, so that the system is suitable for different occasions.

Drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:

fig. 1 is a schematic structural diagram illustrating an implementation manner of a 3D information acquisition apparatus according to an embodiment of the present invention;

fig. 2 is a schematic structural diagram illustrating another implementation manner of a 3D information acquisition apparatus according to an embodiment of the present invention.

Fig. 3 shows a schematic structural diagram of a third implementation manner of a 3D information acquisition apparatus according to an embodiment of the present invention.

Fig. 4 shows a schematic structural diagram of a fourth implementation manner of the 3D information acquisition apparatus according to the embodiment of the present invention.

Fig. 5 is a schematic diagram illustrating the acquisition matching of the 3D information acquisition device located outside the target object according to the embodiment of the present invention.

Fig. 6 shows another schematic diagram of the 3D information acquisition device provided by the embodiment of the invention, which is located outside the target object to acquire matching.

Fig. 7 is a schematic diagram illustrating that a 3D information acquisition device provided by an embodiment of the present invention is located on a target object to acquire a match.

The correspondence of reference numerals to the various components in the drawings is as follows:

1, an image acquisition device;

2, a rotating device;

3, carrying device;

4, a telescopic device;

5 a pitching device;

63D information acquisition device.

Detailed Description

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

3D information acquisition device structure

In order to solve the above technical problem, the present invention provides a 3D information collecting device, as shown in fig. 1, including an image collecting device 1, a rotating device 2, and a carrying device 3.

The image acquisition device 1 is connected with a rotating shaft of the rotating device 2 and is driven to rotate by the rotating device 2. The collection direction of the image collection device 1 is a direction away from the rotation center. I.e. the acquisition direction is pointing outwards with respect to the centre of rotation. The optical axis of the image acquisition apparatus 1 may be parallel to the rotation plane or may form an angle with the rotation plane, for example, in a range of-90 ° to 90 ° with respect to the rotation plane. Usually the axis of rotation or its extension (i.e. the centre line of rotation) passes through the image acquisition device, i.e. the image acquisition device is still rotating in a spinning manner. This is fundamentally different from the conventional image capturing apparatus in the capturing manner (circling manner) of rotating around a certain object, i.e., completely different from the circling manner of rotating around the object. The optical acquisition ports (such as lenses) of the image acquisition devices face away from the direction of the rotation axis, that is, the acquisition area of the image acquisition devices does not intersect with the rotation center line. Optionally, the optical axis of the image capturing device forms an angle with the horizontal plane, so that this method is also greatly different from a general rotation method, and particularly can capture an object whose surface is not perpendicular to the horizontal plane.

Of course, the rotation shaft of the rotation device may also be connected to the image capturing device through a reduction device, such as a gear set. When the image capturing device makes a 360 ° rotation in the horizontal plane, it captures an image of the corresponding object at a specific position (the specific capturing position will be described later in detail). The shooting can be performed synchronously with the rotation action, or shooting can be performed after the rotation of the shooting position is stopped, and the rotation is continued after the shooting is finished, and so on. The rotating device can be a motor, a stepping motor, a servo motor, a micro motor and the like. The rotating device (e.g., various motors) can rotate at a prescribed speed under the control of the controller and can rotate at a prescribed angle, thereby achieving optimization of the acquisition position, which will be described in detail below. Of course, the image acquisition device can be mounted on the rotating device in the existing equipment.

The bearing device 3 is used for bearing the weight of the whole equipment, and the rotating device 2 is connected with the bearing device 3. The carrying device may be a tripod, a base with a support device, etc. Typically, the rotating means is located in the central part of the carrying means to ensure balance. But in some special cases it can be located anywhere on the carrier. And the carrier is not necessary. The rotating device may be mounted directly in the application, for example, may be mounted on the roof of a vehicle.

In another embodiment, as shown in fig. 2, the device comprises an image acquisition device 1, a rotating device 2, a carrying device 3 and a telescopic device 4.

The image acquisition device 1 is connected with a rotating shaft of the rotating device 2 and is driven to rotate by the rotating device 2. Of course, the rotation shaft of the rotation device may also be connected to the image capturing device through a speed reduction device, such as a gear set. When the image capturing device makes a 360 ° rotation in the horizontal plane, it captures an image of the corresponding object at a specific position (the specific capturing position will be described later in detail). The shooting can be performed synchronously with the rotation action, or shooting can be performed after the rotation of the shooting position is stopped, and the rotation is continued after the shooting is finished, and the like. The rotating device can be a motor, a stepping motor, a servo motor, a micro motor and the like. The rotating device (e.g., various motors) can rotate at a specified speed under the control of the controller and can rotate at a specified angle, so as to optimize the acquisition position, which will be described in detail below. Of course, the image acquisition device can be mounted on the rotating device in the existing equipment.

One end of the telescopic device 4 is connected with the rotating device 2, and the other end of the telescopic device is connected with the bearing device 3 and is used for stretching in the direction vertical to the optical axis of the image acquisition device, so that the image acquisition device can be positioned at different positions. At each position, the rotating scanning is carried out under the driving of a rotating device, so that a 3D model of the target object at the position can be constructed. After a certain position is scanned, the telescopic device moves again, so that the image acquisition device moves to another position, the scanning is repeated, and the analogy is repeated, and the construction of the 3D model in the slender target object can be realized. The method can also be used for scanning in different height levels when the peripheral object is higher, so as to construct a 3D model of the whole object.

The telescopic device can be a telescopic sleeve, a telescopic slide rail and other telescopic structures. The expansion and contraction of the telescopic device can be manually adjusted and also can be expanded and contracted under the control of the control unit. The telescopic device may further comprise a telescopic motor for driving the telescopic unit (e.g. telescopic tube) to lengthen or shorten. After telescoping to the right position, the length of the telescoping device can be locked by the locking unit, so that stable support is provided for the rotating device. The locking unit may be a mechanical locking unit, such as a locking pin or the like, or an electrical locking unit, such as a locking of the telescopic device under control of the control unit.

The carrying device 3 is used for carrying the weight of the whole equipment. The carrying device may be a tripod, a base with a support device, etc. Typically, the rotating means is located in the central part of the carrying means to ensure balance. But in some special cases it can be located anywhere on the carrier. And the carrier is not necessary. The rotating means may be mounted directly in the application device, for example may be mounted on top of the walking robot.

Through the telescoping device design, can be so that image acquisition device can be at the not information of co-altitude collection, like this to the higher building of indoor choosing, collection that can be comprehensive accurate.

In another embodiment, as shown in fig. 3, the 3D information acquisition device includes an image acquisition device 1, a rotation device 2, a carrying device 3, and a tilting device 5. Wherein the image acquisition device 1 is arranged on the pitching device 5, so that the image acquisition device 1 can be pitched and rotated along a vertical plane. The pitching device can be a roller, a gear, a bearing, a ball joint and the like. The optical axis of the image capturing device 1 is usually parallel to the pitch direction, but may be at a certain angle in some special cases. The pitch device can be manually adjusted and can also be driven by a motor to pitch and rotate, so that the precise pitch angle adjustment can be realized according to program control. The pitching device also comprises a locking mechanism which is used for locking the pitching device after the pitching angle is adjusted in place and the optical axis of the image acquisition device and the horizontal plane form a preset angle, so that the pitching device is prevented from rotating in the vertical direction again.

The pitching device 5 is connected with the rotating shaft of the rotating device 2 and is driven to rotate by the rotating device 2. Of course, the rotation shaft of the rotation device may also be connected to the pitching device through a reduction device, such as a gear set.

Due to the adjustment of the pitching device, the optical axis of the image acquisition device and the horizontal plane form a certain included angle under normal conditions. This allows scanning of objects whose surface is not perpendicular to the horizontal. The pitching device is adjusted according to the condition that the surface of the target object and the horizontal plane form an approximate included angle, so that the optical axis of the image acquisition device is perpendicular to the surface of the target object as much as possible, and the acquisition accuracy of the details of the target object is improved. Of course, it may also be parallel to the horizontal plane in special cases.

Through the design of the pitch angle, when the view field of the image acquisition device can not cover all indoor spaces, particularly the upper space, the image acquisition device can be properly lifted upwards, so that the acquisition range is larger.

In another embodiment, as shown in fig. 4, both the pantograph 4 and the luffing device 5 may be included. Namely, the image acquisition device 1 is installed on the pitching device 5, the pitching device 5 is connected with the rotating device 2, the rotating device 2 is installed at one end of the telescopic device 4, and the other end of the telescopic device 4 is installed on the bearing device 3. Like this, when meetting that the interior space is great, higher (for example church), can scan the collection through the telescopic link with image acquisition device location in different height positions in proper order again, also can be through adjusting the angle of pitch for image acquisition device gathers more upper spatial information. Of course both may be used simultaneously, depending on the circumstances.

For all the above embodiments, the capturing direction of the image capturing device is a direction away from the rotation center. I.e. the acquisition direction is pointing outwards with respect to the centre of rotation. The optical axis of the image acquisition device may be parallel to the plane of rotation or may be at an angle to the plane of rotation, for example in the range-90 ° to 90 ° with respect to the plane of rotation. Usually the axis of rotation or its extension (i.e. the centre line of rotation) passes through the image acquisition device, i.e. the image acquisition device is still rotating in a spinning manner. This is fundamentally different from the conventional image capturing apparatus in the capturing manner (circling manner) of rotating around a certain object, i.e., completely different from the circling manner of rotating around the object. The optical acquisition ports (such as lenses) of the image acquisition devices face away from the direction of the rotation axis, that is, the acquisition area of the image acquisition devices does not intersect with the rotation center line. Optionally, the optical axis of the image capturing device forms an angle with the horizontal plane, so that this method is also greatly different from a general rotation method, and particularly can capture an object whose surface is not perpendicular to the horizontal plane.

The device can further comprise a distance measuring device, the distance measuring device is fixedly connected with the image acquisition device, and the pointing direction of the distance measuring device is the same as the direction of the optical axis of the image acquisition device. Of course, the distance measuring device can also be fixedly connected to the rotating device, as long as the distance measuring device can synchronously rotate along with the image acquisition device. Preferably, an installation platform can be arranged, the image acquisition device and the distance measurement device are both positioned on the platform, and the platform is installed on a rotating shaft of the rotating device and driven to rotate by the rotating device. The distance measuring device can use a laser distance measuring instrument, an ultrasonic distance measuring instrument, an electromagnetic wave distance measuring instrument and other various modes, and can also use a traditional mechanical measuring tool distance measuring device. Of course, in some applications, the 3D acquisition device is located at a specific position, and its distance from the target object is calibrated, and no additional measurement is needed.

The device also comprises a light source which can be arranged on the periphery of the image acquisition device, the rotating device and the mounting platform. Of course, the light source may be separately provided, for example, a separate light source may be used to illuminate the target. Even when the lighting conditions are good, no light source is used. The light source can be an LED light source or an intelligent light source, namely, the light source parameters are automatically adjusted according to the conditions of the target object and the ambient light. Typically, the light sources are distributed around the lens of the image capturing device, for example, the light sources are ring-shaped LED lamps around the lens. Since in some applications it is desirable to control the intensity of the light source. In particular, a light softening means, for example a light softening envelope, may be arranged in the light path of the light source. Or the LED surface light source is directly adopted, so that the light is soft, and the light is more uniform. Preferably, an OLED light source can be adopted, the size is smaller, the light is softer, and the flexible OLED light source has the flexible characteristic and can be attached to a curved surface.

In order to facilitate the actual size measurement of the target, a plurality of marking points may be provided at the target. And the coordinates of these marked points are known. The absolute size of the 3D synthetic model is obtained by collecting the mark points and combining the coordinates thereof. These marking points may be previously set points or may be laser light spots. The method of determining the coordinates of the points may comprise: using laser to measure distance: and emitting laser towards the target object by using the calibration device to form a plurality of calibration point light spots, and obtaining the coordinates of the calibration points through the known position relation of the laser ranging units in the calibration device. And emitting laser towards the target by using the calibration device, so that the light beam emitted by the laser ranging unit in the calibration device falls on the target to form a light spot. Since the laser beams emitted from the laser ranging units are parallel to each other, the positional relationship between the respective units is known. The two-dimensional coordinates in the emission plane of the plurality of light spots formed on the target object can be obtained. The distance between each laser ranging unit and the corresponding light spot can be obtained by measuring the laser beam emitted by the laser ranging unit, namely the depth information equivalent to a plurality of light spots formed on the target object can be obtained. I.e. the depth coordinate perpendicular to the emission plane, can be obtained. Thereby, three-dimensional coordinates of each spot can be obtained. Secondly, distance measurement and angle measurement are combined: and respectively measuring the distances of the plurality of mark points and the included angles between the mark points, thereby calculating respective coordinates. Using other coordinate measuring tools: such as RTK, global coordinate positioning systems, satellite-sensitive positioning systems, position and pose sensors, etc.

3D information acquisition matching process

The two objects are matched, that is, the two objects respectively occupy the space with an intersection, or the two objects respectively occupy the space with a stable relationship. Such as insertion of a fuel nozzle into a fuel filler opening, alignment of a marine helicopter with a vessel to be rescued on the sea surface, landing of an unmanned aerial vehicle on top of a traveling vehicle, etc.

1. The collecting device is positioned outside the target object

As shown in fig. 5-6:

(1) the 3D acquisition device 6 is placed at a certain position in space to ensure that the object to be matched (hereinafter referred to as the target) is within its scanning field of view.

(2) At time T0, the acquisition device 6 acquires images of a plurality of objects and performs 3D synthesis, respectively, to thereby obtain 3D models of the plurality of objects in a short time. The specific method comprises the following steps:

the rotating device drives the image acquisition device to rotate at a certain speed, and the image acquisition device acquires images at a set position in the rotating process. At the moment, the rotation can not be stopped, namely, the image acquisition and the rotation are synchronously carried out; or stopping rotation at the position to be acquired, acquiring images, and continuing to rotate to the next position to be acquired after acquisition is finished. The rotating means may be driven by a program in a control unit set in advance. The device can also communicate with an upper computer through a communication interface, and the rotation is controlled through the upper computer. In particular, the rotating device can be connected with a mobile terminal in a wired or wireless mode, and the rotating device is controlled to rotate through the mobile terminal (such as a mobile phone). The rotating device can set rotating parameters through the remote platform, the cloud platform, the server, the upper computer and the mobile terminal, and the rotating start and stop of the rotating device are controlled.

The image acquisition device acquires a plurality of images of the target object, sends the images into a remote platform, a cloud platform, a server, an upper computer and/or a mobile terminal through the communication device, synthesizes a 3D model of the target object by using a 3D model synthesis method, and finally synthesizes the 3D model of the object to be matched. It will be appreciated that the overall acquisition and synthesis speed is very fast.

(3) At time T1, the acquisition device acquires images of a plurality of objects and performs 3D synthesis on the images, thereby acquiring 3D models of the plurality of objects again. It is understood that the time intervals of T0 and T1 may be shorter or longer as desired. The shortest interval is: the collecting device rotates ceaselessly all the time, thereby collecting uninterruptedly.

(4) By analogy, images of a plurality of target objects are respectively collected at different moments and are respectively subjected to 3D synthesis, so that 3D models of the plurality of target objects are continuously obtained. And in the process, the 3D model information of the target object is continuously utilized to carry out matching operation on different target objects. For example, when two trucks are used for goods handover, the 3D models of the two vehicles are continuously obtained, so that the actions of the vehicles are continuously adjusted according to the latest 3D models of the vehicles and the position and posture information of the vehicles, the containers of the vehicles are aligned, and the goods handover is smoothly completed.

In the above-described acquisition process, for a space that needs to be acquired at a plurality of heights, the following procedure may be adopted.

The length of the telescopic device is controlled to enable the image acquisition device to be located at a preset position, the rotating device drives the image acquisition device to rotate at a certain speed, and the image acquisition device acquires images at a set position in the rotating process. At the moment, the rotation can not be stopped, namely, the image acquisition and the rotation are synchronously carried out; or stopping rotation at the position to be acquired, acquiring images, and continuing to rotate to the next position to be acquired after acquisition is finished. The rotation device may be driven by a program in a control unit set in advance. The device can also communicate with an upper computer through a communication interface, and the rotation is controlled through the upper computer. Particularly, the rotating device can be connected with a mobile terminal in a wired or wireless mode, and the rotating device is controlled to rotate through the mobile terminal (such as a mobile phone). The rotating device can set rotating parameters through the remote platform, the cloud platform, the server, the upper computer and the mobile terminal, and the rotating start and stop of the rotating device are controlled.

And controlling the length of the telescopic device to enable the image acquisition device to be located at another preset position, and repeating the action of the rotating device, so that the image acquisition device can acquire the images of the target object surrounding the position, and repeating the steps in the same way, so as to acquire the images at a plurality of height positions, thereby constructing a corresponding 3D model.

The image acquisition device acquires a plurality of images of the target object, sends the images to a remote platform, a cloud platform, a server, an upper computer and/or a mobile terminal through the communication device, and carries out 3D synthesis on the target object by using a 3D model synthesis method.

Of course, the pitching device can also be controlled, so that the image acquisition device can be rotationally acquired after being pitched to a certain angle.

In particular, the distance measuring device may be used to measure the corresponding distance parameters in the relevant formula conditions, i.e. the distance from the center of rotation to the target object and the distance from the sensor element to the target object, before or simultaneously with the acquisition. And calculating the acquisition position according to a corresponding condition formula, and prompting a user to set rotation parameters or automatically setting the rotation parameters.

When the distance measurement is carried out before the collection, the rotating device can drive the distance measurement device to rotate, so that the two distances at different positions can be measured. And respectively averaging two distances measured by a plurality of measuring points, and taking the average value as a uniform distance value acquired at this time to be introduced into a formula. The average value can be obtained by using a sum average, a weighted average, other averaging methods, or a method of discarding outliers and then averaging.

When distance measurement is carried out in the acquisition process, the rotating device rotates to the first position to carry out image acquisition, the two distance values are measured at the same time, the two distance values are brought into a condition formula to calculate the interval angle, and the next acquisition position is determined according to the angle.

2. The collecting device is positioned on the target object

In many cases, there is no extra space or position for placing the acquisition device, and the invention proposes a solution (which is also one of the points of the invention) as shown in fig. 7 for this purpose:

(1) the 3D information acquisition device 6 is disposed on an object to be matched (referred to as a target object) so that the object matched therewith is within the scanning field of view thereof. Usually, at least one 3D information acquisition device is disposed on each target. However, in some cases, if the 3D model or information of one of the objects is known or is available in real time, it is only necessary to provide an acquisition device thereon.

(2) At time T0, the acquisition device acquires images corresponding to the object and performs 3D synthesis to obtain a 3D model of the object. If the two targets to be matched are provided with the acquisition devices, each acquisition device acquires the image of the target opposite to the acquisition device. If one of the two objects to be matched is provided with a collecting device, the collecting device collects the image of the object opposite to the collecting device, and the 3D model of the object collected by the collecting device is known and can be given by an inquiry system. The specific collection and synthesis method is the same as the above description, and is not described again.

(3) At time T1, the acquisition device acquires images corresponding to the object and performs 3D synthesis to obtain a 3D model of the object. If the two targets to be matched are provided with the acquisition devices, each acquisition device acquires the image of the target opposite to the acquisition device. If one of the two objects to be matched is provided with a collecting device, the collecting device collects the image of the object opposite to the collecting device, and the 3D model of the object collected by the collecting device is known and can be given by an inquiry system. The specific collection and synthesis method is the same as that described above and is not described in detail. It is understood that the time intervals of T0 and T1 may be shorter or longer as desired. The shortest interval is: the collecting device rotates ceaselessly all the time, thereby collecting uninterruptedly.

(4) By analogy, images of the target object are respectively collected at different moments and are respectively subjected to 3D synthesis, so that 3D models of a plurality of target objects are continuously obtained. And in the process, the 3D model information of the target object is continuously utilized to carry out matching operation on different target objects. For example, when two trucks are used for goods handover, the 3D models of the two vehicles are continuously obtained, so that the actions of the vehicles are continuously adjusted according to the latest 3D models of the vehicles and the position and posture information of the vehicles, the containers of the vehicles are aligned, and the goods handover is smoothly completed.

Different from the above, the matching process may be respectively completed in the acquisition devices of the two target objects or one of the devices, or may be completed in the server or the cloud platform. That is, any one of the acquisition devices can independently obtain the 3D model of the opposite target (synthesized in the device, or synthesized in a third-party server or a cloud platform and then sent to the device), and can receive or query the 3D model of the target where the acquisition device is located, so that the actions of the acquisition device are continuously adjusted according to the 3D models of the two targets and the position and posture information of the two targets in the device, and the matching of the two targets is realized. In another case, each device may independently synthesize the 3D models of the objects it faces, and send them to the third-party server and the cloud platform, so as to match the two objects in the third-party server and the cloud platform. In another case, each device sends the acquired picture to a third-party server and a cloud platform, so that the synthesis of the 3D model of the two target objects is realized in the third-party server and the cloud platform, and the matching is completed simultaneously.

In each acquisition, the images can be time-stamped with the acquisition time, so as to determine which images are acquired at the same time, and only the images with the same (or very similar) time stamp are used for synthesizing the 3D model of the target object at the time.

It is understood that the matching may be between two objects, one object may be matched with two objects, or a plurality of objects may be matched with a plurality of objects.

Optimization of camera position

In order to ensure that the device can take the effect and efficiency of 3D synthesis into account, a method for optimizing the acquisition position of a camera can be used in addition to a conventional method for optimizing a synthesis algorithm. Especially in the case of 3D acquisition synthesis devices in which the acquisition direction of the camera and the direction of its axis of rotation deviate from each other, the prior art does not mention how to perform a better optimization of the camera position for such devices. Even if some optimization methods exist, they are different empirical conditions obtained under different experiments. In particular, some existing position optimization methods require obtaining the size of the target, which is feasible in the wrap-around 3D acquisition, and can be measured in advance. However, it is difficult to measure in advance in an open space. It is therefore desirable to propose a method that can be adapted to camera position optimization when the acquisition direction of the camera of the 3D acquisition composition device and its rotation axis direction deviate from each other. This is the problem to be solved by the present invention, and a technical contribution is made.

For this reason, the present invention has performed a large number of experiments, and it is concluded that empirical conditions preferably satisfied by the intervals of camera acquisition when acquisition is performed are as follows.

When 3D acquisition is carried out, the included angle alpha of the optical axis of the image acquisition device at two adjacent positions meets the following condition:

wherein the content of the first and second substances,

r is the distance from the center of rotation to the surface of the target,

t is the sum of the object distance and the image distance during acquisition, namely the distance between the photosensitive unit of the image acquisition device and the target object.

d is the length or width of a photosensitive element (CCD) of the image acquisition device, and when the two positions are along the length direction of the photosensitive element, the length of the rectangle is taken as d; when the two positions are along the width direction of the photosensitive element, d takes a rectangular width.

And F is the focal length of the lens of the image acquisition device.

u is an empirical coefficient.

Usually, a distance measuring device, for example a laser distance meter, is arranged on the detection device. The optical axis of the distance measuring device is parallel to the optical axis of the image acquisition device, so that the distance from the acquisition device to the surface of the target object can be measured, and R and T can be obtained according to the known position relation between the distance measuring device and each part of the acquisition device by using the measured distance.

When the image acquisition device is at any one of the two positions, the distance from the photosensitive element to the surface of the target object along the optical axis is taken as T. In addition to this method, multiple averaging or other methods can be used, the principle being that the value of T should not deviate from the sum of the image distances from the object at the time of acquisition.

Similarly, when the image pickup device is in any one of the two positions, the distance from the rotation center to the surface of the object along the optical axis is defined as R. In addition to this method, multiple averaging or other methods can be used, with the principle that the value of R should not deviate from the radius of rotation at the time of acquisition.

In general, the size of an object is adopted as a method for estimating the position of a camera in the prior art. Since the object size will vary with the measurement object. For example, when a large object is acquired 3D information and then a small object is acquired, the size needs to be measured again and reckoning needs to be performed again. The inconvenient measurement and the repeated measurement bring errors in measurement, thereby causing errors in camera position estimation. According to the scheme, the experience conditions required to be met by the position of the camera are given according to a large amount of experimental data, and the size of an object does not need to be directly measured. In the empirical condition, d and F are both fixed parameters of the camera, and corresponding parameters can be given by a manufacturer when the camera and the lens are purchased without measurement. R, T is only a straight line distance that can be easily measured by conventional measuring methods such as a ruler and a laser rangefinder. Meanwhile, in the apparatus of the present invention, the capturing direction of the image capturing device (e.g., camera) and the direction of the rotation axis thereof are away from each other, that is, the lens is oriented substantially opposite to the rotation center. At the moment, the included angle alpha of the optical axis for controlling the image acquisition device at the two positions is easier, and only the rotation angle of the rotary driving motor needs to be controlled. Therefore, it is more reasonable to use α to define the optimal position. Therefore, the empirical formula of the invention enables the preparation process to be convenient and fast, and simultaneously improves the arrangement accuracy of the camera position, so that the camera can be arranged in an optimized position, thereby simultaneously considering the 3D synthesis precision and speed.

According to a number of experiments, u should be less than 0.498 in order to ensure the speed and effect of the synthesis, and for better synthesis effect, u is preferably <0.411, especially preferably <0.359, in some applications u <0.281, or u <0.169, or u <0.041, or u < 0.028.

Experiments were carried out using the apparatus of the invention, and some experimental data are shown below, in mm. (the following data are given by way of example only)

The above data are obtained by experiments for verifying the conditions of the formula, and do not limit the invention. Without these data, the objectivity of the formula is not affected. Those skilled in the art can adjust the equipment parameters and the step details as required to perform experiments, and obtain other data which also meet the formula conditions.

3D model synthesis method

A plurality of images acquired by the image acquisition device are sent to the processing unit, and a 3D model is constructed by using the following algorithm. The processing unit can be located in the acquisition equipment or remotely, such as a cloud platform, a server, an upper computer and the like.

The specific algorithm mainly comprises the following steps:

step 1: and performing image enhancement processing on all input photos. The following filters are used to enhance the contrast of the original picture and simultaneously suppress noise.

In the formula: g (x, y) is the gray value of the original image at (x, y), f (x, y) is the gray value of the original image at the position after being enhanced by the Wallis filter, and m _g Is the local gray average value, s, of the original image _g Is the local standard deviation of gray scale of the original image, m _f For the transformed image local gray scale target value, s _f The target value is the standard deviation of the local gray scale of the image after transformation. c belongs to (0, 1) as the expansion constant of the image variance, and b belongs to (0, 1) as the image brightness coefficient constant.

The filter can greatly enhance the image texture modes of different scales in the image, so the number and the precision of the feature points can be improved when the point features of the image are extracted, and the reliability and the precision of the matching result are improved in the photo feature matching.

Step 2: and extracting feature points of all input photos, and matching the feature points to obtain sparse feature points. And extracting and matching feature points of the photos by adopting a SURF operator. The SURF feature matching method mainly comprises three processes of feature point detection, feature point description and feature point matching. The method uses a Hessian matrix to detect characteristic points, a Box filter (Box Filters) is used for replacing second-order Gaussian filtering, an integral image is used for accelerating convolution to improve the calculation speed, and the dimension of a local image characteristic descriptor is reduced to accelerate the matching speed. The method mainly comprises the steps of firstly, constructing a Hessian matrix, generating all interest points for feature extraction, and constructing the Hessian matrix for generating stable edge points (catastrophe points) of an image; secondly, establishing scale space characteristic point positioning, comparing each pixel point processed by the Hessian matrix with 26 points in a two-dimensional image space and a scale space neighborhood, preliminarily positioning a key point, filtering the key point with weak energy and the key point with wrong positioning, and screening out a final stable characteristic point; and thirdly, determining the main direction of the characteristic points by adopting the harr wavelet characteristics in the circular neighborhood of the statistical characteristic points. In a circular neighborhood of the feature points, counting the sum of horizontal and vertical harr wavelet features of all points in a sector of 60 degrees, rotating the sector at intervals of 0.2 radian, counting the harr wavelet feature values in the region again, and taking the direction of the sector with the largest value as the main direction of the feature points; and fourthly, generating a 64-dimensional feature point description vector, and taking a 4-by-4 rectangular region block around the feature point, wherein the direction of the obtained rectangular region is along the main direction of the feature point. Each subregion counts haar wavelet features of 25 pixels in both the horizontal and vertical directions, where both the horizontal and vertical directions are relative to the principal direction. The haar wavelet features are in 4 directions of the sum of the horizontal direction value, the vertical direction value, the horizontal direction absolute value and the vertical direction absolute value, the 4 values are used as feature vectors of each subblock region, and therefore a total 4 × 4-64-dimensional vector is used as a descriptor of the Surf features; and fifthly, matching the characteristic points, wherein the matching degree is determined by calculating the Euclidean distance between the two characteristic points, and the shorter the Euclidean distance is, the better the matching degree of the two characteristic points is.

And step 3: inputting matched feature point coordinates, resolving the sparse three-dimensional point cloud of the target object and the position and posture data of the photographing camera by using a light beam method adjustment, namely obtaining model coordinate values of the sparse three-dimensional point cloud of the target object model and the position; and performing multi-view photo dense matching by taking the sparse feature points as initial values to obtain dense point cloud data. The process mainly comprises four steps: stereo pair selection, depth map calculation, depth map optimization and depth map fusion. For each image in the input data set, we select a reference image to form a stereo pair for use in computing the depth map. Therefore, we can get rough depth maps of all images, which may contain noise and errors, and we use its neighborhood depth map to perform consistency check to optimize the depth map of each image. And finally, carrying out depth map fusion to obtain the three-dimensional point cloud of the whole scene.

And 4, step 4: and reconstructing the curved surface of the target object by using the dense point cloud. The method comprises the steps of defining an octree, setting a function space, creating a vector field, solving a Poisson equation and extracting an isosurface. And obtaining an integral relation between the sampling point and the indicating function according to the gradient relation, obtaining a vector field of the point cloud according to the integral relation, and calculating the approximation of the gradient field of the indicating function to form a Poisson equation. And (3) solving an approximate solution by using matrix iteration according to a Poisson equation, extracting an isosurface by adopting a moving cube algorithm, and reconstructing a model of the measured point cloud.

And 5: full-automatic texture mapping of object models. And after the surface model is constructed, texture mapping is carried out. The main process comprises the following steps: texture data is obtained to reconstruct a surface triangular surface grid of a target through an image; and secondly, reconstructing the visibility analysis of the triangular surface of the model. Calculating a visible image set and an optimal reference image of each triangular surface by using the calibration information of the image; and thirdly, clustering the triangular surface to generate a texture patch. Clustering the triangular surfaces into a plurality of reference image texture patches according to the visible image set of the triangular surfaces, the optimal reference image and the neighborhood topological relation of the triangular surfaces; and fourthly, automatically sequencing the texture patches to generate texture images. And sequencing the generated texture patches according to the size relationship of the texture patches to generate a texture image with the minimum surrounding area, and obtaining the texture mapping coordinate of each triangular surface.

It should be noted that the above algorithm is an algorithm used by the present invention, and the algorithm is matched with the image acquisition condition, and the time and quality of the synthesis are considered by using the algorithm. It will be appreciated that conventional 3D synthesis algorithms known in the art may be used with the solution of the invention.

Examples of the applications

In order to realize goods transfer without stopping of two transport vehicles, the 3D information acquisition device can be arranged on the roofs of the two transport vehicles. The method comprises the steps that images of opposite vehicles are obtained through scanning and acquisition respectively and are transmitted to a server, the server synthesizes 3D models of the two vehicles at different moments according to a plurality of images, and meanwhile, the running tracks and the postures of the two trucks are adjusted continuously according to the 3D models, the positions and the posture information of the two vehicles, and finally matching alignment of containers is achieved.

The three-dimensional model of the object is imported into the processor and the optimal alignment is performed according to the existing algorithm, i.e. the alignment of the two is realized according to the three-dimensional shape of the object. The above operations can be implemented in a remote platform, a cloud platform, a server, an upper computer and/or a mobile terminal.

The above is that each scanning device obtains the 3D model respectively and then leads it into the processor for matching, or the pictures of each scanning device can be directly led into the processor, and the synthesis of the spatial 3D model, the synthesis of the object 3D model, and the matching between them are completed in the processor in a centralized manner. This simplifies the structure and cost of the acquisition hardware. For example, a scanner does not need to be provided with a processor capable of processing large data, and only needs to perform simple control. The collected images can be transmitted to a cloud platform (equivalent to a processor) for centralized processing through a 4G or 5G or other communication networks. This is also one of the points of the present invention.

The target object, and the object all represent objects for which three-dimensional information is to be acquired. The object may be a solid object or a plurality of object components. The three-dimensional information of the target object comprises a three-dimensional image, a three-dimensional point cloud, a three-dimensional grid, a local three-dimensional feature, a three-dimensional size and all parameters with the three-dimensional feature of the target object. Three-dimensional in the present invention means having XYZ three-direction information, particularly depth information, and is essentially different from only two-dimensional plane information. It is also fundamentally different from some definitions, which are called three-dimensional, panoramic, holographic, three-dimensional, but actually comprise only two-dimensional information, in particular not depth information.

The capture area in the present invention refers to a range in which an image capture device (e.g., a camera) can capture an image. The image acquisition device can be a CCD, a CMOS, a camera, a video camera, an industrial camera, a monitor, a camera, a mobile phone, a tablet, a notebook, a mobile terminal, wearable equipment, intelligent glasses, an intelligent watch, an intelligent bracelet and all equipment with an image acquisition function.

In the description provided herein, numerous specific details are set forth. It is understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be interpreted as reflecting an intention that: that the invention as claimed requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.

Those skilled in the art will appreciate that the modules in the device in an embodiment may be adaptively changed and disposed in one or more devices different from the embodiment. The modules or units or components of the embodiments may be combined into one module or unit or component, and furthermore they may be divided into a plurality of sub-modules or sub-units or sub-components. All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or elements of any method or apparatus so disclosed, may be combined in any combination, except combinations where at least some of such features and/or processes or elements are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Furthermore, those skilled in the art will appreciate that while some embodiments herein include some features included in other embodiments, rather than other features, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the claims, any of the claimed embodiments may be used in any combination.

The various component embodiments of the invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It will be appreciated by those skilled in the art that a microprocessor or Digital Signal Processor (DSP) may be used in practice to implement some or all of the functionality of some or all of the components in an apparatus in accordance with embodiments of the present invention. The present invention may also be embodied as apparatus or device programs (e.g., computer programs and computer program products) for performing a portion or all of the methods described herein. Such programs implementing the present invention may be stored on computer-readable media or may be in the form of one or more signals. Such a signal may be downloaded from an internet website or provided on a carrier signal or in any other form.

It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The usage of the words first, second and third, etcetera do not indicate any ordering. These words may be interpreted as names.

Thus, it should be appreciated by those skilled in the art that while a number of exemplary embodiments of the invention have been illustrated and described in detail herein, many other variations or modifications consistent with the principles of the invention may be directly determined or derived from the disclosure of the present invention without departing from the spirit and scope of the invention. Accordingly, the scope of the invention should be understood and interpreted to cover all such other variations or modifications.

Claims

1. A device for realizing 3D matching between moving objects is characterized in that: the system comprises a target object 3D information acquisition device and a first processor;

the first processor is used for matching three-dimensional models of at least two objects to be matched at the same moment;

the 3D information acquisition device comprises an image acquisition device and a rotating device;

2. The apparatus of claim 1, wherein: the first processor is also used for synthesizing a three-dimensional model of the object to be matched.

3. The apparatus of claim 1, wherein: the system also comprises a second processor, which is positioned on the target object 3D information acquisition device and is used for synthesizing a three-dimensional model of the object to be matched relative to the target object.

4. The apparatus of claim 1, wherein: when the target object 3D information acquisition device acquires the images of the object to be matched at any time, the images are stamped.

5. The apparatus of claim 1, wherein: u <0.498, or u <0.411, or u <0.359, or u <0.281, or u <0.169, or u <0.041, or u < 0.028.

6. The apparatus of claim 1, wherein: the optical acquisition ports of the image acquisition devices are back to the direction of the rotating shaft.

7. The apparatus of claim 1, wherein: the target object 3D information acquisition device is arranged on the object to be matched.

8. The apparatus of claim 1, wherein: the target object 3D information acquisition device is arranged separately from the object to be matched.

9. The apparatus of claim 1, wherein: after the matching is completed, the first processor outputs the matching result to the display device, the printing device and/or the action execution device.

10. A method for realizing 3D matching between moving objects is characterized in that: the system comprises a target object 3D information acquisition device and a first processor;

11. The method of claim 10, wherein: the first processor is also used for synthesizing a three-dimensional model of the object to be matched.

12. The method of claim 10, wherein: the system also comprises a second processor, which is positioned on the target object 3D information acquisition device and is used for synthesizing a three-dimensional model of the object to be matched relative to the target object.

13. The method of claim 10, wherein: when the target object 3D information acquisition device acquires the images of the object to be matched at any time, the images are stamped.

14. The method of claim 10, wherein: u <0.498, or u <0.411, or u <0.359, or u <0.281, or u <0.169, or u <0.041, or u < 0.028.

15. The method of claim 10, wherein: the optical acquisition ports of the image acquisition devices are back to the direction of the rotating shaft.

16. The method of claim 10, wherein: the target object 3D information acquisition device is arranged on an object to be matched.

17. The method of claim 10, wherein: the target object 3D information acquisition device is arranged separately from the object to be matched.

18. The method of claim 10, wherein: after the matching is completed, the first processor outputs the matching result to the display device, the printing device and/or the action execution device.