JP6293049B2 - Point cloud data acquisition system and method - Google Patents

Description

  The present invention relates to a system and method for acquiring point cloud data representing the outline of an object by non-contact measurement from a plurality of measurement positions.

  Using a 3D measurement device such as a laser scanner, measure the point cloud data representing the outline including the position and shape of the surface of the measurement object, and create a panoramic image of the measurement object from this point cloud data Data processing devices are known. This 3D data processing device facilitates the display of point cloud data by creating a panoramic image of a virtual spherical surface with the measurement position of the 3D measurement device as the origin, for example, based on the point cloud data. Is known to improve (see, for example, Patent Document 1).

Also, using a three-dimensional measuring device such as a laser scanner, point cloud data that represents the contour of a non-planar measurement object having irregularities on an existing building (for example, a plant such as a nuclear power plant or a cultural property) It is known to measure point cloud data representing the contour of the measurement object from a plurality of viewpoints by varying the measurement position.
The position information of each point cloud data measured from a plurality of different measurement positions is represented by a coordinate system (hereinafter also referred to as “inherent coordinate system”) based on the position and orientation of each measurement position at which the point cloud data is measured. .
Therefore, the position information represented by each unique coordinate system of each point cloud data measured from these different measurement positions is converted into the position information represented by the same coordinate system (hereinafter also referred to as “reference coordinate system”). Need to convert. By converting the coordinate system, it is known that the position information of all point cloud data is represented by the reference coordinate system, and each point cloud data represented by the same coordinate system is partially matched and combined. Yes. By doing so, the blind spot is reduced, and three-dimensional data for grasping the contour of the measurement object is obtained (see, for example, Patent Document 2).
Here, the term “matching” refers to the integration of the plurality of point cloud data obtained from different measurement positions by aligning the partially overlapping portions while maintaining the consistency.

  In the above-described prior art, when measuring point cloud data representing the outline of the measurement object from a plurality of measurement positions, it is necessary to set the position and orientation of each three-dimensional measurement device in advance. For this reason, for example, when a three-dimensional measurement device such as a laser scanner is changed from a preset measurement position to another measurement position and installed, the position, orientation, etc. It is necessary to measure accurately. Changing the position of the three-dimensional measuring device takes time and is not easy. Further, for example, if the measurement object is a non-stationary object whose outline changes dynamically with the passage of time such as, for example, the movement of a person or the movement of an object, a single measurement device can be used. It is impossible to capture from various directions.

  For matching, it is known that a plurality of markers are installed (for example, bonded or applied) around the target structure and its surroundings, and measurement data is synthesized based on the plurality of markers installed. (For example, patent document 3). However, in order to use a plurality of markers as a reference, it is necessary to install a marker, and it takes time to perform operations such as aligning the marker to an appropriate position, and it takes time to prepare for measurement work. There was a problem.

JP 2010-097419 A JP 2013-161133 A JP 2010-066169 A

In the above-described conventional technology, when the point cloud data representing the outline of the measurement object is measured from a plurality of measurement positions, it is necessary to accurately measure the position, orientation, and the like of the three-dimensional measurement apparatus in advance. It was.
For this reason, there existed a subject that the setting operation | work of the position of a three-dimensional measuring apparatus took time.
Moreover, since it is necessary to install each three-dimensional measuring device at a position accurately measured in advance, there is a problem that the measurement position cannot be changed immediately in accordance with the situation of the measurement object.
For example, when an object whose shape changes over time, such as a moving human, a moving animal, or a moving object, is measured, even if a plurality of measuring devices are used, the object is deformed. In the case where a blind spot occurs due to occlusion (a state in which the object in the front covers the object behind), the conventional technique has a problem that it is not possible to immediately change the measurement position and acquire a three-dimensional image with few missing portions. there were.

In addition, when a plurality of markers are installed on the object and its surroundings, there is a problem that the operation of installing the markers is necessary, and it takes time to perform operations such as aligning the marker to an appropriate position.
In addition, in the operation | work which installs a marker etc. in a target object (for example, adhesion | attachment or application | coating), there also existed a subject that a measurement target object might be damaged.

  The present invention relates to a system or method for acquiring point cloud data representing an outline of a measurement object from a plurality of measurement positions, when the three-dimensional measurement apparatus is moved to measure the measurement object. Providing a system for acquiring a plurality of point cloud data representing the contour of the measurement object from an angle of 360 degrees and acquiring a three-dimensional image with few missing portions by acquiring the position and orientation of the measurement object in real time. Objective.

  The present invention includes a first electronic device group having one first electronic device and one or more movable second electronic devices, and has depth information for each point on the surface of a subject in units of pixels. A point cloud data acquisition system for acquiring group data, wherein the first electronic device includes a first coordinate system, and includes a first communication unit and a plurality of ultrasonic transmission units that transmit ultrasonic waves. Each of the second electronic devices has a unique second coordinate system, a second communication unit, a plurality of ultrasonic reception units that receive ultrasonic waves, and the point cloud data in the second coordinate system. A point measurement data acquisition system, wherein each of the plurality of ultrasonic reception units receives an ultrasonic wave transmitted from each of the plurality of ultrasonic transmission units. An ultrasonic wave propagation time measurement unit for measuring the ultrasonic wave propagation time required to A second electronic device position information calculation unit that calculates coordinate values in the first coordinate system of the plurality of ultrasonic reception units based on the ultrasonic propagation times measured by the ultrasonic propagation time measurement unit; A second electronic device position information storage unit that stores coordinate values in the first coordinate system of the plurality of ultrasonic reception units for each of the second electronic devices; and the point cloud data measured by the three-dimensional measurement unit. A point cloud data acquisition system provided with a source point cloud data storage part memorized for every said 2nd electronic equipment.

  Further, each of the plurality of ultrasonic transmission units is provided at an arbitrary position that is linearly independent of the first coordinate system, and each of the plurality of ultrasonic reception units is an arbitrary one that is linearly independent of the second coordinate system. Can be provided in position.

  The second electronic device further includes a plurality of timer counters corresponding to the plurality of ultrasonic reception units, and the point cloud data acquisition system further includes the plurality of ultrasonic transmission units. An infrared light emitting unit that emits infrared rays simultaneously with transmitting ultrasonic waves, and an infrared light receiving unit that receives the infrared rays and simultaneously starts the plurality of timer counters, each of the plurality of ultrasonic reception units, Further, at the time when the ultrasonic wave is received, each of the plurality of timer counters is stopped, and the ultrasonic wave propagation time measurement unit is based on the time from when each of the plurality of timer counters is started until it is stopped. Each of the plurality of ultrasonic reception units is transmitted from each of the plurality of ultrasonic transmission units. It can be measured ultrasonic wave propagation time required to receive the waves.

  Each of the first electronic device and the second electronic device further includes a time synchronization unit that synchronizes an internal time with a reference time, and the ultrasonic propagation time measurement unit includes the plurality of ultrasonic transmission units. Each of the plurality of ultrasonic reception units is based on the transmission time information at which the ultrasonic waves are transmitted and the reception time information at which each of the plurality of ultrasonic reception units has received the ultrasonic waves. It is possible to measure the ultrasonic propagation time required to receive the ultrasonic waves transmitted from each of the transmission units.

  In addition, the second electronic device position information storage unit further includes a single ultrasonic receiving unit among the plurality of ultrasonic receiving units, based on coordinate values in the first coordinate system of the plurality of ultrasonic receiving units. Second electronic device position information associated with a time stamp corresponding to the time at which the ultrasonic wave is received is stored for each second electronic device, and the source point cloud data storage unit is further measured by the three-dimensional measurement unit Point cloud data information in which a time stamp that is a measurement time of the point cloud data is associated with the point cloud data is stored for each second electronic device, and the point cloud data acquisition system further includes the second electronic device. Based on the position information and the point cloud data information, the coordinate value in the second coordinate system of the data of each point in the point cloud data included in the point cloud data information is converted into the coordinate value in the first coordinate system. By aligning overlapping portions based on the coordinate values in the first coordinate system converted by the point group data coordinate value conversion unit and the point group data coordinate value conversion unit. A composite point cloud data creation unit that creates one composite point cloud data and a composite point cloud data storage unit that stores the composite point cloud data created by the composite point cloud data creation unit can be provided.

  Further, the composite point cloud data creation unit further aligns overlapping portions based on the coordinate values in the first coordinate system obtained by converting the plurality of point cloud data by the point cloud data coordinate value conversion unit. And a point group for matching the plurality of point group data based on curvatures of curved surfaces according to a three-dimensional shape respectively extracted from a plurality of different point group data having portions having the same coordinate value in the first coordinate system A data matching unit can be provided.

  Further, the composite point cloud data creation unit further, based on the coordinate values in the first coordinate system converted by the point cloud data coordinate value conversion unit, a plurality of point cloud data measured at the same time, One composite point cloud data can be created by aligning overlapping portions.

  The point cloud data acquisition system further includes a second electronic device group including one first electronic device 10 and one or more second electronic devices 20 that do not belong to the first electronic device group. The first electronic device belonging to the second electronic device group further includes a plurality of ultrasonic reception units that receive ultrasonic waves, and the ultrasonic propagation time measurement unit further includes the first electronic device. Ultrasonic waves transmitted from each of the plurality of ultrasonic transmission units of the first electronic device belonging to the group of the plurality of ultrasonic reception units provided in the first electronic device belonging to the second electronic device group Each of the second electronic equipment position information calculation unit measures the ultrasonic propagation time required for each reception, and the second electronic device position information calculation unit further determines the first propagation time based on the ultrasonic propagation time measured by the ultrasonic propagation time measurement unit. 1st electronic machine belonging to 2 electronic equipment groups Coordinate values in the first coordinate system of the first electronic devices 10 belonging to the first electronic device group of the plurality of ultrasonic receiving units are calculated, respectively, and the second electronic device position information storage unit further includes the second electronic device. Coordinate values in the first coordinate system of the first electronic device 10 belonging to the first electronic device group of the plurality of ultrasonic receivers of the first electronic device belonging to the second electronic device group are the second electronic device. Stored in correspondence with the first electronic device belonging to a group, the source point group data storage unit further includes point cloud data measured by a three-dimensional measuring instrument of the second electronic device belonging to the second electronic device group Can be stored for each of the second electronic devices belonging to the second electronic device group.

The present invention is a point cloud data acquisition method for acquiring point cloud data having depth information for each point on the surface of a subject in a pixel unit, and a plurality of ultrasonic transmissions of a first electronic device having a first coordinate system. The plurality of ultrasonic waves transmitted by the ultrasonic wave transmitting step by the ultrasonic wave transmitting step of transmitting a plurality of ultrasonic waves by the unit and the plurality of ultrasonic receiving units of one or more second electronic devices each having the second coordinate system. An ultrasonic reception step of receiving each of the ultrasonic waves, and a three-dimensional measurement step of measuring point cloud data based on the second coordinate system by the three-dimensional measurement unit of the one or more second electronic devices, Ultrasound required to receive each of the plurality of ultrasonic waves transmitted in the ultrasonic transmission step by each of the plurality of ultrasonic reception units of the second electronic device in the ultrasonic reception step. And ultrasonic wave propagation time measuring step of measuring the wave propagation time,
Second electronic device positions that respectively calculate coordinate values in the first coordinate system of the plurality of ultrasonic receivers of the second electronic device based on the ultrasonic propagation times measured in the ultrasonic propagation time measurement step The coordinate values in the first coordinate system of the plurality of ultrasonic receiving units of the one or more second electronic devices calculated in the information calculating step and the second electronic device position information calculating step are calculated for each second electronic device. A second electronic device position information storing step stored in
And a source point cloud data storage step for storing the point cloud data measured in the three-dimensional measurement step for each of the second electronic devices.

  The present invention relates to a computer program for causing a computer including a plurality of ultrasonic transmission units and a computer including a three-dimensional measurement unit and a plurality of ultrasonic reception units to execute each step of the point cloud data acquisition method described above.

According to the point cloud data acquisition system of the present invention, in order to measure an object, the measurement position of the three-dimensional measurement apparatus is immediately changed without installing the position and orientation of the three-dimensional measurement apparatus in a preset location. When it becomes necessary to move, the three-dimensional measuring device can be moved and the position and orientation of the three-dimensional measuring device can be acquired in real time.
By doing so, for example, when an object whose shape changes with the passage of time, such as a moving human, a moving animal, or a moving object, is an occlusion (an object in the foreground) that occurs along with the deformation of the object. When a blind spot occurs due to a situation in which the object is behind, the measurement position of the three-dimensional measurement device can be changed immediately in order to minimize the missing portion, and measurement is performed from an angle of 360 degrees. A plurality of point cloud data representing the contour of the object can be acquired.

It is a block diagram showing roughly a point cloud data acquisition system concerning an embodiment of the present invention. It is the schematic which shows the relationship between the ultrasonic wave propagation time between the ultrasonic generator which concerns on embodiment of this invention, and an ultrasonic receiver, and the linear distance based on it. It is a chart figure which shows the sequence between an infrared rays light emission part and an infrared rays light-receiving part, and between an ultrasonic transmission part and an ultrasonic wave reception part. It is a chart figure which shows the sequence between an ultrasonic transmission part and an ultrasonic reception part. It is a figure which shows the flow of a series of processes in embodiment of this invention. It is the schematic which shows the hierarchization by arrange | positioning the electronic device group comprised from a 1st electronic device and one or more 2nd electronic devices hierarchically.

  Hereinafter, a preferred embodiment of the point cloud data acquisition system of the present invention will be described with reference to the drawings. Here, all of the following embodiments are examples of the present invention, and the present invention is not limited thereto.

[First Embodiment]
As shown in FIG. 1, the point cloud data acquisition system 100 includes a first electronic device 10, one or more movable second electronic devices 20, a three-dimensional data storage unit 30, and a three-dimensional data processing unit 40. And a three-dimensional data display operation unit 50.

<First electronic device 10>
The first electronic device 10 can be configured by a personal computer having a wired and / or wireless communication function, for example, a personal computer including a notebook personal computer, or an arithmetic board on which an arithmetic chip such as a CPU or DSP is mounted.

  The first electronic device 10 has a first coordinate system (hereinafter also referred to as “reference coordinate system”), and includes a first communication unit 11, a plurality of (for example, four) ultrasonic wave transmission units 12, and a second electronic device. A position information calculation unit 13. In addition, the first electronic device 10 can further include an infrared light emitting unit 14 and / or a time synchronization unit 15. Hereinafter, the functional units included in the first electronic device 10 are collectively referred to as a “first functional unit”. Note that a plurality of (for example, four) ultrasonic transmission units 12 and the infrared light emitting unit 14 may be referred to as “first GPS units”.

  The first electronic device 10 is installed with a program for causing the first electronic device 10 to function as a first functional unit, such as the second electronic device position information calculating unit 13 described above, and the second electronic device position information is calculated. It can be made to function as each functional unit such as the unit 13. The program may be installed in the first electronic device 10 in advance. The program may be installed from a computer-readable recording medium as necessary or downloaded from a server set in advance as needed. Hereinafter, the “computer-readable recording medium” refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, and a CD-ROM, and a storage device such as a hard disk built in the computer system.

<Second electronic device 20>
The second electronic device 20 includes a three-dimensional measuring unit 22 and a wired and / or wireless communication function. For example, a movable electronic device such as a personal computer including a notebook computer or a tablet terminal, or an operation of a CPU, DSP, or the like. It can be composed of an arithmetic board equipped with a chip.

  The second electronic device 20 has a second coordinate system unique to the second electronic device (hereinafter also referred to as “unique coordinate system”), and includes a second communication unit 21, a three-dimensional measurement unit 22, and a plurality of (for example, 3) an ultrasonic receiving unit 23 and an ultrasonic propagation time measuring unit 26. In addition, the second electronic device 20 can further include an infrared light receiving unit 24 and / or a time synchronization unit 25. Hereinafter, the respective functional units included in the second electronic device 20 are collectively referred to as “second functional units”. Note that a plurality of (for example, three) ultrasonic wave receiving units 23 and the infrared light receiving unit 24 may be referred to as “second GPS units”.

  The second electronic device 20 is installed in the second electronic device 20 by installing a program for causing the second electronic device 20 to function as a second functional unit such as the three-dimensional measuring unit 22 and the ultrasonic wave receiving unit 23. Can be made to function as functional units such as the three-dimensional measuring unit 22 and the ultrasonic receiving unit 23. Note that the program may be installed in advance. The program may be installed from a computer-readable recording medium as necessary or downloaded from a server set in advance as needed.

<Three-dimensional data storage unit 30>
The three-dimensional data storage unit 30 includes a source point group data storage unit 31, a second electronic device position information storage unit 32 (child device coordinate information storage unit), and a composite point group data storage unit 33.

<Three-dimensional data processing unit 40>
The three-dimensional data processing unit 40 includes a point group data coordinate value conversion unit 41 and a synthesized point group data creation unit 42.
The composite point cloud data creation unit 42 further includes a first composite point cloud data creation unit 421 and a point cloud data matching unit 422.

<Three-dimensional data display control unit 50>
The three-dimensional data display control unit 50 includes a viewpoint operation unit 51, a data display method selection unit 52, a data reproduction / storage instruction unit 53, and a three-dimensional data display processing unit 54.

The three-dimensional data storage unit 30, the three-dimensional data processing unit 40, and the three-dimensional data display control unit 50 have a wired and / or wireless communication function, for example, one or more personal computers, tablet terminals, smartphones, servers, etc. Or an arithmetic board on which an arithmetic chip such as a CPU or DSP is mounted.
Further, the 3D data storage unit 30, the 3D data processing unit 40, and the 3D data display control unit 50 are each a distributed system including electronic devices such as different personal computers, tablet terminals, smartphones, and servers. Can do.
In addition, a part or all of the three-dimensional data storage unit 30, the three-dimensional data processing unit 40, and the three-dimensional data display control unit 50 can be configured by the first electronic device 10 or the second electronic device 20.
The three-dimensional data storage unit 30 and the three-dimensional data processing unit 40 may be on a virtual server on the cloud.

  The electronic devices constituting the three-dimensional data storage unit 30, the three-dimensional data processing unit 40, and the three-dimensional data display control unit 50 are connected to the electronic device. By installing a program for causing the three-dimensional data display control unit 50 to function, the electronic device is caused to function as the three-dimensional data storage unit 30, the three-dimensional data processing unit 40, and the three-dimensional data display control unit 50. Can do. Note that the program may be installed in the electronic device in advance. The program may be installed from a computer-readable recording medium as necessary or downloaded from a server set in advance as needed.

Next, each electronic device and the like included in the point cloud data acquisition system 100 will be described in detail.
First, the second electronic device 20 will be described.

<About the second electronic device 20>
As shown in FIG. 1, the second electronic device 20 includes a second communication unit 21, a three-dimensional measurement unit 22, a plurality of (for example, three) ultrasonic reception units 23, an ultrasonic propagation time measurement unit 26, Is provided. In addition, the second electronic device 20 can further include an infrared light receiving unit 24 and / or a time synchronization unit 25.

[Second communication unit 21]
A communication module using a wired or wireless line. It faces the first communication unit 11 of the first electronic device 10 and exchanges control signals and various data between the first electronic device 10 and the second electronic device 20.

[Three-dimensional measurement unit 22]
As a three-dimensional measuring means similar to a three-dimensional digitizer, laser scanner, etc., there is a “depth sensor”. “Depth sensor” mainly includes (1) Time-of-flight method, (2) Triangulation method, (3) Depth of Defense method, and (1) Time-of-flight method is Calculates based on the flight time to reach the sensor by irradiating and reflecting light such as laser and infrared rays on the measurement object, and calculating the linear distance to each point on the measurement object surface (hereinafter referred to as “depth”) It is also a sensor that can be obtained instantaneously. The depth sensor can grasp the depth of each point of the measurement object in units of pixels.
Here, the measurement object is not only a non-planar shape measurement object having unevenness of an existing building (for example, a plant such as a nuclear power plant or a cultural property), but also a moving human, a moving animal, An object such as an object whose shape changes over time can be included.
The three-dimensional measuring unit 22 included in the second electronic device 20 can be configured by combining a depth sensor with, for example, a normal image camera. Hereinafter, the three-dimensional measurement unit 22 configured by combining the depth sensor with a normal image camera is also referred to as a “depth camera”. By measuring the measurement object using the depth camera, it is possible to generate image data of the measurement object having both the depth information together with, for example, RGB color information for each point on the measurement object surface in pixel units.

The three-dimensional measuring unit 22 includes a three-dimensional second coordinate system (hereinafter also referred to as “unique coordinate system”) based on a unique reference position. The position of each point (pixel unit) on the surface of the measurement object measured by the three-dimensional measurement unit 22 is represented by a three-dimensional coordinate (x, y, z) by the inherent coordinate system.
Therefore, when the measurement is performed by the three-dimensional measurement unit 22, the three-dimensional coordinates (x, y, z) representing the position of each point (pixel unit) on the surface of the measurement target and the measurement target at the three-dimensional coordinate position. A set of point data composed of RGB color information on the surface can be acquired as measurement data.

  The three-dimensional measuring unit 22 can move in the three-dimensional space by moving the second electronic device 20. A user (for example, a measurement operator or a cameraman) changes the measurement position of the three-dimensional measurement unit 22 by moving the second electronic device 20 according to the state of the measurement object, and measures the measurement object. It can be performed. In addition, about the movement of the 2nd electronic device 20, the moving apparatus carrying the 2nd electronic device 20 can also be moved by self-propelled or remote operation, for example. For example, the quad copter on which the second electronic device 20 is mounted can be moved by remote control.

The three-dimensional measurement unit 22 can acquire information on each point on the surface of the measurement target in units of pixels. In addition, the three-dimensional measurement unit 22 can acquire information on each point on the surface of the measurement target in units of pixels at preset time intervals. For example, by setting 1/30 seconds as the time interval, the three-dimensional measurement unit 22 can acquire 30 frames of image data (30 fps) per second. Each measurement data measured by the three-dimensional measurement unit 22 can be given a time stamp as a measurement time, and the measurement data can be associated with each other based on the time stamp.
That is, when N (2 ≦ N) three-dimensional measuring units 22 are identified by indexes 1, 2, 3,... N, t is the measurement time, and the three-dimensional measuring unit with index (identification number) n. The point cloud data acquired by 22 (hereinafter also referred to as “three-dimensional measuring unit 22n”) can be expressed as Qn (t).

<Synchronous control of measurement time by the three-dimensional measurement unit 22>
For example, it is possible to configure the measurement object to be measured at the same time by synchronizing the measurement operation with respect to all of the plurality of three-dimensional measurement units 22 by the synchronization control by the trigger of the first electronic device 10. it can. As a trigger, for example, a measurement instruction signal can be issued from the first electronic device 10 to the three-dimensional measurement unit 22.

Here, when the image data of the measurement object is acquired by each three-dimensional measurement unit 22 at a preset time interval (for example, 30 frames per second), the point cloud data is obtained by the three-dimensional measurement unit 22n (1 ≦ 1). n ≦ N) is generated every measurement time (t) in units of 1/30 seconds, and stored in real time in the source point group data storage unit 31 of the three-dimensional data storage unit 30 via the communication network.
That is, the source point cloud data storage unit 31 stores a set of point cloud data {Qn (t)} (1 ≦ n ≦ N) (t: time stamp) measured by all the three-dimensional measuring units 22n. Is done.
The point cloud data Qn (t) represents each point on the surface of the measurement object, for example, in the data format {the identification number n of the three-dimensional measurement unit 22, the coordinate value (x in the second coordinate system) of each point (x , Y, z), point cloud acquisition time t, R (red component), G (green component), B (blue component), etc.}.

The point group data Qn (t) is obtained by matching the measurement time of the position information in the reference coordinate system (first coordinate system) of the second electronic device 20 measured by the second electronic device position information calculation unit 13 described later. ) Is measured, the measurement position of the three-dimensional measurement unit 22n can be specified.
If the measurement times do not necessarily match, linear interpolation can be performed, for example, using the measurement position of the three-dimensional measurement unit 22n near the time.

[Ultrasonic receiver 23]
The second electronic device 20 includes at least three ultrasonic receivers 23 (C1, C2, C3). The positions of the three ultrasonic wave receivers 23 (C1, C2, C3) are linear when three vectors having a start point C0 and end points C1, C2, C3, respectively, when the origin C0 of the second coordinate system is set. If they are independent, the positions of C1, C2, and C3 may be arbitrary positions.
The ultrasonic receiving unit 23 can be a non-directional receiving unit.

[Infrared light receiving unit 24]
The infrared light receiving unit 24 receives infrared rays emitted from an infrared light emitting unit 14 of the first electronic device 10 to be described later, and at the same time, a plurality of timers respectively corresponding to the three ultrasonic receiving units 23 (C1, C2, C3). Start the counter. Each of the plurality of ultrasonic reception units 23 (C1, C2, C3) receives an ultrasonic wave transmitted by each of the ultrasonic transmission units 12 (P1 to P4) of the first electronic device 10 to be described later. The corresponding timer counter is stopped by interrupt control.
Thus, infrared rays can be used as a trigger for synchronizing the timer counter between the first electronic device 10 and the second electronic device 20.
The infrared light receiving unit 24 can be, for example, a non-directional light receiving unit.

[Time synchronization unit 25]
The time synchronization unit 25 synchronizes the internal time of the second electronic device 20 with the reference time using, for example, a protocol defined in IEEE 1588. As will be described later, the first electronic device 10 similarly has a time synchronization unit 15 and synchronizes the internal time of the first electronic device 10 with a reference time using, for example, a protocol defined in IEEE 1588.
By doing so, the internal time of the 1st electronic device 10 and each internal time of the 1 or more 2nd electronic device 20 can be synchronized.

[Ultrasonic propagation time measurement unit 26]
The ultrasonic wave propagation time measuring unit 26 transmits each ultrasonic wave receiving unit 23 (C1) after ultrasonic waves are transmitted from the respective ultrasonic wave transmitting units 12 (P1, P2, P3, P4) of the first electronic device 10 described later. , C2, C3) to measure the time until reception.
A detailed procedure for measuring the ultrasonic propagation time will be described later.

<About the first electronic device 10>
As illustrated in FIG. 1, the first electronic device 10 includes a first communication unit 11, a plurality of (for example, four) ultrasonic transmission units 12, and a second electronic device position information calculation unit 13. In addition, the first electronic device 10 can further include an infrared light emitting unit 14 and / or a time synchronization unit 15.
Note that the number of the ultrasonic transmitters 12 may theoretically be three or more. However, the number of the ultrasonic transmitters 12 is four in the present embodiment because subsequent calculation processing is simplified by using four. However, the number is not limited to four.

[First communication unit 11]
A communication module using a wired or wireless line. It faces the second communication unit 21 of the second electronic device 20 and exchanges control signals and various data between the first electronic device 10 and the second electronic device 20.

[Ultrasonic transmitter 12]
The first electronic device 10 includes at least four ultrasonic transmission units 12 (P1, P2, P3, P4). The positions of the four ultrasonic transmission units 12 (P1, P2, P3, P4) may be arbitrary positions that are linearly independent.
The ultrasonic transmission unit 12 can be a non-directional transmission unit. For example, a “tubular ultrasonic transceiver” can also be applied (for example, refer to JP 2005-530370 A). Note that the present invention is not limited to the “tubular ultrasonic transceiver”. For example, a plurality of groups of four ultrasonic transmission units 12 may be provided, and an arbitrary area within the movement range of the second electronic device 20 may be covered by ultrasonic waves transmitted from at least one of the groups.
The first electronic device 10 sequentially transmits ultrasonic waves of, for example, about 40 KHz from the four ultrasonic wave transmission units 12 (P1, P2, P3, P4) at a preset time interval (for example, an interval of 6 milliseconds). can do. In this case, the position information of the ultrasonic receivers 23 (C1, C2, C3) of all the second electronic devices 20 is measured in 24 milliseconds. That is, the turnaround time required for one position information acquisition can be 24 milliseconds.

  Since the ultrasonic waves transmitted from the respective ultrasonic transmission units 12 (P1, P2, P3, P4) of the first electronic device 10 are non-directional, they are transmitted toward all the second electronic devices 20. Therefore, the time until the ultrasonic wave receiving unit 23 (C1, C2, C3) of each second electronic device 20 receives the ultrasonic wave transmitted from each ultrasonic wave transmitting unit 12 (P1, P2, P3, P4). Can be measured independently. As described above, the turnaround time required for one position information acquisition does not depend on the number of second electronic devices 20.

For this reason, when setting, for example, 1/30 second as a time interval of the measurement time of the three-dimensional measurement unit 22 in advance, the measurement time of the measurement object by the three-dimensional measurement unit 22 and the position information of the second electronic device 20 The measurement time can be synchronized.
Specifically, for example, a measurement instruction signal is transmitted to the three-dimensional measurement units 22 in all the second electronic devices 20 at the ultrasonic output time P1 of the ultrasonic transmission unit 12. This is sequentially performed as P1, P2, P3, and P4, waits after the output of P4 of the ultrasonic transmission unit 12 is finished, and synchronizes both by adjusting the ultrasonic output time of the ultrasonic transmission unit P1. Can do.

[Second Electronic Device Position Information Calculation Unit 13]
As described above, the ultrasonic wave propagation time measurement unit 26 receives the ultrasonic waves transmitted from the respective ultrasonic transmission units 12 (P1, P2, P3, P4) by the respective ultrasonic reception units 23 (C1, C2, C3). It is possible to measure the time until.
The second electronic device position information calculation unit 13 acquires the ultrasonic wave propagation time measured by the ultrasonic wave propagation time measurement unit 26 of the second electronic device 20 via, for example, a network, and each ultrasonic wave transmission unit 12 (P1 , P2, P3, P4) and the distances between the ultrasonic receiving units 23 (C1, C2, C3) can be calculated.
By doing so, the second electronic device position information calculation unit 13 is configured to perform coordinate values (hereinafter referred to as “first coordinate system”) in the reference coordinate system (first coordinate system) of the ultrasonic reception unit 23 (C1, C2, C3) of the second electronic device 20. It is also possible to calculate “second electronic device position measurement result”.
Detailed processing regarding the calculation of coordinate values in the reference coordinate system (first coordinate system) of the ultrasonic receiver 23 (C1, C2, C3) will be described later.

The second electronic device position measurement result calculated by the second electronic device position information calculation unit 13 can be attached with a time stamp indicating the reception time when the ultrasonic wave reception unit 23 (for example, C1) receives the ultrasonic wave. Based on the time stamp, the second electronic device position measurement results can be associated with each other.
That is, when N (2 ≦ N) second electronic devices 20 are identified by the same indexes 1, 2, 3,... N as the three-dimensional measuring unit 22, t is a measurement of the position of the second electronic device 20. Assuming the time, if the position information of each second electronic device 20 is represented by L, the position information at the time t of the second electronic device 20 with the identification number n can be expressed as Ln (t) or the like.

Here, the second electronic device position measurement result calculated by the second electronic device position information calculation unit 13 associates the time stamp, and as the second electronic device position information, sets the network for each second electronic device 20n. Through the second electronic device position information storage unit 32 of the three-dimensional data storage unit 30 in real time.
That is, in the second electronic device position information storage unit 32, a set {Ln (t)} (1 ≦ n ≦ N) of second electronic device position information measured at time t of all the second electronic devices 20n. (T: time stamp) is stored in real time.

As described above, the second electronic device position information set {Ln (t)} (1 ≦ n ≦ N) (t: time stamp) and the set of point cloud data measured by the three-dimensional measurement unit 22 {Qn (t )} (1 ≦ n ≦ N) (t: time stamp) By matching, the measurement position of the point cloud data can be specified.
That is, the second electronic device 20n (3) when each point group data Qn (t) is measured by matching the time t with a three-dimensional data processing unit 40 (point group data coordinate value conversion unit 41) described later. The measurement position Ln (t) of the dimension measurement unit 22n) can be specified, and each point represented by the coordinate value of the unique coordinate system (second coordinate system) of each point group data Qn (t). Can be expressed by coordinate values in the reference coordinate system (first coordinate system).
For detailed processing for representing each point represented by the coordinate value of the unique coordinate system (second coordinate system) of each point group data Qn (t) with the coordinate value of the reference coordinate system (first coordinate system), It will be described later.

[Infrared light emitting unit 14]
The infrared light emitting unit 14 emits infrared rays simultaneously when at least four ultrasonic wave transmitting units 12 (P1, P2, P3, and P4) sequentially transmit ultrasonic waves at predetermined time intervals.
The infrared light emitting unit 14 can be, for example, a non-directional light emitting unit. The infrared light emitting unit 14 includes, for example, a plurality of light emitting units, and each light emitting unit emits infrared light in a different direction at the same time, so that an arbitrary area within the moving range of the second electronic device 20 emits infrared light. You may make it cover with the infrared rays which the part 14 light-emits.

[Time synchronization unit 15]
For example, the time synchronization unit 15 synchronizes the internal time of the first electronic device 10 with the reference time using a protocol defined in IEEE 1588. As described above, the second electronic device 20 similarly includes the time synchronization unit 25, and synchronizes the internal time of the first electronic device 10 with the reference time using, for example, a protocol defined in IEEE 1588.
By doing so, the internal time of the first electronic device 10 and the internal times of the plurality of second electronic devices 20 can be synchronized.

  The first functional unit included in the first electronic device 10 and the second functional unit included in the second electronic device 20 have been described above.

  Next, a method for calculating the second electronic device position information of each second electronic device 20 will be described.

<Measurement of ultrasonic propagation time>
First, referring to FIG. 2 to FIG. 4, after the ultrasonic wave is transmitted from each ultrasonic wave transmitting unit 12 (P 1, P 2, P 3, P 4) of the first electronic device 10 by the ultrasonic propagation time measuring unit 26. A process for measuring the time until reception by each ultrasonic receiving unit 23 (C1, C2, C3) of each second electronic device 20 will be described.
In the following, the calculation of the second electronic device position information of one second electronic device 20 will be described, but all the second electronic devices 20 can be similarly calculated.

Here, in order to simplify the calculation formula, each ultrasonic receiving unit 23 (C1, C2, C3) coincides with the unique coordinate axis (second coordinate axis) of the three-dimensional measuring unit 22, and the unique coordinate axis (second coordinate axis). It is assumed that it is installed at a distance of 10 cm from the origin C0.
Referring to FIG. 2, for example, if the focal point of the lens (or the center of gravity of the camera) is the origin C0, the optical axis direction is the Z ′ axis, the horizontal direction is the X ′ axis, and the vertical direction is the Y ′ axis, then C1 , C2 and C3 are located at a fixed distance, for example, 10 cm away from the origin C0, corresponding to the Z ′ axis, the X ′ axis, and the Y ′ axis.
That is, the coordinate values in the intrinsic coordinate system (second coordinate system) of C0, C1, C2, and C3 are as follows (unit: cm).
Coordinate value of C0 = (0,0,0),
Coordinate value of C1 = (0, 0, 10),
Coordinate value of C2 = (10, 0, 0)
Coordinate value of C3 = (0, 10, 0)

Similarly, in order to simplify the calculation formula, among the four ultrasonic wave transmission units 12, P1 is installed at the origin of the reference coordinate system (first coordinate system), and P2, P3, and P4 are set as reference coordinate axes (first It is assumed that they are installed at a distance of 10 cm from the origin P1 of the reference coordinate axis (first coordinate axis) in accordance with the coordinate axes.
Referring to FIG. 2, for example, the four ultrasonic transmission units 12 (P1, P2, P3, and P4) are respectively connected to the origin, X axis, Y axis, and Z axis of the reference coordinate system (first coordinate system). The P1 to P3 are installed at the same distance, and are installed at a certain distance, for example, 10 cm away from the origin P1.
That is, the coordinate values in the reference coordinate system (first coordinate system) of P1, P2, P3, and P4 are as follows (unit: cm).
P1 coordinate value = (0,0,0),
Coordinate value of P2 = (10, 0, 0),
Coordinate value of P3 = (0, 10, 0),
P4 coordinate value = (0, 0, 10)

  FIG. 2 shows the positional relationship between the ultrasonic transmission units 12 (P1, P2, P3, P4) and the ultrasonic reception units 23 (C1, C2, C3). FIG. 3 is a sequence chart for explaining the operations of the ultrasonic transmission units 12 (P1, P2, P3, P4) and the ultrasonic reception units 23 (C1, C2, C3).

  As shown in FIG. 2, the coordinate values of the points C0, C1, C2, and C3 in the reference coordinate system (first coordinate system) are (x0, y0, z0), (x1, y1, z1), (x2, y2, z2) and (x3, y3, z3).

  As shown in FIG. 2, there are a total of 12 linear distances between each of the ultrasonic transmitters 12 (P1, P2, P3, P4) and each of the ultrasonic receivers 23 (C1, C2, C3). (1 ≦ j ≦ 4, 1 ≦ k ≦ 3). These distances Rjk can be calculated from the ultrasonic propagation time Tjk (1 ≦ j ≦ 4, 1 ≦ k ≦ 3) between the two points and the ultrasonic velocity.

  As illustrated in FIG. 3, the first electronic device 10 sequentially transmits ultrasonic waves at predetermined time intervals (for example, 6 millisecond intervals) sequentially from the ultrasonic transmission units 12 (P1, P2, P3, P4). To send. At that time, infrared light is emitted from the infrared light emitting unit 14 at the same time.

The infrared light receiver 24 receives the infrared light emitted from the infrared light emitter 14 and simultaneously starts a plurality of timer counters corresponding to the three ultrasonic receivers 23 (C1, C2, C3).
Next, each ultrasonic receiving unit 23 (C1, C2, C3) stops the corresponding timer counter by interrupt control when the ultrasonic wave transmitted by P1 of the ultrasonic transmitting unit 12 is received.
In this way, the ultrasonic wave propagation time measuring unit 26 transmits each ultrasonic wave from P1 of the ultrasonic wave transmitting unit 12 until each ultrasonic wave receiving unit 23 (C1, C2, C3) receives the ultrasonic wave. Ultrasonic propagation time measurements T11, T12, T13 can be obtained.
Similarly, as shown in FIG. 3, by repeating the measurement of the ultrasonic propagation time with P2, P3, and P4 of the ultrasonic transmission unit 12, a total of 12 ultrasonic propagation time measurement values {Tjk} (1 ≦ j ≦ 4, 1 ≦ k ≦ 3).
At this time, as described above, the time stamp which is the ultrasonic reception time of the ultrasonic receiver 23 (for example, C1) is used as the ultrasonic propagation time measurement value {Tjk} (1 ≦ j ≦ 4, 1 ≦ k ≦ 3). The second electronic device position measurement results can be associated with each other based on the time stamp.

<Modification 1 of Ultrasonic Propagation Time Measurement Unit 26>
In this embodiment, in order to measure the ultrasonic propagation time, a unidirectional wave is used instead of a reflected wave. Therefore, the ultrasonic transmission time is synchronized between the first electronic device 10 and the second electronic device 20. It is necessary to take. For this reason, infrared rays are used as a trigger for synchronizing the ultrasonic transmission time between the infrared light emitting unit 14 of the first electronic device 10 and the infrared light receiving unit 24 of the second electronic device 20.
On the other hand, for example, the first electronic device 10 and all the second electronic devices 20 include time synchronization units 15 and 25 that synchronize their internal times with the reference time using a protocol defined in IEEE 1588, for example. Thus, for example, as shown in FIG. 4, the infrared light emitting unit 14 of the first electronic device 10 and the infrared light receiving unit 24 of the second electronic device 20 can be omitted.
More specifically, for example, the ultrasonic wave propagation time measurement unit 26 is configured to transmit the ultrasonic waves from the ultrasonic transmission units 12 (P1, P2, P3, P4), respectively (s 1 (j ≦ 4)). Is acquired via a network, for example, and stored in real time. Alternatively, the ultrasonic transmission time of the ultrasonic transmission unit 12 (P1, P2, P3, P4) can be set in advance and shared with the ultrasonic propagation time measurement unit 26, for example.
The ultrasonic propagation time measuring unit 26 receives 12 ultrasonic waves received by the ultrasonic receiving units 23 (C1, C2, C3) from the ultrasonic waves transmitted by the ultrasonic transmitting units 12 (P1, P2, P3, P4). Time tjk (1 ≦ j ≦ 4, 1 ≦ k ≦ 3) is acquired.
The ultrasonic wave propagation time measurement unit 26 subtracts sj (1 ≦ j ≦ 4) from the 12 reception times tjk (1 ≦ j ≦ 4, 1 ≦ k ≦ 3), whereby each ultrasonic wave transmission unit 12 ( Ultrasonic propagation time measured values Tjk (1 ≦ j ≦ 4, 1 ≦ k) until the ultrasonic waves transmitted by P1, P2, P3, and P4) are received by the respective ultrasonic receivers 23 (C1, C2, and C3). ≦ 3) is obtained.

<Calculation of the distance between the ultrasonic transmitter 12 and the ultrasonic receiver 23>
The second electronic device position information calculation unit 13 multiplies the twelve ultrasonic propagation time measurement values (Tjk) measured by the ultrasonic propagation time measurement unit 26 by the ultrasonic velocity to thereby generate each ultrasonic transmission unit 12. A linear distance Rjk (1 ≦ j ≦ 4, 1 ≦ k ≦ 3) between (P1, P2, P3, P4) and each ultrasonic receiving unit 23 (C1, C2, C3) can be calculated.

<Calculation of position information of the three-dimensional measuring unit 22>
Next, the second electronic device position information calculation unit 13 sets each linear distance Rjk () between each ultrasonic transmission unit 12 (P1, P2, P3, P4) and each ultrasonic reception unit 23 (C1, C2, C3). 1 ≦ j ≦ 4, 1 ≦ k ≦ 3), the process of calculating the coordinate values in the reference coordinate system (first coordinate system) of each point C0, C1, C2, C3 is based on the above example. explain.
Here, the coordinate values of the points C0, C1, C2, and C3 in the reference coordinate system (first coordinate system) are (x0, y0, z0), (x1, y1, z1), (x2, y2, z2), And (x3, y3, z3).

Then, based on the three-square theorem, the following expressions 1 to 4 are established for the distance between P1 and C1, the distance between P2 and C1, the distance between P3 and C1, and the distance between P4 and C1.
x1 ² + y1 ² + z1 ² = R11 ² (Formula 1)
(X1-10) ² + y1 ² + z1 ² = R21 ² (Formula 2)
x1 ² + (y1−10) ² + z1 ² = R31 ² (Formula 3)
x1 ² + y1 ² + (z1−10) ² = R41 ² (Formula 4)

The second electronic device position information calculation unit 13 calculates the coordinate values (x1, y1, z1) in the reference coordinate system (first coordinate system) of C1 based on Equations 1 to 4.
x1 = (R21 ² −R11 ² +100) / 20
y1 = (R31 ² −R11 ² +100) / 20
z1 = (R41 ² −R11 ² +100) / 20
... (Formula 5)

Similarly, the second electronic device position information calculation unit 13 uses the coordinate values (x2, y2, z2) in the reference coordinate system (first coordinate system) of C2 as the distance between P1 and C2, and the distance between P2 and C2. Based on the distance between P3 and C2 and the distance between P4 and C2, the coordinate values (x2, y2, z2) in the reference coordinate system (first coordinate system) of C2 are calculated by Expression 6.
x2 = (R22 ² −R12 ² +100) / 20
y2 = (R32 ² −R12 ² +100) / 20
z2 = (R42 ² −R12 ² +100) / 20
... (Formula 6)

Similarly, the second electronic device position information calculation unit 13 uses the coordinate values (x3, y3, z3) in the reference coordinate system (first coordinate system) of C3 as the distance between P1 and C3, and the distance between P2 and C3. Based on the distance between P3 and C3 and the distance between P4 and C3, the coordinate value (x3, y3, z3) in the reference coordinate system (first coordinate system) of C3 is calculated by Equation 7.
x3 = (R23 ² −R13 ² +100) / 20
y3 = (R33 ² −R13 ² +100) / 20
z3 = (R43 ² −R13 ² +100) / 20
... (Formula 7)

Next, the second electronic device position information calculation unit 13 calculates the coordinate value (x0, y0, z0) in the reference coordinate system (first coordinate system) of C0 based on (Expression 8) to (Expression 10). .
(X0−x1) ² + (y0−y1) ² + (z0−z1) ² = 10 ² (Formula 8)
(X0−x2) ² + (y0−y2) ² + (z0−z2) ² = 10 ² (Equation 9)
(X0−x3) ² + (y0−y3) ² + (z0−z3) ² = 10 ² (Formula 10)

In this way, the second electronic device position information calculation unit 13 can generate the origins C0 (0, 0, 0), Z ′ in the inherent coordinate system (second coordinate system) of the three-dimensional measurement unit 22 of the second electronic device 20. A reference coordinate system (the first coordinate system of the point C1 (0, 0, 10) on the axis, the point C2 (10, 0, 0) on the X ′ axis, and the point C3 (0, 10, 0) on the Y ′ axis Each coordinate value in the coordinate system can be calculated.
In order to measure the positions of C1, C2, and C3, three ultrasonic transmitters 12 on the first electronic device 10 side are sufficient, but in this embodiment, four of P1, P2, P3, and P4 are used. did. By doing so, the calculation can be simplified and the measurement accuracy can be improved.

  As described above, the second electronic device position information calculation unit 13 can calculate the position information in the reference coordinate system (first coordinate system) of each three-dimensional measurement unit 22 included in the second electronic device 20.

  As described above, the positions of the four ultrasonic transmission units 12 (P1, P2, P3, P4) can be arbitrary positions that are linearly independent. In addition, regarding the positions of the three ultrasonic receiving units 23 (C1, C2, C3), when the origin C0 of the second coordinate system is used, three vectors having a start point C0 and end points C1, C2, and C3, respectively. However, if they are linearly independent, the positions of C1, C2, and C3 can be arbitrary positions.

  Also in this case, the second electronic device position information calculation unit 13 determines the coordinate values in the reference coordinate system (first coordinate system) of each Cj (1 ≦ j ≦ 3) as P1 and Cj (1), as described above. ≦ j ≦ 3), P2 and Cj (1 ≦ j ≦ 3), P3 and Cj (1 ≦ j ≦ 3), P4 and Cj (1 ≦ j ≦ 3) And the coordinate values in the reference coordinate system (first coordinate system) of P1, P2, P3, and P4.

  Similarly to the above, the second electronic device position information calculation unit 13 calculates the coordinate value in the reference coordinate system (first coordinate system) of the origin C0 of the unique coordinate system (second coordinate system) for each Cj (1 ≦ 1). It can be calculated based on the coordinate value in the reference coordinate system (first coordinate system) of j ≦ 3) and the coordinate value in the unique coordinate system (second coordinate system) of each Cj (1 ≦ j ≦ 3). .

  As described above, the second electronic device position information calculation unit 13 uses the reference coordinates of each ultrasonic reception unit 23 (C0, C1, C2, C3) that represents the position of the second electronic device 20 (three-dimensional measurement unit 22). A coordinate value (second electronic device position measurement result) in the system (first coordinate system) can be calculated.

As described above, the coordinate values in the reference coordinate system (first coordinate system) of each ultrasonic wave reception unit 23 (C0, C1, C2, C3) calculated by the second electronic device position information calculation unit 13 are: The second electronic device position information Ln (t) (where n is a second value) is attached to each second electronic device 20n with a time stamp that is the ultrasonic reception time of the ultrasonic receiving unit 23 (for example, C1). The identification number of the electronic device 20 and t is a time stamp) can be stored in the second electronic device position information storage unit 32 of the three-dimensional data storage unit 30 in real time.
A series of processes of the ultrasonic propagation time measurement unit 26 and the second electronic device position information calculation unit 13 can be processed in real time.

Thus, by measuring the propagation time of a plurality of (for example, four) ultrasonic waves transmitted from different positions, the three ultrasonic receiving units 23 (C1) of the second electronic device 20 (three-dimensional measuring unit 22) are measured. , C2, C3) can accurately calculate coordinate values in the reference coordinate system (first coordinate system). By doing so, the line-of-sight direction and the rotation angle around the line-of-sight of the three-dimensional measuring unit 22 are determined, and for example, more accurate than calculating the line-of-sight direction and the rotation angle around the line of sight by using, for example, an acceleration sensor or a gyro sensor as in the past. Position information can be obtained.
Next, the three-dimensional data processing unit 40 will be described.

As shown in FIG. 1, the three-dimensional data processing unit 40 includes a point group data coordinate value conversion unit 41 and a synthesized point group data creation unit 42. The composite point cloud data creation unit 42 further includes a first composite point cloud data creation unit 421 and a point cloud data matching unit 422.
<About the three-dimensional data processing unit 40>
[Point cloud data coordinate value conversion unit 41]
When n is the index (identification number) of the second electronic device 20 (three-dimensional measurement unit 22) and the measurement time is t, the point cloud data coordinate value conversion unit 41 sets the point cloud data {Qn (t)} (1 ≦ n ≦ N) (t: time stamp) A coordinate transformation matrix {Fn (t)} (1 ≦ n ≦ N) (t: time stamp) for transforming coordinates into the same coordinate system (for example, reference coordinate system) ) And a coordinate conversion from a coordinate value in the unique coordinate system (second coordinate system) representing each point of each point group data Qn (t) to a coordinate value in the reference coordinate system (first coordinate system) is performed.

Below, the point group data coordinate value conversion unit 41 performs each point group based on the second electronic device position information Ln (t) (where n is the identification number of the second electronic device 20 and t is a time stamp). Processing for calculating a transformation matrix Fn (t) for transforming coordinate values in the unique coordinate system (second coordinate system) indicating each point of the data Qn (t) into coordinate values in the reference coordinate system (first coordinate system) Will be described based on the above example.
First, the point group data coordinate value conversion unit 41 matches the time t to measure the measurement position Ln (2n) of the second electronic device 20n (three-dimensional measurement unit 22n) when each point group data Qn (t) is measured. t) is specified. Note that the measurement time when the point cloud data Qn (t) is measured and the measurement time when the measurement position Ln (t) of the second electronic device 20n (three-dimensional measurement unit 22n) is not necessarily coincident. Can be linearly complemented using, for example, the measurement position of the three-dimensional measurement unit 22n at a time near the measurement time when the point cloud data Qn (t) is measured.

  As described above, in the second electronic device 20n with the identification number (n), the coordinate value in the unique coordinate system (second coordinate system) of the ultrasonic receiving unit 23 (C1, C2, C3), C1 = (0, 0 , 10), C2 = (10, 0, 0), and point C3 = (0, 10, 0) in the reference coordinate system (first coordinate system) calculated by the second electronic device position information calculation unit 13 The coordinate values (that is, the second electronic device position information Ln (t)) are represented by C0 = (x0, y0, z0), C1 = (x1, y1, z1), C2 = (x2, y2, z2), C3, respectively. = (X3, y3, z3).

(式１１) When a coordinate transformation matrix (4 × 4 matrix) is represented by Fn (t), the following Expression 11 is established.

(Formula 11)

の逆行列Ｃ^−１を算出する。

The point cloud data coordinate value conversion unit 41

The inverse matrix C ⁻¹ is calculated.

(式１２)

このように、点群データ座標値変換部４１は、各第２電子機器位置情報Ｌｎ（ｔ）に基づいて、点群データ{Ｑｎ（ｔ）}の各点を示す固有座標系(第２座標系)における座標値を基準座標系(第１座標系)における座標値に変換するための座標変換行列Ｆｎ（ｔ）を算出することができる。 By doing so, the point group data coordinate value conversion unit 41 converts the coordinate values in the unique coordinate system (second coordinate system) indicating each point of each point group data {Qn (t)} to the reference coordinate system (first coordinate system). A transformation matrix Fn (t) for transformation into coordinate values in the system can be calculated by multiplying both sides of Equation 11 by C- ¹ .

(Formula 12)

As described above, the point group data coordinate value conversion unit 41 is based on each second electronic device position information Ln (t), and a unique coordinate system (second coordinate) indicating each point of the point group data {Qn (t)}. The coordinate conversion matrix Fn (t) for converting the coordinate values in the system) into the coordinate values in the reference coordinate system (first coordinate system) can be calculated.

  Note that the calculation of the coordinate transformation matrix Fn (t) is such that the positions of C1, C2, and C3 are arbitrary positions if the three vectors having C1, C2, and C3 as end points and the start point as C0 are linearly independent. It's okay. Also in this case, the point cloud data coordinate value conversion unit 41 can calculate the coordinate conversion matrix Fn (t) by the same procedure as described above.

(式１３)
このようにして、点群データ座標値変換部４１は、点群データＱｎ（ｔ）の各点を基準座標系(第１座標系)での座標値に変換した点群データ{Ｑ´ｎ（ｔ）}を算出することができる。 The coordinates in the intrinsic coordinate system (second coordinate system) of the point Qn (t) of the point group data are (x, y, z), and the coordinates in the reference coordinate system (first coordinate system) are (xg, yg, zg). ), The coordinate transformation is expressed by Equation 13.

(Formula 13)
In this way, the point cloud data coordinate value conversion unit 41 converts the point cloud data Qn (t) into the coordinate values in the reference coordinate system (first coordinate system) {Q′n ( t)} can be calculated.

[Composite Point Cloud Data Creation Unit 42]
The composite point cloud creation unit 42 includes a first composite point cloud data creation unit 421 and a point cloud data matching unit 422.

[First Composite Point Cloud Data Creation Unit 421]
The first composite point group data creation unit 421 uses the point group data coordinate value conversion unit 41 to calculate the coordinate values in the unique coordinate system (second coordinate system) of each point group data Qn (t) at time t. Coordinates in the reference coordinate system (first coordinate system) between the point group data {Q′n (t)} (1 ≦ n ≦ N) converted into coordinate values in the reference coordinate system (first coordinate system) By combining the points having the same value as the same point, the synthesized point group data Q ′ (t) can be created.

[Point cloud data matching unit 422]
However, even if the coordinate values in the reference coordinate system (first coordinate system) between the respective point group data {Q′n (t)} (1 ≦ n ≦ N) are the same, the position of the point group data {Q′n (t)} Deviation may occur.
For this reason, the point group data matching unit 422 uses the point group data coordinate value conversion unit 41 to convert each point group data {Q′n (t)} into coordinates in the reference coordinate system (first coordinate system). Based on the coordinate values of the points, feature parameters (for example, curvature of a curved surface (Gaussian curvature)) relating to a three-dimensional shape extracted from each point group data {Q′n (t)} (1 ≦ n ≦ N) Based on the alignment (matching) of the overlapping portions based on this, the positional deviation can be corrected.
By doing so, using a plurality of point group data {Q′n (t)} (1 ≦ n ≦ N) represented by coordinates in the same reference coordinate system (first coordinate system), more accurate It is possible to acquire composite point cloud data (hereinafter referred to as “Q (t)”) that is a three-dimensional solid with few missing portions.

It should be noted that the point cloud data Qn (ti) and Qn (ti + 1) measured by the same three-dimensional measuring unit 22n at adjacent measurement times (for example, ti, ti + 1) may be misaligned as necessary. Thus, the positional deviation can be corrected by aligning (matching) the overlapping portions.
In addition, about correction | amendment of position shift, it is good also as sampling about once per second as background processing, for example.

For positioning (matching) of the point cloud data, for example, a well-known technique called an ICP method can be used. In the ICP method, alignment is performed by minimizing (converging) the sum of squares of the distances between nearest vertices for each point group data to be aligned.
However, since the ICP method is a brute force method, an enormous amount of time is required when the amount of data increases.
There is also a method of using color information as a determination index for alignment. However, even if the color is the same, the color may appear different depending on the viewing angle and the degree of light hitting. Therefore, the point cloud data Qn (t) and Qm (t) (n ≠ n) measured from different positions. Matching using m) as a determination index was unstable.

Therefore, the point cloud data matching unit 422 uses “curvature of curved surface” indicating the characteristics of each point on the shape surface as a determination index. Consider the “degree of coincidence” between the point cloud data by considering the “curved surface” rather than the feature points such as vertices and sharp points, and using the curvature of the curved surface. Therefore, it is set as a determination index.
The curvature of the curved surface is a physical mathematical characteristic that does not depend on the coordinate system, and the degree of unevenness of the same portion of the same curved surface is the same regardless of the position (that is, which three-dimensional measurement unit 22). , Can be used as an index for determining matching. By using “curvature of curved surface”, it is assumed that the curvature of “curved surface” takes the same value regardless of which three-dimensional measurement unit 22 measures, so that the occurrence of errors during alignment is reduced. be able to. Further, since there is no need to install markers and light is not required unlike color information, it is not affected by the condition of the light source (sun, etc.).
The curvature of the curved surface includes, for example, a Gaussian curvature, an average curvature, etc. In the first embodiment, a Gaussian curvature can be applied.

In addition, preprocessing of matching processing is also important.
Specifically, the point cloud data matching unit 422 first matches a plurality of point cloud data with coordinate values in the reference coordinate system, and obtains a product set thereof, thereby limiting the range of the point cloud data to be matched. To do. As a result, the amount of calculation required for the matching process can be greatly reduced, which can contribute to an improvement in the speed of the entire system. If the product set is an empty set, the matching process does not have to be performed in the first place.
Next, the point cloud data matching unit 422 performs a point overlap of the point cloud data A and the point cloud data B on each point on the “surface with a certain extent” on the “both curved surfaces”. Curvatures are obtained and compared, and for example, an optimal conversion that minimizes the sum of the differences in curvatures is obtained while repeating a series of processes.

Here, the point cloud data matching unit 422 can calculate the curvature as follows, for example. First, the point (x ₀ , y ₀ , z ₀ ) of one point group data Qn (t) for obtaining the curvature of the curved surface is used as the origin, and the plane X including the point, for example, the reference coordinate system (first coordinate system) And a local coordinate system (x ′, y ′, z ′) including the plane X′Y ′ that is parallel to Y is set. Next, the local coordinates of the “curved surface” at a point near the point (x ₀ , y ₀ , z ₀ ) are obtained. The local coordinate value of the “curved surface” can be calculated from the point (x, y, z) of the point group data in the vicinity of the point (x ₀ , y ₀ , z ₀ ). By doing so, the relationship between the z value on the “curved surface” and (x, y) can be obtained.
For example near points adjacent value, for example _x -2 of _{x 0, x -1, x 0} , x +1, expressed as _{x +2,} etc.,
_{(X +1, y 0),} (x +2, y 0), ···
(X ₋₁ , y ₀ ), (x ₋₂ , y ₀ ),...
(X ₀ , y ₊₁ ), (x ₀ , y ₊₂ ), ...
(X ₀ , y ₋₁ ), (x ₀ , y ₋₂ ),...
And so on.

Next, a quadratic surface patch p (u, v) = (u, v, au ² + bv ² + cuv + du + ev + f) is locally applied to the measurement object surface by the least square method, and from the partial derivative of the two-dimensional surface patch, Calculate the curvature of a curved surface.
Finally, using the smooth surface of the object as a constraint, the accuracy of curvature calculation is improved by iterative calculation.
However, the method using this curvature is accurate, but requires a large amount of calculation. Or, depending on the location of the object (such as an edge portion), it may not be applied well. In that case, the conventional ICP method, a method using color information, and the like can be combined.

  As described above, the point cloud data matching unit 422 performs a series of operations for each point cloud data and performs matching by combining a plurality of point cloud data while observing the degree of matching. be able to. The result is stored in the synthesized point cloud data storage unit 33 of the three-dimensional data storage unit 30.

In this way, the point cloud data matching unit 422 uses a plurality of point cloud data (for example, {Qn (t)} (1 ≦ n ≦ N) measured by the point cloud data coordinate value conversion unit 41 at the same time. ), After aligning the overlapping portions based on the coordinate values in the reference coordinate system (first coordinate system) converted by the point cloud data coordinate value conversion unit 41, if necessary, due to measurement error or the like, Even if the coordinate values in the reference coordinate system (first coordinate system) between the respective point group data {Qn (t)} (1 ≦ n ≦ N) are the same, correction is performed in the event of positional deviation. Thus, it is possible to acquire more accurate single composite point cloud data with few missing portions.
By doing so, for example, it is possible to synthesize point cloud data at the same time of multiple viewpoints of a measurement object whose shape changes.

  In the case of a stationary measurement object whose shape does not change, the second electronic device 20n is not moved, that is, the measurement is performed at the same position where the coordinate values in the reference coordinate system (first coordinate system) are the same. By combining the plurality of point group data that are overlapped based on the coordinate values in the reference coordinate system (first coordinate system) converted by the point group data coordinate value conversion unit 41, one composition is performed. Point cloud data can be created. The result is stored in the synthesized point cloud data storage unit 33 of the three-dimensional data storage unit 30.

  In the case of a stationary measurement object whose shape does not change, a plurality of point cloud data obtained by measuring one second electronic device 20 while moving around the measurement object is represented by point cloud data. Based on the coordinate values in the reference coordinate system (first coordinate system) converted by the coordinate value conversion unit 41, one overlapping point group data can be created by aligning overlapping portions. The result is stored in the synthesized point cloud data storage unit 33 of the three-dimensional data storage unit 30.

Next, the three-dimensional data storage unit 30 will be described.
As shown in FIG. 1, the three-dimensional data storage unit 30 includes a source point group data storage unit 31, a second electronic device position information storage unit 32, and a composite point group data storage unit 33.

[Source Point Cloud Data Storage Unit 31]
The source point cloud data storage unit 31 stores, for each second electronic device 20n, point cloud data information in which the time stamp corresponding to the measurement time is associated with the point cloud data measured by each three-dimensional measurement unit 22.
As described above, for example, when each point on the surface of the measurement object is acquired by each three-dimensional measurement unit 22n in a pixel unit at a preset time interval (for example, 30 frames per second), the source point group data is stored. The unit 31 stores, for example, point group data {Qn (t)} generated at each measurement time (t) in units of 1/30 seconds for each three-dimensional measurement unit 22n (1 ≦ n ≦ N) ( t: time stamp).

[Second Electronic Device Position Information Storage Unit 32]
The second electronic device position information storage unit 32 includes coordinate values in the reference coordinate system of the plurality of ultrasonic reception units 23 (C1, C2, C3) of the second electronic device 20 and the plurality of ultrasonic reception units 23. The second electronic device position information associated with the time stamp corresponding to the time at which one ultrasonic receiving unit 23 receives the ultrasonic wave is stored for each second electronic device 20n.
As described above, the second electronic device position information storage unit 32 stores the set {Ln (t)} (1 ≦ n ≦) of the second electronic device position information measured at the time t of all the second electronic devices 20n. N) (t: time stamp) is stored.

[Combined Point Cloud Data Storage Unit 33]
As described above, the synthesized point group data storage unit 33 is, for example, a plurality of point group data (for example, {Qn (t)} (1 ≦ 1) measured by the first synthesized point group data creating unit 421 at the same time. One composite point cloud data synthesized from n ≦ N)) is stored for each time. Note that the synthesized point cloud data corrected by the point cloud data matching unit 422 can be stored as necessary.

<Three-dimensional data display control unit 50>
Next, the three-dimensional data display control unit 50 will be described.
As shown in FIG. 1, the three-dimensional data display control unit 50 includes a viewpoint operation unit 51, a data display method selection unit 52, a data reproduction / save instruction unit 53, and a three-dimensional data display processing unit 54. .

The point cloud data can be rendered in the same format, but the data format is often not suitable for various three-dimensional processes.
For this reason, the point cloud data can be converted into a surface format and can be handled as, for example, a polygon, an irregular triangular mesh, and a CAD model.
Moreover, since each point data has a three-dimensional coordinate value, the point cloud data can realize data browsing and spatial reference from a free viewpoint. Specifically, the point cloud data can be projected from the 3D space to, for example, a panoramic space using the 3D coordinate values of the point cloud data itself.

[Viewpoint operation unit 51]
The viewpoint operation unit 51 can receive input from the user such as the user's viewpoint direction and viewpoint coordinates. This enables operations similar to camera work such as zooming of the display image.

[Data display method selection unit 52]
The data display method selection unit 52 can receive an instruction regarding the display method of the display image from the user. For example, it is possible to select on which projection plane the point cloud data in the three-dimensional space is projected. For example, a panoramic model can select a projection surface such as a spherical surface, a cylinder, or a cube.

[Data playback / save instruction unit 53]
The data reproduction / storage instruction unit 53 can receive an instruction regarding reproduction or storage of the three-dimensional model data generated from the point cloud data from the user.

[Three-dimensional data display processing unit 54]
The three-dimensional data display processing unit 54 reads the composite point cloud data stored in the composite point cloud data storage unit 33, performs mapping processing based on the virtual viewpoint of the user input by the viewpoint operation unit 51, The resulting display image is displayed three-dimensionally on the display screen.
In addition, the three-dimensional data display processing unit 54 can draw a moving image in which the three-dimensional space is viewed with the viewpoint parameter designated by the viewpoint operation unit 51 using a dynamic three-dimensional model.

  FIG. 5 simply shows a series of processing flows in the embodiment of the present invention described so far.

Referring to FIG. 5, in ST1, first electronic device 10 is set at a predetermined location.
In ST2, the power of each second electronic device 20n (1 ≦ n ≦ N) is turned on. For example, the measurement operator moves the measurement object by the second electronic device 20n (three-dimensional measurement unit 22n) while moving. Start measuring.
In ST3, the point group data is acquired by each three-dimensional measuring unit 22n, and at the same time, the position acquisition distance data of each second electronic device 20n (three-dimensional measuring unit 22n) is acquired in ST4.
In ST5, position information of each second electronic device 20n (three-dimensional measurement unit 22n) in the reference coordinate system (first coordinate system) is calculated.
In ST6, the point group data Qn (t) acquired by the three-dimensional measuring unit 22n by the three-dimensional data processing unit 40 and the second electronic device position information Ln (t) calculated by the second electronic device position information calculating unit 13 are used. ).
In ST7, the three-dimensional data processing unit 40 performs matching / combination (combination) of the point group data Qn (t) at the same time t.
In ST8, the 3D display control unit 50 displays a 3D moving image viewed from the virtual viewpoint according to the user's designation of the virtual viewpoint.

  As described above, according to the first embodiment, each three-dimensional measurement unit 22 can be freely and dynamically moved, and even when a blind spot due to occlusion caused by deformation of the measurement object occurs, By changing the measurement position of the 3D measurement device immediately and responding. A three-dimensional image with few missing portions can be acquired.

  Further, according to the present invention, by such a series of processes, the synthetic point cloud data is clipped as a measurement target viewed from the viewpoint position designated and input by the viewpoint operation unit 51, and a three-dimensional object with few missing portions is obtained. Can be viewed in real time from any viewpoint.

  For example, in the medical field, an image of the condition of an affected part during surgery may be taken in order to leave it as a document video. At that time, even if it is moved to another position because the camera interferes with the operation, the viewer can always see the image from the same viewpoint regardless of the actual camera position. . Therefore, the video created as a document by this method is extremely useful.

  It can also be displayed as a video from a viewpoint hidden by the surgeon's body. In laparoscopic surgery, the state of the internal organs can be seen from the viewpoint of the outside of the patient's skin.

  In addition, by applying the present invention to a shooting camera, by moving the camera to the optimal position each time in sports live broadcasting etc., the number of blind spots is reduced with fewer blind spots, and the movement of the player from an impossible viewpoint, In various scenes, such as being able to display on each image device, it is possible to “can see an image display target object from a viewpoint different from the shooting position”.

  In addition, the composite point cloud data is placed on a server, and a plurality of users can view the same object from any viewpoint using, for example, a smartphone, a tablet terminal, a laptop computer, or the like via a network.

[Second Embodiment]
Next, the point cloud data acquisition system 100 according to the second embodiment of the present invention will be described. In the description of the second embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and the description thereof is omitted or simplified.
In the first embodiment, the first electronic device 10 includes the second electronic device position information calculation unit 13 and the second electronic device 20 includes the ultrasonic propagation time measurement unit 26. However, the present invention is limited to this. Not.
For example, the second electronic device position information calculation unit 13 may be included in the second electronic device 20. In the point cloud data acquisition system 100, the second electronic device position information calculation unit 13 may be provided separately from the first electronic device 10 and the second electronic device 20.

  When the second electronic device 20 includes the second electronic device position information calculation unit 13, the second electronic device 20 adds a time stamp to the second electronic device position measurement result calculated by the second electronic device position information calculation unit 13. Are stored in the second electronic device position information storage unit 32 of the three-dimensional data storage unit 30 for each second electronic device 20 as the second electronic device position information.

  In the point cloud data acquisition system 100, when the second electronic device position information calculation unit 13 is provided separately from the first electronic device 10 and the second electronic device 20, the second electronic device position information calculation unit 13 The ultrasonic propagation time measurement value {Tjk (1 ≦ j ≦ 4, 1 ≦ k ≦ 3)} is acquired from the time measurement unit 26 via, for example, a network, and the ultrasonic propagation time measurement value {Tjk (1 ≦ j ≦ 3) is obtained. 4, 1 ≦ k ≦ 3)}, the second electronic device position measurement result is measured, the time stamp is associated, and the second electronic device position information is stored for each second electronic device 20 as three-dimensional data. The information is stored in the second electronic device position information storage unit 32 of the unit 30.

  Similarly, the ultrasonic propagation time measurement unit 26 may be included in the first electronic device 10. Further, in the point cloud data acquisition system 100, the ultrasonic propagation time measurement unit 26 may be provided separately from the first electronic device 10 and the second electronic device 20.

  When the first electronic device 10 includes the ultrasonic wave propagation time measurement unit 26, the first electronic device 10 transmits the ultrasonic waves transmitted from the second electronic device 20 by P1 to P4 of the ultrasonic wave transmission unit 12, for example, via a network. Twelve reception times tjk (1 ≦ j ≦ 4, 1 ≦ k ≦ 3) at which the ultrasonic wave receiving units 23 (C1, C2, and C3) receive the sound waves are acquired, and the ultrasonic propagation time measurement value {Tjk ( 1 ≦ j ≦ 4, 1 ≦ k ≦ 3)}.

In the point cloud data acquisition system 100, when the ultrasonic propagation time measurement unit 26 is provided separately from the first electronic device 10 and the second electronic device 20, the ultrasonic propagation time measurement unit 26 is connected via a network, for example. From the first electronic device 10, P <b> 1 to P <b> 4 of the ultrasonic transmission unit 12 acquire transmission times sj (1 ≦ j ≦ 4) at which ultrasonic waves are transmitted, and from the second electronic device 20, the ultrasonic transmission unit 12. 12 reception times tjk (1 ≦ j ≦ 4, 1 ≦ k ≦ 3) at which the ultrasonic reception units 23 (C1, C2, C3) received the ultrasonic waves transmitted from P1 to P4 are obtained, The sound wave propagation time measurement value {Tjk (1 ≦ j ≦ 4, 1 ≦ k ≦ 3)} can be measured.
Thus, in the point cloud data acquisition system 100 of the present invention, the first electronic device 10, the second electronic device 20, the three-dimensional data storage unit 30, the three-dimensional data processing unit 40, and the three-dimensional data display control unit 50. It is a design matter that can be appropriately performed for the user to appropriately disperse the respective functional units.

[Third Embodiment]
In the first embodiment or the second embodiment, the point cloud data acquisition system 100 includes a first electronic device group having one first electronic device 10 and one or more second electronic devices 20.
On the other hand, the point cloud data acquisition system 100 has a hierarchy of a second electronic device group that does not belong to the first electronic device group and includes one first electronic device 10 and one or more second electronic devices 20. Can be prepared.

More specifically, an electronic device group having one first electronic device 10 and one or more second electronic devices 20 can form a hierarchical structure.
Referring to FIG. 6, the first electronic device 10 is provided in the (n + 1) th layer (n ≧ 1), and the first electronic device 10 in the (n + 1) th layer is viewed from the first electronic device 10 in the nth layer. It acts like the second electronic device 20.

  That is, the (n + 1) -th layer (n ≧ 1) first electronic device 10 includes an ultrasonic wave receiver 23 (C1, C2, C3) and, if necessary, an infrared light receiver 24, and the n-th layer. The ultrasonic waves transmitted from P1 to P4 of each ultrasonic wave transmission unit 12 of the first electronic device 10 are received, and the ultrasonic wave propagation time measurement unit 26 measures the ultrasonic wave propagation time measurement value {Tjk} (1 ≦ j ≦ 4). , 1 ≦ k ≦ 3), and the second electronic device position information calculation unit 13 calculates 12 linear distances {Rjk} (1 ≦ j ≦ 4, 1 ≦ k ≦ 3).

  The second electronic device position information calculation unit 13 calculates the first of the (n + 1) th layer (n ≧ 1) based on the 12 straight line distances {Rjk} (1 ≦ j ≦ 4, 1 ≦ k ≦ 3). The position information of the electronic device 10 in the first coordinate system of the first electronic device 10 in the nth layer is calculated.

  Then, the point group data coordinate value conversion unit 41 converts the coordinate value in the first coordinate system of the first electronic device 10 in the (n + 1) th layer to the coordinate value in the first coordinate system of the first electronic device 10 in the nth layer. A coordinate transformation matrix to be transformed can be calculated.

  It should be noted that the ultrasonic waves transmitted from P1 to P4 of each ultrasonic wave transmitting unit 12 of the first electronic device 10 in the nth layer and the P1 to P4 of each ultrasonic wave transmitting unit 12 of the first electronic device 10 in the (n + 1) th layer. Each of the ultrasonic waves transmitted by can transmit ultrasonic waves having different frequencies. Further, the first electronic device 10 in the nth layer (n ≧ 2) can be appropriately moved in the same manner as the second electronic device 20.

By doing so, for example, the point cloud data coordinate value conversion unit 41 converts the coordinate value in the unique coordinate system of the second electronic device 20 of the (n + 2) th layer to the first electronic device 10 of the (n + 1) th layer. Conversion into coordinate values in the coordinate system, and conversion of coordinate values in the first coordinate system of the first electronic device 10 in the (n + 1) th layer into coordinate values in the first coordinate system of the first electronic device 10 in the nth layer Can do.
By iteratively processing in this way, the coordinate value in the first coordinate system of the first electronic device 10 and the coordinate value in the unique coordinate system of the second electronic device 20 in the (n + 1) th layer or more (n ≧ 1) are obtained. It can convert into the coordinate value in the reference | standard coordinate system (1st coordinate system) of the 1st electronic device 10 in a 1st layer.

  As described above, the point cloud data acquisition system 100 is configured by hierarchically configuring a plurality of electronic device groups including one first electronic device 10 and one or more second electronic devices 20, thereby providing a three-dimensional measurement unit 22. It is possible to flexibly expand the number and the measurement position. That is, by increasing the hierarchy of the electronic device group including one first electronic device 10 and one or more second electronic devices 20, it is possible to expand the area that can be measured by the three-dimensional measuring unit 22.

  As mentioned above, although preferable embodiment of the point cloud data acquisition system 100 of this invention was described, this invention is not restrict | limited to the above-mentioned embodiment, It can change suitably.

In the above embodiment, in order to represent all measured point cloud data {Qn (t)} (1 ≦ n ≦ N) (t: time stamp) by coordinates in a unified coordinate system, the first electronic device 10 The first coordinate system is used as a unified coordinate system, but the present invention is not limited to this.
For example, a global coordinate system different from the first coordinate system of the first electronic device 10 may be applied. In this case, each point of all point group data can be represented by a coordinate value in the global coordinate system.

  In the above embodiment, it is assumed that the depth camera is applied as the three-dimensional measurement unit 22, but the present invention is not limited to this. The same applies to laser scanners, digitizers, electronic cameras, and the like.

  In the first embodiment, the measurement object is measured at the same time by synchronizing the measurement operation with respect to all of the plurality of three-dimensional measurement units 22 by the synchronous control by the trigger of the first electronic device 10. However, each three-dimensional measuring unit 22 may be configured to measure the measurement object at different times.

Each of the plurality of three-dimensional measuring units 22 can measure an object at its own timing, capture point cloud data, and store the result in an allocated area in the source point cloud data storage unit 31 in real time. it can. At that time, a time stamp is given to the point cloud data, but the time is given to each three-dimensional measuring unit 22. The performance may vary depending on the three-dimensional measuring unit 22, and even if the same target is viewed by the three-dimensional measuring unit 22 having the same performance, the timing of data acquisition is shifted if the amount of point cloud data to be acquired differs depending on the viewing angle. This is because there are cases.
However, at the stage of performing data synthesis and matching processing using a plurality of point cloud data, it is necessary to set the time of the data. Specifically, when the measurement times do not necessarily match, linear interpolation can be performed, for example, by the measurement position of the three-dimensional measurement unit 22n near the time. In addition, among a plurality of point cloud data, data having similar times can be taken out and combined and matched.
By configuring in this way, the three-dimensional measuring unit 22 can measure the measurement object at its own timing without the synchronous control by the trigger of the first electronic device 10.

  As a natural extension of this system, the addition of special cameras other than depth sensors can be considered. Cameras that can acquire temperature data such as thermography, or that can acquire various image information such as water content, sugar content, separation of cancer cells from normal cells, enhancement of blood vessels, etc. by analyzing spectral graphs, such as spectral sensing cameras Is present. By combining such a special camera and the second electronic device 20, special information can be added to the point cloud data. This enables “segmentation (significant separation of images)” necessary for recognition of the outside world by a computer.

  Accordingly, for example, recognition processing such as separation of a machine and a human body and separation of normal cells and cancer cells can be performed using a realistic three-dimensional image. These are useful techniques when performing surgery using a robot.

DESCRIPTION OF SYMBOLS 10 1st electronic device 11 1st communication part 12 Ultrasonic transmission part 13 2nd electronic device position information calculation part 14 Infrared light emission part 15 Time synchronization part 20 2nd electronic device 21 2nd communication part 22 3D measurement part 23 Ultrasonic wave Reception unit 24 Infrared light receiving unit 25 Time synchronization unit 26 Ultrasonic propagation time measurement unit 30 Three-dimensional data storage unit 31 Source point group data storage unit 32 Second electronic device position information storage unit 33 Composite point group data storage unit 40 Three-dimensional data Processing unit 41 Point cloud data coordinate value conversion unit 42 Composite point cloud data creation unit 421 First composite point cloud data creation unit 422 Point cloud data matching unit 50 3D data display control unit 51 Viewpoint operation unit 52 Data display method selection unit 53 Data reproduction / storage instruction unit 54 3D data display processing unit 100 Point cloud data acquisition system

Claims (10)

A first electronic device group including one first electronic device and a plurality of second electronic devices that can move at an arbitrary time, the surface of a subject including an object whose shape changes at least over time A point cloud data acquisition system for acquiring point cloud data having depth information for each point in pixel units,
The first electronic device is
Having a first coordinate system;
A first communication unit;
A plurality of ultrasonic transmitters for transmitting ultrasonic waves;
With
The second electronic device is
Each having a unique second coordinate system;
A second communication unit;
A plurality of ultrasonic receivers for receiving ultrasonic waves;
A time stamp indicating the measured time by continuously measuring the point cloud data of the subject based on the second coordinate system at a predetermined cycle set to have a plurality of frequencies set in advance per second. A three-dimensional measurement unit for acquiring the point cloud data with
With
The point cloud data acquisition system further includes
For each of said second electronic device, with each of the plurality of ultrasonic receiver measures the ultrasonic wave propagation time required to receive the ultrasonic wave transmitted from each of the plurality of ultrasonic wave transmission unit An ultrasonic propagation time measurement unit for obtaining the measured time;
Based on the ultrasonic wave propagation time measured by the ultrasonic wave propagation time measuring unit , for each of the second electronic devices, the second electron that is a coordinate value in the first coordinate system of the plurality of ultrasonic wave receiving units. A second electronic device position information calculating unit for calculating device position information,
For each of the second electronic devices, the second electronic device position information calculated by the second electronic device position information calculation unit and the measured time are stored corresponding to each of the second electronic devices. A second electronic device position information storage unit;
For each of the second electronic devices , a source for storing the point cloud data continuously measured at the predetermined period by the three-dimensional measuring unit corresponding to each of the second electronic devices for each time stamp A point cloud data storage unit;
Based on the second electronic device position information and the point cloud data, each point in the point cloud data included in the point cloud data continuously measured at the predetermined period for each of the second electronic devices. A point cloud data coordinate value conversion unit that converts the coordinate value of the data in the second coordinate system into a coordinate value in the first coordinate system for each time stamp;
The first coordinate system obtained by converting the plurality of point group data continuously measured at the predetermined period by the three-dimensional measurement unit included in each of the second electronic devices by the point group data coordinate value conversion unit. A composite point cloud data creation unit that creates one composite point cloud data by aligning the overlapping portions for each predetermined period based on the coordinate values at
A point cloud data acquisition system. The plurality of ultrasonic transmitters included in the first electronic device are provided at arbitrary positions that are linearly independent of the first coordinate system, respectively.
2. The point cloud data acquisition system according to claim 1, wherein each of the plurality of ultrasonic reception units included in the second electronic device is provided at an arbitrary position that is linearly independent of the second coordinate system. The second electronic device further includes:
A plurality of timer counters corresponding to each of the plurality of ultrasonic receiving units;
The point cloud data acquisition system further includes:
An infrared light emitting unit that emits infrared rays at the same time as each of the plurality of ultrasonic transmission units transmits ultrasonic waves;
An infrared receiver that starts the plurality of timer counters simultaneously with receiving the infrared light,
Each of the plurality of ultrasonic receivers further stops each of the plurality of timer counters at the time of receiving the ultrasonic wave,
The ultrasonic wave propagation time measuring unit is configured so that each of the plurality of ultrasonic wave reception units is connected to each of the plurality of ultrasonic wave transmission units based on a time from when each of the plurality of timer counters is started to when it is stopped. The point cloud data acquisition system of Claim 1 or Claim 2 which measures the ultrasonic propagation time required in order to receive the ultrasonic wave transmitted from each. Each of the first electronic device and the second electronic device further includes:
A time synchronization unit that synchronizes the internal time with the reference time,
The ultrasonic propagation time measuring unit is based on transmission time information when each of the plurality of ultrasonic transmission units transmits ultrasonic waves and reception time information when each of the plurality of ultrasonic reception units receives the ultrasonic waves. Each of the plurality of ultrasonic reception units measures an ultrasonic propagation time required to receive an ultrasonic wave transmitted from each of the plurality of ultrasonic transmission units. The point cloud data acquisition system described. The point cloud data acquisition system further includes:
The point cloud data acquisition system of any one of Claim 1 thru | or 4 provided with the synthetic | combination point cloud data storage part which memorize | stores the synthetic | combination point cloud data created by the said synthetic | combination point cloud data preparation part. The point cloud data acquisition system further includes:
  Three-dimensional drawing of the moving image in which the subject is viewed from the viewpoint set by the user by inputting the synthesized point group data corresponding to each of the predetermined periods stored in the synthesized point group data storage unit The point cloud data acquisition system according to claim 5, comprising a data display control unit. The composite point cloud data creation unit further includes:
A point group data matching unit that matches the plurality of point group data using curvatures of curved surfaces according to a three-dimensional shape respectively extracted from a plurality of different point group data having portions having the same coordinate value in the first coordinate system A point cloud data acquisition system according to any one of claims 1 to 6 . The point cloud data acquisition system further includes:
A second electronic device group comprising one first electronic device and one or more second electronic devices not belonging to the first electronic device group;
The first electronic device belonging to the second electronic device group further includes:
A plurality of ultrasonic receivers for receiving ultrasonic waves are provided.
The ultrasonic propagation time measurement unit further transmits ultrasonic waves transmitted from each of the plurality of ultrasonic transmission units of the first electronic device belonging to the first electronic device group to the second electronic device group. Measuring the ultrasonic propagation time required for each of the plurality of ultrasonic receivers included in the first electronic device to belong,
The second electronic device position information calculation unit further includes a plurality of ultrasonic waves of the first electronic device belonging to the second electronic device group based on the ultrasonic propagation time measured by the ultrasonic propagation time measurement unit. Calculating each second electronic device position information which is a coordinate value in the first coordinate system in the first electronic device belonging to the first electronic device group of the receiving unit;
The second electronic device position information storage unit further stores the second electronic device position information of the first electronic device belonging to the second electronic device group in the first electronic device belonging to the second electronic device group. Correspondingly remembered,
The source point group data storage unit further stores the point group data measured by the three-dimensional measuring instrument of the second electronic device belonging to the second electronic device group as the second electronic device belonging to the second electronic device group. The point cloud data acquisition system according to any one of claims 1 to 7, wherein the point cloud data acquisition system is stored in each of the above. A point cloud data acquisition method for acquiring point cloud data having depth information in units of pixels to each point on the surface of a subject including an object whose shape changes at least over time ,
An ultrasonic transmission step of transmitting a plurality of ultrasonic waves by a plurality of ultrasonic transmission units of the first electronic device having the first coordinate system;
A plurality of ultrasonic receiver comprises a plurality of second electronic device movable at any time having a second coordinate system, respectively, said plurality being Oite originating in the ultrasonic wave transmission step by said first electronic device An ultrasonic receiving step for receiving each of the ultrasonic waves;
The point cloud data of the subject is based on the second coordinate system at a predetermined cycle set to have a plurality of preset frequencies per second by a three-dimensional measuring unit of each of the plurality of second electronic devices. A three-dimensional measurement step of acquiring the point cloud data with a time stamp indicating the measured time ,
Each originating on the plurality of ultrasound in the ultrasonic wave transmission step, wherein in the ultrasonic receiving step, required to receive a plurality of second electronic devices each comprise a plurality of respective ultrasonic receiver Ultra An ultrasonic wave propagation time measurement step for measuring the sound wave propagation time and obtaining the measured time;
Based on the ultrasonic wave propagation time measured in the ultrasonic wave propagation time measuring step, the second electron that is a coordinate value in the first coordinate system of the plurality of ultrasonic wave receivers for each of the second electronic devices. the second electronic device position information calculated in the calculation step and the second electronic device position information calculation step, the second electronic device the measured time and the second electronic device location information for each of calculating the device location information, respectively And a second electronic device position information storing step for storing the corresponding to each of the second electronic devices;
For each of said second electronic device, the three-dimensional measurement step Oite each corresponding memory of the predetermined period in the second electronic device the point group data measured continuously for each of the time stamp A source point cloud data storing step,
Based on the second electronic device position information and the point cloud data, each point in the point cloud data included in the point cloud data continuously measured at the predetermined period for each of the second electronic devices. A point cloud data coordinate value conversion step for converting the coordinate value of the data in the second coordinate system into a coordinate value in the first coordinate system for each time stamp;
In the three-dimensional measurement step, the first coordinate system obtained by converting the plurality of point group data continuously measured at the predetermined period in each of the second electronic devices in the point group data coordinate value conversion step. A composite point cloud data creating step for creating one composite point cloud data by aligning the overlapping portions for each predetermined period based on the coordinate values at
A point cloud data acquisition method comprising:   A computer program for causing a computer including a plurality of ultrasonic transmission units and a computer including a three-dimensional measurement unit and a plurality of ultrasonic reception units to execute each step of the method according to claim 9.

Images