WO2023223916A1

WO2023223916A1 - Three-dimensional reconfiguration system and three-dimensional reconfiguration method

Info

Publication number: WO2023223916A1
Application number: PCT/JP2023/017602
Authority: WO
Inventors: 叡一松元; 泰輔橋本
Original assignee: 株式会社Preferred Networks
Priority date: 2022-05-20
Filing date: 2023-05-10
Publication date: 2023-11-23

Abstract

Provided is a three-dimensional reconfiguration system capable of easily photographing a marker irrespective of a photographing direction. This three-dimensional reconfiguration system comprises: one or more cameras that photograph an object; and a control device configured to output reconfiguration information on the object on the basis of a stereoscopic marker, a plurality of photographed images of the object photographed from various directions by the cameras, and attitude information of the cameras based on an image of the stereoscopic marker included in the photographed images. The stereoscopic marker include a marker portion that is disposed at an angle with respect to a mount surface for the stereoscopic marker.

Description

Three-dimensional reconstruction system and three-dimensional reconstruction method

The present disclosure relates to a three-dimensional reconstruction system and a three-dimensional reconstruction method.

Conventionally, three-dimensional reconstruction systems are known that generate a three-dimensional model of an object based on a plurality of images obtained by photographing the object from different directions with a camera. Such a three-dimensional reconstruction system is used in 3DCG (3 Dimensional Computer Graphics) and the like.

Furthermore, a method is disclosed in which camera posture information is acquired based on an image taken by a camera of a marker provided on the mounting surface of an object, and is used for three-dimensional reconstruction.

International Publication No. 2020/075768

However, with conventional three-dimensional reconstruction methods, when the marker is photographed at a shallow angle with respect to the mounting surface using a camera, it is not possible to identify the marker image from the photographed image, and as a result, the posture information of the camera is not obtained. There are things I can't do. If camera orientation information cannot be obtained, the accuracy of three-dimensional reconstruction of the object will be reduced.

An object of the present disclosure is to provide a three-dimensional reconstruction system in which markers are easily photographed regardless of the photographing direction of the camera.

A three-dimensional reconstruction system according to one aspect of an embodiment of the present disclosure includes one or more cameras for photographing an object, a three-dimensional marker, and a plurality of images of the object photographed from different directions by the one or more cameras. a control device configured to output reconstruction information of the object based on a captured image and posture information of the camera based on an image of the three-dimensional marker included in the plurality of captured images; The three-dimensional marker has a marker portion arranged at an angle with respect to a mounting surface of the three-dimensional marker.

According to the present disclosure, it is possible to provide a three-dimensional reconstruction system in which markers are easily photographed.

1 is a diagram illustrating the overall configuration of a three-dimensional reconstruction system according to an embodiment. FIG. 3 is a diagram illustrating a two-dimensional code formed on a three-dimensional marker according to an embodiment. It is a figure which illustrates the corner of the marker part based on embodiment. FIG. 3 is a flow diagram showing an example of processing by the control device according to the embodiment. FIG. 3 is a diagram illustrating the overall configuration of a three-dimensional reconstruction system according to a modified example. FIG. 2 is a block diagram of an example hardware configuration of a control device according to an embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In order to facilitate understanding of the description, the same components in each drawing are given the same reference numerals as much as possible, and overlapping descriptions are omitted as appropriate.

In each figure, the x direction, y direction, and z direction are directions perpendicular to each other. The z direction is a normal direction to the mounting surface, and is typically a vertical direction. The z positive direction side is written as the upper side, and the z negative direction side is written as the lower side. The x direction and the y direction are directions in which the mounting surface extends, and are typically horizontal directions.

<Configuration example of three-dimensional reconstruction system 100>
The configuration of a three-dimensional reconstruction system 100 according to the first embodiment will be described with reference to FIGS. 1 to 3. FIG. 1 is a diagram illustrating the overall configuration of a three-dimensional reconstruction system 100. FIG. 2 is a diagram illustrating a two-dimensional code formed on the three-dimensional marker 6 included in the three-dimensional reconstruction system 100. FIG. 3 is a diagram illustrating the corners 61 to 64 of the marker portion 6b of the three-dimensional marker 6.

As shown in FIG. 1, the three-dimensional reconstruction system 100 includes a support base 2,

upper cameras

3A and 3B,

lower cameras

4A and 4B, a rotating section 5, four three-dimensional markers 6, and two reference markers. 6R,

lights

8A and 8B, and a control device 9.

The three-dimensional reconstruction system 100 generates a three-dimensional model of an object based on a plurality of images obtained by photographing the object 10 from different directions using an upper camera 3A, an upper camera 3B, a lower camera 4A, and a lower camera 4B. . Examples of the object 10 include, for example, a container-shaped object such as a cup as shown in FIG. Can be mentioned.

In this specification, "object" includes both an object that actually exists and is the target of three-dimensional reconstruction, and an object that is reconstructed based on a plurality of images taken of this object. Whether the term "object" refers to an actually existing object or a reconstructed object can be appropriately distinguished depending on the context. For example, when referring to the handling of an object in real space, such as "placing an object on a support stand," "object" means an object that actually exists. On the other hand, when referring to the handling of an object in a virtual space such as "memorizing an object" or "reconstructing an object", the "object" means a reconstructed object that actually exists.

The support stand 2 is an example of a support member that supports the object 10. The support base 2 includes an upper surface 2A. In this embodiment, the support base 2 is a base on which the object 10 is placed and on which the placed object 10 is rotated. The support base 2 is, for example, a transparent plate such as an acrylic plate. The upper surface 2A corresponds to a placement surface on which the object 10 is placed. Since the support base 2 is a transparent plate, it is possible to photograph the object 10 and the three-dimensional marker 6 placed on the upper surface 2A from the lower surface 2B, which is the back side of the upper surface 2A, as shown in FIG. Since the three-dimensional reconstruction system 100 uses a transparent plate for the support base 2 and can photograph the object 10 from all directions, it is possible to obtain three-dimensional reconstruction information of the object 10 without missing information.

The upper camera 3A, the upper camera 3B, the lower camera 4A, and the lower camera 4B are examples of one or more cameras for photographing an object. Each of the upper camera 3A, the upper camera 3B, the lower camera 4A, and the lower camera 4B can photograph the object 10 placed on the upper surface 2A from different directions. The

upper cameras

3A and 3B are installed in the direction of the upper surface 2A of the support base 2 (above the mounting surface), and can photograph the object 10 by being directed toward the object 10 from diagonally above the upper surface 2A. The

upper cameras

3A and 3B are installed at different inclination angles. The

lower cameras

4A and 4B are installed toward the lower surface 2B side of the support base 2 (below the mounting surface), and can photograph the object 10 by being directed toward the object 10 from diagonally below the upper surface 2A. The

lower cameras

4A and 4B are installed at different inclination angles.

Each of the upper camera 3A, upper camera 3B, lower camera 4A, and lower camera 4B is, for example, an RGB-D camera. The RGB-D camera can capture an RGB image including each color of R (Red), G (Green), and B (Blue) of the object 10 and a depth image. The depth image is an image that includes depth information (depth) up to the object 10. Note that, hereinafter, the

upper cameras

3A and 3B will be collectively referred to as "upper camera 3", and the

lower cameras

4A and 4B will also be collectively referred to as "lower camera 4". Note that the positions of the upper camera 3 and the lower camera 4 may be rotated.

In this embodiment, the "camera" means an element that can capture an RGB image or a depth image of the object 10. This "camera" includes the entire camera device such as an RGB-D camera and a depth camera, sensors such as a CMOS sensor and a depth sensor built into the camera device, and sensors used alone.

The upper camera 3A, the upper camera 3B, the lower camera 4A, and the lower camera 4B each have an internal parameter and an external parameter. The internal parameters include information related to distortion of the lens included in each camera. In this embodiment, the internal parameters are known based on simulation results and the like. The external parameters include posture information of each camera. The attitude of the camera includes a relative attitude of the camera with respect to the three-dimensional marker 6 and an absolute attitude in the world coordinate system. Further, the attitude of the camera corresponds to the inclination of the optical axis of an optical system such as a lens included in each camera. The external parameters are acquired using the three-dimensional marker 6.

The rotating unit 5 is an example of a rotating mechanism that rotates the object 10 by rotating the support base 2. The rotating unit 5 rotates the support base 2 in the direction of an arrow 50, for example. Each of the upper camera 3A, the upper camera 3B, the lower camera 4A, and the lower camera 4B photographs the object 10 at each of a plurality of rotation angles by the rotation unit 5.

A known power transmission system can be used for the mechanism of the rotating part 5. For example, the rotating section 5 includes a motor and a gear mechanism. The rotating part 5 may be configured such that the driving force of the motor is transmitted to the rotating shaft of the support base 2 via a gear mechanism. Alternatively, the rotating unit 5 may be configured to apply a driving force to the outer edge of the support base 2 to rotate the support base 2 .

The three-dimensional marker 6 is arranged around the object 10 on the upper surface 2A. The three-dimensional marker 6 is a mark used to recognize the relative position of the object 10 and each of the upper camera 3A, upper camera 3B, lower camera 4A, and lower camera 4B. The three-dimensional marker 6 is placed on the upper surface 2A. The three-dimensional marker 6 is not fixed to the upper surface 2A, and its position and orientation with respect to the upper surface 2A are variable. However, since the three-dimensional marker 6 is not moved while the support base 2 is rotating to photograph the object 10 by the upper camera 3 and the lower camera 4, the position and orientation of the three-dimensional marker 6 with respect to the upper surface 2A remain unchanged. becomes.

There is no particular limit to the number of three-dimensional markers 6. However, if the number of 3D markers 6 is small, the 3D marker 6 may be hidden behind the object 10 during shooting, and the posture information of the upper camera 3 and lower camera 4 may not be properly acquired. The larger the number, the better. Therefore, it is preferable that a plurality of three-dimensional markers 6, including two three-dimensional markers 6 placed before and after the object 10, be arranged around the object 10 in the photographing direction of the upper camera 3 or the lower camera 4. In this embodiment, the number of three-dimensional markers 6 is four, as an example. Note that the three-dimensional markers 6 in FIG. 1 each have a different two-dimensional code.

The three-dimensional marker 6 includes a three-dimensional member 6a and a marker portion 6b arranged at an angle with respect to the mounting surface of the three-dimensional marker 6. In this embodiment, the mounting surface of the object 10 may be the same as the mounting surface of the three-dimensional marker 6. That is, the mounting surface of the three-dimensional marker 6 may be the upper surface 2A. The above-mentioned angle may be an angle that is not parallel to the upper surface 2A, and may be an angle determined according to the shooting direction of each camera so that the marker portion 6b can be detected from the RGB image captured by each camera. The three-dimensional member 6a is a three-dimensional member having a predetermined height with respect to the upper surface 2A, and has a shape for arranging the marker portion 6b at an angle with respect to the upper surface 2A. There is no particular restriction on the material of the three-dimensional member 6a, and the three-dimensional member 6a can be made of resin, metal, wood, paper, or the like. The three-dimensional member 6a may be a hollow box-like member or may be a solid member that is not hollow. In this embodiment, as an example, the three-dimensional member 6a is a polyhedron including six sides, and is a cubic box-shaped member in which all six sides have the same shape.

The marker portions 6b are provided on each of six surfaces (one top surface, four side surfaces, and one bottom surface) of the three-dimensional member 6a. The marker portion 6b provided on the side surface of the three-dimensional member 6a is arranged at an angle (90° in this embodiment) with respect to the upper surface 2A. The marker portion 6b provided on the bottom surface of the three-dimensional member 6a is arranged so that its surface is in contact with the top surface 2A, but since the support base 2 is transparent, the marker portion 6b on the bottom surface can be detected by the lower camera 4. You can take pictures through it. As shown in FIG. 2, the marker unit 6b uses a two-dimensional code such as an AR marker, which is an image of a fixed pattern that serves as a marker in an image recognition type AR (Augmented Reality) system. Alternatively, a two-dimensional code such as a QR code (registered trademark) or another two-dimensional code may be used. This embodiment will be described assuming that an AR marker is used for the marker portion 6b.

A separate marker ID is assigned to each surface of the marker portion 6b, and the marker ID can be recognized by a known image recognition technique. A planar AR marker is used. Further, the AR marker is used to calculate the distance, angle, etc. with respect to the camera from the shape of the image, such as the degree of distortion when it is captured by the camera. Alternatively, the AR marker is used to display 3DCG or the like at the marker position based on the acquired information.

The marker section 6b is rectangular and includes a two-dimensional code (two-dimensional bit pattern) in two colors, black and white. However, the marker section 6b may include a one-dimensional code such as a barcode instead of the two-dimensional code. In the four three-dimensional markers 6 arranged around the object 10, the marker portions 6b provided on the six surfaces of the three-dimensional member 6a each have a different two-dimensional code. Therefore, the four three-dimensional markers 6 include a total of 24 types of AR markers with different bit patterns.

The marker part 6b may be one in which an AR marker is printed on each surface of the three-dimensional member 6a, or a plate-like member or a sheet-like member on which an AR marker is formed is attached to each surface of the three-dimensional member 6a with an adhesive member. It may be fixed by, for example. Note that the three-dimensional marker 6 may be shaped using a 3D printer so that an AR marker appears on each surface.

Identification information of the marker section 6b is recorded in the two-dimensional code. The identification information of the marker portion 6b may be, for example, an identification number indicating which marker portion of which three-dimensional marker it is, or may be, for example, a serial identification number of the marker portion 6b that does not distinguish between three-dimensional markers. It may be.

In each marker portion 6b, the bit pattern of the two-dimensional code and the orientation of the marker portion 6b are associated in advance, and the orientation of the bit pattern directly or indirectly indicates the orientation of the marker portion 6b. That is, as shown in FIG. 3, each marker portion 6b has corners 61 to 64 at its four corners that can be distinguished from each other and represent its orientation. In the identification of the marker portion 6b, each of the corners 61 to 64 is recognized as being different from each other. For example, the four corners of each marker portion 6b can be identified by mutually different identification numbers 1 to 4, such as “the corner with identification number 4 in the marker portion 6b with identification number 1”.

The two reference markers 6R are an example of two flat markers that are fixed to the upper surface 2A and are flat markers parallel to the upper surface 2A. The reference marker 6R also has a marker part having a two-dimensional code in which its own identification information is recorded, and the marker part of the reference marker 6R also has four corners that indicate its orientation, similar to the marker part 6b of the three-dimensional marker 6. has. The reference marker 6R is also an AR marker, for example. The two reference markers 6R may be AR markers printed at predetermined positions on the support base 2, or plate-like members or sheet-like members on which AR markers are formed are placed on each surface of the three-dimensional member 6a. It may be fixed with an adhesive member or the like.

The two reference markers 6R are provided to define a world coordinate system used in three-dimensional reconstruction. The distance between the two reference markers 6R is predetermined. This distance is used to determine the length of object 10. The upper surface 2A on which the two reference markers 6R are provided corresponds to the xy plane in the world coordinate system. The center position between the two reference markers 6R becomes the origin of the world coordinate system.

In this embodiment, the reference marker 6R, which is a planar marker, is illustrated, but the reference marker may be two or more of the three-dimensional markers 6. That is, at least two of the plurality of three-dimensional markers 6 included in the three-dimensional reconstruction system 100 may be fixed to the upper surface 2A and may be reference markers having a predetermined distance between them.

The illumination 8A is arranged in the direction toward the upper surface 2A of the support base 2. The illumination 8B is arranged toward the lower surface 2B of the support base 2. The

lights

8A and 8B each illuminate the object 10 with light. The illumination lights 8A and 8B are arranged according to the installation positions of the upper camera 3 and the lower camera 4 so that there is no shadow on the surface of the object 10, especially in the images taken by the upper camera 3 and the lower camera 4.

The control device 9 controls the operation of the three-dimensional reconstruction system 100. In the present embodiment, the control device 9 captures a plurality of captured images of the object 10 captured from different directions by the upper camera 3A, the upper camera 3B, the lower camera 4A, and the lower camera 4B, and a plurality of captured images included in the plurality of captured images. It is configured to output reconstruction information of the object 10 based on the posture information of each camera based on the image of the three-dimensional marker 6. Further, the control device 9 uses the RGB image and the depth image from the upper camera 3 and the lower camera 4, and the posture information of the upper camera 3 and the lower camera 4 based on the image of the three-dimensional marker 6 included in the RGB image. Reconstruction information of the object 10 can be output.

Specifically, the control device 9 controls photographing of the object 10 by the upper camera 3 and the lower camera 4. Further, the control device 9 reconstructs the object 10 by generating a three-dimensional model of the object 10 based on the captured image of the object 10. The control device 9 includes an imaging control section 91, an attitude calculation section 92, and a model generation section 93 as functions related to these.

The photographing control section 91 controls the rotation so that the upper camera 3 and the lower camera 4 photograph the object 10 during rotation by the rotating section 5 to obtain a plurality of photographed images (in this embodiment, an RGB image and a depth image). 5, the upper camera 3, and the lower camera 4.

The attitude calculation unit 92 calculates the attitude information of the upper camera 3A, the upper camera 3B, the lower camera 4A, and the lower camera 4B based on the images of the three-dimensional marker 6 included in the plurality of captured images by the upper camera 3 and the lower camera 4. .

The model generation unit 93 generates a three-dimensional image of the object 10 based on a plurality of captured images captured by the upper camera 3 and the lower camera 4 and the attitude information of the upper camera 3 and the lower camera 4 acquired by the attitude calculation unit 92. The object 10 is reconstructed by generating the original model. The model generation unit 93 outputs the generated three-dimensional model of the object 10 as reconstruction information of the object 10.

For example, the model generation unit 93 can reconstruct the object 10 by generating a model shape using a depth image and generating a texture using an RGB image. However, the three-dimensional reconstruction method by the model generation unit 93 is not limited to this method using depth images. For example, the model generation unit 93 uses a three-dimensional reconstruction method of the object 10 using so-called photogrammetry, or an error between an image (two-dimensional image) obtained by rendering current three-dimensional information of the object 10 and a photographed image. The object 10 can also be reconstructed using a three-dimensional reconstruction method using machine learning that updates the three-dimensional information based on the following.

<Example of processing by control device 9>
FIG. 4 is a flowchart showing an example of processing by the control device 9. For example, the control device 9 starts the process shown in FIG. 4 when receiving an operation input from the user to instruct the start of reconfiguration through its operation unit. Before the process in FIG. 4 is started, the object 10 is placed on the upper surface 2A of the support base 2. The mounting position of the object 10 is preferably the center of rotation of the support base 2. In this embodiment, as shown in FIG. 1, the top surface 2A has a substantially circular shape, and the three-dimensional marker 6 is arranged approximately concentrically with the outer edge of the top surface 2A, so that the object 10 is placed at the center of the top surface 2A. It is preferable that

First, in step S41, the control device 9 causes the photographing control section 91 to drive the rotation section 5 and cause the support base 2 to start rotating.

Subsequently, in step S42 (photographing step), the control device 9 causes the photographing control unit 91 to operate the upper camera 3 and the lower camera 4 to photograph the rotating object 10 from both the upper and lower sides. Each camera of the lower camera 4 acquires an RGB image and a depth image. The photographing control unit 91 causes the upper camera 3 and the lower camera 4 to perform photographing at the same timing or at different timings while the object 10 is rotating. Thereby, the photographing control unit 91 can acquire RGB images of each camera in parallel when the object 10 is at a position of an arbitrary rotation angle. At this time, the photographing control unit 91 also acquires depth images from each camera. The depth image is represented by a group of points that are colored according to the distance from the camera to the surface of the object 10 at each pixel.

In step S42, the photographing control unit 91 continues the photographing operation and acquires an image while the rotating unit 5 rotates the object 10 once. Thereby, each of the upper camera 3 and the lower camera 4 can photograph the object 10 from directions inclined at different angles with respect to the upper surface 2A, and from a plurality of directions along the rotation direction. Here, the upper camera 3 whose photographing range includes the upper surface of the three-dimensional marker 6 can photograph the marker portion 6b on the upper surface of the three-dimensional marker 6. Moreover, since the three-dimensional marker 6 is used, even if the shooting direction of the upper camera 3 and the lower camera 4 is at a shallow angle with respect to the upper surface 2A, the upper camera 3 and the lower camera 4 The same marker portion 6b arranged at an angle with respect to the upper surface 2A can be photographed. Further, the lower camera 4 can photograph the marker portion 6b on the bottom surface of the three-dimensional marker 6 from below the transparent support base 2. In this way, each camera can acquire an RGB image including an image of the marker section 6b and a depth image in each photographing direction. That is, the photographing control unit 91 can simultaneously photograph the object 10 from different directions using a plurality of cameras. The photographing control section 91 outputs the photographed image to each of the posture calculation section 92 and the model generation section 93.

Subsequently, in step S43, the control device 9 causes the photographing control section 91 to stop the rotation section 5 and stop the rotation of the support base 2.

Subsequently, in step S44 (posture information estimation step), the control device 9 uses the posture calculation section 92 to determine the position relative to the reference marker 6R, that is, the world coordinate system, based on the image of the three-dimensional marker 6 (marker section 6b) in the RGB image. Estimate the posture information of each of the upper camera 3 and the lower camera 4 in .

The procedure for estimating the posture information of each camera in step S44 will be explained. The posture calculation unit 92 detects the marker portions (the marker portion 6b of the three-dimensional marker 6 and the marker portion of the reference marker 6R) photographed by the camera. In this detection, the posture calculation section 92 specifies the position of each of the four corners 61 to 64 in the camera coordinate system for each of the marker sections being photographed. In addition, in this detection, the posture calculation section 92 specifies the identification number of the marker section being photographed.

The posture calculation unit 92 calculates the position of the marker portion based on the positions in the camera coordinate system of the corners 61 to 64 of the marker portion photographed by the camera, the known size of the marker portion, and the internal parameters of the camera. The relative attitude (relative position and relative rotation angle) of the camera relative to the camera can be calculated. That is, the attitude calculation unit 92 can calculate the relative attitude of the camera with respect to the marker part based on how the marker part looks. The positions of the corners 61 to 64 in each marker portion in the camera coordinate system are information representing how the marker portion is viewed.

The attitude calculation unit 92 calculates the attitude (absolute attitude) of each camera based on the relative attitude of each camera with respect to the marker unit, for example, using one reference marker 6R as a reference, thereby calculating the attitude indicating the absolute attitude of all the cameras. Estimate information. An example of this calculation will be explained below. First, the attitude calculation unit 92 regards the relative attitude of a certain camera (herein referred to as camera 1) that is photographing (detecting) the marker portion of the reference marker 6R as the absolute attitude of the camera 1. This makes it possible to estimate the absolute orientation of the camera 1 with reference to the reference marker 6R. Next, the posture calculation unit 92 identifies another camera (herein referred to as camera 2) that is photographing the marker portion 6b of the three-dimensional marker 6 photographed by the camera 1, and determines the absolute posture of the camera 1. Based on the relative postures of the

cameras

1 and 2 with respect to the marker portion 6b, the absolute posture of the camera 2 with respect to the reference marker 6R is calculated. By performing such calculations on all cameras, the pose calculation unit 92 can estimate the pose information indicating the absolute poses of all the cameras. Note that when a plurality of marker parts 6b are included in the RGB image taken by one camera, the attitude calculation part 92 calculates the absolute attitude of the camera that minimizes the error with respect to the relative attitude with respect to each marker part 6b. The accuracy of estimating the absolute orientation of the camera may be improved by estimating the absolute orientation of the camera, that is, by optimizing the absolute orientation of the camera. The posture calculation section 92 outputs the estimated posture information to the model generation section 93.

Subsequently, in step S45 (model generation step), the control device 9 causes the model generation unit 93 to use the information acquired in step S42 based on the posture information of each of the upper camera 3 and the lower camera 4 with respect to the reference marker 6R. The depth images of the object 10 taken in each shooting direction are combined to generate a three-dimensional model of the object 10. The attitude information of each camera here is information on the absolute attitude of each camera. The model generation unit 93 can generate a three-dimensional model with a pattern or color by creating a three-dimensional mesh model based on the depth image, for example, and pasting a texture created based on the RGB image on the surface of the mesh model.

Subsequently, in step S46, the control device 9 causes the model generation unit 93 to output the generated three-dimensional model as reconstruction information, and stores it in the ROM, auxiliary storage device, etc. of the control device 9. Note that the model generation unit 93 can also output the reconstruction information to an external device such as a display device or a PC (Personal Computer). When the process of step S46 is completed, this process flow ends.

Note that the control device 9 may collectively execute each process of steps S44 and S45 using the RGB image and depth image of the object 10 at each position in the rotational direction acquired in step S42. That is, the functions of the attitude calculation section 92 and the model generation section 93 in the control device 9 may be integrated.

<Effects of the three-dimensional reconstruction system 100>
As explained above, the three-dimensional reconstruction system 100 includes an upper camera 3 and a lower camera 4 (cameras) that can photograph the object 10 placed on the upper surface 2A (placement surface) from different directions, and The three-dimensional marker 6 is arranged around the object 10. Furthermore, the three-dimensional reconstruction system 100 uses the RGB image and the depth image (a plurality of captured images) by the upper camera 3 and the lower camera 4, and the image of the three-dimensional marker 6 included in the RGB image and the depth image. It has a control device 9 configured to output reconstruction information of the object 10 based on posture information of the camera 4 . The three-dimensional marker 6 has a marker portion 6b arranged at an angle with respect to the upper surface 2A. The three-dimensional marker 6 has a three-dimensional member 6a having a predetermined height with respect to the upper surface 2A, and the marker portion 6b is provided on the three-dimensional member 6a.

Conventional three-dimensional reconstruction methods use planar markers parallel to the mounting surface, so when the marker is photographed with a camera at a shallow angle to the mounting surface, the marker can be easily detected from the photographed image. Images may not be identified and, as a result, camera pose information may not be obtained. The shallow angle with respect to the mounting surface means, for example, an angle in which the inclination angle with respect to the mounting surface is 15 degrees or less. If camera orientation information cannot be obtained, the accuracy of three-dimensional reconstruction of the object will be reduced.

In this embodiment, a photographed image of the marker portion 6b arranged at an angle with respect to the upper surface 2A is used. Therefore, even when photographing the three-dimensional marker 6 at a shallow angle with respect to the upper surface 2A, the three-dimensional reconstruction system 100 can capture an image of the marker portion 6b provided on either surface of the three-dimensional marker 6. can be included in the camera, and camera pose information can be obtained. Thereby, in this embodiment, it is possible to provide the three-dimensional reconstruction system 100 in which markers are easily photographed regardless of the photographing direction of each of the upper camera 3 and the lower camera 4.

Moreover, in this embodiment, it has a support base 2 (support member) and a rotating part 5 (rotating mechanism), and each camera of the upper camera 3 and the lower camera 4 is Object 10 is photographed. With this configuration, the object 10 placed on the upper surface 2A (placing surface) can be photographed from different directions, and the object 10 can be three-dimensionally reconstructed using the photographed images. Note that in this embodiment, the three-dimensional reconstruction system 100 only needs to have one or more cameras.

Furthermore, in this embodiment, the three-dimensional member 6a is a cube having six planes (a polyhedron including a plurality of planes). The marker portions 6b are provided on two or more of the six planes of this cube. With this configuration, it is possible to increase the accuracy with which the image of the marker section 6b is included in the photographed image, so that three-dimensional reconstruction allows the marker to be easily photographed, regardless of the photographing directions of the upper camera and the lower camera 4. A system 100 can be provided.

Furthermore, in this embodiment, the marker section 6b has a two-dimensional code that records identification information of the marker section 6b. With this configuration, camera attitude information can be acquired no matter which marker portion 6b included in the three-dimensional marker 6 is included in the captured image. Thereby, it is possible to provide the three-dimensional reconstruction system 100 in which markers can be easily photographed regardless of the photographing directions of the upper camera 3 and the lower camera 4. Note that the marker section 6b may include a one-dimensional code instead of a two-dimensional code.

Furthermore, in this embodiment, the three-dimensional reconstruction system 100 has at least two reference markers 6R (planar markers) that are fixed to the upper surface 2A and are planar markers parallel to the upper surface 2A. The distance between the two reference markers 6R is predetermined. The three-dimensional reconstruction system 100 can calibrate the size information of the marker section 6b based on the distance information between the two reference markers 6R included in the photographed image. As a result, camera posture information can be acquired accurately, so that the reconstruction accuracy by the three-dimensional reconstruction system 100 can be improved.

Note that instead of the reference marker 6R, which is a flat marker, one of the three-dimensional markers 6 may be used as the reference marker, or two three-dimensional markers 6 with a predetermined distance between them may be used as the reference marker. may be used as a pair of reference markers. When using a flat marker as the reference marker, the reference marker can be formed on the support base 2 in advance by printing or the like, which is advantageous in that the reference marker can be easily provided. When a three-dimensional marker is used as a reference marker, it is advantageous in that the reference marker can be photographed regardless of the photographing direction of the camera.

<Modified example>
A three-dimensional reconstruction system 100a according to a modification will be described. Note that the same components as those in the embodiment described above are given the same reference numerals, and redundant explanations will be omitted as appropriate.

FIG. 5 is a diagram illustrating the overall configuration of the three-dimensional reconstruction system 100a. The three-dimensional reconstruction system 100a uses an opaque floor 11 as a mounting surface, and images an object 10 mounted on the floor 11 from different directions using

upper cameras

3A and 3B. In other words, the three-dimensional reconstruction system 100a includes the

upper cameras

3A and 3B as a plurality of cameras each capable of photographing the object 10 from different directions.

The control device 9 determines the object based on the plurality of images taken by the

upper cameras

3A and 3B and the attitude information of each of the

upper cameras

3A and 3B based on the image of the three-dimensional marker 6 included in these plurality of images. 10 reconstruction information is output.

The three-dimensional reconstruction system 100a uses a photographed image of a marker section 6b provided on a three-dimensional member 6a having a predetermined height with respect to the floor 11. For this reason, in the three-dimensional reconstruction system 100a, even when photographing the three-dimensional marker 6 at a shallow angle with respect to the floor 11, the image of the marker section 6b provided on either surface of the three-dimensional marker 6 is reflected in the photographed image. It is possible to obtain camera pose information. Thereby, in this modification, it is possible to provide a three-dimensional reconstruction system 100 that can acquire posture information of each camera, regardless of the shooting direction of each of the

upper cameras

3A and 3B.

Moreover, since the three-dimensional reconstruction system 100a does not include the support base 2, the rotating part 5, etc., the object 10 can be three-dimensionally reconstructed with a simple configuration. Further, the three-dimensional reconstruction system 100a can three-dimensionally reconstruct the object 10 placed on an arbitrary mounting surface such as the floor 11, and can improve the usability of the three-dimensional reconstruction system 100a.

<Example of hardware configuration in the embodiment and modification described above>
Part or all of each device (the three-

dimensional reconstruction systems

100 and 100a) in the embodiments described above may be configured with hardware, or may include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc. It may be configured by information processing of software (program) to be executed. In the case where the information processing is configured by software, the software that realizes at least some of the functions of each device in the above-described embodiments can be stored in a CD-ROM (Compact Disc-Read Only Memory), USB (Universal Serial Bus) memory, etc. The information processing of the software may be executed by storing the information in a non-temporary storage medium (non-temporary computer readable medium) such as the following, and reading it into a computer. Further, the software may be downloaded via a communication network. Furthermore, all or part of the software processing may be implemented in a circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array), so that the information processing by the software may be executed by hardware. .

The storage medium that stores the software may be a removable one such as an optical disk, or a fixed storage medium such as a hard disk or memory. Further, the storage medium may be provided inside the computer (main storage device, auxiliary storage device, etc.) or may be provided outside the computer.

FIG. 6 is a block diagram showing an example of the hardware configuration of each device (three-

dimensional reconstruction systems

100 and 100a) in the embodiment described above. Each device includes, for example, a processor 71, a main storage device 72 (memory), an auxiliary storage device 73 (memory), a network interface 74, and a device interface 75, which are connected via a bus 76. It may also be realized as a computer 7.

Although the computer 7 in FIG. 6 includes one of each component, it may include a plurality of the same components. Furthermore, although one computer 7 is shown in FIG. 6, the software may be installed on multiple computers, and each of the multiple computers may execute the same or different part of the software. Good too. In this case, it may be a form of distributed computing in which each computer communicates via the network interface 74 or the like to execute processing. In other words, each device (three-

dimensional reconstruction system

100 and 100a) in the embodiment described above is a system that realizes functions by one or more computers executing instructions stored in one or more storage devices. It may be configured as Alternatively, the information transmitted from the terminal may be processed by one or more computers provided on the cloud, and the processing results may be sent to the terminal.

Various operations of each device (three-

dimensional reconstruction systems

100 and 100a) in the embodiments described above are executed in parallel using one or more processors or multiple computers via a network. Good too. Further, various calculations may be distributed to a plurality of calculation cores within the processor and executed in parallel. Further, a part or all of the processing, means, etc. of the present disclosure may be realized by at least one of a processor and a storage device provided on a cloud that can communicate with the computer 7 via a network. In this way, each device in the embodiments described above may be in the form of parallel computing using one or more computers.

The processor 71 may be an electronic circuit (processing circuit, processing circuit, CPU, GPU, FPGA, ASIC, etc.) that performs at least one of computer control or calculation. Further, the processor 71 may be any of a general-purpose processor, a dedicated processing circuit designed to execute a specific operation, or a semiconductor device including both a general-purpose processor and a dedicated processing circuit. Further, the processor 71 may include an optical circuit or may include an arithmetic function based on quantum computing.

The processor 71 may perform calculation processing based on data and software input from each device in the internal configuration of the computer 7, and may output calculation results and control signals to each device. The processor 71 may control each component constituting the computer 7 by executing the OS (Operating System) of the computer 7, applications, and the like.

Each device (the three-

dimensional reconstruction systems

100 and 100a) in the embodiments described above may be realized by one or more processors 71. Here, the processor 71 may refer to one or more electronic circuits arranged on one chip, or one or more electronic circuits arranged on two or more chips or two or more devices. You can also point. When using multiple electronic circuits, each electronic circuit may communicate by wire or wirelessly.

The main memory device 72 may store instructions and various data to be executed by the processor 71, and the information stored in the main memory device 72 may be read by the processor 71. The auxiliary storage device 73 is a storage device other than the main storage device 72. Note that these storage devices are any electronic components capable of storing electronic information, and may be semiconductor memories. Semiconductor memory may be either volatile memory or nonvolatile memory. The storage device for storing various data in each device (three-

dimensional reconstruction systems

100 and 100a) in the embodiments described above may be realized by the main storage device 72 or the auxiliary storage device 73, and may be implemented by the main storage device 72 or the auxiliary storage device 73. It may also be realized by a built-in memory.

Each of the devices (three-

dimensional reconstruction systems

100 and 100a) in the embodiments described above includes at least one storage device (memory) and at least one processor connected to (coupled with) this at least one storage device. In this case, at least one processor may be connected to one storage device. Furthermore, at least one storage device may be connected to one processor. Furthermore, the present invention may include a configuration in which at least one processor among the plurality of processors is connected to at least one storage device among the plurality of storage devices. Further, this configuration may be realized by a storage device and a processor included in a plurality of computers. Furthermore, a configuration in which the storage device is integrated with the processor (for example, a cache memory including an L1 cache and an L2 cache) may be included.

The network interface 74 is an interface for connecting to the communication network 8 wirelessly or by wire. As the network interface 74, an appropriate interface such as one that complies with existing communication standards may be used. Information may be exchanged with an external device 9A connected via the communication network 8 through the network interface 74. The communication network 8 may be any one or a combination of WAN (Wide Area Network), LAN (Local Area Network), PAN (Personal Area Network), etc., and can be used to communicate between the computer 7 and the external device 9A. It may be anything that involves the exchange of information. Examples of WAN include the Internet, examples of LAN include IEEE802.11 and Ethernet (registered trademark), and examples of PAN include Bluetooth (registered trademark) and NFC (Near Field Communication).

The device interface 75 is an interface such as a USB that is directly connected to the external device 9B.

The external device 9A is a device connected to the computer 7 via a network. The external device 9B is a device directly connected to the computer 7.

The external device 9A or the external device 9B may be an input device, for example. The input device is, for example, a camera, a microphone, a motion capture device, various sensors, a keyboard, a mouse, a touch panel, or other devices, and provides the acquired information to the computer 7. Alternatively, the device may be a device including an input unit, a memory, and a processor, such as a personal computer, a tablet terminal, or a smartphone.

Additionally, the external device 9A or the external device 9B may be an output device, for example. The output device may be, for example, a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) panel, or may be a speaker that outputs audio or the like. Alternatively, the device may be a device including an output unit, a memory, and a processor, such as a personal computer, a tablet terminal, or a smartphone.

Furthermore, the external device 9A or the external device 9B may be a storage device (memory). For example, the external device 9A may be a network storage or the like, and the external device 9B may be a storage such as an HDD.

Further, the external device 9A or the external device 9B may be a device that has some functions of the components of each device (three-

dimensional reconstruction systems

100 and 100a) in the above-described embodiments. In other words, the computer 7 may transmit some or all of the processing results to the

external device

9A or 9B, or may receive some or all of the processing results from the

external device

9A or 9B. .

In this specification (including claims), the expression "at least one (one) of a, b, and c" or "at least one (one) of a, b, or c" (including similar expressions) When used, it includes either a, b, c, a-b, a-c, b-c or a-b-c. Further, each element may include multiple instances, such as a-a, a-b-b, a-a-b-b-c-c, etc. Furthermore, it also includes adding other elements other than the listed elements (a, b and c), such as having d as in a-b-c-d.

In this specification (including claims), when expressions such as "using data as input/based on/according to data" (including similar expressions) are used, there is a special disclaimer. If there is no data, this includes cases where the data itself is used, or data that has been processed in some way (e.g., noise added, normalized, features extracted from the data, intermediate representation of the data, etc.) . In addition, if it is stated that a certain result is obtained "using data as input/based on/according to data" (including similar expressions), unless otherwise specified, the relevant This includes cases in which the relevant results are obtained based solely on data, and cases in which the relevant results are obtained under the influence of other data, factors, conditions, and/or states other than the relevant data. In addition, if it is stated that "data will be output" (including similar expressions), if there is no special notice, if the data itself is used as output, or if the data has been processed in some way (for example, noise This includes cases in which outputs (additions, normalizations, feature quantities extracted from data, intermediate representations of various data, etc.) are used as outputs.

In this specification (including the claims), when the terms "connected" and "coupled" are used, the terms "connected" and "coupled" refer to direct connection/coupling and indirect connection/coupling. , electrically connected/coupled, communicatively connected/coupled, functionally connected/coupled, physically connected/coupled, etc., without limitation. intended as a term. The term should be interpreted as appropriate depending on the context in which the term is used, but forms of connection/coupling that are not intentionally or naturally excluded are not included in the term. Should be construed in a limited manner.

In this specification (including the claims), when the expression "A configured to B" is used, it means that the physical structure of element A is capable of performing operation B. configuration, and includes a permanent or temporary setting/configuration of element A being configured/set to actually perform operation B. good. For example, if element A is a general-purpose processor, the processor has a hardware configuration that can execute operation B, and can perform operation B by setting a permanent or temporary program (instruction). It only needs to be configured to actually execute. In addition, when element A is a dedicated processor, dedicated arithmetic circuit, etc., the circuit structure of the processor is designed to actually execute operation B, regardless of whether control instructions and data are actually attached. It is sufficient if it is implemented in

In this specification (including claims), when terms meaning inclusion or ownership (e.g., "comprising/including", "having", etc.) are used, the object of the term It is intended as an open-ended term, including the case of containing or possessing something other than the object indicated. If the object of a term meaning inclusion or possession is an expression that does not specify a quantity or suggests a singular number (an expression with a or an as an article), the expression shall be interpreted as not being limited to a specific number. It should be.

In this specification (including the claims), expressions such as "one or more" and "at least one" are used in some places, and quantities are specified in other places. Even if an expression is used that suggests no or singular (an expression with the article a or an), it is not intended that the latter expression means "one". In general, expressions that do not specify a quantity or imply a singular number (expressions with the article a or an) should be construed as not necessarily being limited to a particular number.

In this specification, when it is stated that a specific advantage/result can be obtained with a specific configuration of a certain embodiment, unless there is a reason to the contrary, one or more other components having the configuration are described. It should be understood that this effect can also be obtained with the embodiment of . However, it should be understood that the presence or absence of such an effect generally depends on various factors, conditions, and/or states, and that the configuration does not necessarily provide the effect. The effect is only obtained by the configuration described in the embodiment when various factors, conditions, and/or states are satisfied, and in the claimed invention that defines the configuration or a similar configuration, Effects are not always obtained.

In this specification (including claims), when terms such as "minimize" or "minimization" are used, it refers to finding a global minimum value or an approximate value of the global minimum value. This term includes determining, determining a local minimum, and determining an approximation of a local minimum, and should be interpreted as appropriate depending on the context in which the term is used. It also includes finding approximate values of these minimum values probabilistically or heuristically. Similarly, when terms such as "optimize" or "optimization" are used, it refers to finding a global optimum, finding an approximation of a global optimum, or calculating a local optimum. This term includes determining and approximating a local optimum, and should be interpreted accordingly depending on the context in which the term is used. It also includes finding approximate values of these optimal values probabilistically or heuristically.

In this specification (including claims), when multiple pieces of hardware perform a predetermined process, each piece of hardware may cooperate to perform the predetermined process, or some of the hardware may perform the predetermined process. You may do all of the above. Further, some hardware may perform part of a predetermined process, and another piece of hardware may perform the rest of the predetermined process. In this specification (including claims), expressions such as "one or more hardware performs a first process, and the one or more hardware performs a second process" (including similar expressions) are used. ), the hardware that performs the first processing and the hardware that performs the second processing may be the same or different. In other words, the hardware that performs the first processing and the hardware that performs the second processing may be included in the one or more pieces of hardware. Note that the hardware may include an electronic circuit, a device including an electronic circuit, and the like.

In this specification (including claims), when multiple storage devices (memories) store data, each storage device among the multiple storage devices may store only part of the data. , the entire data may be stored. Further, a configuration may be included in which some of the plurality of storage devices store data.

Although the embodiments of the present disclosure have been described in detail above, the present disclosure is not limited to the individual embodiments described above. Various additions, changes, substitutions, partial deletions, etc. can be made without departing from the conceptual idea and spirit of the present invention derived from the content defined in the claims and equivalents thereof. For example, in the embodiments described above, when numerical values or formulas are used for explanation, these are shown for illustrative purposes and do not limit the scope of the present disclosure. Further, the order of each operation shown in the embodiments is also an example, and does not limit the scope of the present disclosure.

This application is based on and claims priority to Japanese Patent Application No. 2022-082762 filed with the Japan Patent Office on May 20, 2022, and the entire content of this Japanese patent application is including.

Claims

one or more cameras for photographing the object;
three-dimensional marker,
of the object based on a plurality of captured images of the object taken from different directions by the one or more cameras, and posture information of the camera based on images of the three-dimensional marker included in the plurality of captured images. a control device configured to output reconfiguration information;
A three-dimensional reconstruction system, wherein the three-dimensional marker has a marker section arranged at an angle with respect to a mounting surface of the three-dimensional marker.
The three-dimensional reconstruction system according to claim 1, wherein the mounting surface of the object is the same as the mounting surface of the three-dimensional marker.
The three-dimensional marker has a three-dimensional member having a predetermined height with respect to the placement surface,
The three-dimensional reconstruction system according to claim 1, wherein the marker section is provided on the three-dimensional member.
a support member that includes the placement surface and supports the object;
a rotation mechanism that rotates the object by rotating the support member,
The three-dimensional reconstruction system according to claim 1, wherein the camera photographs the object at each of a plurality of rotation angles by the rotation mechanism.
The three-dimensional reconstruction system according to claim 1, wherein the camera includes a plurality of cameras each capable of photographing the object from different directions.
The three-dimensional reconstruction system according to claim 5, wherein the plurality of cameras are capable of photographing the same marker section arranged at an angle with respect to the mounting surface together with the object.
The plurality of cameras include an upper camera installed above the mounting surface and capable of photographing the object from diagonally above, and an upper camera installed below the mounting surface capable of photographing the object diagonally from below. The three-dimensional reconstruction system according to claim 6, comprising: a lower camera.
The three-dimensional member is a polyhedron including a plurality of planes,
The three-dimensional reconstruction system according to claim 3, wherein the marker portion is provided on two or more of the plurality of planes in the polyhedron.
The three-dimensional reconstruction system according to claim 1, wherein the marker section has a one-dimensional code or a two-dimensional code that records identification information of the marker section.
The three-dimensional reconstruction system according to claim 1, wherein a plurality of the three-dimensional markers are arranged around the object.
The three-dimensional reconstruction system according to claim 10, wherein the plurality of three-dimensional markers include two three-dimensional markers placed before and after the object in the photographing direction of the camera.
The three-dimensional reconstruction system according to claim 10, wherein at least two of the plurality of three-dimensional markers are fixed to the placement surface, and a distance between them is predetermined.
further comprising at least two planar markers fixed to the placement surface and parallel to the placement surface;
The three-dimensional reconstruction system according to claim 1, wherein the distance between the two planar markers is predetermined.
The three-dimensional marker further includes another marker portion having a surface that contacts the placement surface,
The placement surface is a placement surface for a transparent member,
The three-dimensional reconstruction system according to claim 1, wherein the camera is capable of photographing the other marker section together with the object from below the placement surface.
A three-dimensional reconstruction method using a three-dimensional reconstruction system, the three-dimensional reconstruction system comprising:
The camera allows the object placed on the mounting surface to be photographed from different directions.
outputting reconstruction information of the object by a control device based on a plurality of images taken by the camera and posture information of the camera based on images of three-dimensional markers included in the plurality of images;
A three-dimensional reconstruction method, wherein the three-dimensional marker has a marker portion arranged at an angle with respect to the placement surface.