KR20200135131A - 3d camera system - Google Patents

Abstract

Disclosed is a three-dimensional camera system capable of three-dimensionally generating a photographed object by using a depth sensor module having a partial shape of a sphere so as to have versatility and expandability. According to an aspect of the present invention, the three-dimensional camera system three-dimensionally configuring the photographed object comprises: a depth sensor module formed in a partial shape of a sphere having a predetermined curvature, including a plurality of unit cells which include a light emitting unit for emitting a beam and a light receiving unit for receiving the beam reflected by a photographed object, and having a center of curvature of each unit cell which is a center of the sphere; and a control unit calculating three-dimensional coordinates of points constituting the photographed object on an orthogonal coordinate system having the center of curvature as an origin using information measured by the depth sensor module, and using the calculated three-dimensional coordinates to three-dimensionally configure the photographed object.

Description

3D camera system {3D CAMERA SYSTEM}

The present invention relates to a 3D camera system, and more particularly, to a 3D camera system capable of generating a photographed object in 3D by using a depth sensor module having a partial shape of a sphere.

A current 3D camera system is a camera system that acquires depth information of an object to be photographed. After calculating the depth of the object (eg, a face or object), it is reconstructed into three dimensions. However, since the depth sensor module of the current 3D camera system is implemented in a flat shape, it is difficult to be compatible with an image camera that generates a 2D image.

In addition, the current 3D camera system consists of three-dimensional spaces and objects with only color information and depth information separated from color information, and thus lacks versatility and scalability due to no color concept. A separate 3D application can be used to add color to a 3D space or object formed by photographing with a 3D sensing camera, but the color at this time is not a color of an actual space or an object, but a virtual color, which is less realistic. In other words, it is not universal and lacks scalability. In particular, the depth sensor module of the current 3D camera system has a flat shape, and the 2D image camera uses a convex lens to obtain mutual information, so the color information of the 2D image is acquired by the 3D camera system. It is difficult to apply to an object or to apply depth information acquired by a 3D camera system to a 2D image.

The present invention has been proposed to solve the above-described problem, and provides a 3D camera system capable of generating a photographed object in 3D by using a depth sensor module having a partial shape of a sphere so as to have versatility and expandability. There is a purpose.

In addition, another object of the present invention is to provide a 3D camera system capable of generating a 3D space or object in which color information is added to depth information by applying an image generation method of an image camera to a 3D sensing function.

In a three-dimensional camera system that configures the object to be photographed according to one side in three dimensions, it is formed in a shape of a sphere having a predetermined curvature, and a light emitting unit that emits a beam and the object reflected by the object to be photographed A depth sensor module including a plurality of unit cells including a light receiving unit for receiving a beam, wherein a center of curvature of each unit cell is a center of the sphere; And calculating 3D coordinates of points constituting the object to be photographed on a Cartesian coordinate system having the center of curvature as an origin using the information measured by the depth sensor module, and using the calculated three-dimensional coordinates to determine the photographing object. It includes a control unit configured in three dimensions.

The 3D camera system further includes a storage unit for storing a coordinate value in a spherical coordinate system having the center of curvature of each unit cell of the depth sensor module as an origin and a radius of the sphere, and the control unit comprises: the center of curvature The linear distance to each point constituting the object to be photographed is calculated from, and the value of (φ, θ) in the spherical coordinate system of each unit cell is checked in the storage unit, and the calculated linear distance and the (φ) , θ) is used to calculate the coordinate value of each point on the spherical coordinate system, and the calculated coordinate value on the spherical coordinate system is (X, Y, Z) on an orthogonal coordinate system with the center of curvature as the origin. Can be converted to coordinate values.

The 3D camera system may include an image sensor that generates an image of the object to be photographed; A camera lens disposed in front of the image sensor may be further included, and the depth sensor module may be disposed in front of the camera lens, and a second main point of the camera lens and the center of curvature may coincide with each other.

Each unit cell of the depth sensor module may be located at a position 180 degrees opposite to the direction of each pixel of the image sensor centering on the second main point and at a position distant from the second main point by a radius of the sphere.

The control unit may include depth information of each point of the object to be photographed measured in each unit cell of the depth sensor module, and color information of each pixel of the image sensor corresponding to each unit cell around the second main point. By using, the three-dimensional can be constructed.

The 3D camera system further includes a moving trajectory arranged in parallel along the surface of the camera lens, the width of the depth sensor module is smaller than the width of the camera lens, and the control unit determines the moving trajectory. Accordingly, the depth sensor module may be moved and 3D coordinates of each point constituting the object to be photographed may be calculated.

The depth sensor module is composed of two separate, and each depth sensor module is disposed on the left and right sides of the moving orbit, and the control unit moves the depth sensor module to the center along the moving orbit. I can.

The control unit may activate the image sensor when the depth sensor modules are opened apart from each other around the center, and deactivate the image sensor when the depth sensor modules contact each other at the center and are closed. have.

The control unit is calculated by dividing the moving time or moving distance of the depth sensor module, measurement information of the depth sensor module, and the total moving time or total moving distance of the depth sensor module by the number of horizontal pixels of the image sensor. The time or distance per pixel may be matched and stored in the storage unit.

The 3D camera system further includes a rotating body for moving the depth sensor module along the surface of the camera lens, the width of the depth sensor module is smaller than the width of the camera lens, and the controller comprises: By controlling the rotating body, the depth sensor module may be moved, and 3D coordinates of each point constituting the object to be photographed may be calculated.

The control unit includes a rotation angle per pixel calculated by dividing a moving rotation angle of the depth sensor module, measurement information of the depth sensor module, and a total rotation rotation angle of the depth sensor module by the number of horizontal pixels of the image sensor. It can be matched and stored in the storage.

The depth sensor module has an area overlapping the horizontal and vertical angles of view of the image sensor, and the control unit controls the depth sensor module in an inactive transparent state in a default mode, and activates the depth sensor module when a trigger occurs. I can make it.

When the position of the second main point is changed according to a change in the focal length of the camera lens, the control unit may move the depth sensor module so that the center of curvature of the depth sensor module coincides with the changed position of the second main point. .

The image sensor is a line scan image sensor, and a scan area of the line scan image sensor and a depth measurement area of the depth sensor module may overlap or be adjacent to each other.

The depth sensor module may be fixed to the camera lens in a filter manner.

According to the present invention, since depth information of an object to be photographed can be obtained by using a depth sensor module having a partial shape of a sphere, it is easily compatible with an image camera using a convex lens, and thus has versatility.

In addition, according to the present invention, by applying an image generation method of an image camera to a three-dimensional sensing function, a three-dimensional space or an object in which color information is added to depth information can be created, thereby creating a realistic three-dimensional space or object.

1 is a block diagram showing a 3D camera system according to an embodiment of the present invention.
2 is a diagram showing a spherical coordinate system.
3 is a diagram illustrating a method of calculating 3D coordinates of points constituting an object to be photographed in a 3D camera system according to an exemplary embodiment of the present invention.
4 is a block diagram showing a 3D camera system according to another embodiment of the present invention.
5 is a diagram schematically illustrating an arrangement structure of an image sensor and a depth sensor module according to an embodiment of the present invention.
6 is a diagram schematically illustrating an arrangement structure of an image sensor and a depth sensor module according to another embodiment of the present invention.
7 is a diagram schematically illustrating an installation structure of an image sensor and a depth sensor module according to another embodiment of the present invention.
8 is a diagram schematically illustrating an installation structure of an image sensor and a depth sensor module according to another embodiment of the present invention.
9 is a diagram schematically illustrating an installation structure of a depth sensor module according to another embodiment of the present invention.
10 is a diagram schematically illustrating an installation structure of an image sensor and a depth sensor module according to another embodiment of the present invention.
11 is a diagram showing an example in which the 3D camera system of the present invention is applied to a line scan camera.

The above-described objects, features, and advantages will become more apparent through the following detailed description in connection with the accompanying drawings, whereby those of ordinary skill in the technical field to which the present invention pertains can easily implement the technical idea of the present invention. There will be. In addition, in describing the present invention, when it is determined that a detailed description of a known technology related to the present invention may unnecessarily obscure the subject matter of the present invention, a detailed description thereof will be omitted. Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram showing a 3D camera system according to an embodiment of the present invention. Referring to FIG. 1, a 3D camera system according to the present embodiment includes a depth sensor module 130, a control unit 140, and a storage unit 150.

The depth sensor module 130 is manufactured in a shape of a sphere. That is, any part of the depth sensor module 130 is manufactured to have a constant curvature (1/radius). The curvature of the depth sensor module 130 is determined according to the installation position of the depth sensor module 130. In the depth sensor module 130, unit cells including a light emitting unit that emits a beam and a light receiving unit that receives the beam reflected by the object to be photographed and converts it into an electric signal are continuously arranged in a checkerboard shape, or the depth sensor module 130 ) Is arranged and configured in a radioactive form from the center. The depth sensor module 130 may consist of a matrix of M×1 (M is a natural number) or a matrix of M×N (M, N is a natural number) when the unit cells are continuously arranged in a checkerboard shape. have. The unit cell may include one light emitting unit and one light receiving unit, or may include one light emitting unit and a plurality of light receiving units. A laser device or an infrared device may be used for the light emitting unit, and a photodiode may be used for the light receiving unit. Preferably, a surface emission laser (VCSEL) may be used for the light emitting unit. The unit cells of the depth sensor module 130 have a center of curvature as an origin, form a spherical coordinate system in which the distance from the origin to each unit cell is constant, and coordinate values in the spherical coordinate system of each unit cell are determined during manufacture.

The light emitting units inside the cells of the depth sensor module 130 simultaneously emit a beam, and the beams reflected from each portion of the object to be photographed are received by the light receiving units of each unit cell of the depth sensor module 130 and converted into electrical signals. The depth sensor module 130 is manufactured in a shape of a part of a sphere. Accordingly, the extension line of the beam emitted from the light emitting unit of the unit cells of the depth sensor module 130 is connected to the center of curvature and the extension line of the beam received from the light receiving unit is also connected to the center of curvature. The depth sensor module 130, specifically, from the unit cell to a specific point of the object to which the beam is reflected, using the time of flight between the emission of the beam from the light emitting unit and the reception of the beam from the light receiving unit. Distance can be calculated. The beam emitted from the light emitting unit in the unit cell is reflected by the object to be photographed and received by the light receiving unit in the same unit cell. In this case, the beam emitted from the light emitting unit in another cell may be reflected and received, but the magnitude of the electric signal by this beam Is less than the reference value and can be ignored. Alternatively, it is possible to calculate the distance by selecting the time of flight of the most frequent beam by repeating the emission and reception of the beam for a predetermined period of time.

The storage unit 150 includes a distance from the center of curvature of the depth sensor module 130 to the depth sensor module 130 (that is, the radius of the sphere because it is a partial shape of a sphere), and the curvature of each unit cell of the depth sensor module 130 The position coordinates on the spherical coordinate system with the center as the origin, and the result value processed by the controller 140 are stored. In addition, the storage unit 150 stores a program for the operation of the control unit 140. The storage unit 150 may be a memory, and such a memory may be a storage device located far from one or more processors, such as a communication circuit, the Internet, an intranet, a local area network (LAN), a wide LAN (WLAN), and a SAN. (ST0rage Area Network), etc., or a suitable combination thereof, such as a network attached storage device that is accessed through a communication network (not shown).

The controller 140 calculates three-dimensional coordinates of points constituting the object to be photographed using information measured by the depth sensor module 130, and uses the calculated three-dimensional coordinates to determine the three-dimensional space of the object to be photographed. Or create an object. The controller 140 may be a separate and independent computing device, or may be an image processing processor. A process of calculating the three-dimensional coordinates of the object to be photographed by the control unit 140 will be described below with reference to FIGS. 2 and 3.

2 is a diagram showing a spherical coordinate system. Referring to FIG. 2, a point P on a spherical surface can be represented by a spherical coordinate system (ρ, φ, θ). Here, ρ means the distance from the origin to the point P, and ρ≥0. φ is the angle from the positive Z axis to the point P, and 0≤φ≤π. θ is the angle formed from the positive X axis to the point P projected on the XY plane, and 0≤θ≤2π. Such a spherical coordinate system (ρ, φ, θ) can be converted into a Cartesian coordinate system (x, y, z) as follows. In the present invention, the origin shown in FIG. 2 corresponds to the center of curvature of the depth sensor module 130 and the positive Z axis may be an axis from the center of curvature of the depth sensor module 130 toward the center of the depth sensor module 130. .

x=ρsinφcosθ,

y=ρsinφsinθ,

z=ρcosφ

FIG. 3 is a diagram illustrating a method of calculating 3D coordinates of points constituting an object in a 3D camera system according to an exemplary embodiment of the present invention. 3 shows the depth sensor module 130 (S0) and the object surface (S2). The subject plane S2 is a point T2 constituting the object to be photographed, that is, an object plane in which the subject exists. The beam emitted from the light emitting unit of the depth sensor module 130 is reflected from the subject T2 and is directed toward the center of curvature O of the depth sensor module 130 to the light receiving unit at a specific point T0 of the depth sensor module 130. Enter. By calculating the three-dimensional coordinates of all points constituting the object to be photographed, the object to be photographed can be configured in three dimensions. Here, the criteria, such as the size of the dot, vary depending on the setting. The center of the depth sensor module 130 is P0, the center of the object surface S2 is P2, and the center of curvature of the depth sensor module 130 is indicated by O.

Specifically, a method of calculating the three-dimensional coordinates of the subject T2 with reference to FIG. 3 is as follows. The beam emitted from the light emitting unit of the depth sensor module 130 is reflected by the subject T2 and returns to the light receiving unit at point T0 of the depth sensor module 130. Therefore, when the point T0 of the depth sensor module 130 is expressed in a spherical coordinate system with the center of curvature O as the origin, it is as follows.

T0 = (R0, φ0, θ0)

Where R0 is the distance from the center of curvature (O) to the point T0, φ0 is the angle from the positive Z axis to the point T0, and θ0 is the angle from the positive X axis to the point T0 is projected on the XY plane. .

Next, the subject T2 is expressed in a spherical coordinate system as follows.

T2 = (R2, φ0, θ2)

Here, R2 is the distance from the center of curvature (O) to the point T2, φ0 is the angle from the positive Z axis to the point T2, and θ2 is the angle from the positive X axis to the point where T2 is projected on the XY plane.

Since the beam reflected from the subject T2 is incident to the point T0 of the depth sensor module 130, the value of φ in the spherical coordinate system of the point T0 of the depth sensor module 130 and the value of φ in the spherical coordinate system of the subject T2 The value is the same as φ0. Further, θ0 in the spherical coordinate system of the point T0 and θ2 in the spherical coordinate system of the subject T2 are also the same. Accordingly, in the spherical coordinate system having the center of curvature 0 of the depth sensor module 130 as the origin, the point T0 and the subject T2 differ only in the distance, that is, R0 and R2. Since (R, φ, θ) values in the spherical coordinate system of each unit cell of the depth sensor module 130 are stored in the storage unit 150, the beam reflected from the subject T2 is transmitted from the depth sensor module 130. When incident to the T0 point, the (φ, θ) value of the T0 point, that is, (φ0, θ0) is extracted, and if only the distance R2 from the center of curvature (0) to the subject (T2) is known, the subject ( (X, Y, Z) coordinate values in the Cartesian coordinate system with the center of curvature (0) of T2) as the origin can be calculated. The (X, Y, Z) coordinate values in the Cartesian coordinate system are obtained by converting the (R2, φ0, θ2) coordinate values of the spherical coordinate system of the subject T2 into the coordinate values of the Cartesian coordinate system. That is, (X2, Y2, Z2), which is the (X, Y, Z) coordinate value of the subject T2, is obtained as follows through the spherical coordinate system (R2, φ0, θ2).

X2 = R2sinφ0cosθ0

Y2 = R2sinφ0sinθ0

Z2 = R2cosφ

As described above, when (X, Y, Z) coordinate values of points constituting a three-dimensional object having the center of curvature O as an origin are obtained, a three-dimensional shape of the object to be photographed can be finally obtained. When the (X, Y, Z) coordinate values in the Cartesian coordinate system of the points constituting the object to be photographed are obtained, the controller 140 can obtain a three-dimensional shape without color information of the object to be photographed using this coordinate value. have. The controller 140 may determine a range according to a depth value, that is, a Z value, for each point of the object to be photographed, and express the range with a designated color. The controller 140 may calculate an actual size or distance of a space or an object. For example, a robot equipped with a 3D camera system according to the present invention calculates the actual size of the door, that is, the width and height value at a certain distance to pass through the door in the space, and then compares it with its width and height to pass through. You can decide whether or not.

4 is a block diagram showing a 3D camera system according to another embodiment of the present invention. Referring to FIG. 4, the 3D camera system according to the present embodiment includes an image sensor 410, a camera lens 420, a depth sensor module 130, a control unit 140, and a storage unit 150.

The image sensor 410 receives light from an object to be photographed through the camera lens 420 and converts the received light into an electrical image signal. The surface of the image sensor 410 is an imaging surface and pixels are arranged in two dimensions. The camera lens 420 collects light from an object to be photographed and transmits it to the image sensor 410. The camera lens 420 is disposed in front of the image sensor 410. The point where the light of the camera lens 420 collects is referred to as a second main point, and the distance from the second main point to the image sensor 410 is referred to as a focal length or focal length.

The depth sensor module 130 is disposed in front of the camera lens 420. Accordingly, the image sensor 410, the camera lens 420, and the depth sensor module 130 are sequentially arranged in the direction of the object to be captured. The depth sensor module 130 may be attached to the lens of the camera by attaching a filter to the lens of the camera. The depth sensor module 130 may be coupled with the support member and the moving means so that the center of curvature coincides with the second main point according to the change in the position of the second main point according to the change of the focal length of the camera lens 420. The support member may be made of a transparent material, and various known means such as a front and rear moving means of the camera lens 420 may be used as the moving means. The depth sensor module 130 is manufactured in a shape of a sphere. That is, any part of the depth sensor module 130 is manufactured to have a constant curvature (1/radius). The curvature of the depth sensor module 130 is determined according to the installation position of the depth sensor module 130. When the depth sensor module 130 is installed, the depth sensor module 130 is fabricated to form a part of a sphere whose radius is the distance to the second main point of the camera lens 420. When the depth sensor module 130 is installed close to the outer surface of the camera lens 420, the curvature of the depth sensor module 130 increases. Conversely, when the depth sensor module 130 is installed away from the outer surface of the camera lens 420, the depth sensor module 130 The curvature becomes smaller. The depth sensor module 130 is supported in front of the camera lens 420 by a support.

The light emitting units inside the cells of the depth sensor module 130 simultaneously emit a beam, and the beams reflected from each portion of the object to be photographed are received by the light receiving units of each unit cell of the depth sensor module 130 and converted into electrical signals. The depth sensor module 130 is manufactured in a shape of a part of a sphere. Accordingly, the extension line of the beam emitted from the light emitting unit of the unit cells of the depth sensor module 130 is connected to the second main point of the camera lens 420, that is, the center of curvature of the depth sensor module 130, and is also received from the light receiving unit. The extension line of is also connected to the second main point of the camera lens 420, that is, the center of curvature of the depth sensor module 130. The depth sensor module 130, specifically, from the unit cell to a specific point of the object to which the beam is reflected, using the time of flight between the emission of the beam from the light emitting unit and the reception of the beam from the light receiving unit. Distance can be calculated. The beam emitted from the light emitting unit in the unit cell is reflected by the object to be photographed and received by the light receiving unit in the same unit cell. In this case, the beam emitted from the light emitting unit in another cell may be reflected and received, but the magnitude of the electric signal by this beam Is less than the reference value and can be ignored. Alternatively, it is possible to calculate the distance by selecting the time of flight of the most frequent beam by repeating the emission and reception of the beam for a predetermined period of time.

The storage unit 150 includes a second main point of the camera lens 420, that is, the distance from the center of curvature to the depth sensor module 130 (that is, the radius of the sphere because it is a part of a sphere), and the depth sensor module 130 The position coordinates on the spherical coordinate system with the center of curvature as the origin of each unit cell of, and a result value processed by the controller 140 are stored. In addition, the storage unit 150 stores a program for the operation of the control unit 140. The storage unit 150 may be a memory, and such a memory may be a storage device located far from one or more processors, such as a communication circuit, the Internet, an intranet, a local area network (LAN), a wide LAN (WLAN), and a SAN. (Storage Area Network) or the like, or a suitable combination thereof, such as a network attached storage device that is accessed through a communication network (not shown).

The controller 140 calculates 3D coordinates of points constituting the object to be photographed using the information measured by the depth sensor module 130, and generates the calculated 3D coordinates and the image sensor 110 A 3D space or object to which color information of the object to be captured is added is created by using the 2D image of the object to be captured. The controller 140 may be a separate and independent computing device, or may be an image processing processor. The control unit 140, depending on the setting, a two-dimensional image capture mode using the camera lens 420, a depth measurement mode using only the depth sensor module 130, and both the camera lens 420 and the depth sensor module 130 It can operate in the 3D object measurement mode used. The method of calculating the 3D coordinates of points constituting the object to be captured by the controller 140 is the same as described above. The controller 140 controls the moving means so that the center of curvature of the depth sensor module 130 coincides with the second main point according to a change in the position of the second main point according to the change of the focal length of the camera lens 420, (130) can be moved.

5 is a diagram schematically illustrating an arrangement structure of an image sensor and a depth sensor module according to an embodiment of the present invention. FIG. 5 shows an imaging surface S1, a depth sensor module 130 (S0), and an object surface S2. The imaging surface S1 is a surface of the image sensor 410 and is composed of pixels. The subject plane S2 is a point T2 constituting the object to be photographed, that is, an object plane in which the subject exists. The light reflected from the subject T2 passes through the second main point Q2 and is incident on a specific point T1 of the imaging surface S1. The subject surface S2 is parallel to the imaging surface S1. By calculating the three-dimensional coordinates of all points constituting the object to be photographed, the object to be photographed can be configured in three dimensions. Here, the criteria, such as the size of the dot, vary depending on the setting. The center of the imaging surface S1 is indicated by P1, the center of the depth sensor module 130 is indicated by P0, and the center of the object surface (S2) is indicated by P2. In addition, the second main point of the camera lens 420 is denoted as Q2. The second main point Q2 of the camera lens 420 is also the center of curvature of the depth sensor module 130. The focal length of the camera lens 420 is a distance from the center point P1 on the imaging surface S1 to the second main point Q2. The range in which the camera lens 420 can focus (focus) on the subject is from the minimum focal length to the infinity focal length. Within this range, the focal lengths have different values, and the change in the focal length is of the second main point Q2. It changes the position. Therefore, as in the case of the camera lens 420 with an automatic focus control function, whenever a change in the position of the second main point Q2 is detected, the depth sensor module 130 is moved back and forth under the control of the controller 140 to center the curvature. (O) is quickly moved to the position of the second main point (Q2). The moving means of the depth sensor module 130 may be installed in conjunction with an automatic focus control function. Positions of unit cells constituting the depth sensor module 130 in the depth sensor module 130 may be appropriately manufactured on the imaging surface S1 of the image sensor 410. This is explained as follows.

First, when the point T1 of the imaging surface S1 of the image sensor 410 is displayed in a spherical coordinate system, it is as follows.

T1 = (R1, φ1, θ1)

Where R1 is the distance from the second main point (Q2) to the point T1, φ1 is the angle from the positive Z axis to the point T1, and θ1 is the angle from the positive X axis to the point T1 projected on the XY plane. to be. More specifically, the values of (R1, φ1, θ1) can be obtained as follows.

R1=√(F ² + S ² ), where F is the focal length and S is the distance from P1 to T1.

φ1=cos ^-1 (F/R1)

θ1=cos ^-1 (X1/S)

Next, the point T0 of the depth sensor module 130 is expressed in a spherical coordinate system as follows.

T0 = (R0, φ0, θ0)

Where R0 is the distance from the second main point (Q2) to the point T0, φ0 is the angle from the positive Z axis to the point T0, and θ0 is the angle from the positive X axis to the point T0 projected on the XY plane. to be.

Based on the same spherical coordinate system with the second main point (Q2) as the origin, there is a difference of 180 degrees between the point T1 and the point T0, but consider two spherical coordinate systems that make only the origin the same and change the direction of the axis 180 degrees. , Φ1 of T1 and φ0 of T0 are the same, and θ1 of T1 and θ0 of T0 are the same. Therefore, when viewed in a spherical coordinate system based on the second main point Q2, T1 and T0 have the same φ and θ, and only R1 and R0, which are distances from the second main point Q2, are different. Therefore, in order to manufacture the depth sensor module S0 suitable for the imaging surface S1 of the image sensor 410, (1) the pixels of the imaging surface S1 of the image sensor 410 are converted into coordinate values of the spherical coordinate system. , (2) The distance from the second main point (Q2) to each pixel (i.e., R1) is replaced by the distance (R0) from the second main point (Q2) to the depth sensor module 130, and (3) the By using the coordinate values of the spherical coordinate system in which the distance is replaced, each unit cell of the depth sensor module 130 is positioned and manufactured. That is, the position of each pixel on the imaging surface S1 of the image sensor 410 may be rotated 180 degrees around the second main point Q2 and the distance may be set to R0. Each unit cell of the depth sensor module is located at a position 180 degrees opposite to the direction of each pixel of the image sensor centering on the second principal point and at a location that is separated by a radius of the sphere from the second principal point. In this way, each unit cell of the depth sensor module 130 corresponds to each pixel of the imaging surface S1 of the image sensor 410, so that a depth value can be applied to each pixel of the imaging surface S1. This embodiment is preferable when the depth sensor module 130 has an area covering the angle of view of the image sensor 410. For example, it is the same as the embodiment of FIG. 10 described below.

The control unit 140 calculates 3D coordinates of points constituting the object to be photographed corresponding to each unit cell through each unit cell of the depth sensor module 130, and calculates the 3D coordinates of each point, especially the depth. The Z value, which is information, may be applied to a value of a pixel corresponding to each unit cell of the imaging surface S1. Alternatively, when configuring a three-dimensional space or an object through the depth sensor module 130, the controller 140 acquires color information from pixels of the imaging surface S1 corresponding to each unit cell of the depth sensor module 130 Color information can be added to a three-dimensional space or object.

Meanwhile, the coordinates of each pixel of the imaging surface S1 of the image sensor 410 and the coordinates of each unit cell of the depth sensor module 130 may not correspond to each other. In this case, after obtaining the coordinates, that is, (φ, θ) in the spherical coordinate system of each pixel of the imaging surface S1 of the image sensor 410, and (φ, θ) of each unit cell of the depth sensor module 130 By comparison, a unit cell having a matching or approximate value can be viewed as a unit cell corresponding to a corresponding pixel.

6 is a diagram schematically illustrating an arrangement structure of an image sensor and a depth sensor module according to another embodiment of the present invention. Referring to FIG. 6, in the 3D camera system, the width W of the depth sensor module 130 is smaller than the width of the camera lens 420. That is, the depth sensor module 130 does not cover all the surfaces of the camera lens 420. The depth sensor module 130 is composed of a matrix of unit cells M×1 (M is a natural number), that is, one column. In addition, the three-dimensional camera system includes a moving trajectory 610 arranged in parallel along the surface of the camera lens 420. Like the depth sensor module 130, the moving trajectory 610 creates an arc-shaped path centered on the second main point Q2. The depth sensor module 130 emits a beam while moving in the direction B3 by installing a photodiode on the left and a surface emitting laser on the right, while configuring one surface emission laser and a photodiode each inside the unit cell. The beam reflected from the subject and returned may be received. Alternatively, the depth sensor module 130 may install one surface emitting laser and photodiodes on both left and right sides in the unit cell, and may emit and receive a beam while moving left and right.

In the embodiment of FIG. 6, the depth sensor module 130 has a very narrow width because unit cells are configured in one row. In addition, the depth sensor module 130 is installed as close as possible to the surface of the camera lens 420, and thus is located within the shortest photographing distance of the camera. Therefore, even if the depth sensor module 130 moves left and right, it is difficult for the image sensor 410 to detect the depth sensor module 130. Preferably, the depth sensor module 130 may be matte. Although the beam may be emitted from the light emitting unit of the entire column of the depth sensor module 130, the beams are sequentially emitted with a short parallax from row 1 to row M, thereby preventing interference from the beams emitted from adjacent cells or other It is possible to minimize a phenomenon in which the beam emitted from the cell is reflected from the object to be photographed together and received.

7 is a diagram schematically illustrating an installation structure of an image sensor and a depth sensor module according to another embodiment of the present invention. Referring to FIG. 7, the 3D camera system includes two separate depth sensor modules 130:130-1 and 130-2 and a moving trajectory 610 for moving the two depth sensor modules 130. Include. The width of each depth sensor module 130 is smaller than the width of the camera lens 420. That is, the depth sensor module 130 does not cover all the surfaces of the camera lens 420. The depth sensor module 130 is composed of a matrix of unit cells M×1 (M is a natural number), that is, one column. Like the depth sensor module 130, the moving trajectory 610 creates an arc-shaped path centered on the second main point Q2. One depth sensor module 130-1 is disposed on the left side of the moving track 610, and the other depth sensor module 130-2 is disposed on the right side of the moving track 610, and is controlled by the controller 140. Accordingly, the two depth sensor modules 130 are moved to the center 710 along the moving trajectory 610 and measure the depth of the object to be photographed. Each depth sensor module 130 may include one surface emitting laser and one photodiode in the unit cell. In each unit cell of the depth sensor module 130-1 installed on the left side of the moving orbit 610, a photodiode is installed on the left side, a surface emitting laser is installed on the right side, and a depth installed on the right side of the moving track 610 In each unit cell of the sensor module 130-2, a photodiode is installed on the right side and a surface emission laser is installed on the left side. The system of the embodiment with reference to FIG. 7 can shorten the depth measurement time compared to the system of the embodiment with reference to FIG. 6.

8 is a diagram schematically illustrating an installation structure of an image sensor and a depth sensor module according to another embodiment of the present invention. Referring to FIG. 8, similar to the embodiment described with reference to FIG. 7, the 3D camera system includes two separate depth sensor modules 130:130-1 and 130-2, and the two depth sensor modules ( 130) and a moving orbit 610 to move. However, each depth sensor module 130 is composed of a matrix of unit cells M×N (M, N is a natural number of 2 or more), and the moving trajectory 610 is more than the moving trajectory 610 described with reference to FIG. It is extended longer. Each depth sensor module 130 moves along the movement trajectory 610 under the control of the controller 140. The controller 140 activates the image sensor 410 when each depth sensor module 130 is opened apart from each other by the center, and the depth sensor module 130 contacts each other at the center and is closed. In this case, the image sensor 410 may be deactivated. It is effective for shooting sporadic images and continuous depth of field shooting.

9 is a diagram schematically illustrating an installation structure of a depth sensor module according to another embodiment of the present invention. In this embodiment with reference to FIG. 9, like the embodiment with reference to FIG. 6, the width of the depth sensor module 130 is smaller than the width of the camera lens 420, and the depth sensor module 130 has unit cells M×1 ( M is a natural number), that is, it is composed of one column. In the embodiment with reference to FIG. 6, the depth sensor module 130 moves along the moving trajectory 610, but in this embodiment with reference to FIG. 9, the depth sensor module 130 is a rotating body 910 that rotates at a fixed position. It is connected to and moves along the surface of the camera lens 420. The controller 140 controls the rotating body 910 to move the depth sensor module 130 and calculates 3D coordinates of each point constituting the object to be photographed. The movement trajectory of the depth sensor module 130 moving by the rotating body 910 creates an arc-shaped path centered on the second main point Q2. That is, the rotation center 911 of the rotating body 910 is the second main point Q2 of the camera lens 420. The depth sensor module 130 may install one surface emitting laser and photodiodes on both left and right sides inside the unit cell, and may emit and receive a beam while moving left and right.

In the embodiment described with reference to FIGS. 6 to 9 above, the controller 140 of the 3D camera system moves the depth sensor module 130 and stores information about the movement in the storage unit 150, and the movement By matching the information on and the pixel information of the image sensor 410, depth information and color information of the object to be captured can be accurately matched. Here, the information on the movement is a movement time, a movement distance, or a rotation angle. It is preferable that the moving time or the moving distance is stored in the embodiment with reference to FIGS. 6 to 8 and the rotation angle is stored in the embodiment with reference to FIG. 9.

In the embodiment with reference to FIGS. 6 to 8, the controller 140 includes a moving time or a moving distance of the depth sensor module 130, measurement information of the depth sensor module 130, and the entire depth sensor module 130. The time or position per pixel, calculated by dividing the moving time or the total moving distance by the number of horizontal pixels of the image sensor 410, is matched and stored in the storage unit 150. The total travel time can be calculated from the departure time and arrival time. In addition, the moving distance may be a physical moving distance, or may be a difference in position on the moving trajectory 610 of the depth sensor module 130. When the measurement information of the depth sensor module 130 at the corresponding time or distance is matched and stored for each unit movement time or movement distance of the depth sensor module 130, measurement information at a specific movement time or movement distance can be searched. I can. In addition, a pixel of the image sensor 410 corresponding to the searched measurement information may be specified as a pixel having a time or location corresponding to the specific movement time or movement distance.

In the embodiment with reference to FIG. 9, the control unit 140 calculates the moving rotation angle of the depth sensor module 130, measurement information of the depth sensor module 130, and the total rotation movement angle of the depth sensor module 130. The rotation angle per pixel calculated by dividing by the number of horizontal pixels of the image sensor 110 is matched and stored in the storage unit 150. When measurement information of the depth sensor module 130 at a corresponding rotation angle is matched and stored for each unit movement rotation angle of the depth sensor module 130, measurement information at a specific rotation angle may be searched. In addition, a pixel of the image sensor 410 corresponding to the searched measurement information may be specified as a pixel having a rotation angle corresponding to the specific rotation angle.

The controller 140 may record measurement information of the depth sensor module 130, that is, depth information, in each pixel of an image file generated by the image sensor 410. For example, among the ARGB values of each pixel, measurement information is recorded instead of the A (transparent) value, or measurement information is recorded in ARGBD (D: measurement information) format by adding, for example, 1 byte to the ARGB value. Can be recorded. If 1 byte is added, up to 256 decimal numbers are recorded. In this case, the remaining depth value is recorded by subtracting the minimum depth value from the depth value, and the minimum depth value can be recorded separately. For example, a minimum depth value may be recorded in meta information (EXIF) of an image.

10 is a diagram schematically illustrating an installation structure of an image sensor and a depth sensor module according to another embodiment of the present invention. Unlike the embodiments of FIGS. 6 to 9, the depth sensor module 130 of the embodiment with reference to FIG. 10 is a part of a sphere with the second main point Q2 of the camera lens 420 as the center point, and the image sensor 410 is It has an area overlapping with an area composed of a horizontal angle of view and a vertical angle of view corresponding to. The depth sensor module 130 includes cells in an M×N matrix, and, like a transparent display, is deactivated in a default state to maintain a transparent state. When a specific trigger is given, the depth sensor module 130 is activated to emit and receive a beam. Preferably, when a specific trigger occurs, the control unit 140 activates the depth sensor module 130 in the depth measurement mode and performs depth measurement sequentially from the left column 1 column by column 1 or the left column 1 row, 1 column 2 row type. Depth measurement can be performed in units of cells. The embodiment of FIG. 10 is suitable for a case in which the object to be photographed is dynamic, such as a case in which the object to be photographed is continuously imaged or a depth of field is to be measured.

The 3D camera system according to the embodiment described above may be used in face recognition, generation of a 3D space using a panoramic image, VR fitting, and autonomous driving vehicles. For example, in the case of VR fitting, a user creates and stores his/her human body as a 3D object using a 3D camera system of a smartphone, and then receives 3D clothing from an online shopping ball through an application and stores it in his 3D object. By applying it, you can check whether the clothes look good. As another example, in the case of face recognition, the same person is determined by recognizing faces in a two-dimensional image. When the 3D camera system according to the present invention is installed on the smartphone, the user registers by recognizing his/her face using the 3D camera system of the smartphone. At this time, the 3D camera system measures and stores the distance between the centers of the two eyes, and can use the distance between the centers of the two eyes as a reference for face recognition. When measuring the distance between the centers of two eyes, the 3D camera system of the present invention does not simply measure a straight line distance in a two-dimensional image, but a three-dimensional curved distance using a depth value. Authentication will fail.

The 3D camera system of the present invention can be applied not only to an area scan camera, but also to a line scan camera. When shooting 360 degrees, several area scan cameras are required, but in the case of a line scan camera, 360 degrees can be taken with a single line scan camera. In the image sensor of a line scan camera, in a state in which CCD or CMOS elements are arranged only in a row, photographing is continuously continuously performed according to the moving speed of the object to be photographed, realizing ultra-high quality, and the scanning speed is also very fast.

11 is a diagram showing an example in which the 3D camera system of the present invention is applied to a line scan camera. Referring to FIG. 11, the depth sensor module 130 is installed in front of the camera lens 420 as in the other embodiments described above. The depth sensor module 130 includes unit cells arranged in a row, and the unit cells include a surface emission laser and a photodiode. In the image sensor 410 of the line scan camera 1110, CCD or CMOS elements are arranged in a row. As shown in FIG. 11, the depth measurement area 1120 of the depth sensor module 130 and the scan area 1130 of the line scan camera, that is, the scan area 1130 of the line scan image sensor 410 are adjacent to each other. They can be placed or overlapped with each other. The depth sensor module 130 is installed as close as possible to the surface of the camera lens 420, so that it is located within the shortest shooting distance of the line scan camera, and thus the depth measurement area 1120 of the depth sensor module 130 and the line scan image sensor ( Even if the scan areas 1130 of 410 overlap, it is difficult for the line scan image sensor 410 to detect the depth sensor module 130. Since the line scan camera 1110 rotates at a fixed position, the depth of the object to be photographed can be measured without the need to separately move the depth sensor module 130. The line scan camera 1110 may be designed to rotate in a fixed position as shown in FIG. 11, but may be designed to move without rotating. For example, there is a case in which the depth sensor module 130 must be moved in a straight line to measure the surface of a structure that is not easy to move, as opposed to the case of installing on a conveyor to inspect a product.

While this specification includes many features, such features should not be construed as limiting the scope or claims of the invention. In addition, features described in separate embodiments herein may be combined and implemented in a single embodiment. Conversely, various features described in a single embodiment herein may be individually implemented in various embodiments, or may be properly combined and implemented. The present invention described above is capable of various substitutions, modifications, and changes without departing from the technical spirit of the present invention for those of ordinary skill in the technical field to which the present invention belongs. It is not limited by the drawings.

130: depth sensor module
140: control unit
150: storage unit
410: image sensor
420: camera lens
610: moving orbit
910: rotating body
1110: line scan camera

Claims (15)

In a three-dimensional camera system constituting a photographed object in three dimensions,
A plurality of unit cells including a light-emitting unit that emits a beam and a light-receiving unit that receives the beam reflected by the object, each unit cell having a shape of a sphere having a predetermined curvature A depth sensor module in which the center of curvature of the spheres is the center of the sphere and the origin of the spherical coordinate system; And
Using the information measured by the depth sensor module, three-dimensional coordinates of points constituting the object to be photographed on the spherical coordinate system having the center of curvature as an origin are calculated, and the object is determined using the calculated three-dimensional coordinates. A 3D camera system including a control unit configured in 3D. The method of claim 1,
Further comprising a storage unit for storing a coordinate value on a spherical coordinate system having the center of curvature of each unit cell of the depth sensor module as an origin, and a radius of the sphere,
The control unit,
Calculate a linear distance from the center of curvature to each point constituting the object to be photographed, check values of (φ, θ) in the spherical coordinate system of each unit cell in the storage unit, and the calculated linear distance and 3D camera system, characterized in that calculating a coordinate value of each point on the spherical coordinate system by using the (φ, θ) value. The method of claim 2,
An image sensor generating an image of the object to be photographed;
Further comprising a camera lens disposed in front of the image sensor,
The depth sensor module is disposed in front of the camera lens, and the second main point of the camera lens and the center of curvature coincide with each other. The method of claim 3,
Each unit cell of the depth sensor module is located at a position 180 degrees opposite to the direction of each pixel of the image sensor centering on the second main point and at a position distant by the radius of the sphere from the second main point. 3D camera system. The method of claim 4,
The control unit,
Using depth information of each point of the object to be photographed, measured in each unit cell of the depth sensor module, and color information of each pixel of the image sensor corresponding to each unit cell centering on the second main point, the A three-dimensional camera system, characterized in that the three-dimensional configuration. The method of claim 3,
Further comprising a moving trajectory disposed in parallel along the surface of the camera lens,
The width of the depth sensor module is smaller than the width of the camera lens,
The control unit,
3D camera system, characterized in that the depth sensor module is moved along the movement trajectory to calculate 3D coordinates of each point constituting the object to be photographed. The method of claim 6,
The depth sensor module is composed of two separate,
Each depth sensor module is disposed on the left and right sides of the moving trajectory, respectively,
The control unit, 3D camera system, characterized in that for moving the depth sensor module to the center along the movement trajectory. The method of claim 7,
The control unit,
3 characterized in that the image sensor is activated when the depth sensor modules are opened apart from each other around the center, and the image sensor is deactivated when the depth sensor modules are closed by contacting each other at the center. Dimensional camera system. The method according to any one of claims 6 to 8,
The control unit,
Time per pixel calculated by dividing the moving time or moving distance of the depth sensor module, measurement information of the depth sensor module, and the total moving time or total moving distance of the depth sensor module by the number of horizontal pixels of the image sensor, or A 3D camera system, characterized in that the distance is matched and stored in a storage unit. The method of claim 3,
Further comprising a rotating body for moving the depth sensor module along the surface of the camera lens,
The width of the depth sensor module is smaller than the width of the camera lens,
The control unit,
3D camera system, characterized in that by controlling the rotating body to move the depth sensor module, and calculating 3D coordinates of each point constituting the object to be photographed. The method of claim 10,
The control unit,
A storage unit by matching the rotation angle of the depth sensor module, measurement information of the depth sensor module, and the rotation angle per pixel calculated by dividing the total rotation angle of the depth sensor module by the number of horizontal pixels of the image sensor. 3D camera system, characterized in that to store in. The method of claim 3,
The depth sensor module has an area overlapping the horizontal and vertical angles of view of the image sensor,
The control unit,
In a default mode, the depth sensor module is controlled in an inactive transparent state, and when a trigger occurs, the depth sensor module is activated. The method of claim 3,
The control unit,
When the position of the second main point is changed according to the change of the focal length of the camera lens, the depth sensor module is moved so that the center of curvature of the depth sensor module coincides with the changed position of the second main point. Camera system. The method of claim 3,
The image sensor is a line scan image sensor,
3D camera system, characterized in that the scan area of the line scan image sensor and the depth measurement area of the depth sensor module overlap or are adjacent. The method of claim 3,
The depth sensor module, 3D camera system, characterized in that fixed to the camera lens in a filter method.

Images