KR20220060891A - Apparatus for LIDAR - Google Patents

Abstract

The present application relates to a lidar device. According to an embodiment of the present invention, the lidar device may include: a laser light source unit which outputs line laser beams into a target space; and an image processing unit which generates a detection image including the line laser beams reflected in the target space and generates a 3D coordinate map of the target space by using the detection image. Accordingly, processing speed for generating the 3D coordinate map of the target space can be improved.

Description

LIDAR device {Apparatus for LIDAR}

The present application relates to an image-based lidar device, and to a lidar device capable of generating a three-dimensional map of a target space through image processing.

Laser (Light Amplification by the Stimulated Emission of Radiation, LASER) is light amplified by stimulated emission and is used as a core technology in many fields such as electronics, optical communication, medicine, and national defense.

In addition, a LiDAR (Light Detection and Ranging, LiDAR) device uses a laser to measure the distance to an object, and has recently attracted attention as a core technology for autonomous vehicles, mobile robots, and drones.

Since the conventional lidar device applies the TOF (Time of Flight) method that calculates the distance through the flight time of the laser, it has the advantage of being able to measure with high precision and high resolution, but it is expensive, the overall size is large, etc. disadvantages exist.

An object of the present application is to provide a lidar device capable of generating a three-dimensional coordinate map for a target space by using a detection image acquired for the target space.

An object of the present application is to provide a lidar device capable of reducing manufacturing cost and reducing product size compared to the case of using the TOF method.

An object of the present application is to provide a lidar device capable of detecting an object located in a target space and recognizing the motion of the object.

A lidar device according to an embodiment of the present invention relates to an image-based three-dimensional lidar (LiDAR) device, comprising: a laser light source unit for outputting a line laser in a target space; and an image processing unit generating a detection image including the line lasers reflected in the target space, and generating a 3D coordinate map of the target space using the detection image.

Here, the laser light source unit, according to a trigger signal, a laser output unit for generating a point-shaped laser light; a scanning unit converting the laser light into a single-line or multi-line laser and outputting it in the target space; and a control unit controlling operations of the laser output unit and the scanning unit to output the line laser in the target space.

Here, the scanning unit may include a diffraction grating unit configured to generate the multi-line line laser by dividing the laser light into a plurality of line lasers traveling in different optical paths.

Here, the scanning unit may include: a reflector that reflects the single-line or multi-line line laser; And it may further include a driving unit for adjusting the reflection angle of the reflector.

Here, the control unit may control the operation of the driving unit to control a light projection angle at which the line laser is irradiated into the target space.

Here, the scanning unit may further include a galvanometer or a micro-electro mechanical systems (MEMS) mirror.

Here, the control unit transmits the trigger signal upon receiving the frame start signal from the image processing unit, controls the laser output unit to emit laser light for a duration, and changes the projection angle when the duration ends. The drive unit can be controlled.

Here, the controller may control the line lasers having a plurality of light transmission angles to be output within one frame exposure time.

Here, the control unit, when receiving an odd-numbered frame start signal from the image processing unit, transmits the trigger signal to control the laser output unit to emit laser light for a duration, and when receiving an even-numbered frame start signal, the trigger By omitting signal transmission, it is possible to control the laser output unit not to emit laser light.

Here, the image processing unit may include: an image sensor unit generating a detection image including the line laser; and a calculating unit that generates a lidar frame using the detected images and generates the three-dimensional coordinate map by using the three-dimensional coordinates of the target space extracted from the lidar frame.

Here, the image sensor unit adopts a global shutter method, so that all pixels included in the image sensor unit may sense the line laser during the same frame exposure time.

Here, the calculation unit generates one lidar frame by combining N detection images (N is a natural number greater than or equal to 1) received from the image sensor unit, and extracts a plurality of line lasers included in the lidar frame. can

Here, the calculator may set the three-dimensional coordinates of each line laser displayed in the lidar frame by using a triangulation technique.

Here, the calculation unit, when the line laser includes a reference pattern and a target pattern of a shape that is short-circuited from the reference pattern and spaced in a vertical direction, the target pattern is a vertical position value that is vertically spaced from the reference pattern, the A target pattern is horizontally spaced apart from a reference point by calculating a horizontal position value and a height position value corresponding to the reference pattern, respectively, and using the vertical position value, horizontal position value and height position value, the three-dimensional coordinate value can create

Here, the calculating unit may generate the three-dimensional coordinate value by further applying the projection angles of the line lasers to the vertical position value, the horizontal position value, and the height position value.

Here, the image processing unit may further include a recognition unit that detects objects located in the target space using the three-dimensional coordinate map and recognizes the type of the object.

A 3D map generation method according to an embodiment of the present invention relates to a 3D map generation method using a lidar device including a laser light source unit and an image processing unit, comprising: outputting a line laser in a target space by the laser light source unit; and generating, by an image processing unit, a detection image including the line lasers reflected in the target space, and generating a 3D coordinate map of the target space by using the detection image.

Incidentally, the means for solving the above problems do not enumerate all the features of the present invention. Various features of the present invention and its advantages and effects may be understood in more detail with reference to the following specific embodiments.

Since the lidar device according to an embodiment of the present invention can irradiate by changing the light projection angle of the line laser, it is possible to easily increase the number of line lasers irradiated into the target space. Through this, it is possible to improve the processing speed for generating a three-dimensional coordinate map for the target space.

Since the lidar device according to an embodiment of the present invention applies the global shutter method and synchronizes the frame exposure time and the laser duration, it is possible to minimize interference by external light. That is, the signal to noise ratio (SNR) of the detected image may be improved. In addition, since a plurality of line lasers can be included in one frame, it is possible to increase the frame rate of the lidar frame.

According to the lidar device according to an embodiment of the present invention, since a three-dimensional map of the target space can be generated using image processing, manufacturing cost reduction and product size reduction can be achieved compared to the case of using TOF, etc. can be implemented

1 is a schematic diagram showing the generation of a three-dimensional coordinate map using a lidar device according to an embodiment of the present invention.
2 is a block diagram illustrating a lidar device according to an embodiment of the present invention.
3 is a block diagram illustrating a laser light source unit according to an embodiment of the present invention.
4 is a schematic diagram showing a laser output unit according to an embodiment of the present invention.
5 to 7 are schematic diagrams showing the adjustment of the light transmission angle of the scanning unit according to an embodiment of the present invention.
8 is a schematic diagram illustrating generation of a multi-line laser using a diffraction grating unit of a scanning unit according to an embodiment of the present invention.
9 to 10 are exemplary views showing generation of a line laser by a laser light source according to an embodiment of the present invention.
11 and 12 are graphs illustrating synchronization between a laser light source unit and an image processing unit according to an embodiment of the present invention.
13 is a block diagram illustrating an image processing unit according to an embodiment of the present invention.
14 is a graph showing the operation of the image sensor unit according to the global shutter method and the rolling shutter method, respectively.
15 is an exemplary diagram illustrating a detection image according to an embodiment of the present invention.
16 and 17 are schematic diagrams illustrating three-dimensional coordinate generation according to an embodiment of the present invention.
18 is a schematic diagram showing a three-dimensional map using a lidar device according to an embodiment of the present invention.
19 is a flowchart illustrating a 3D map generation method using a lidar device according to an embodiment of the present invention.

Hereinafter, preferred embodiments will be described in detail so that those of ordinary skill in the art can easily practice the present invention with reference to the accompanying drawings. However, in describing the preferred embodiment of the present invention in detail, if it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. In addition, the same reference numerals are used throughout the drawings for parts having similar functions and functions.

In addition, throughout the specification, when a part is 'connected' with another part, it is not only 'directly connected' but also 'indirectly connected' with another element interposed therebetween. include In addition, 'including' a certain component means that other components may be further included, rather than excluding other components, unless otherwise stated. In addition, terms such as "~ unit" and "module" described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software, or a combination of hardware and software.

1 is a schematic diagram illustrating the generation of a three-dimensional coordinate map using a lidar device according to an embodiment of the present invention, and FIG. 2 is a block diagram illustrating a lidar device using a lidar device according to an embodiment of the present invention. .

1 and 2 , the lidar apparatus 1000 according to an embodiment of the present invention may include a laser light source unit 100 and an image processing unit 200 .

Hereinafter, a lidar device 1000 according to an embodiment of the present invention will be described with reference to FIGS. 1 and 2 .

The laser light source unit 100 may output a line laser in the target space (A). Specifically, referring to FIG. 3 , the laser light source unit 100 may include a laser output unit 110 , a scanning unit 120 , and a control unit 130 .

The laser output unit 110 may generate a point-shaped laser light according to a trigger signal, and as shown in FIG. 4 , may include a laser diode 111 and a collimator 112 .

The laser diode 111 may be excited by a laser driver (not shown) to generate laser light, and the generated laser light may be collected at a point through the collimator 112 to generate point-shaped laser light. Here, the laser driver may generate laser light by exciting the laser diode 111 according to a trigger signal.

Specifically, the laser diode 111 may generate laser light having a set wavelength, and in this case, the set wavelength may be set to an infrared wavelength region. That is, the infrared laser is irradiated so that the laser light is not visually confirmed in the target space (A).

Here, the laser diode 111 may be energy designed to meet the Eye-safety 1 class standard. The laser diode 111 may emit laser light in a pulsed form according to a set energy design. In this case, the duration of the laser pulse to satisfy the eye-safety 1 class standard may be determined according to the set wavelength of the infrared laser. there is.

On the other hand, the target space (A) to which the line laser is output may be a security area such as a house or a store, or a space around a vehicle being driven or parked. However, the present invention is not limited thereto, and any space may correspond to a space in which the lidar device 1000 is installed to irradiate the line laser.

The scanning unit 120 may convert the laser light into a single-line or multi-line line laser, and output the converted laser light in the target space. Here, the method in which the scanning unit 120 converts the laser light may largely include three embodiments.

According to the first embodiment, the scanning unit 120 may generate a single-line line laser and then irradiate the single-line laser beam into the target space A while changing the light-transmitting angle of the single line. In this case, the scanning unit 120 may include a reflector 121 , a driving unit 122 , and a lens unit 124 as shown in FIG. 5A .

The reflector 121 is a member that reflects a single-line line laser such as a mirror, and the driving unit 122 corresponds to a configuration that adjusts the reflection angle of the reflector 121 by rotating the reflector 121 . That is, the driving unit 122 may rotate or move the reflector 121 according to the control signal received from the control unit 130 , and through this, controlling the light projection angle at which the line laser is irradiated into the target space A is It is possible.

The lens unit 124 may spread the point-shaped laser light incident from the laser output unit 110 and convert it into a line-shaped line laser. In some embodiments, the lens unit 124 may be formed as a semi-cylindrical cylinder lens, and through this, a point-shaped laser light may be converted into a single-line line laser.

Here, the scanning unit 120 may be implemented using a galvanometer or a MEMS (Micro-Electro Mechanical Systems) mirror, depending on the embodiment. FIG. 5(a) is a case in which the galvanometers 121 and 122 are used, and FIG. 5(b) corresponds to an example in which the MEMS mirror 123 is used.

In this case, as shown in FIGS. 9(a) to 9(c), the line laser may be sequentially output from top to bottom in the target space A. Here, the movement order or number of line lasers may be variously modified according to embodiments. In addition, although it is shown here that one line laser is generated for each frame, according to an embodiment, a plurality of line lasers are displayed in one frame image by changing the light emission angle of the line laser several times within the frame exposure time. It is also possible to make

Also, in the second embodiment, after the scanning unit 120 generates a multi-line line laser, the multi-line light transmission angle may be changed while irradiating the multi-line laser beam into the target space (A). In this case, the scanning unit 120 may include a diffraction grating unit 125 as shown in FIG. 6 .

The laser light emitted from the laser diode 111 may form a laser light having a point shape in cross section through the collimator 112 , and then a multi-line laser including a plurality of line laser shapes by the diffraction grating unit 125 . can be processed into That is, the diffraction grating unit 125 may process the collimated laser light in various shapes, and according to an embodiment, it is also possible to divide the collimated laser light into a plurality of line laser shapes.

Here, the diffraction grating unit 125 may generate a plurality of line lasers parallel to the horizontal direction, and in this case, the plurality of line lasers may be divided to proceed in optical paths of different angles in the vertical direction. According to an embodiment, it is also possible to generate a plurality of line lasers parallel to the vertical direction, and divide the line lasers so as to proceed in optical paths of different angles in the horizontal direction. In addition, the number of multi-lines generated by the diffraction grating unit 125 or an interval between the multi-lines may be variously set according to embodiments.

Thereafter, the reflector 121 may reflect the multi-line line laser incident from the diffraction grating unit 125, and at this time, the light transmission angle of the line lasers may be set by adjusting the reflection angle of the reflector by the driving unit (not shown). there is. That is, the multi-line laser incident through the diffraction grating unit 125 may be emitted to the target space A after being reflected by the reflector 121 , and in this case, the light transmission angle of the multi-line laser according to the reflection angle of the reflector can be determined. Therefore, as shown in FIGS. 6 and 7 , the position at which the multi-line lasers M1 and M2 are irradiated in the target space A can be changed, and the multi-line lasers M1 and M2 are sequentially arranged in the vertical direction. can be irradiated in the target space (A).

In this case, as shown in FIGS. 10(a) and 10(b), a multi-line laser may appear. That is, after outputting a line laser having three multi-lines at the upper end of the target space as shown in Fig. 10(a), a line laser having three multi-lines at the bottom as shown in Fig. can be printed out.

According to an embodiment, it is also possible to display a plurality of multi-line lasers in one frame image by changing the light emission angle of the line laser several times within the frame exposure time.

In the third embodiment, as shown in FIG. 8 , the scanning unit 120 may output multi-line line lasers into the target space A using the diffraction grating unit 125 . That is, the scanning unit 120 uses the diffraction grating unit 125 to divide the laser light received from the laser output unit 110 into a plurality of line lasers that travel through different optical paths to generate a multi-line line laser. can create In this case, a multi-line laser may be output in the target space A as shown in FIG. 10( c ).

In some embodiments, as shown in FIG. 8(b), a plurality of combinations including the laser output unit 110 and the diffraction grating unit 125 are provided, and then sequentially irradiated to the target space (A) It is also possible to output a multi-line line laser within.

The controller 130 may control the operation of the laser output unit 110 and the scanning unit 120 to output a line laser in the target space. Specifically, as shown in FIGS. 5 to 7 , in the case of adjusting the light emission angle of the line laser, the control unit 130 operates the laser output unit 110 and the scanning unit 120 as shown in FIG. 11 . can be controlled.

11 , the control unit 130 may receive a signal of start frame from the image processing unit 200 and transmit a trigger signal to the laser output unit 110 in response to the frame start signal. there is. In this case, the laser output unit 110 may emit laser light for a duration (IR Laser On/Off), and in this case, the laser light duration of the laser output unit 110 may be preset. According to an embodiment, the duration may be set to satisfy eye-safety 1 class.

Thereafter, when the duration is over, the controller 130 may control the scanning unit 120 to change the light transmission angle of the line laser. That is, the controller 130 may control to change the light projection angle for the line laser within the frame exposure time of the image processing unit 200 . Therefore, when controlling as shown in FIG. 11, it is possible to generate a detection image so that a multi-line laser is included in one frame even when a single-line laser is output.

Specifically, the control unit 130 may control the scanning unit 120 to sequentially change the light projection angle by a set angle change amount in the vertical or horizontal direction from the reference position. For example, after setting the reference position to 30 degrees vertically upward from the horizontal line, the light projection angle can be controlled to descend by 15 degrees. Although FIG. 11 exemplifies the adjustment of the light transmission angle twice within one frame exposure time, the number of times the light transmission angle is adjusted may be variously modified according to embodiments.

Meanwhile, the controller 130 may repeatedly perform the operation until the number of generated frames reaches a set number. For example, when the target space A is divided into three regions, the line laser may be repeatedly output until each detection image corresponding to the three regions is generated. Thereafter, a set number of detection images can be combined to create one lidar frame.

In addition, as shown in FIG. 8( a ), in the case of an embodiment in which a multi-line laser is output in the target space A in a fixed state, the control unit 130 operates the laser output unit 110 as shown in FIG. 12 . can be controlled.

Specifically, the control unit 130 may receive a frame start signal from the image processing unit 200 , and the control unit 130 may transmit a trigger signal to the laser output unit 110 . In this case, the laser output unit 110 may emit laser light for a duration, and in this case, the duration may be synchronized to match the frame exposure time. When the frame exposure time is over, the image processing unit 200 may generate a detection image, and in this case, the detection image may correspond to a lidar frame.

Here, in order to efficiently detect the line laser, a case in which the line laser is output and a case in which the line laser is not output may be compared. To this end, the controller 130 may control to continuously generate a detection image to which the line laser is output and a detection image to which the line laser is not output in the target space A. That is, it is possible to more accurately detect the line laser by comparing the detected images according to whether the line laser is output or not.

Specifically, when receiving the odd-numbered frame start signal from the image processing unit 200 , the control unit 130 may transmit a trigger signal to control the laser output unit 110 to emit laser light for a duration. On the other hand, when the even-numbered frame start signal is received, transmission of the trigger signal may be omitted to control the laser output unit not to emit laser light. Although it is assumed here that the trigger signal is transmitted at the odd-numbered time, it is also possible to transmit the trigger signal at the even-numbered time and not transmit the trigger signal at the odd-numbered time according to an embodiment. In addition, various modifications are possible, such as not transmitting a trigger signal for every frame corresponding to a multiple of 3 or a multiple of 4.

Additionally, as shown in Fig. 8 (b), when a plurality of combinations including the laser output unit 110 and the diffraction grating unit 125 are provided, the control unit 130 controls each of the assemblies sequentially in the target space (A). It can be controlled to output a multi-line line laser inside. That is, when there are N binders, N detection images including multi-line lasers can be generated by each combination, and then N detection images can be combined to generate one lidar frame. .

The image processing unit 200 may generate a detection image including line lasers reflected in the target space, and generate a 3D coordinate map of the target space by using the detection image. Specifically, as shown in FIG. 13 , the image processing unit 200 may include an image sensor unit 210 , an operation unit 220 , and a recognition unit 230 .

The image sensor unit 210 may generate a detection image including a line laser. Depending on the embodiment, the image sensor unit 210 may be implemented as a sensor array such as a charge-coupled device (CCD) sensor or a complementary metal-oxide semiconductor (CMOS) sensor, and the image sensor unit 210 may include a target space ( A) It is possible to measure the luminance of the multi-line lasers reflected within and convert it into an electrical signal. That is, a two-dimensional detection image may be generated by setting the measured luminance value of the multi-line laser to the pixel value of each pixel.

Here, the image sensor unit 210 may adopt a global shutter method, and in this case, all pixels included in the image sensor unit 210 may detect a multi-line laser during the same frame exposure time. . Specifically, FIG. 14(a) is a graph illustrating an operation in a global shutter method and FIG. 14(b) is a rolling shutter method.

First, confirming the rolling shutter method of FIG. 14(b), pixel rows may be individually reset for each row, and frame exposure ( Exposure start, exposure stop, and line readout may be sequentially performed.

On the other hand, in the global shutter method of FIG. 14A , all pixel rows may be simultaneously exposed after initializing all pixel rows (global reset). Thereafter, when the frame exposure time elapses, readout is sequentially performed from the upper pixel row. That is, the frame exposure time for all the pixel rows is applied equally, and it is possible to secure the time until all the pixel rows are read out after the frame exposure time as a processing time or the like.

Comparing the rolling shutter method with the global shutter method, in the case of the global shutter, the exposure time of the pixel rows is relatively short, so the time for receiving external light may be reduced compared to the rolling shutter method, and accordingly, the signal to noise ratio (SNR) ) can be improved, and it is possible to secure the processing time after the frame exposure time.

Accordingly, the image sensor unit 210 may adopt a global shutter method so that all pixels included in the image sensor unit 210 detect the line laser during the same frame exposure time. However, in order for the image sensor unit 210 to generate a detected image according to the global shutter method, the line laser starts to be projected at the time frame exposure starts, and emission of the line laser stops exactly when the frame exposure stops. need to be To this end, the controller 130 may synchronize the duration of the laser light to the frame exposure time.

Additionally, the image sensor unit 210 may further include a band-pass filter (not shown), a polarizing filter (not shown), and a receiving optical system (not shown) for blocking an external light source and effectively receiving a line laser. . That is, the laser light that has passed through the receiving optical system, the polarization filter, and the bandpass filter may be incident on the CCD sensor or the CMOS sensor. Here, the polarizing filter may remove existing natural light, etc., and the band-pass filter may block a frequency corresponding to external light such as the sun or indoor lighting, and pass a frequency corresponding to the line laser. By moving the lens, the receiving optical system can automatically or manually adjust the focal length or FOV (Field of View) for the reflected multi-line laser.

According to an exemplary embodiment, it is also possible for the image sensor unit 210 to set an ROI (Region of Interest) to perform exposure only for pixels in a corresponding area for operational efficiency. For example, by using the detection images generated in the target space A, pixel positions where the line laser is expected to be located can be confirmed in advance, and the corresponding positions can be set as ROIs.

The operation unit 220 may generate a LIDAR frame using the detected images, and may generate a 3D coordinate map using 3D coordinates of the target space extracted from the LIDAR frame.

First, the operation unit 220 may generate one lidar frame by combining the N detection images (N is a natural number greater than or equal to 1) received from the image sensor unit. Here, the number N of detection images combined to generate a lidar frame may be set differently for each embodiment.

After generating the lidar frame, the operation unit 220 may extract a plurality of line lasers included in the lidar frame. In general, since the line lasers have a high luminance value in the lidar frame, the line lasers can be extracted by binarizing them with a fixed threshold value. However, due to environmental factors such as surface reflectance and illuminance change of various objects that may exist in the target space (A), laser blurring, blurring, and non-detection or erroneous detection of the line laser due to the influence of direct sunlight occur, and the correct line Detection of the laser can be difficult. In order to solve this, the operation unit 220 may use a deep learning learning technique to extract a line laser.

After that, when the line laser is extracted from within the lidar frame, the calculating unit 220 uses the geometric distance and angle between the laser light source unit 110 and the image sensor unit 210, and the distance to the line laser within the target space. can be measured. Here, according to an embodiment, the calculating unit 220 may measure the distance to the line laser by applying a triangulation technique.

Specifically, the projection angle at which the laser light source unit 110 irradiates the line laser into the target space and the separation distance between the laser light source unit 110 and the image sensor unit 210 are determined in the design process of the lidar device 1000 . It corresponds to a value known in advance. In addition, the operation unit 220 may check which pixel among the pixel arrays in the image sensor unit 210 is detected in which laser light at a specific point arbitrarily set among the line lasers is detected, and using this, the image sensor unit 210 and a specific The angle of incidence between points can be determined.

Here, by connecting the laser light source unit 110, the image sensor unit 210, and a specific point, a triangle can be formed. applicable if known. Accordingly, by using the triangulation method, the distance between the laser light source unit 110 and a specific point and the distance between the image sensor unit 210 and the specific point can be easily obtained.

In addition, since the length of the two sides of the triangle and the included angle therebetween are known, the distance between the specific point and the lidar device 1000 can be easily derived by using a trigonometric function.

That is, using the triangulation method, it is possible to measure the distance to each line laser displayed in the lidar frame, and based on this, it is possible to set 3D coordinates for each point included in the line laser.

Thereafter, the calculator 220 may generate a three-dimensional coordinate map for the target space A by collecting the three-dimensional coordinates for each line laser. According to an embodiment, it is also possible to generate a three-dimensional coordinate map for each lidar frame.

However, when the object S is present in the target space A, the line laser in the lidar frame may include a reference pattern and a target pattern of a shape that is short-circuited from the reference pattern and spaced in the vertical direction. That is, as shown in FIG. 15 or the like, when an object exists in the target space, the line laser may appear in a form including the target pattern spaced apart from the reference pattern.

In this case, it may be difficult to measure the distance to the object S using the triangulation method, and the operation unit 220 may set the three-dimensional coordinates for the object S by using the relationship between the target pattern and the reference patterns. there is. That is, the calculating unit 220 may calculate a vertical position value at which the target pattern is vertically spaced from the reference pattern, a horizontal position value at which the target pattern is horizontally spaced apart from the reference point, and a height position value corresponding to the reference pattern, respectively. , can be used to set three-dimensional coordinates for an object.

Specifically, as shown in FIG. 16( a ), when the line laser is horizontally irradiated into the target space, the reference pattern P1 is located at the center of the lidar frame. Here, the height position value can be set based on the position of the center line (RL) of the lidar frame, and when the reference pattern (P1) is located on the center line (RL) as shown in FIG. 16 (a), the reference pattern (P1) You can set the height position value to 0.

On the other hand, when the object S is located in the target space as shown in FIG. 16( b ), the target pattern P2 may appear spaced apart from the reference pattern P1 in the lidar frame. However, the target pattern P2 appears distorted on the lidar frame due to the perspective relationship with the object S, and the actual line laser is horizontally irradiated to the same height in the target space A. Accordingly, the height position values of the corresponding line lasers are all the same, and the height position value of the target pattern P2 may be set to the same value as the height position value of the reference pattern P1 . Here, the height position value corresponds to the z-axis coordinates of each point corresponding to the line laser.

In addition, as shown in Fig. 16(b), when the object S is located in the target space A, the target pattern P2 and the reference pattern P1 are displayed together in the lidar frame. In this case, the center point of the lidar frame may be set as the reference point RP, and the distance from the reference point RP to the target pattern P2 spaced apart in the horizontal direction may be measured and set as a horizontal position value. That is, for each point included in the target pattern P2, horizontal position values spaced apart from the reference point RP in the horizontal direction can be measured, and in this case, each horizontal position value is the x-axis coordinate for each point. respond to According to the embodiment, it is also possible to obtain the width of the object (S) by measuring the horizontal length of the target pattern (P2).

On the other hand, the distance between the lidar device 100 and the object S is proportional to the vertical position value p at which the target pattern P2 in the lidar frame is vertically spaced from the reference pattern P1. Accordingly, the calculator 220 may calculate the distance D1 to the object S by using the vertical position value p. Here, a function or table table for obtaining the distance value to the object S corresponding to each vertical position value p may be preset, and the operation unit 220 uses this to determine the vertical position value p ) to set the y-axis coordinates.

Accordingly, the calculator 220 may generate a three-dimensional coordinate value for the object by using the vertical position value, the horizontal position value, and the height position value.

Additionally, as shown in FIG. 17, when the emission angle of the line laser is changed, the calculation unit 220 further applies the emission angle θ of the line lasers to the vertical position value, the horizontal position value, and the height position value, 3D coordinate values can be generated.

Specifically, when irradiating a line laser into the target space at a light transmission angle θ as shown in FIG. 17( a ), the reference pattern P1 is positioned below the center line RL of the lidar frame.

As shown in Fig. 17(b), the distance D2 between the lidar device 1000 and the object S is a vertical position value at which the target pattern P2 in the lidar frame is vertically spaced apart from the reference pattern P1. It is proportional to (p). Accordingly, the calculator 220 may calculate the distance D2 to the object S by using the vertical position value p. Here, a function or table table for obtaining the distance D2 to the object S corresponding to each vertical position value p may be preset, and the operation unit 220 uses this to determine the vertical position value The distance D2 from (p) to the object S can be calculated. Thereafter, the y-axis coordinate of the object S can be obtained as D2*cos(θ).

In addition, since the height position value is based on the center line RL of the lidar frame, the height position value can be calculated using the interval between the reference pattern P1 and the center line RL. Here, the height position value corresponds to the z-axis coordinate of the reference pattern P1. However, as shown in FIG. 17(b), since the object S is located at the upper end in the z-axis direction than the line laser, additional calculation is performed on the height position value to obtain the z-axis coordinates of the object S. Needs to be. Here, it is possible to obtain the z-axis coordinates by using various trigonometric functions. For example, it is also possible to obtain the z-axis coordinates of the object using D2*cos(θ)*tan(θ).

On the other hand, the horizontal position value can be measured in the same way as in the case of horizontally irradiating the line laser. That is, the target pattern P2 from the reference point RP may measure a horizontal position value spaced apart in the horizontal direction to measure a distance, and in this case, the reference point RP may be the center point of the lidar frame. Here, since the horizontal position value corresponds to the x-axis coordinate, it is possible to set the x-axis coordinate for each point by using the horizontal position values of each point included in the target pattern P2.

Accordingly, the calculator 220 may calculate the three-dimensional coordinates for the target space A by using the line laser extracted from the lidar frame, and may generate a three-dimensional coordinate map based on this. That is, as shown in FIG. 18, a three-dimensional coordinate map for the target space can be generated.

The recognition unit 230 may detect objects located in the target space using the 3D coordinate map and recognize the type of the object. Specifically, it is possible to detect an object by determining at which location the object of interest exists among a plurality of 3D point cloud clusters included in the 3D coordinate map. That is, by using the pre-trained model, the degree of similarity between candidate 3D point cloud clusters over a certain density and the object of interest is measured, and when the candidate 3D point cloud clusters have a similarity greater than or equal to a set value with the object of interest, Object detection may be performed by detecting 3D coordinate information.

In addition, the recognition unit 230 recognizes the object in a manner such as classifying a type of the detected object of interest, such as a person, an animal, a vehicle, or inferring a pose when the detected object is a person. can do. According to an embodiment, the recognition of each object may be performed by inputting the 3D point cloud of the object of interest to a deep neural network. Here, in the deep neural network, learning about the type or pose of each object may be performed in advance.

19 is a flowchart illustrating a 3D map generation method using a lidar device according to an embodiment of the present invention. Here, the lidar device may include a laser light source unit and an image processing unit.

Referring to FIG. 19 , first, the laser light source may output a line laser in the target space ( S10 ). Here, the laser light source unit may generate a single-line laser or a multi-line laser including a plurality of line lasers traveling in different optical paths, and may irradiate the generated line laser into the target space by changing the emitting angle. In this case, the LIDAR device may change the emission angle of the line laser by using a galvanometer or MEMS mirror, and the angular change amount and change direction of the emission angle may be preset. However, since the light projection angle and the emission timing control of the line laser have been described above, a detailed description thereof will be omitted.

Thereafter, the image processing unit may generate a detection image including the line lasers reflected in the target space, and may generate a 3D coordinate map of the target space using the detection image ( S20 ).

Specifically, an image sensor unit such as a CCD sensor or a CMOS sensor may generate a detection image by detecting a reflected multi-line laser, and the operation unit may generate a lidar frame by collecting the generated detection images.

When the lidar frame is generated, the operation unit may extract a line laser from the lidar frame, and may generate 3D coordinate information in a target space for the line laser. Here, the line laser may be extracted using a deep learning model, etc., and 3D coordinate information for each line laser displayed in the lidar frame may be generated using triangulation. Thereafter, each 3D coordinate information may be collected to generate a 3D map for the target space. According to an embodiment, it is possible to detect objects located in the target space using the generated 3D coordinate map, and perform an operation such as recognizing the type of the object.

On the other hand, the lidar device 1000 according to an embodiment of the present invention is applicable to an unmanned mobile robot such as an AGV (Automated Guided Vehicle), a cooperative robot, an industrial robot, and the like. That is, it can be used as a sensor for detecting the surrounding environment around the robot or detecting the size or shape of an object.

In addition, the lidar device 1000 according to an embodiment of the present invention detects a nearby environment of the vehicle to prevent collision in the autonomous driving assistance system of the vehicle, or a night environment that cannot be detected by a camera, a dark indoor environment, or Surveillance for security and safety in places where cameras cannot be used due to privacy concerns can be performed.

In addition, the lidar device 1000 according to an embodiment of the present invention can be utilized in an entertainment system such as a game or VR that requires a function of recognizing a human body motion, shape, and the like.

The embodiments described above may be implemented by a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, the apparatus, methods, and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA) array), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

Software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

The components described in the embodiments are one or more of a digital signal processor (DSP), a processor, a controller, an application specific integrated circuit (ASIC), a programmable logic device such as a field programmable gate array (FPGA). Logic Element), other electronic devices, and hardware components including one or more of combinations thereof. At least some of the functions or processes described in the embodiments may be implemented by software, and the software may be recorded on a recording medium. Components, functions and processes described in the embodiments may be implemented by a combination of hardware and software.

As described above, although the embodiments have been described with reference to the limited drawings, those skilled in the art may apply various technical modifications and variations based on the above. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

1000: lidar device 100: laser light source unit
110: laser output unit 111: laser diode
112: collimator 120: scanning unit
121: reflector 122: driving unit
123: MEMS mirror 124: lens unit
125: diffraction grating unit 130: control unit
200: image processing unit 210: image sensor unit
220: calculation unit 230: recognition unit

Claims (17)

In the image-based three-dimensional LiDAR (LiDAR) device,
a laser light source for outputting a line laser in the target space; and
and an image processing unit for generating a detection image including the line lasers reflected in the target space, and generating a 3D coordinate map for the target space by using the detection image.
According to claim 1, wherein the laser light source unit
a laser output unit that generates a point-shaped laser light according to a trigger signal;
a scanning unit converting the laser light into a single-line or multi-line laser and outputting it in the target space; and
and a controller for controlling operations of the laser output unit and the scanning unit to output the line laser in the target space.
The method of claim 2, wherein the scanning unit
and a diffraction grating unit configured to divide the laser light into a plurality of line lasers traveling in different optical paths to generate the multi-line lasers.
The method of claim 2, wherein the scanning unit
a reflector for reflecting the single-line or multi-line line laser; and
Lidar device, characterized in that it further comprises a driving unit for adjusting the reflection angle of the reflector.
5. The method of claim 4, wherein the control unit
LiDAR device, characterized in that by controlling the operation of the driving unit to control a light projection angle at which the line laser is irradiated into the target space.
The method of claim 2, wherein the scanning unit
Lidar device, characterized in that it further comprises a galvanometer (galvanometer) or MEMS (Micro-Electro Mechanical Systems) mirror (mirror).
According to claim 5, wherein the control unit
When receiving the frame start signal from the image processing unit, transmitting the trigger signal, controlling the laser output unit to emit laser light for a duration, and controlling the driving unit to change the projection angle when the duration ends Features a lidar device.
The method of claim 7, wherein the control unit
A lidar device, characterized in that the line lasers having a plurality of light transmission angles are controlled to be output within one frame exposure time.
According to claim 2, wherein the control unit
When an odd-numbered frame start signal is received from the image processing unit, the trigger signal is transmitted to control the laser output unit to emit laser light for a duration, and when an even-numbered frame start signal is received, transmission of the trigger signal is omitted. to control the laser output unit not to emit laser light.
The method of claim 1, wherein the image processing unit
an image sensor unit generating a detection image including the line laser; and
LiDAR device, comprising: a arithmetic unit for generating a lidar frame using the detection images, and generating the three-dimensional coordinate map by using the three-dimensional coordinates of the target space extracted from the lidar frame.
11. The method of claim 10, wherein the image sensor unit
By adopting a global shutter method, all pixels included in the image sensor unit detect the line laser during the same frame exposure time.
11. The method of claim 10, wherein the operation unit
Ra, characterized in that by combining N detection images (N is a natural number greater than or equal to 1) received from the image sensor unit to generate one lidar frame, and extracting a plurality of line lasers included in the lidar frame is the device.
11. The method of claim 10, wherein the operation unit
A lidar device, characterized in that by using a triangulation technique, to set the three-dimensional coordinates of each line laser displayed in the lidar frame.
11. The method of claim 10, wherein the operation unit
When the line laser includes a reference pattern and a target pattern in a shape that is vertically spaced apart from the reference pattern by short circuit,
A vertical position value in which the target pattern is vertically spaced apart from the reference pattern, a horizontal position value in which the target pattern is horizontally spaced apart from a reference point, and a height position value corresponding to the reference pattern are calculated, respectively, and the vertical position value , using the horizontal position value and the height position value, the lidar device, characterized in that for generating the three-dimensional coordinate value.
15. The method of claim 14, wherein the operation unit
The three-dimensional coordinate value is generated by further applying the projection angles of the line lasers to the vertical position value, the horizontal position value, and the height position value.
11. The method of claim 10, wherein the image processing unit
LiDAR device, characterized in that it detects objects located in the target space using the three-dimensional coordinate map, and further comprises a recognition unit for recognizing the type of the object.
A 3D map generation method using a lidar device including a laser light source unit and an image processing unit, the method comprising:
The laser light source unit outputting a line laser in the target space; and
and generating, by an image processing unit, a detection image including the line lasers reflected in the target space, and generating a 3D coordinate map of the target space by using the detection image.

Images