KR20210122155A - Integrated fusion sensor apparatus including detachable light source with built-in intelligent camera - Google Patents

Abstract

Disclosed is an integrated fusion sensor apparatus. According to an embodiment of the present invention, the integrated fusion sensor apparatus comprises: a detachable lidar sensor wherein a transmission module transmitting a lidar light signal and a reception module receiving the lidar light signal are detached; an intelligent image sensor acquiring a camera image of an object; and a control unit integrating and controlling object recognition using the detachable lidar sensor and object recognition using the intelligent image sensor. The transmission module comprises a transmission unit generating and outputting the lidar light signal and a transmission mirror reflecting the light signal output from the transmission unit. The reception module comprises a reception unit receiving the lidar light signal reflected and returning from the object and a reception mirror reflecting the received light signal from the reception unit.

Description

Integrated fusion sensor apparatus including detachable light source with built-in intelligent camera}

The present invention relates to a LIDAR light sensor.

LIDAR (Light Detection And Ranging) is a device that detects an object using light and measures the distance to the object. Lidar is similar to radar RADAR (Radio Detection And Ranging) in function, but it is different from radar using radio waves in that it measures using light. Due to the difference in the Doppler effect between light and microwaves, lidar has superior azimuth resolution and distance resolution compared to radar.

LiDAR devices emit laser pulses from satellites or aircraft, and aerial lidars that receive pulses backscattered by particles in the atmosphere from a ground observatory have been mainstream. It has been used to measure the presence and movement of smoke, aerosols, and cloud particles, and to analyze the distribution of dust particles in the atmosphere or air pollution.

Recently, land lidar surveillance and reconnaissance robots, combat robots, unmanned surface ships, unmanned helicopters, drones, etc. that perform obstacle detection, terrain modeling, and location acquisition functions are used in defense fields, civil mobile robots, intelligent cars, unmanned vehicles, Research is being actively carried out with the application of heavy equipment, subway, and road toll collection in mind.

According to an embodiment, an integrated fusion sensor device in which a separate lidar sensor and an intelligent image sensor are integrated is proposed.

An integrated fusion sensor device according to an embodiment includes a separate lidar sensor in which a transmission module for transmitting a lidar optical signal and a reception module for receiving a lidar optical signal are separated, and an intelligent image sensor for acquiring a camera image of an object; , a control unit that integrates and controls object recognition using a separate lidar sensor and deep learning-based object recognition using an intelligent image sensor, and the transmitting module includes a transmitter that generates and outputs a LiDAR optical signal and a light output through the transmitter It includes a transmitting mirror that reflects the signal, and the receiving module includes a receiving unit for receiving the LIDAR optical signal reflected from the object and returning, and a receiving mirror for reflecting the optical signal received through the receiving unit.

The integrated fusion sensor device may further include a connection unit for coupling the transmission mirror and the reception mirror, and a two-axis motor coupled to the connection unit and rotating the transmission mirror and the reception mirror.

The two-axis motor may have an encoder that separates and operates the transmitting module and the receiving module to create a separate optical structure.

The control unit controls the driving of both axis motors, but any one of a first mode in which the transmitting mirror and the receiving mirror are rotated in the same direction at the same time, a second mode in which the transmitting mirror and the receiving mirror are rotated in different directions, and the transmitting mirror and the receiving mirror The third mode of rotating one mirror and the fourth mode of rotating at least one of the transmitting mirror and the receiving mirror according to a user setting may be entered to control the presence or absence of rotation and the rotation direction of each mirror.

The control unit detects each object using raw data of the separate lidar sensor and the intelligent image sensor, calculates the distance and angle of the object in space from the object detection result of the separate lidar sensor, and the intelligent image sensor The IoU of the object on the image is calculated from the object detection result obtained through

The integrated fusion sensor device detects an object within a scan range of 270 degrees within a preset distance through a separate lidar sensor, and can integrally detect an object with a scan range of 90 degrees in front through a separate lidar sensor and an intelligent image sensor. .

Each mirror may have a polygonal columnar shape including a side surface or a cylindrical shape.

The integrated fusion sensor device further includes a transmitting cover and a receiving cover, and each cover may allow a light source to be constantly output and received.

According to the integrated fusion sensor device according to an embodiment, the problem of the existing lidar light sensor (the problem of signal interference in various bad weather conditions caused difficulties in commercialization, and the lidar characteristic limited to product characteristics rather than user requirements) Many problems could not be solved).

The integrated fusion sensor device is capable of transmitting and receiving signals at any position, up, down, left, or right as the transmitting module and the receiving module of the lidar sensor are separated and driven integrally.

Various sensor functions can be created according to the driving mode of the two-axis motor, and can be customized according to the user's convenience.

By installing an intelligent image sensor and providing an integrated sensor that combines a lidar signal and an image signal, the problems of the conventional lidar sensor can be supplemented and can be used as a safety device for autonomous vehicles and industrial use in the future.

The intelligent image sensor can solve the problem of installing at a distance of more than ±5 degrees from the sensing plane due to interference from interfering light sources (incandescent, fluorescent, stroboscope light, flashing sign, sunlight, infrared light source).

A lidar sensor that detects obstacles and informs the distance and angle, and an intelligent image sensor that transmits images of obstacles and executes deep learning learning with the images Combined with the ability to sense angles, integrated control is possible.

While minimizing the influence of external interference and internal signal crosstalk according to the characteristics of the lidar sensor, the original characteristics of lidar can be exhibited and lidar characteristics can be utilized in various locations.

The role of the LiDAR light sensor to accurately output signals and to properly receive these signals to increase the accuracy of accurate location information of the location scanned by the lidar and zone setting and object recognition, and to create position information from various angles This is important, and can be overcome with discrete optical sensor technology.

By grafting the intelligent image sensor to the lidar sensor, the problem of the bad weather environment of the lidar can be improved by learning through the deep learning of the intelligent image sensor by using the detection ability of the lidar and the image processing ability of the camera.

It is possible to provide a precise integrated sensor with an algorithm that supplements the object detection ability lacking in the intelligent image sensor with the distance and detection ability of the lidar sensor, and improves the object recognition and object recognition ability with the intelligent image sensor. Based on this, an integrated safety sensor industry can be built with composite image technology.

1 is a view showing the configuration of an integrated fusion sensor device according to an embodiment of the present invention;
2 is a view showing a detailed configuration of an integrated fusion sensor device according to an embodiment of the present invention;
3 is a view showing an appearance of an integrated fusion sensor device according to an embodiment of the present invention;
4 is a diagram illustrating a scan range of an integrated fusion sensor device according to an embodiment of the present invention.

Advantages and features of the present invention, and a method for achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only these embodiments allow the disclosure of the present invention to be complete, and common knowledge in the art to which the present invention pertains. It is provided to fully inform the possessor of the scope of the invention, and the present invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout.

In the description of the embodiments of the present invention, if it is determined that a detailed description of a known function or configuration may unnecessarily obscure the gist of the present invention, the detailed description will be omitted, and the terms to be described later are used in the embodiment of the present invention. These terms are defined in consideration of the function of Therefore, the definition should be made based on the content throughout this specification.

Each block in the accompanying block diagram and combinations of steps in the flowchart may be executed by computer program instructions (execution engine), which computer program instructions are executed by the processor of a general-purpose computer, special-purpose computer, or other programmable data processing device. It may be mounted so that its instructions, which are executed by the processor of a computer or other programmable data processing device, create means for performing the functions described in each block of the block diagram or in each step of the flowchart.

These computer program instructions may also be stored in a computer-usable or computer-readable memory that may direct a computer or other programmable data processing device to implement a function in a particular manner, and thus the computer-usable or computer-readable memory. It is also possible for the instructions stored in the block diagram to produce an article of manufacture containing instruction means for performing a function described in each block of the block diagram or each step of the flowchart.

And since the computer program instructions may be mounted on a computer or other programmable data processing device, a series of operational steps is performed on the computer or other programmable data processing device to create a computer-executable process to create a computer or other programmable data processing device. It is also possible that the instructions for performing the data processing apparatus provide steps for executing the functions described in each block of the block diagram and each step of the flowchart.

In addition, each block or step may represent a module, segment, or portion of code comprising one or more executable instructions for executing specified logical functions, and in some alternative embodiments the blocks or steps referred to in the block or steps. It should be noted that it is also possible for functions to occur out of sequence. For example, it is possible that two blocks or steps shown one after another may be performed substantially simultaneously, and also the blocks or steps may be performed in the reverse order of the corresponding functions, if necessary.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the embodiments of the present invention illustrated below may be modified in various other forms, and the scope of the present invention is not limited to the embodiments described below. The embodiments of the present invention are provided to more completely explain the present invention to those of ordinary skill in the art to which the present invention pertains.

1 is a diagram illustrating a configuration of an integrated fusion sensor device according to an embodiment of the present invention.

Referring to FIG. 1 , the integrated fusion sensor device 1 includes a detachable lidar sensor 10 , an intelligent image sensor 20 , and a controller 30 .

The detachable lidar sensor 10 has a form in which a transmitting module and a receiving module are separated. On the other hand, the integrated lidar sensor has a single cover, so that light output and light reception are performed in one, so internal crosstalk and signal interference cannot be avoided. Due to this problem, it is not possible to reduce the error rate in short-distance measurement, and it is difficult to solve problems caused by adverse conditions (eg rain, snow, hail, color, sunlight, etc.). In order to solve this structural problem, the present invention provides a detachable lidar sensor 10 in which a transmission module for transmitting a lidar optical signal to an object and a reception module for receiving a lidar optical signal reflected back from the object are separated. suggest The detachable lidar sensor 10 has a separate transmit optical structure and a receive optical structure, and as it has a dual-axis motor, 270 degree optical scanning is possible, and signals can be detected in various directions up, down, left, and right.

When the transmitting module and the receiving module are separated as described above, problems caused by external influences and internal influences can be solved, and a more precise measurement value can be obtained. In addition, it is possible to miniaturize and make complex products.

The integrated fusion sensor device (1) is an industrial fusion sensor to which deep learning technology is attracting attention in the field of artificial intelligence. It uses an intelligent image sensor (20) that has more accurate situational judgment than the existing simplified sensor and can be used more safely by users. do. The intelligent image sensor 20 acquires a camera image of the object. If only the lidar sensor is used, there is a problem that the probability of malfunction is high due to performance degradation in bad weather and the price of the product is high. In order to improve this, it is intended to increase the accuracy of object recognition by using the intelligent image sensor 20 .

The controller 30 controls the overall operation of the integrated fusion sensor device 1 . For example, it controls the operation and synchronization of the transmitting module and the receiving module of the separate lidar sensor 10 , and object recognition using the separate lidar sensor 10 and deep learning-based object recognition using the intelligent image sensor 20 . integrated and controlled. An object is recognized using the detachable lidar sensor 10 and the intelligent image sensor 20, but the distance and angle of the object can be measured using the detachable lidar sensor 10, and the intelligent image sensor 20 It can increase the accuracy of object recognition and improve intelligent learning ability.

It detects the angle, distance, and score information of an object obtained through object detection using a support vector machine (SVM) of the detachable lidar sensor 10, and is displayed on the image using CNN through the intelligent image sensor 20 Detects an existing object. The control unit 30 fuses the results detected through the separate lidar sensor 10 and the intelligent image sensor 20 .

The control unit 30 detects an obstacle using the detachable lidar sensor 10 and calculates the distance and angle, and transmits an image of the obstacle using the intelligent image sensor 20 and executes deep learning learning with this image. . It is possible to combine object recognition through field learning of the camera image of the intelligent image sensor 20 and the sensing ability of the distance and angle to the object of the detachable lidar sensor 10 . The two sensors 10 and 20 may be controlled in an integrated manner using an interface module and a display module.

The controller 30 detects each object by using the raw data of the separate lidar sensor 10 and the intelligent image sensor 20 . The distance and angle of the object in space are calculated from the object detection result of the separate lidar sensor 10 , and Intersection over (IoU) of the object on the image from the object detection result obtained through the intelligent image sensor 20 Union) is calculated. Then, the distance and angle of the object in space and the IoU of the object in the image are fused.

2 is a diagram illustrating a detailed configuration of an integrated fusion sensor device according to an embodiment of the present invention.

1 and 2 , the detachable lidar sensor 10 includes a transmission module 12 and a reception module 14 , and may include a connection unit 15 and a two-axis motor 16 . The transmitting module 12 includes a transmitting unit 121 , a transmitting mirror 122 and a transmitting cover 123 , and the receiving module 14 includes a receiving unit 142 , a receiving mirror 141 and a receiving cover 143 . do.

The transmitter 121 generates and outputs the LIDAR optical signal, and the transmission mirror 122 reflects the optical signal output through the transmitter 121 . The receiver 142 receives the LIDAR optical signal reflected from the object and returns, and the receiving mirror 141 reflects the optical signal received through the receiver 142 .

In more detail, the transmission module 12 is a module configured to emit a laser beam of a uniform pattern through an optical system with a modulated laser beam, and includes an LD (Laser Diode) as a light source, an Illumination Board for controlling a laser modulation signal, and an optical system. is composed of LD (Laser Diode) is a very important component for measuring the distance of LiDAR. Collimator Lens is a lens that integrates laser light signals and outputs them uniformly. The size of the transmission mirror 122 may be changed according to a laser beam, transmission signal quality, and the like. The transmission mirror 122 may have a polygonal columnar shape including a side surface or a cylindrical shape. In this embodiment, an elliptical shape is shown.

The receiving module 14 includes a PD (Photodiode) Array that recognizes a plurality of laser point groups that are reflected back from the object, an optical system used to collect the reflected and returned light, and a signal processing unit that processes signals transmitted from the PD Array. includes The shape of the receiving mirror 141 may be of various shapes, the side surface may be coated with a mirror, and may be a flat polygonal column shape or a cylindrical shape. In this embodiment, an elliptical shape is shown.

The detachable lidar sensor 10 includes a transmitting cover 123 and a receiving cover 143 in which a laser light is output to detect an object at a wide angle, protects internal optical components, and has a cylindrical shape and a comb-like structure. . The transmitting cover 123 and the receiving cover 143 may be made of a material through which a laser can pass.

In the detachable lidar sensor 10 according to an embodiment, the cover is composed of a detachable cover in which the transmitting cover 123 and the receiving cover 143 are separated, and separating the transmitting mirror 122 and the receiving mirror 141, The transmission mirror 122 and the reception mirror 141 are rotated using a biaxial motor 16 having two axes. It has a connection part 15 for connecting and fixing the transmission cover 123 and the reception cover 143, and the angle of the internal mirror can be changed according to the use of the product. By rotating this mirror in the same direction and in different directions, it is possible to sense a wide-angle sensing area. The transmission mirror 122 transmits the light source according to rotation through the lower shaft of the both-axis motor 16, and the transmission of the light source can be configured differently depending on the transmission angle, and the transmission unit 121 for generating the transmitted light source is mounted together. . The reception mirror 141 receives the light source according to rotation through the upper shaft of the both-axis motor 16, and the reception of the light source may be configured differently depending on the reception angle, and the reception unit 142 is mounted together.

In the case of an integrated lidar sensor, since the arrangement of optical components greatly influences the performance, it cannot be transformed into various systems depending on the product use. However, since the detachable lidar sensor 10 according to an embodiment uses a separate light source, the arrangement of optical components is easy, problems caused by optical signals generated from outside and inside can be solved, and there is a restriction on the vertical and horizontal positions of the product. can be used without

The receiving cover 143 receives a sensing signal to the receiving mirror 141 , and the transmitting cover 123 outputs the sensing signal to the transmitting mirror 122 . The transmission cover 123 and the reception cover 143 may be distinguished through a separate mark, and may be divided without a separate distinction mark. The transmission cover 123 is configured such that the light source is constantly output, and the reception cover 143 is configured so that the light source is constantly received.

The two-axis motor 15 has two axes, ie, two axes, and supports the transmitting cover 123 and the receiving cover 143 . The two-axis motor 16 may have a built-in encoder that separates the transmission module 12 and the reception module 14 to operate. The two-axis motor 16 may be positioned between the receiving mirror 141 and the transmitting mirror 122 . One surface of the two-axis motor 16 may be connected to the receiving mirror 141 through the connection unit 15 , and the other surface of the bi-axial motor 16 may be connected to the transmitting mirror 122 through the connection unit 15 . The connection part 15 may be configured by one shaft, and may be configured by using two shafts.

The control unit 30 controls the driving of the two-axis motor 16 . For example, the first mode in which the transmission mirror 122 and the reception mirror 141 are simultaneously rotated in the same direction through the driving of the two-axis motor 16, the transmission mirror 122 and the reception mirror 141 are rotated in different directions A second mode of rotating to , a third mode of rotating any one of the transmitting mirror 122 and the receiving mirror 141, rotating at least one of the transmitting mirror 122 and the receiving mirror 141 according to a user setting and enters any one of the fourth modes to control the presence or absence of rotation of each of the mirrors 122 and 141 and the direction of rotation.

When the transmission mirror 122 and the reception mirror 141 are dynamically coupled to the two-axis motor 16, when the two-axis motor 16 rotates, the reception mirror 141 rotates together. In addition, when the transmission mirror 122 is also coupled to the two-axis motor 16 by the connection unit 15 , it rotates like the reception mirror 141 .

The intelligent image sensor 20 includes cameras 18-1, 18-2, and 18-3. In FIG. 2 , the positions of the cameras 18 - 1 , 18 - 2 and 18 - 3 are not limited thereto. The cameras 18-1, 18-2, 18-3 can use different distances, such as for short-range, medium-range, etc., increase precision with an intelligent algorithm, establish the establishment and diversity of the integrated fusion algorithm, and, if necessary, It can be applied to necessary uses in industry such as cameras. It is possible to solve the problem of the lidar sensor through the cameras 18-1, 18-2, 18-3. For example, it is possible to solve problems such as obstacle detection errors and disturbance light (solar light) interference according to deep learning learning in bad weather of the lidar sensor, and it is possible to evolve into an autonomous driving system.

3 is a view illustrating an external appearance of an integrated fusion sensor device according to an embodiment of the present invention.

In more detail, (a) is a side view, (b) is a front view of the front, (c) is a reverse image of the front, (d) is a side view, respectively.

4 is a diagram illustrating a scan range of an integrated fusion sensor device according to an embodiment of the present invention.

1 and 4 , the integrated fusion sensor device 1 detects all 270° within a preset distance (eg, 50 m) while the detachable lidar sensor 10 detects all of them, and in the case of a front 90°, separate type The lidar sensor 10 and the intelligent image sensor 20 may be integrated to detect.

So far, the present invention has been looked at focusing on the embodiments thereof. Those of ordinary skill in the art to which the present invention pertains will understand that the present invention may be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the present invention is indicated in the claims rather than the foregoing description, and all differences within the scope equivalent thereto should be construed as being included in the present invention.

Claims (8)

a detachable lidar sensor in which a transmitting module for transmitting a lidar optical signal and a receiving module for receiving a lidar optical signal are separated;
an intelligent image sensor for acquiring a camera image of an object; and
a controller for integrating and controlling object recognition using the separate lidar sensor and deep learning-based object recognition using the intelligent image sensor; including,
The sending module is
a transmitter for generating and outputting a lidar optical signal; and
a transmitting mirror that reflects the optical signal output through the transmitting unit; includes,
The receiving module is
a receiver for receiving the LIDAR optical signal reflected from the object; and
a receiving mirror for reflecting the optical signal received through the receiving unit; An integrated fusion sensor device comprising a. The method of claim 1, wherein the integrated fusion sensor device is
a connection unit for coupling the transmit mirror and the receive mirror; and
a two-axis motor coupled to the connector and rotating the transmitting mirror and the receiving mirror;
Integrated fusion sensor device, characterized in that it further comprises. 3. The two-axis motor according to claim 2,
An integrated fusion sensor device, characterized in that an encoder is built-in to separate and operate the transmitting module and the receiving module in order to create a separate optical structure. The method of claim 2, wherein the control unit
A first mode for controlling the driving of both axis motors at the same time rotating the transmitting mirror and the receiving mirror in the same direction, a second mode for rotating the transmitting mirror and the receiving mirror in different directions, any one of the transmitting mirror and the receiving mirror Integrated fusion characterized in that it enters into any one of the third mode of rotating the , and the fourth mode of rotating at least one of the transmitting mirror and the receiving mirror according to user settings to control the presence or absence of rotation and the direction of rotation of each mirror sensor device. The method of claim 1, wherein the control unit
Each object is detected using the raw data of the separate LiDAR sensor and the intelligent image sensor,
Calculate the distance and angle of the object in space from the object detection result of the separate lidar sensor,
Calculate the IoU of the object on the image from the object detection result obtained through the intelligent image sensor,
An integrated fusion sensor device characterized in that it fuses the distance and angle of the object in space and the IoU of the object in the image. The method of claim 1, wherein the integrated fusion sensor device is
Detects an object within a scan range of 270 degrees within a preset distance through a detachable lidar sensor,
An integrated fusion sensor device, characterized in that it integrally detects an object in a scan range of 90 degrees in front through a separate lidar sensor and an intelligent image sensor. The method of claim 1, wherein each mirror is
An integrated fusion sensor device, characterized in that it has a polygonal columnar shape or a cylindrical shape including a side surface. The method of claim 1, wherein the integrated fusion sensor device is
transmission cover; and
receiving cover; further comprising,
An integrated fusion sensor device, characterized in that each cover allows the light source to be output and received uniformly.

Images