KR20220092746A - Method and system for detecting unlearned objects based on stereo camera - Google Patents

Abstract

A method and system for detecting an unlearned object based on a stereo camera are disclosed. The present invention relates to a method for detecting an unlearned object using a first image acquisition part and a second image acquisition part for photographing a plurality of targets. The method may include the steps of: acquiring a first image from the first image acquisition part; acquiring a second image from the second image acquisition part; recognizing the plurality of targets based on the first image and the second image; and distinguishing a pre-learned first target and an unlearned second target among the plurality of targets.

Description

Stereo camera-based unlearned object detection method and system

The present invention relates to a method and system for detecting unlearned objects based on a stereo camera. More particularly, it relates to a method and system for recognizing an object using two cameras, but detecting an unlearned object among a plurality of objects.

The single camera-based vision recognition system has limitations in improving the detection speed because the search area cannot be reduced without prior information or additional information.

Accordingly, a stereo camera-based vision recognition system has emerged. A general stereo camera-based vision recognition system is a vision recognition system for generating a depth map using the disparity of two cameras and extracting each object using this.

As such, vision recognition technology has been focused on contrasting with the features learned to recognize objects. However, since the unlearned object is not detected through deep learning, many problems have occurred in the actual driving environment.

Looking at a technology such as Korean Patent Registration Publication KR101699014B1, the object is recognized based on a stereo camera, but the focus is on recognizing an object based on a pre-learned feature.

Accordingly, there is an increasing need for a technology capable of separately detecting an object requiring additional learning later, except for an object having already learned and knowing a characteristic among a plurality of objects.

KR 101699014 B1

An object of the present invention is to provide a method and system for extracting an unlearned object from among a plurality of objects.

Another object of the present invention is to provide a stereo camera-based unlearned object detection method and system, which can be utilized not only in daily life but also in industrial settings, in a stereo camera-based unlearned object detection method and system.

In order to solve the above problem, the present invention provides a method for detecting an unlearned object using a first image acquisition unit and a second image acquisition unit for photographing a plurality of targets, wherein the first image is obtained from the first image acquisition unit. acquiring, acquiring a second image from the second image acquiring unit, recognizing the plurality of targets based on the first image and the second image, and a pre-learned first target among the plurality of targets and discriminating the non-learned second target.

In addition, recognizing the plurality of targets based on the first image and the second image may include: obtaining a depth map including depth information based on the first image; and binarizing the depth map for each frame. , removing noise by obtaining mode data based on the binarized depth map, and obtaining a plurality of coordinate values for the plurality of targets based on the mode data.

In addition, the step of recognizing the plurality of targets based on the first image and the second image may include performing a first learning algorithm based on the second image and selecting the first target based on the learning result. It may include obtaining a coordinate value.

In addition, the step of discriminating the pre-learned first target and the non-learned second target among the plurality of targets includes the plurality of coordinate values obtained from the first image, and the first obtained from the second image. Based on the coordinate value of the target, the first target and the second target may be distinguished.

In addition, the first image acquisition unit includes a camera module for generating an image image from an optical image, the camera module includes a housing including a through hole in a sidewall, a lens installed in the through hole, and a driving unit for driving the lens can do.

In addition, in order to solve the above problems, the present invention, a first image acquisition unit for photographing a first image of a plurality of targets, a second image acquisition unit for photographing a second image of the plurality of targets, and the first and a controller configured to recognize the plurality of targets based on an image and the second image, and to distinguish a pre-learned first target from a non-learned second target among the plurality of targets.

In addition, the first image acquisition unit includes a camera module that generates an image image from an optical image, wherein the camera module includes a housing including a through hole in a sidewall, a lens installed in the through hole, and a driving unit for driving the lens may include

In addition, the controller obtains a depth map including depth information based on the first image, binarizes the depth map for each frame, and removes noise by obtaining mode data based on the binarized depth map, , it is possible to obtain a plurality of coordinate values for the plurality of targets based on the mode data.

In addition, the control unit recognizes the plurality of targets based on the first image and the second image, performs a first learning algorithm based on the second image, and based on the learning result, the first target It is possible to obtain the coordinate values of

In addition, the controller may distinguish the first target from the second target based on the plurality of coordinate values obtained from the first image and the coordinate values of the first target obtained from the second image have.

In addition, in order to solve the above problem, the present invention provides a non-transitory computer-readable medium storing instructions, which, when executed by a processor, cause the processor to perform the above-described method, a non-transitory computer It may be a readable medium.

The present invention has the effect of effectively detecting an unlearned object using a stereo camera-based system.

In addition, the present invention has the effect of effectively detecting an unlearned object regardless of the pollutants even when the stereo camera is installed in a place with a lot of pollutants in daily life or industrial sites.

The effects obtainable according to the present invention are not limited to the effects mentioned above, and other effects not mentioned may be clearly understood by those of ordinary skill in the art to which this specification belongs from the description below. will be.

1 is a view showing a stereo camera-based unlearned object detection method according to the present invention.
2 is a view showing steps S1300 and S1400 according to the present invention.
3 is a view showing a stereo camera-based unlearned object detection system according to the present invention.
4 is a view showing a control unit according to the present invention.
5 is a diagram illustrating an image acquisition unit according to the present invention.
6 is a view showing a specific embodiment according to the present invention.
BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are included as a part of the detailed description to help the understanding of the present specification, provide embodiments of the present specification, and together with the detailed description, explain the technical features of the present specification.

Hereinafter, the embodiments disclosed in the present specification will be described in detail with reference to the accompanying drawings, but the same or similar components are assigned the same reference numbers regardless of reference numerals, and redundant description thereof will be omitted.

In addition, in describing the embodiments disclosed in the present specification, if it is determined that detailed descriptions of related known technologies may obscure the gist of the embodiments disclosed in this specification, the detailed description thereof will be omitted.

In addition, the accompanying drawings are only for easy understanding of the embodiments disclosed in this specification, and the technical idea disclosed herein is not limited by the accompanying drawings, and all changes included in the spirit and scope of the present invention , should be understood to include equivalents or substitutes.

Terms including an ordinal number such as 1st, 2nd, etc. may be used to describe various elements, but the elements are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

The singular expression includes the plural expression unless the context clearly dictates otherwise.

In the present application, terms such as “comprises” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Hereinafter, a stereo camera-based unlearned object detection method and system according to a preferred embodiment of the present invention will be described in detail based on the above-mentioned contents.

1 is a view showing a stereo camera-based unlearned object detection method according to the present invention. The subject performing the unlearned object detection method according to the present invention may be the unlearned object detection system or the control unit according to the present invention.

1 , the method for detecting an unlearned object according to the present invention includes: acquiring a first image from a first image acquiring unit (S1100), acquiring a second image from a second image acquiring unit (S1200); Recognizing a plurality of targets based on the first image and the second image (S1300) and separating a pre-learned first target from a non-learned second target among the plurality of targets (S1400) may be included. . The first image acquisition unit and the second image acquisition unit may be cameras.

The first image may be an image for extracting a depth map, and the second image may be an image for applying a vision recognition algorithm. The present invention can detect an unlearned object by classifying the unlearned second target.

2 is a view showing steps S1300 and S1400 according to the present invention.

According to FIG. 2 , steps S1300 and S1400 according to the present invention include: obtaining a depth map including depth information based on a first image (S1311); binarizing the depth map for each frame (S1312); The method may include removing noise by obtaining mode data based on the depth map ( S1313 ) and obtaining a plurality of coordinate values for a plurality of targets based on the mode data ( S1314 ).

In addition, the steps S1300 and S1400 according to the present invention further include the steps of performing the first learning algorithm based on the second image (S1321) and obtaining the coordinate values of the first target based on the learning result (S1322). may include

In addition, in steps S1300 and S1400 according to the present invention, the first target and the second target are divided based on the plurality of coordinate values obtained from the first image and the coordinate values of the first target obtained from the second image. It may further include a step (S1410) of doing. Step S1410 may use an IoU (Intersection Of Union) algorithm. Specifically, the IoU algorithm may be an algorithm for obtaining an overlapping ratio based on the ratio of the union and intersection of two Bounding Boxes.

That is, as a result of performing the IoU algorithm, if the overlapping ratio exceeds the threshold value, it is determined that the coordinate values overlap each other, and if the overlapping ratio does not exceed the threshold value, it can be determined that the coordinate values do not overlap each other. Based on this, in the present invention, it is possible to obtain a coordinate value determined not to overlap among a plurality of coordinate values obtained from the first image and a plurality of coordinate values obtained from the second image. In this case, the object corresponding to the obtained coordinate value corresponds to the unlearned object.

3 is a view showing a stereo camera-based unlearned object detection system according to the present invention.

Referring to FIG. 3 , the present invention may include a first image acquisition unit 110 , a second image acquisition unit 120 , and a controller 130 . The first image acquisition unit 110 may photograph a first image of the targets 10 and 11 , and the second image acquisition unit 120 may photograph a second image of the targets 10 and 11 . The control unit 130 recognizes the plurality of targets 10 and 11 based on the first image and the second image, and among the plurality of targets 10 and 11 , the pre-learned first target 10 and the unlearned The second target 11 can be distinguished.

The plurality of targets 10 and 11 may include a first target 10 and a second target 11 . The first target 10 may be an object previously learned through vision recognition. The second target 11 may be an object that has not been previously learned. The unlearned object detection system according to the present invention unlearns a target that does not overlap the coordinate values of the target 10 in the second image from among the plurality of coordinate values for the plurality of targets 10 and 11 in the first image. It can be determined as the second target 11 that is the target.

The controller 130 obtains a depth map including depth information based on the first image, binarizes the depth map for each frame, obtains mode data based on the binarized depth map, removes noise, and collects the mode data. A plurality of coordinate values for a plurality of targets may be obtained based on the plurality of coordinate values. The mode data may mean data having a high frequency appearing in an image for each frame. The depth map may be an image type for vision recognition based on depth information.

In addition, the controller 130 recognizes a plurality of targets based on the first image and the second image, performs a first learning algorithm based on the second image, and coordinates values of the first target based on the learning result can be obtained.

Also, the controller 130 may distinguish the first target from the second target based on the plurality of coordinate values obtained from the first image and the coordinate values of the first target obtained from the second image.

4 is a view showing a control unit according to the present invention.

Referring to FIG. 4 , the control unit 130 may include a processor 131 , a memory, and a communication unit. The processor 131 is a component capable of performing calculations and controlling other devices. Mainly, it may mean a central processing unit (CPU), an application processor (AP), a graphics processing unit (GPU), or the like. In addition, the CPU, AP, or GPU may include one or more cores therein, and the CPU, AP, or GPU may operate using an operating voltage and a clock signal. However, a CPU or AP may consist of a few cores optimized for serial processing, whereas a GPU may consist of thousands of smaller and more efficient cores designed for parallel processing.

The memory 132 stores data supporting various functions of the controller 130 . The memory 132 may store a plurality of application programs (or applications) and commands driven by the controller 130 . At least some of these application programs may be downloaded from an external server through wireless communication.

The memory 132 is a flash memory type, a hard disk type, a solid state disk type (SSD), a silicon disk drive type (SDD), and a multimedia card micro type. ), card-type memory (such as SD or XD memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read (EEPROM) -only memory), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk may include at least one type of storage medium. Also, the memory 132 may include a web storage that performs a storage function on the Internet.

The communication unit 133 transmits/receives information to and from a base station or a vehicle including a communication function through an antenna. The communication unit 133 may include a modulator, a demodulator, a signal processor, and the like.

Wireless communication may refer to communication using a communication facility installed by telecommunication companies and a wireless communication network using a frequency of the communication facility. At this time, the communication unit 133, CDMA (code division multiple access), FDMA (frequency division multiple access), TDMA (time division multiple access), OFDMA (orthogonal frequency division multiple access), SC-FDMA (single carrier frequency division multiple) access), etc., may be used in various wireless communication systems, and the communication unit 133 may also be used in 3rd generation partnership project (3GPP) long term evolution (LTE) and the like. In addition, not only 5G communication that is being commercialized recently, but also 6G, which is scheduled to be commercialized in the future, may be used. However, in the present specification, a pre-installed communication network may be utilized without being limited by such a wireless communication method.

5 is a diagram illustrating an image acquisition unit according to the present invention.

Referring to FIG. 5 , the image acquisition unit of the present invention may refer to the first image acquisition unit 110 and the second image acquisition unit 120 . For convenience of description, the first image acquisition unit 110 is described, but the second image acquisition unit 120 may also have the same configuration.

The first image acquisition unit 110 may include a camera module 111 as a camera. The camera module 111 may generate a video image from the optical image. The camera module 111 may include a housing including a through hole in a sidewall, a lens 113 installed in the through hole, and a driving unit 112 for driving the lens 113 . The through hole may be formed in a size corresponding to the diameter of the lens 113 . The lens 113 may be inserted into the through hole.

The driving unit 112 may be configured to control the lens 113 to move forward or backward. The lens 113 and the driving unit 112 may be connected in a conventionally known manner, and the lens 113 may be controlled by the driving unit 112 in a conventionally known manner.

In order to obtain various image images, the lens 113 needs to be exposed to the outside of the housing constituting the camera module 111 or the image acquisition unit 110 .

Preferably, the lens 113 is a siloxane-based compound represented by the following [Formula 1] on its surface; It may be coated with a coating composition containing an organic solvent, inorganic particles and a dispersing agent.

[Formula 1]

(Where n is an integer from 1 to 100.)

When the lens 1321 is coated with the coating composition, it can exhibit excellent water repellency and stain resistance, so even if the lens 113 installed outside the vehicle is exposed to a polluted environment for a long time, images or images that can be used as road information are collected. can do.

The inorganic particles may be selected from the group consisting of silica, alumina, and mixtures thereof. The average diameter of the inorganic particles is 70 to 100 μm, but is not limited to the above example. After the inorganic particles are formed as a coating layer 114 on the surface of the lens 113 , physical strength may be improved, and the viscosity may be maintained within a certain range to increase moldability.

The organic solvent is selected from the group consisting of methyl ethyl ketone (MEK), toluene, and mixtures thereof, and preferably methyl ethyl ketone may be used, but is not limited thereto.

As the dispersant, a polyester-based dispersant may be used, and specifically, as a polyester-based dispersion stabilizer composed of a copolymer of 2-methoxypropyl acetate and 1-methoxy-2-propyl acetate, TEGO-Disperse 670 (manufacturer : EVONIK), but is not limited to the above examples and any dispersant obvious to those skilled in the art can be used without limitation.

The coating composition may further include a stabilizer as other additives, and the stabilizer may include a UV absorber, an antioxidant, and the like, but is not limited to the above examples and may be used without limitation.

For forming the coating layer 114, the coating composition may include a siloxane-based compound represented by Chemical Formula 1; organic solvents, inorganic particles and dispersants.

The coating composition may include 40 to 60 parts by weight of the siloxane-based compound represented by Formula 1, 20 to 40 parts by weight of inorganic particles, and 5 to 15 parts by weight of a dispersant based on 100 parts by weight of the organic solvent. In the case of the above range, a synergistic effect is expressed to the extent that the water repellency effect due to the interaction of each component has a critical significance, and when it is out of the above range, the synergistic effect is rapidly reduced or almost absent.

More preferably, the viscosity of the coating composition is 1500 to 1800 cP, and when the viscosity is less than 1500 cP, when applied to the surface of the lens 113, there is a problem that the formation of the coating layer 114 is not easy because it flows down, and exceeds 1800 cP In this case, there is a problem in that the formation of a uniform coating layer 114 is not easy.

[Preparation Example 1: Preparation of coating layer]

1. Preparation of coating composition

A coating composition was prepared by mixing methyl ethyl ketone with the siloxane-based compound represented by the above [Formula 1], inorganic particles and a dispersant:

A more specific composition of the antistatic composition is shown in Table 1 below.

TX1 TX2 TX3 TX4 TX5 organic solvent 100 100 100 100 100 polysiloxane 30 40 50 60 70 inorganic particles 10 20 30 40 50 dispersant One 5 10 15 20

(unit parts by weight) 2. Preparation of coating layer

The coating composition of DX1 to DX5 was applied to one surface of the lens 113 and cured to form a coating layer 114 .

[Experimental example]

1. Evaluation of surface appearance

Due to the difference in viscosity of the coating composition, after the coating layer 114 was prepared, a sensory evaluation was performed as to whether a uniform surface was formed. Whether or not a uniform coating layer 114 was formed was evaluated, and evaluation was performed according to the following criteria.

○: uniform coating layer formation

×: Formation of a non-uniform coating layer

TX1 TX2 TX3 TX4 TX5 sensory evaluation Х ○ ○ ○ Х

When the coating layer 114 is formed, if the viscosity is less than a certain level, a flow occurs on the surface of the lens 113 , and after the curing process, it is difficult to form a uniform coating layer 114 . Accordingly, there may be a problem in that the production yield is lowered. In addition, even when the viscosity is too high, it is difficult to uniformly apply the composition to form a uniform coating layer 114 .

2. Measurement of water repellency angle

After forming the coating layer 114 on the surface of the lens 113, the results of measuring the water repellency angle are shown in Table 3 below.

) advancing contact angle (

) stop contact angle (

) receding contact angle (

) TX1 117.1±2.9 112.1±4.1 < 10 TX2 132.4±1.5 131.5±2.7 141.7±3.4 TX3 138.9±3.0 138.9±2.7 139.8±3.7 TX4 136.9±2.0 135.6±2.6 140.4±3.4 TX5 116.9±0.7 115.4±3.0 < 10

As shown in Table 3, after forming the coating layer 114 using the coating composition of TX1 to TX5, the result of measuring the contact angle was confirmed. For TX1 and TX5, the receding contact angle was measured to be less than 10 degrees. That is, when it is out of the optimal range for preparing the coating composition, it was confirmed that the phenomenon of pinning water droplets occurs. On the other hand, it was confirmed that the peening phenomenon did not occur in TX2 to 4, thereby exhibiting an excellent waterproof effect.

3. Pollution resistance evaluation

The lens 113 having the coating layer 114 formed according to the above embodiment on the outside of the facility was attached to the model camera, and exposed to the general road driving environment for 4 days. As the comparative example (Con), the same lens 113 on which the coating layer 114 was not formed was used, and in each embodiment, the model camera was attached to the same position of the vehicle.

Thereafter, the degree of contamination of the lens 113 before and after the experiment was evaluated as related, and for objective comparison, the result was evaluated as an index of 1 to 10 in comparison with the comparative example in which the coating layer 114 was not formed. shown in In the following index, the lower the number, the better the stain resistance.

Con TX1 TX2 TX3 TX4 TX5 stain resistance 10 7 3 3 3 8

(Unit: Index) Referring to Table 4, when the coating layer 114 is formed on the lens 113, the lidar sensor or camera is installed on the outside of the vehicle and the lens 113 is exposed to the outside, but the high pollution resistance It can be seen that image data can be collected in a form that is easy to analyze over a long period of time. In particular, in the case of TX2 to TX4, it can be seen that the stain resistance by the coating layer 114 is very excellent.

6 is a view showing a specific embodiment according to the present invention.

Referring to FIG. 6 , the depth map images a and c on the left may be the above-described first images, and the RGB image images b and d on the right may be the above-described second images. In addition, in the case of a depth map image, by using Mode-ROI, the background of the depth map image can be deleted and the invention can be detected by displacing the contour of the object.

According to FIG. 6, all three objects were detected using Mode-ROI in the figure (c). However, in (d), two objects were detected through the YOLO algorithm. However, the can in the middle was not detected by the algorithm in the figure (d). Therefore, as a result of applying the IoU algorithm by combining (a, c) and (b, d), the Coke can in the middle was detected as an unlearned object.

The present invention described above can be modeled as computer readable code on a medium in which a program is recorded. The computer-readable medium includes all kinds of recording devices in which data readable by a computer system is stored. Examples of computer-readable media include Hard Disk Drive (HDD), Solid State Disk (SSD), Silicon Disk Drive (SDD), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. It also includes modeling in the form of a carrier wave (eg, transmission over the Internet). Accordingly, the above detailed description should not be construed as restrictive in all respects but as exemplary. The scope of the present specification should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present specification are included in the scope of this specification.

Any or other embodiments of the present invention described above are not mutually exclusive or distinct. Any of the above-described embodiments or other embodiments of the present invention may be combined or combined with each configuration or function.

The above detailed description should not be construed as restrictive in all respects and should be considered as exemplary. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are included in the scope of the present invention.

10: learned target
11: Untrained target
110: first image acquisition unit
120: second image acquisition unit
130: control unit

Claims (11)

In the unlearned object detection method using a first image acquisition unit and a second image acquisition unit for photographing a plurality of targets,
acquiring a first image from the first image acquiring unit;
acquiring a second image from the second image acquiring unit;
recognizing the plurality of targets based on the first image and the second image; and
Distinguishing a pre-learned first target from among the plurality of targets and a non-learned second target; will include, a stereo camera-based unlearned object detection method.
According to claim 1,
Recognizing the plurality of targets based on the first image and the second image includes:
obtaining a depth map including depth information based on the first image;
binarizing the depth map for each frame;
removing noise by obtaining mode data based on the binarized depth map; and
Acquiring a plurality of coordinate values for the plurality of targets based on the mode data; will include, a stereo camera-based unlearned object detection method.
3. The method of claim 2,
Recognizing the plurality of targets based on the first image and the second image includes:
performing a first learning algorithm based on the second image; and
Acquiring the coordinates of the first target based on the learning result; comprising a, stereo camera-based unlearned object detection method.
4. The method of claim 3,
The step of distinguishing the pre-learned first target from the non-learned second target among the plurality of targets,
Based on the plurality of coordinate values obtained from the first image and the coordinate values of the first target obtained from the second image, the first target and the second target are distinguished from each other, stereo camera-based A method for detecting unlearned objects.
The method of claim 1,
The first image acquisition unit,
A camera module that generates a video image from an optical image; includes,
The camera module is
a housing including a through hole in the side wall;
a lens installed in the through hole; and
A method for detecting unlearned objects based on a stereo camera comprising a; a driving unit for driving the lens.
a first image acquisition unit for capturing a first image of a plurality of targets;
a second image acquisition unit for capturing second images of the plurality of targets; and
A control unit that recognizes the plurality of targets based on the first image and the second image, and distinguishes a pre-learned first target from a non-learned second target among the plurality of targets; stereo camera-based including; Unlearned object detection system.
7. The method of claim 6,
The first image acquisition unit,
A camera module for generating a video image from the optical image; includes,
The camera module,
a housing including a through hole in the sidewall;
a lens installed in the through hole; and
A stereo camera-based unlearned object detection system that includes; a driving unit for driving the lens.
7. The method of claim 6,
The control unit is
Obtaining a depth map including depth information based on the first image, binarizing the depth map for each frame, obtaining mode data based on the binarized depth map to remove noise, and based on the mode data to obtain a plurality of coordinate values for the plurality of targets, a stereo camera-based unlearned object detection system.
9. The method of claim 8,
The control unit is
Recognizing the plurality of targets based on the first image and the second image, performing a first learning algorithm based on the second image, and obtaining the coordinate values of the first target based on the learning result , a stereo camera-based unlearned object detection system.
10. The method of claim 9,
The control unit is
Based on the plurality of coordinate values obtained from the first image and the coordinate values of the first target obtained from the second image, the first target and the second target are distinguished from each other, stereo camera-based Unlearned object detection system.
A non-transitory computer-readable medium having stored thereon instructions, comprising:
The non-transitory computer-readable medium, wherein the instructions, when executed by a processor, cause the processor to perform the method of claim 1 .

Images