KR102589433B1 - Method and system for detecting unlearned objects based on stereo camera - Google Patents

Abstract

A stereo camera-based untrained object detection method and system are disclosed. The present invention provides a method for detecting an untrained object using a first image acquisition unit and a second image acquisition unit for photographing a plurality of targets, comprising the steps of acquiring a first image from the first image acquisition unit, and acquiring the second image. Obtaining a second image from a unit, recognizing the plurality of targets based on the first image and the second image, and distinguishing between a pre-learned first target and an unlearned second target among the plurality of targets. It may include steps.

Description

Stereo camera-based unlearned object detection method and system {METHOD AND SYSTEM FOR DETECTING UNLEARNED OBJECTS BASED ON STEREO CAMERA}

The present invention relates to a method and system for detecting untrained objects based on a stereo camera. More specifically, it relates to a method and system for recognizing an object using two cameras and detecting an untrained object among a plurality of objects.

A single camera-based vision recognition system had limitations in improving detection speed because the search area could not be reduced without prior information or additional information.

Accordingly, a vision recognition system based on a stereo camera emerged. A general stereo camera-based vision recognition system is a vision recognition system that uses the disparity of two cameras to create a depth map and uses it to extract each object.

In this way, vision recognition technology has focused on comparing features that have been learned to recognize objects. However, because untrained objects cannot be detected through deep learning, many problems have occurred in actual operating environments.

Looking at technologies such as Korean Patent Registration Publication KR101699014B1, the focus is on recognizing objects based on features learned in advance while recognizing objects based on stereo cameras.

Accordingly, the need for technology that can separately detect objects that require additional learning later, excluding objects whose characteristics are already learned and known among a plurality of objects, is increasing.

KR 101699014 B1

The purpose of the present invention is to provide a method and system for extracting untrained objects from a plurality of objects.

Additionally, the purpose of the present invention is to provide a stereo camera-based unlearned object detection method and system that can be used not only in daily life but also in industrial settings.

In order to solve the above-described problem, the present invention provides a method for detecting an untrained object using a first image acquisition unit and a second image acquisition unit for photographing a plurality of targets, and the first image is obtained from the first image acquisition unit. acquiring a second image from the second image acquisition unit, recognizing the plurality of targets based on the first image and the second image, and selecting a pre-learned first target among the plurality of targets. It may include a step of distinguishing between the target and the unlearned second target.

In addition, recognizing the plurality of targets based on the first image and the second image includes obtaining a depth map including depth information based on the first image, and binarizing the depth map for each frame. , It may include obtaining mode data based on the binarized depth map to remove noise, and obtaining a plurality of coordinate values for the plurality of targets based on the mode data.

In addition, recognizing the plurality of targets based on the first image and the second image includes performing a first learning algorithm based on the second image and selecting the first target based on the learning result. It may include the step of acquiring coordinate values.

In addition, the step of distinguishing between a pre-learned first target and an unlearned second target among the plurality of targets includes the plurality of coordinate values obtained from the first image, and the first target obtained from the second image. Based on the coordinate value of the target, the first target and the second target can be distinguished.

In addition, the first image acquisition unit includes a camera module that generates an image image from an optical image, and the camera module includes a housing including a through hole in a side wall, a lens installed in the through hole, and a driver that drives the lens. can do.

In addition, in order to solve the above-described problem, the present invention includes a first image acquisition unit for capturing first images of a plurality of targets, a second image acquisition unit for capturing second images of the plurality of targets, and the first image acquisition unit for capturing the first images of the plurality of targets. It may include a control unit that recognizes the plurality of targets based on the image and the second image, and distinguishes between a pre-learned first target and an unlearned second target among the plurality of targets.

In addition, the first image acquisition unit includes a camera module that generates an image image from an optical image, and the camera module includes a housing including a through hole in a side wall, a lens installed in the through hole, and a driver that drives the lens. It can be included.

In addition, the control unit obtains a depth map including depth information based on the first image, binarizes the depth map for each frame, and obtains mode data based on the binarized depth map to remove noise. , a plurality of coordinate values for the plurality of targets can be obtained based on the mode data.

In addition, the control unit recognizes the plurality of targets based on the first image and the second image, performs a first learning algorithm based on the second image, and selects the first target based on the learning result. The coordinate values of can be obtained.

In addition, the control unit may distinguish the first target and the second target based on the plurality of coordinate values obtained from the first image and the coordinate value of the first target obtained from the second image. there is.

In addition, in order to solve the above-mentioned problem, the present invention is a non-transitory computer-readable medium storing instructions, which, when executed by a processor, cause the processor to perform the above-described method. It may be a readable medium.

The present invention has the effect of effectively detecting untrained objects using a stereo camera-based system.

In addition, the present invention has the effect of effectively detecting untrained objects regardless of the contaminants, even when a stereo camera is installed in a place with a lot of contaminants in daily life or industrial sites.

The effects that can be obtained according to the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below. will be.

Figure 1 is a diagram showing a stereo camera-based untrained object detection method according to the present invention.
Figure 2 is a diagram showing steps S1300 and S1400 according to the present invention.
Figure 3 is a diagram showing a stereo camera-based untrained object detection system according to the present invention.
Figure 4 is a diagram showing a control unit according to the present invention.
Figure 5 is a diagram showing an image acquisition unit according to the present invention.
Figure 6 is a diagram showing a specific embodiment according to the present invention.
The accompanying drawings, which are included as part of the detailed description to aid understanding of the present specification, provide embodiments of the present specification and explain technical features of the present specification together with the detailed description.

Hereinafter, embodiments disclosed in the present specification will be described in detail with reference to the attached drawings. However, identical or similar components will be assigned the same reference numbers regardless of reference numerals, and duplicate descriptions thereof will be omitted.

Additionally, in describing the embodiments disclosed in this specification, if it is determined that detailed descriptions of related known technologies may obscure the gist of the embodiments disclosed in this specification, the detailed descriptions will be omitted.

In addition, the attached drawings are only for easy understanding of the embodiments disclosed in this specification, and the technical idea disclosed in this specification is not limited by the attached drawings, and all changes included in the spirit and technical scope of the present invention are not limited. , should be understood to include equivalents or substitutes.

Terms containing ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

Singular expressions include plural expressions unless the context clearly dictates otherwise.

In this application, terms such as “comprise” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of one or more other features. It should be understood that this does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof.

Below, based on the above-described contents, a detailed description will be given of the stereo camera-based untrained object detection method and system according to a preferred embodiment of the present invention.

Figure 1 is a diagram showing a stereo camera-based untrained object detection method according to the present invention. The subject performing the unlearned object detection method according to the present invention may be the unlearned object detection system or control unit according to the present invention.

According to Figure 1, the method for detecting an untrained object according to the present invention includes the steps of acquiring a first image from a first image acquisition unit (S1100), acquiring a second image from a second image acquisition unit (S1200), It may include recognizing a plurality of targets based on the first image and the second image (S1300) and distinguishing between the pre-learned first target and the unlearned second target among the plurality of targets (S1400). . The first image acquisition unit and the second image acquisition unit may be cameras.

The first image may be an image for extracting a depth map, and the second image may be an image for applying a vision recognition algorithm. The present invention can detect untrained objects by distinguishing between untrained second targets.

Figure 2 is a diagram showing steps S1300 and S1400 according to the present invention.

According to Figure 2, steps S1300 and S1400 according to the present invention include obtaining a depth map including depth information based on the first image (S1311), binarizing the depth map for each frame (S1312), and binarizing the depth map for each frame (S1312). It may include a step of removing noise by acquiring mode data based on the depth map (S1313) and a step of acquiring a plurality of coordinate values for a plurality of targets based on the mode data (S1314).

In addition, steps S1300 and S1400 according to the present invention further include performing a first learning algorithm based on the second image (S1321) and obtaining coordinate values of the first target based on the learning result (S1322). It can be included.

In addition, steps S1300 and S1400 according to the present invention distinguish the first target and the second target based on a plurality of coordinate values obtained from the first image and the coordinate value of the first target obtained from the second image. A step (S1410) may be further included. Step S1410 may use the IoU (Intersection Of Union) algorithm. Specifically, the IoU algorithm may be an algorithm that obtains the overlap ratio based on the ratio of the union and intersection of two Bounding Boxes.

That is, as a result of performing the IoU algorithm, if the overlapping ratio exceeds the threshold, it can be determined that the coordinate values overlap each other, and if the overlapping ratio does not exceed the threshold, it can be determined that the coordinate values do not overlap. Based on this, the present invention can obtain coordinate values that are determined not to overlap among a plurality of coordinate values obtained from the first image and coordinate values obtained from the second image. At this time, the object corresponding to the acquired coordinate value corresponds to an untrained object.

Figure 3 is a diagram showing a stereo camera-based untrained object detection system according to the present invention.

According to FIG. 3, the present invention may include a first image acquisition unit 110, a second image acquisition unit 120, and a control unit 130. The first image acquisition unit 110 may capture the first images of the targets 10 and 11, and the second image acquisition unit 120 may capture the second images of the targets 10 and 11. The control unit 130 recognizes a plurality of targets 10 and 11 based on the first image and the second image, and selects a pre-learned first target 10 and an unlearned target among the plurality of targets 10 and 11. The second target 11 can be distinguished.

The plurality of targets 10 and 11 may include a first target 10 and a second target 11 . The first target 10 may be an object learned in advance through vision recognition. The second target 11 may be an object, whether or not it has been previously studied. The untrained object detection system according to the present invention unlearns a target that does not overlap with the coordinate value of the target 10 in the second image among the plurality of coordinate values for the plurality of targets 10 and 11 in the first image. It can be determined that the target is the second target 11.

The control unit 130 acquires a depth map including depth information based on the first image, binarizes the depth map for each frame, obtains mode data based on the binarized depth map, removes noise, and generates the mode data. Based on this, multiple coordinate values for multiple targets can be obtained. Mode data may refer to data that appears frequently in an image for each frame. A depth map may be a type of image for vision recognition based on depth information.

In addition, the control unit 130 recognizes a plurality of targets based on the first image and the second image, performs a first learning algorithm based on the second image, and coordinates the first target based on the learning result. can be obtained.

Additionally, the control unit 130 may distinguish the first target and the second target based on a plurality of coordinate values obtained from the first image and the coordinate values of the first target obtained from the second image.

Figure 4 is a diagram showing a control unit according to the present invention.

According to FIG. 4, the control unit 130 may include a processor 131, memory, and a communication unit. The processor 131 is a component that can perform calculations and control other devices. Mainly, it may mean a central processing unit (CPU), an application processor (AP), a graphics processing unit (GPU), etc. Additionally, the CPU, AP, or GPU may include one or more cores therein, and the CPU, AP, or GPU may operate using an operating voltage and clock signal. However, while a CPU or AP consists of a few cores optimized for serial processing, a GPU can consist of thousands of smaller, more efficient cores designed for parallel processing.

The memory 132 stores data supporting various functions of the control unit 130. The memory 132 may store a number of application programs (application programs or applications) and commands running in the control unit 130. At least some of these applications may be downloaded from an external server via wireless communication.

The memory 132 is a flash memory type, hard disk type, solid state disk type, SDD type (Silicon Disk Drive type), and multimedia card micro type. ), card-type memory (e.g. SD or XD memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), EEPROM (electrically erasable programmable read) -only memory), PROM (programmable read-only memory), magnetic memory, magnetic disk, and optical disk may include at least one type of storage medium. Additionally, the memory 132 may include web storage that performs a storage function on the Internet.

The communication unit 133 transmits and receives information to and from a base station or a vehicle including a communication function through an antenna. The communication unit 133 may include a modulation unit, demodulation unit, signal processing unit, etc.

Wireless communication refers to communication using communication facilities already installed by communication companies and a wireless communication network that uses the frequencies of those communication facilities. At this time, the communication unit 133 includes code division multiple access (CDMA), frequency division multiple access (FDMA), time division multiple access (TDMA), orthogonal frequency division multiple access (OFDMA), and single carrier frequency division multiple access (SC-FDMA). It can be used in various wireless communication systems such as access), and in addition, the communication unit 133 can also be used in 3rd generation partnership project (3GPP) long term evolution (LTE), etc. In addition, not only 5G communication, which has recently been commercialized, but also 6G, which is scheduled for commercialization in the future, can be used. However, this specification can utilize a pre-installed communication network without being limited to such wireless communication method.

Figure 5 is a diagram showing an image acquisition unit according to the present invention.

According to FIG. 5, the image acquisition unit of the present invention may mean a first image acquisition unit 110 and a second image acquisition unit 120. For convenience of explanation, the first image acquisition unit 110 is used, but the second image acquisition unit 120 may also have the same configuration.

The first image acquisition unit 110 is a camera and may include a camera module 111. The camera module 111 can generate a video image from an optical image. The camera module 111 may include a housing including a through hole on a side wall, a lens 113 installed in the through hole, and a driving unit 112 that drives the lens 113. The through hole may be formed in a size corresponding to the diameter of the lens 113. The lens 113 may be inserted into the through hole.

The driving unit 112 may be configured to control the lens 113 to move forward or backward. The lens 113 and the driving unit 112 are connected in a conventionally known manner, and the lens 113 can be controlled by the driving unit 112 in a conventionally known manner.

In order to obtain various video images, the lens 113 needs to be exposed to the outside of the housing that constitutes the camera module 111 or the image acquisition unit 110.

Preferably, the lens 113 includes a siloxane-based compound represented by the following [Chemical Formula 1] on its surface; It may be coated with a coating composition containing an organic solvent, inorganic particles, and a dispersant.

[Formula 1]

(Where n is an integer from 1 to 100.)

When the lens 1321 is coated with the coating composition, it can exhibit excellent water repellency and contamination resistance, so even if the lens 113 installed on the outside of the vehicle is exposed to a polluted environment for a long period of time, images or videos that can be used as road information are collected. can do.

The inorganic particles may be selected from the group consisting of silica, alumina, and mixtures thereof. The average diameter of the inorganic particles is 70 to 100 μm, but is not limited to the above example. After forming the coating layer 114 on the surface of the lens 113, the inorganic particles can improve physical strength and maintain viscosity within a certain range to increase moldability.

The organic solvent is selected from the group consisting of methyl ethyl ketone (MEK), toluene, and mixtures thereof, preferably methyl ethyl ketone, but is not limited to the above examples.

As the dispersant, a polyester-based dispersant can be used. Specifically, a polyester-based dispersion stabilizer consisting of a copolymer of 2-methoxypropyl acetate and 1-methoxy-2-propyl acetate is TEGO-Disperse 670 (manufacturer) : EVONIK) can be used, but is not limited to the above example and any dispersant that is obvious to those skilled in the art can be used without limitation.

The coating composition may further include a stabilizer as other additives, and the stabilizer may include an ultraviolet absorber, an antioxidant, etc., but is not limited to the above examples and can be used without limitation.

More specifically, the coating composition for forming the coating layer 114 includes a siloxane-based compound represented by Formula 1; It may include organic solvents, inorganic particles, and dispersants.

The coating composition may include 40 to 60 parts by weight of the siloxane-based compound represented by Formula 1, 20 to 40 parts by weight of inorganic particles, and 5 to 15 parts by weight of a dispersant, based on 100 parts by weight of the organic solvent. If the range is within the above range, a synergistic effect of the water repellent effect due to the interaction of each component is expressed to a critical degree, and if it is outside the above range, the synergistic effect is rapidly reduced or almost non-existent.

More preferably, the viscosity of the coating composition is 1500 to 1800 cP. If the viscosity is less than 1500 cP, there is a problem in that it flows down when applied to the surface of the lens 113, making it difficult to form the coating layer 114, and if the viscosity is less than 1800 cP, it flows down. In this case, there is a problem in that it is not easy to form a uniform coating layer 114.

[Preparation Example 1: Preparation of coating layer]

1. Preparation of coating composition

A coating composition was prepared by mixing methyl ethyl ketone with a siloxane-based compound represented by [Chemical Formula 1], inorganic particles, and a dispersant:

A more specific composition of the antistatic composition is shown in Table 1 below.

TX1 TX2 TX3 TX4 TX5 organic solvent 100 100 100 100 100 polysiloxane 30 40 50 60 70 inorganic particles 10 20 30 40 50 dispersant One 5 10 15 20

(Unit weight part) 2. Preparation of coating layer

The coating compositions DX1 to DX5 were applied to one surface of the lens 113 and then cured to form a coating layer 114.

[Experimental example]

1. Evaluation of surface appearance

Due to the difference in viscosity of the coating composition, after manufacturing the coating layer 114, sensory evaluation was performed to determine whether a uniform surface was formed. An evaluation was conducted on whether a uniform coating layer 114 was formed, and the evaluation was conducted based on the following criteria.

○: Formation of uniform coating layer

×: Formation of uneven coating layer

TX1 TX2 TX3 TX4 TX5 Sensory evaluation Х ○ ○ ○ Х

When forming the coating layer 114, if the viscosity is less than a certain level, flow occurs on the surface of the lens 113, making it difficult to form a uniform coating layer 114 after the curing process. Accordingly, the problem of lowering the production yield may occur. In addition, even when the viscosity was too high, it was difficult to uniformly apply the composition, making it impossible to form a uniform coating layer 114.

2. Measurement of water repellency angle

After forming the coating layer 114 on the surface of the lens 113, the results of measuring the water repellency angle are shown in Table 3 below.

Advancing contact angle ( ) Rest contact angle ( ) Receding contact angle ( ) TX1 117.1±2.9 112.1±4.1 < 10 TX2 132.4±1.5 131.5±2.7 141.7±3.4 TX3 138.9±3.0 138.9±2.7 139.8±3.7 TX4 136.9±2.0 135.6±2.6 140.4±3.4 TX5 116.9±0.7 115.4±3.0 < 10

As shown in Table 3, after forming the coating layer 114 using the coating compositions TX1 to TX5, the results of measuring the contact angle were confirmed. TX1 and TX5 were measured to have receding contact angles of less than 10 degrees. In other words, it was confirmed that pinning of water droplets occurs when the coating composition falls outside the optimal range for manufacturing the coating composition. On the other hand, it was confirmed that no pinning phenomenon occurred in TX2 to 4, showing that excellent waterproofing effects can be achieved.

3. Contamination resistance evaluation

The lens 113 on which the coating layer 114 according to the above embodiment was formed on the outside of the facility was attached to the model camera and exposed to the driving environment on public roads for 4 days. As a comparative example (Con), the same lens 113 without the coating layer 114 was used, and in each example, a model camera was attached to the same location on the vehicle.

Thereafter, the degree of contamination of the lens 113 before and after the experiment was evaluated, and for objective comparison, the results were compared with the comparative example in which the coating layer 114 was not formed, and the results were evaluated with an index of 1 to 10, and are shown in Table 4 below. shown in As for the indices below, the lower the number, the better the contamination resistance.

Con TX1 TX2 TX3 TX4 TX5 Staining resistance 10 7 3 3 3 8

(Unit: Index) Referring to Table 4 above, when forming the coating layer 114 on the lens 113, high contamination resistance is achieved even when the lens 113 is exposed to the outside while installing a lidar sensor or camera on the outside of the vehicle. It can be seen that image data can be collected in a form that is easy to analyze over a long period of time. In particular, in the case of TX2 to TX4, it can be confirmed that the contamination resistance caused by the coating layer 114 is very excellent.

Figure 6 is a diagram showing a specific embodiment according to the present invention.

According to FIG. 6, the depth map images (a, c) on the left may be the above-described first images, and the RGB image images (b, d) on the right may be the above-described second images. Additionally, in the case of depth map images, Mode-ROI can be used to delete the background of the depth map image and detect the invention based on the outline of the object.

According to Figure 6, in figure (c), all three objects were detected using Mode-ROI. However, in figure (d), two objects were detected through the YOLO algorithm. However, the tin can in the middle was not detected by the algorithm in picture (d). Therefore, as a result of applying the IoU algorithm by combining (a, c) and (b, d), the cola can in the middle was detected as an untrained object.

The above-described present invention can be modeled as computer-readable code on a program-recorded medium. Computer-readable media includes all types of recording devices that store data that can be read by a computer system. Examples of computer-readable media include HDD (Hard Disk Drive), SSD (Solid State Disk), SDD (Silicon Disk Drive), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. Includes. Accordingly, the above detailed description should not be construed as restrictive in all respects and should be considered illustrative. The scope of this specification should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent scope of this specification are included in the scope of this specification.

Any or other embodiments of the present invention described above are not exclusive or distinct from each other. In certain embodiments or other embodiments of the present invention described above, each configuration or function may be used in combination or combined.

The above detailed description should not be construed as restrictive in any respect and should be considered illustrative. The scope of the present invention should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

10: Learned target
11: Untrained target
110: First image acquisition unit
120: Second image acquisition unit
130: control unit

Claims (11)

In a method of detecting an untrained object using a first image acquisition unit and a second image acquisition unit for photographing a plurality of targets,
Acquiring a first image as an image for extracting a depth map from the first image acquisition unit;
acquiring a second image as an image for applying a vision recognition algorithm from the second image acquisition unit;
Obtain a depth map including depth information from the first image, binarize the depth map for each frame, obtain mode data based on the binarized depth map to remove noise, and remove noise based on the mode data. Obtaining a plurality of coordinate values for a plurality of targets;
Obtaining coordinate values of a first target based on a learning result of performing a first learning algorithm based on the second image; and
Among the plurality of coordinate values obtained from the first image, determining a coordinate value in which an overlapping ratio of the coordinate values of the first target obtained from the second image does not exceed a threshold value is determined as a second target that is an untrained object; Including,
The first image acquisition unit,
It includes a camera module that generates a video image from an optical image,
The camera module includes a housing including a through hole in a side wall, a lens installed in the through hole, and a driving unit that drives the lens,
The step of distinguishing between the pre-learned first target and the unlearned second target among the plurality of targets,
Based on the plurality of coordinate values obtained from the first image and the coordinate values of the first target obtained from the second image, the first target and the second target are distinguished.
The lens is coated with a coating composition comprising 40 to 60 parts by weight of a siloxane compound represented by the following formula (1), 20 to 40 parts by weight of inorganic particles, and 5 to 15 parts by weight of a dispersant, based on 100 parts by weight of an organic solvent.
Stereo camera-based untrained object detection method.
[Formula 1]

(Where n is an integer from 1 to 100.) delete delete delete delete a first image acquisition unit that captures first images of a plurality of targets;
a second image acquisition unit that captures a second image of the plurality of targets; and
Obtain a depth map including depth information from a first image, binarize the depth map for each frame, obtain mode data based on the binarized depth map to remove noise, and use the plurality of modes based on the mode data. Acquire a plurality of coordinate values for the target, obtain coordinate values of the first target based on a learning result of performing a first learning algorithm based on the second image, and obtain a plurality of coordinate values for the first image. A control unit that determines a coordinate value whose overlapping ratio of the coordinate values of the first target obtained from the second image does not exceed a threshold among the coordinate values as the second target, which is an unlearned object;
The first image acquisition unit,
It includes a camera module that generates a video image from an optical image,
The camera module is,
A housing including a through hole in a side wall;
A lens installed in the through hole; and
It includes a driving unit that drives the lens,
The lens is coated with a coating composition comprising 40 to 60 parts by weight of a siloxane compound represented by the following formula (1), 20 to 40 parts by weight of inorganic particles, and 5 to 15 parts by weight of a dispersant, based on 100 parts by weight of an organic solvent.
A stereo camera-based untrained object detection system.
[Formula 1]

(Where n is an integer from 1 to 100.) delete delete delete delete A computer program stored in a recording medium that executes the stereo camera-based untrained object detection method of claim 1.