KR102368262B1 - Method for estimating traffic light arrangement information using multiple observation information - Google Patents

Abstract

The present invention relates to a method for estimating traffic light arrangement information using multi-observation information, comprising: a step of collecting observation data on a fixed object near a road in a driving process of each of a plurality of vehicles; a step of classifying the observation data in accordance with classification hierarchies defined for each fixed object; a step of clustering the observation data based on a position in the same classification hierarchy; a step of successively sorting the observation data in accordance with the observation conditions and the observation characteristics; a step of applying a numerical analysis model to the observation data and analyzing the observation data; a step of verifying observation reliability based on the analysis results; a step of determining an object position for one or more fixed objects when the observation reliability passes the verification; a step of estimating the type of the relevant traffic lights and the number of lamps based on the observation data when the fixed object falls under a traffic light; a step of detecting the situation of being turned on of each lamp of the relevant traffic light in the observation data and matching the situation of being turned on with each lamp; and a step of determining the lamp configuration and arrangement of the relevant traffic light when the matching has succeeded. The present invention aims to provide the method for estimating traffic light arrangement information using multi-observation information through statistical data processing techniques by collecting a large number of observation results from general image storage devices for vehicles.

Description

Method for estimating traffic light arrangement information using multiple observation information

The present invention relates to image analysis-based object identification technology, and more particularly, using multiple observation information for estimating arrangement information about traffic lights through statistical data processing techniques by collecting a plurality of observation results from a general vehicle image storage device. It relates to a method for estimating traffic light arrangement information.

Fixed facilities on roads, especially traffic signs and traffic lights that serve as guides for safe driving, are important transportation infrastructures that allow drivers to recognize road information and drive safely accordingly.

On the other hand, as autonomous driving technology, which is usually divided into five stages according to the advanced driver assist system (ADAS) or the degree of development of autonomy, is gradually applied, the accuracy of road information judged by autonomous vehicles has become more important. For example, if the speed limit of a section is suddenly changed from 60 km/h to 50 km/h, and the ADAS/autonomous driving system only depends on the stored section information, it will violate traffic laws. There is a risk of this happening because many autonomous vehicles being developed actually rely on high-precision maps and geometric information around the vehicle acquired using Lidar.

In the case of a map for a conventional navigation system, not a high-precision map, a dedicated vehicle equipped with high-precision GPS, lidar, high-resolution camera, and mass storage is operated to maintain and manage map information. Since there are not many, there is a disadvantage that the update cycle of road information for one point is long. Since the normal update cycle takes more than one year, if there is a facility change immediately after the update, incorrect information remains for more than one year.

If the object is on the road surface, the road surface is a two-dimensional plane, so it matches 1:1 with the image taken by the camera, which is also a two-dimensional plane, and location specification is possible from one camera image obtained, but three-dimensional coordinates in the air In the case of signs and traffic lights that need to determine (X, Y, Z), it is impossible to specify only one image.

Korea Patent Publication No. 10-2017-0058640 (2017.06.29)

An embodiment of the present invention is to provide a method for determining the position of a fixed object using multiple observation information for estimating arrangement information about a traffic light through a statistical data processing technique by collecting a plurality of observation results from a general vehicle image storage device.

Among the embodiments, a method for estimating traffic light arrangement information using multiple observation information includes: collecting observation data about a fixed object near a road in the driving process of each of a plurality of vehicles; classifying the observation data according to a classification layer defined for each fixed object; clustering the observation data based on a location within the same classification layer; sequentially selecting the observation data according to observation conditions and observation characteristics; analyzing the observation data by applying a numerical analysis model; verifying observation reliability based on a result of the analysis; determining an object position with respect to the at least one fixed object if the verification is passed; estimating the type of the corresponding traffic light and the number of lamps based on the observation data when the fixed object corresponds to a traffic light; detecting and matching the lighting conditions for each lamp of the corresponding traffic light from the observation data; and determining a lamp configuration and arrangement of the corresponding traffic light when the matching is successful.

The collecting may include: photographing a front image of the vehicle through at least one camera installed in each of the plurality of vehicles; extracting at least one image including the fixed object from the front image; and receiving observation data including the at least one image, an observation number assigned to the image, and the location, driving direction, and time of the corresponding vehicle from the plurality of vehicles.

The selecting corresponds to at least one observation condition among a case in which the degree of conformity to the classification layer is less than or equal to a threshold value, a case in which the corresponding observation time corresponds to a specific time range, and a case in which the weather at the corresponding observation time meets a specific condition. removing it from the observation data; removing a case in which a distance between observation points for the observation data is less than a preset threshold distance; and removing a case in which an angle between direction vectors with respect to the observation data is less than a preset critical angle.

The analyzing may include defining the positional coordinates of the fixed object and generating a vector expression for a distance from a direction vector based on the positional coordinates; converting the vector equation into a quadratic equation relating to the position coordinates and applying the quadratic equation to the observation data to generate a mixed quadratic equation relating to the position coordinates; and calculating the position coordinates based on the mixed quadratic equation.

The verifying may include detecting and removing outliers in which a distance from the observation data to the calculated location coordinates exceeds a verification criterion.

The determining of the object position may include determining the object position as the object position when a predetermined number of observation data passing the verification is accumulated.

The estimating may include: detecting a region of interest (ROI) for the corresponding traffic light from the observation data; and determining the type and the number of lamps, respectively, according to a comparison and a ratio between the width and length of the region of interest.

The determining of the lamp configuration and arrangement may include determining the success of the matching when matching of the lighting conditions for all lamps of the corresponding traffic light is completed.

The disclosed technology may have the following effects. However, this does not mean that a specific embodiment should include all of the following effects or only the following effects, so the scope of the disclosed technology should not be understood as being limited thereby.

The method for determining the location of a fixed object using multiple observation information according to an embodiment of the present invention may collect a plurality of observation results from a general vehicle image storage device and estimate arrangement information regarding a traffic light through a statistical data processing technique.

1 is a view for explaining a placement estimation system according to the present invention.
FIG. 2 is a diagram for explaining a functional configuration of the arrangement estimating apparatus of FIG. 1 .
3 is a flowchart illustrating a method for estimating traffic light arrangement information using multiple observation information according to the present invention.
4 to 6 are diagrams for explaining a camera observation range for a facility near a road.
7 is a view for explaining a camera observation range for facilities near and on the road.
8 to 10 are diagrams for explaining a method for positioning a road facility through multiple observations according to the present invention.
11 is a view for explaining a pre-processing operation of observation data according to the present invention.
12 and 13 are diagrams for explaining a coordinate system transformation process according to the present invention.
14 and 15 are diagrams for explaining a data selection process according to the present invention.
16 is a view for explaining a positioning method according to the present invention.
17 and 18 are diagrams for explaining a method of determining a location of a traffic light and determining arrangement information according to the present invention.

Since the description of the present invention is merely an embodiment for structural or functional description, the scope of the present invention should not be construed as being limited by the embodiment described in the text. That is, since the embodiment may have various changes and may have various forms, it should be understood that the scope of the present invention includes equivalents capable of realizing the technical idea. In addition, since the object or effect presented in the present invention does not mean that a specific embodiment should include all of them or only such effects, it should not be understood that the scope of the present invention is limited thereby.

On the other hand, the meaning of the terms described in the present application should be understood as follows.

Terms such as “first” and “second” are for distinguishing one component from another, and the scope of rights should not be limited by these terms. For example, a first component may be termed a second component, and similarly, a second component may also be termed a first component.

When a component is referred to as being “connected to” another component, it may be directly connected to the other component, but it should be understood that other components may exist in between. On the other hand, when it is mentioned that a certain element is "directly connected" to another element, it should be understood that the other element does not exist in the middle. Meanwhile, other expressions describing the relationship between elements, that is, "between" and "between" or "neighboring to" and "directly adjacent to", etc., should be interpreted similarly.

The singular expression is to be understood as including the plural expression unless the context clearly dictates otherwise, and terms such as "comprises" or "have" refer to the embodied feature, number, step, action, component, part or these It is intended to indicate that a combination exists, and it should be understood that it does not preclude the possibility of the existence or addition of one or more other features or numbers, steps, operations, components, parts, or combinations thereof.

In each step, identification numbers (eg, a, b, c, etc.) are used for convenience of description, and identification numbers do not describe the order of each step, and each step clearly indicates a specific order in context. Unless otherwise specified, it may occur in a different order from the specified order. That is, each step may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the reverse order.

The present invention can be embodied as computer-readable codes on a computer-readable recording medium, and the computer-readable recording medium includes all types of recording devices in which data readable by a computer system is stored. . Examples of the computer-readable recording medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. In addition, the computer-readable recording medium is distributed in a computer system connected to a network, so that the computer-readable code can be stored and executed in a distributed manner.

All terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs, unless otherwise defined. Terms defined in general used in the dictionary should be interpreted as having the meaning consistent with the context of the related art, and cannot be interpreted as having an ideal or excessively formal meaning unless explicitly defined in the present application.

A road facility or fixed object, which is an identifiable object according to the present invention, has a common shape and content widely used under national or administrative standards, local ordinances, industrial technical standards, and other tacit mutual consent. It means an object, and technically, by securing a large number of image data of such an object, machine learning or deep learning techniques are applied to designate a specific classification layer based on common characteristics. means you can

For example, traffic signs in Korea are defined as 'safety signs' in Article 2, No. 16 of the Road Traffic Act, and can usually be used as 'traffic safety signs'. The types and methods of making safety signs are stipulated in Annex 6 of the Enforcement Rules of the Road Traffic Act, an Ordinance of the Ministry of Government Administration and Home Affairs. In the United States, the form and manufacturing method of signs are described in the document called The Manual on Uniform Traffic Control Devices, abbreviation MUTCD, which is designated as a national standard by the Federal Highway Administrator. We are distributing MUTCDs by state by adding signs such as caution and moose crossing caution.

In addition, in the case of traffic lights, a horizontal arrangement and a vertical arrangement are mixed and used, and in Korea, it is common to use a horizontal arrangement for a support frame that crosses a road and a vertical arrangement for a pillar frame along the road. In the United States, the frequency of use for vertical traffic lights is high, such as when vertical traffic lights are hung on a line.

Since the lamps arranged in one traffic light do not turn on at the same time, it is difficult to accurately confirm the shape of an important traffic light, that is, the arrangement of the lamps in one run, and multiple observations may be essential.

Hereinafter, a method for estimating traffic light arrangement information using multiple observation information according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a view for explaining a placement estimation system according to the present invention.

Referring to FIG. 1 , the arrangement estimation system 100 may perform a traffic light arrangement information estimation method using multiple observation information, and may be implemented including a vehicle 110 , an arrangement estimation device 130 , and a database 150 . can

The vehicle 110 may correspond to a vehicle as a means of transportation for transporting passengers or cargo using power produced by a motor or an engine. Here, it is described on the premise that the vehicle 110 basically corresponds to a vehicle traveling on the road, but is not necessarily limited thereto, and may correspond to various transportation means that can be driven on the road, such as a motorcycle, a bicycle, and personal mobility. may be The vehicle 110 may be connected to the arrangement estimator 130 through a network, and the plurality of vehicles 110 may be simultaneously connected to the arrangement estimator 130 .

In one embodiment, the vehicle 110 may be implemented by including a plurality of sensors in order to recognize a situation on the road or monitor a driving state in the driving process. For example, the vehicle 110 may include an acceleration sensor, a brake sensor, a wheel sensor, a global positioning system (GPS) sensor, a steering angle sensor, etc. to collect data related to the driving state, and It may include a black box, a camera sensor, a lidar sensor, and the like for data collection related to the data.

In one embodiment, the vehicle 110 may be implemented to include at least one camera that is mounted forward to capture a front road situation. In this case, the image captured by the camera may be transmitted to the arrangement estimating apparatus 130 . Meanwhile, the vehicle 110 may install and execute a dedicated program or application interworking with the arrangement estimating device 130 .

The arrangement estimating device 130 may be implemented as a server corresponding to a computer or program capable of measuring the absolute position of road facilities (or fixed objects) and estimating arrangement information of traffic lights using observation information taken from multiple viewpoints. there is. The arrangement estimating device 130 may be connected to the vehicle 110 through a wired network or a wireless network such as Bluetooth or WiFi, and may transmit/receive data. In addition, the arrangement estimating apparatus 130 may be implemented to operate in conjunction with a separate external system (not shown in FIG. 1 ) to collect data or provide an additional function.

In an embodiment, the arrangement estimating device 130 may be implemented as a miniaturized computing device to be operably mounted inside the vehicle 110 . More specifically, the arrangement estimating apparatus 130 may be implemented by including a camera that is mounted toward the front of the vehicle 110 to photograph the front road condition of the vehicle 110 . The arrangement estimating apparatus 130 may be implemented by including a processor capable of receiving an image captured by a camera and executing an analysis algorithm. In this case, the processor may include a network processing unit (NPU) or a tensor processing unit (TPU) capable of performing multilayer neural network operation, and time scheduling a plurality of network algorithms ) and can be implemented including control logic that can be processed.

In addition, the batch estimator 130 includes a non-volatile memory that stores a program to be executed by the processor and an algorithm and parameters of a network processing unit, and a removable non-volatile memory that can separately store image data (for example, , micro-SD card, etc.), high-speed volatile memory (eg, DDR SDRAM, etc.) that stores running programs and buffers images before the reference point, GPS that can check the current location of the vehicle A communication module capable of 4G or higher wireless communication and/or WiFi communication that communicates with the server to notify that an event to be observed (recognition of a specific road facility) has occurred, and can transmit stored image data preemptively or at the request of the server It can be implemented including

In an embodiment, the arrangement estimating apparatus 130 may identify road facilities from the front image of the vehicle 110 through image analysis. In this case, the arrangement estimating apparatus 130 may identify the road facility through inference using a learning model in which the result learned in advance is reflected. For example, the arrangement estimation apparatus 130 may identify an object in an image by applying a machine learning technique based on a convolutional neural network (CNN). Hereinafter, a typical reasoning process is briefly described.

First, since convolution linearly calculates the values of one position and the values of neighboring positions (3x3, 5x5, etc.) of data expressed in a two-dimensional array like an image to give correlation, the neural network technique is applied in images. is widely used in (1) An image may be converted into a feature map through a convolution layer and then through various types of multi-layer networks according to network design. (2) A region proposal network (RPN) can propose a region of interest (ROI) in which an object to be found is located. (3) By bringing the part of the feature map corresponding to the region of interest, it is possible to calculate the classification class of the object. In this case, classification does not present only one result, and may output the degree of matching with each classification layer set in advance, and for example, probability information for each object may be generated as a recognized result.

The database 150 may correspond to a storage device for storing various types of information required in the operation process of the arrangement estimating apparatus 130 . For example, the database 150 may store road images collected from a plurality of vehicles 110 , and may store analysis algorithms and program information for image analysis, but is not necessarily limited thereto, and the arrangement estimating device ( 130) may store information collected or processed in various forms in the process of determining the location of a fixed object (eg, a sign, a traffic light, etc.) using multiple observation information and estimating traffic light arrangement information.

Also, in FIG. 1 , the database 150 is illustrated as a device independent of the batch estimating device 130 , but is not limited thereto, and the batch estimating device 130 is a logical storage device of the batch estimating device 130 . Of course, it can be implemented by being included in the .

FIG. 2 is a diagram for explaining a functional configuration of the arrangement estimating apparatus of FIG. 1 .

Referring to FIG. 2 , the arrangement estimating device 130 includes a data collecting unit 210 , a data preprocessing unit 230 , a data analyzing unit 250 , an object position determining unit 270 , and a traffic light arrangement information estimating unit 290 . and a control unit (not shown in FIG. 2 ).

The data collection unit 210 may collect observation data regarding a fixed object near a road in the driving process of each of the plurality of vehicles 110 . The data collection unit 210 may classify the observation data collected for each vehicle 110 according to a predetermined criterion and store it in the database 150 . That is, the observation data may include a front image or image captured by a camera installed in the vehicle 110 , but is not limited thereto, and various types of data may be collected as observation data according to a data acquisition device. In particular, the observation data may include information on various road facilities that can be identified in front of the vehicle 110 . For example, the road facility may include a traffic light or sign installed on a road or nearby.

In an embodiment, the data collection unit 210 may collect a front image of the corresponding vehicle 110 photographed through at least one camera installed in each of the plurality of vehicles 110 , and may collect a fixed object from the front image. At least one image including can be received from The data collection unit 210 may basically acquire observation data based on a front image collected for one vehicle 110 , but may collect observation information from a plurality of vehicles 110 at various viewpoints as necessary. It can also be obtained by consolidation.

Also, the data collection unit 210 may extract an image including a fixed object from the acquired front image in units of frames, and may assign an observation number to the image as identification information. Also, the data collection unit 210 may collect location and driving information and viewpoint information regarding the vehicle 110 in which the corresponding image is captured as observation data.

The data preprocessor 230 may classify the observation data collected by the data collector 210 or perform preprocessing operations such as filtering and clustering. More specifically, the data preprocessor 230 may classify the observation data according to a classification hierarchy defined for each fixed object, and cluster the observation data based on a location within the same classification hierarchy. In addition, the data preprocessor 230 may perform an operation of primary selection of observation data according to observation conditions and secondary selection according to observation characteristics.

In FIG. 11 , the classification layer may correspond to a type of a fixed object that can be identified on a road. For example, the classification layer may include a traffic light, a sign, and the like, and the data preprocessing unit 230 may classify by classification layer based on the observed position and the observed object of the collected observation data (Fig. (b)). ). Also, the data preprocessor 230 may select observations for a specific fixed object by grouping adjacent observations (Fig. (c)). In this case, the data preprocessor 230 may perform a clustering operation by using a clustering algorithm such as a k-clustering algorithm.

In an embodiment, the data preprocessor 230 is configured to perform at least one of a case in which the degree of matching for the classification layer is less than or equal to a threshold value, a case in which the corresponding observation point corresponds to a specific time range, and a case in which the weather at the corresponding observation point meets a specific condition. If it meets the observation conditions of , it can be removed from the observation data. That is, the data preprocessor 230 may select and remove the observed data when the degree of matching with the classification layer is low as a result of the classification. In addition, the data preprocessor 230 may remove obstacles in advance in determining the location of the object by removing it from the observation data when the weather condition of the corresponding observation time is bad, the backlight condition, or a specific time such as night time. .

In an embodiment, the data preprocessor 230 may remove a case in which the distance between observation points for observation data is less than a preset critical distance, and removes a case in which an angle between direction vectors with respect to observation data is less than a preset critical angle can do. This will be described in more detail with reference to FIGS. 14 and 15 .

The data analysis unit 250 may perform data analysis by applying a numerical analysis model to the observation data, and may perform an operation of verifying observation reliability based on a result of the analysis. In one embodiment, the data analysis unit 250 defines the positional coordinates of the fixed object, generates a vector equation regarding the distance from the direction vector based on the positional coordinates, converts the vector equation into a quadratic equation regarding the positional coordinates, By applying the quadratic equation to the observation data, a mixed quadratic equation for position coordinates is generated, and the position coordinates can be calculated based on the mixed quadratic equation. In an embodiment, the data analysis unit 250 may detect and remove an outlier whose distance from the observation data to the calculated position coordinate exceeds the verification criterion. This will be described in more detail with reference to FIG. 16 .

The object position determiner 270 may determine an object position with respect to at least one fixed object if the verification is passed. That is, the object position determiner 270 may more accurately acquire position information by determining the position of the fixed object based on data obtained through the verification operation.

In an embodiment, the object location determiner 270 may determine the object location when a predetermined number of observation data passing the verification is accumulated. For example, when the preset cumulative number is 6, the object position determiner 270 accumulates 6 pieces of observation data that have passed verification according to iterative performance of the data analysis process. location can be determined.

In FIG. 17 , assuming that the position of the fixed object can be finally determined when valid observations are performed 6 or more times (N _min = 6), the object positioning unit 270 generates the first observation (N=1) at point A Even so, location estimation is impossible, so the information can be accumulated and stored in the DB. In addition, in the case of N=6 in the process of repeatedly collecting location information according to data analysis, the object location determination unit 270 may try to estimate the location, but the location fails because the corresponding observation is outside the limit range. can (ie, N _inlier < N _min ).

Contrary to this, in the case of N=7, the object positioning unit 270 may attempt to estimate the location and, if six valid observations are accumulated, the location may be successful. Accordingly, the object position determiner 270 may determine the existence of the corresponding fixed object and determine the position. Thereafter, when a predetermined period or new observation is added, the object position determiner 270 may correct or change the position by updating the position estimate, and may repeatedly perform the operation of removing the outlier.

In an embodiment, the object position determiner 270 may monitor the position of the object to detect the removal, replacement, and movement of the fixed object. That is, the object position determining unit 270 can monitor the position of the fixed object through repeated operations after identifying the position of the fixed object, and can effectively detect whether to dismantle, replace, or move the fixed object according to the change in the position of the fixed object. can

For example, the object positioning unit 270 may accumulate records of vehicles 110 driving and passing a specific point B for a specific point B, but the location of the fixed object is not determined nevertheless, so the road facility If this is not observed, the demolition of the relevant road facility may be decided based on the number or period of driving passages.

In addition, the object location determining unit 270 for a specific point C, while the existing classification hierarchy is no longer observed at the same location, the road facilities of the other classification hierarchy are observed as much as the effective number of observations (that is, more than N _min ). In this case, it is possible to determine the replacement of the fixed object at the corresponding position.

In addition, the object positioning unit 270 for specific points D and D' is no longer observed at the existing point D, whereas the road facilities of the same classification layer are observed at the point D' near the point D by the effective number of observations. In this case, it can be determined that the fixed object has been moved from point D to point D'.

When the fixed object whose location is identified corresponds to a traffic light, the traffic light arrangement information estimator 290 may estimate the type of the corresponding traffic light and the number of lamps based on the observation data, and calculate the lighting situation for each lamp of the corresponding traffic light from the observation data. It can be detected and matched, and when the matching is successful, the lamp configuration and arrangement of the corresponding traffic light can be determined. More specifically, information on the shape of a traffic light can be easily derived through image analysis, but since the lamps of the traffic light do not all light up at the same time, the arrangement of the lamps is not sufficient for single observation or observation in the positioning process. Observational data may be required. The traffic light arrangement information estimator 290 may estimate the traffic light derived through image analysis as a relative ratio through horizontal and vertical size comparison.

3이고, W < H 인 경우 H/W

3이다.In one embodiment, the traffic light arrangement information estimator 290 detects a region of interest (ROI) with respect to the corresponding traffic light from the observation data, and determines the type and number of lamps of the corresponding traffic light according to the comparison and ratio between the width and length of the region of interest. each can be decided. For example, when the horizontal size of the traffic light is larger than the vertical size, it may correspond to a horizontal traffic light, and the number of lamps may be determined through 'round (horizontal/vertical)'. In addition, when the vertical size of the traffic light is larger than the horizontal size, it may correspond to a vertical traffic light, and the number of lamps may be determined through 'round (vertical/horizontal)'. In a typical tri-color traffic light, W/H when W > H

3, and H/W if W < H

3 is

In an embodiment, the traffic light arrangement information estimator 290 may determine the success of matching when matching of lighting conditions for all lamps of the corresponding traffic light is completed. That is, in order to estimate the arrangement information of the traffic light, it is necessary to match all the lighting conditions of each lamp of the traffic light from the front image. Even if the lighting condition of the traffic light is detected through the observation data, if the lighting condition for the same lamp is repeatedly matched, the arrangement information of the remaining unmatched lights inevitably exists in an unconfirmed state, and the traffic light arrangement information estimation unit 290 ) may repeatedly perform the matching operation until the lighting conditions for all lamps are observed.

The controller (not shown in FIG. 2 ) controls the overall operation of the arrangement estimator 130 , and the data collection unit 210 , the data preprocessor 230 , the data analysis unit 250 , and the object position determiner 270 . ) and the traffic light arrangement information estimator 290 may manage a control flow or data flow.

3 is a flowchart illustrating a method for estimating traffic light arrangement information using multiple observation information according to the present invention.

Referring to FIG. 3 , the arrangement estimating apparatus 130 may collect observation data about a stationary object near a road in the driving process of each of the plurality of vehicles 110 through the data collecting unit 210 . The data collection unit 210 may collect and store observation data for each vehicle 110 , and then the data analysis and location determination process may be performed for each vehicle 110 or an integrated operation for a plurality of vehicles 110 . can be performed with

Also, the arrangement estimator 130 may perform a preprocessing operation regarding classification and selection operations on the observed data through the data preprocessor 230 . For example, the data preprocessor 230 classifies the observed data according to a classification hierarchy defined for each fixed object (step S300), and clusters the observed data based on the location within the same classification hierarchy (step S310), and observes the observed data. A selection operation may be performed according to conditions and observation characteristics (steps S320 and S330).

In addition, the arrangement estimation apparatus 130 may apply a numerical analysis model to the observation data through the data analysis unit 250 and analyze it (step S340), and verify the observation reliability based on the result of the analysis (step S350) . To this end, the data analysis unit 250 may utilize various numerical analysis models for the observed data, and data that fails reliability verification may be removed from the observed data (outlier).

Also, the arrangement estimating apparatus 130 may determine an object position with respect to at least one fixed object existing on a road based on the observation data that has passed verification through the object position determiner 270 (step S360). In an embodiment, the object positioning unit 270 may continuously update and monitor the location information of the identified fixed object, and may detect the removal, replacement, and movement of the fixed object according to a change in the location information.

Also, the arrangement estimating apparatus 130 may estimate the arrangement of the traffic lights and the number of lamps through the traffic light arrangement information estimator 290 (step S370). The traffic light arrangement information estimator 290 may detect and match the lighting conditions for each lamp of the corresponding traffic light from the observation data (step S380). If the matching of the lighting conditions for a specific lamp of the corresponding traffic light is not completed, the arrangement estimating apparatus 130 may identify the traffic light from the observation data and repeatedly perform the matching operation for the lighting condition of the traffic light. Finally, when matching is successful, the traffic light arrangement information estimating unit 290 may finally determine the lamp configuration and arrangement of the corresponding traffic light (step S390).

4 to 6 are diagrams for explaining a camera observation range for a facility near a road.

Referring to FIG. 4 , the position of the road facility 410 and the angle of view of the camera may be expressed in three dimensions, but here it will be described with a diagram in two dimensions. In the figure (b), area A is an area in which a sign is visible in a recognizable size and may correspond to an area within a certain distance from the sign. Since the sign is usually installed toward the front of the vehicle 110 traveling, it becomes difficult to recognize the sign as the opposing angle increases, and the area A may be formed in a fan shape. That is, the area X may correspond to an area in which sign recognition is impossible.

Also, in the figure (a), area B may correspond to a recognizable area by a camera mounted on the vehicle 110 . The vertex of the triangle indicating the angle of view may correspond to the position of the camera, and the area Y indicated by a dotted line as an area adjacent to the camera may correspond to an area in which the sign cannot be seen due to the limitation of the angle of view.

Referring to FIG. 5 , the observable range in the vehicle 110 driving in a lane close to the sign may be diagrammed and expressed in two dimensions. Specifically, in the case of figure (a), it may correspond to a case where the sign is not visible due to a long distance, and in the case of figure (b), it may correspond to a case where the sign is visible but not recognized due to the small size of the screen, and in figure (c) ) may correspond to the case where the sign recognizable area starts, and in the case of figure (d), it may correspond to the case just before the sign leaves the screen (upward of the screen).

Referring to FIG. 6 , the observable range in the vehicle 110 driving the lane away from the sign may be diagrammed and expressed in two dimensions. Specifically, in the case of figure (a), it may correspond to the case where observation is started, and in the case of figure (b), it may correspond to the case where the observation is finished (eg, just before moving to the side of the screen). In addition, in the case of Figure (c), it may correspond to a case in which the observable area becomes smaller as driving in a further lane, and in the case of Figure (d), it may correspond to a case where it is impossible to observe a sign outside the sign recognition area.

7 is a view for explaining a camera observation range for facilities near and on the road.

In the case of Figure (a), the range where the sign installed near the road can be observed through the camera installed in the vehicle 110 in motion is formed by four lines, (1), (2), (3) and (4) It may correspond to an area where In the case of Figure (b), the observation range of road facilities installed on the road, such as an intersection, can be expressed in the same way as the observation range of road facilities installed near the road, such as sidewalks. That is, the range in which the traffic light installed on the road can be observed through the camera installed in the vehicle 110 in motion is an area formed by four lines, (1'), (2'), (3') and (4'). may correspond to

Accordingly, when there is a camera mounted on the vehicle 110 in the corresponding area, it may be possible to recognize a traffic light through analysis of the observation data. In this case, lines (1) and (1') may correspond to the sign recognition distance according to the resolution limit of the camera, and the lines (2) and (2') may correspond to the limit of the side view angle range of the camera, , lines (3) and (3') may correspond to the limits of the camera's upper field angle range, and lines (4) and (4') indicate that the camera mounted on the vehicle 110 is laterally moved due to road boundaries. There may be limits to access.

Meanwhile, it goes without saying that the corresponding area shown in FIG. 7 may vary depending on the actual position or height of the sign, detailed specifications of the camera mounted on the vehicle 110 , the mounting height, and the like. In addition, the concept of the scope of the corresponding area may be equally applied to road facilities such as signs and traffic lights.

8 to 10 are diagrams for explaining a method for positioning a road facility through multiple observations according to the present invention.

Referring to FIG. 8 , since the camera projects 3D of the real world onto a 2D plane, a point of the camera (eg, x in Figure (a)) may correspond to a projection of a straight line (xX) in the real world. there is. That is, it is difficult to determine which point is actually projected and expressed on the corresponding straight line in the real world for a point in the image captured by the camera, and as a result, it can be interpreted as meaning that it is difficult to obtain the location information of the point X in the real world.

However, the position of the sign cannot be determined only by single observation through a single camera, but theoretically, if two or more observations are possible from different positions, the position of the target can be specified. As a representative example, in a state in which two cameras are fixed on the same plane, the same object may be observed, and a distance to the object may be measured using a disparity generated in each camera.

On the other hand, in the case of Figure (b), in stereo vision, the distance D can be expressed as D = d·f/δ according to the proportional relationship (here, D: distance to the target object, d: between the two camera centers) Distance, δ (disparity): the distance between pixels where the same reference point of the target object is formed in two cameras, f: focal length). That is, the arrangement estimating apparatus 130 may collect observation information from multiple viewpoints, and may acquire information about the location of road facilities based on a difference between the observation information.

Referring to FIG. 9 , in the case of stereo vision, a condition that the distance (width) between two cameras is sufficiently large must be satisfied in order to measure the distance to the vehicle 110 having a long distance. That is, the image taken by the camera is digitized, and the reference point of the target can be determined in units of each pixel having a certain size. When d) is large enough, it is possible to measure up to a long distance and to increase the accuracy of the measured distance.

As shown in Figure (a), in a single driving, the difference in observation angle does not occur much even if several observations are made. In addition, there is a possibility that observation may not be possible due to the influence of the weather at the time of driving (eg, rain, snow, backlight, night, etc.).

As shown in Figure (b), if a large number of observations are collected from a camera mounted on a number of vehicles 110, a wide observation angle can be secured, and suitable observations such as suitable weather conditions and driving speed can be selectively applied for measurement. can increase the accuracy and reliability of

Referring to FIG. 10 , the arrangement estimating apparatus 130 may collect observation data through multiple observations by multiple vehicles 110 through the data collection unit 210 . In this case, the observation data may include at least one image, an observation number assigned to the image, and the location, driving direction, and time of the vehicle 110 . The data collection unit 210 may collect observation data from each vehicle 110 through wireless communication or copying of a storage medium, and the collected observation data may be stored and stored in the database 150 . . Thereafter, the arrangement estimating device 130 may identify the road facility through analysis of the observation data collected through the data analysis unit 230 and determine the corresponding location, and through this, the replacement, movement, and new installation of the road facility, etc. can be effectively detected.

12 and 13 are diagrams for explaining a coordinate system transformation process according to the present invention.

12 and 13 , the observation of the camera is made in a local coordinate system with respect to the vehicle 110 , and in general, the traveling direction of the vehicle 110 is the Z-axis, the lateral direction is the X-axis, and the upper The direction may correspond to the Y axis.

That is, the position at which the camera is mounted becomes the origin, the camera is mounted at the height h, and the sensor surface of the camera may be formed perpendicular to the moving direction Z-axis. Under such conditions, when a point (X ^(l) , Y ^(l) , Z ^(l) ) of an object is focused on a point (x, y) of an image, the object can be observed through the camera. In FIG. 12 , coordinates in 3D space are indicated by uppercase letters, image coordinates by lowercase letters, and a superscript ^(l) indicates a local coordinate system.

For example, if we explain based on the center of the front image taken through the camera, the position (x, y) of the fixed object is as follows according to the lens formula based on the focal length λ (lambda, focal length) of the camera. It can be expressed as Equation (1).

[Equation 1]

In addition, by modifying Equation 1 above, it can be expressed as a linear equation in a three-dimensional space as shown in Equation 2 below.

[Equation 2]

In this case, since the earth's surface can be locally regarded as a plane, the latitude, longitude, and altitude of GPS can be applied as a global coordinate system corresponding to the three axes.

For example, if the GPS position of the vehicle 110 is (X ₀ , Y ₀ , Z ₀ ) and the traveling direction is (U, V, W), U, V, W are each Z ^(l) , It can correspond to X ^(l) and Y ^(l) , and can be expressed as a linear equation in the global coordinate system through coordinate transformation using an angle formed with the global coordinate system. In addition, observations performed in each vehicle coordinate system (ie, local coordinate system) may be converted into multiple observation results in a unified global coordinate system.

More specifically, Equation 2 above can be expressed as a general expression of a straight line as in Equation 3 below.

[Equation 3]

In addition, Equation 3 above can be converted into a general expression of a straight line in the global coordinate system as in Equation 4 below.

[Equation 4]

Here, (X0, Y0, Z0) may correspond to the position of the vehicle 110 in consideration of the camera height. Based on Equation 4 above, the following Equation 5 regarding the equation of a straight line for each of the multiple (n) observations can be obtained.

[Equation 5]

14 and 15 are diagrams for explaining a data selection process according to the present invention.

Referring to FIG. 14 , the arrangement estimating apparatus 130 may perform a selection operation on observed data according to observation conditions and observation characteristics. For example, in Fig. (a), observations with the same direction vector can correspond to virtually identical observations if they are located close to each other, and in numerical analysis, the determinant is set to 0 to make the solution unstable. so it needs to be removed.

In Figure (b), observations with large errors may be included among observations with different direction vectors at adjacent positions. In Figure (c), it is desirable that the observation points are separated and the direction vectors are different as shown in (1) and (2).

15, Figure (a) shows all observations before selection, Figure (b) shows selection so that the distance between observation points is greater than or equal to a reference value, Figure (c) is an additional observation direction It is shown that the vector direction is removed.

The observations selected in this way must satisfy the conditions for distance and direction vector difference between each other as shown in Equation 6 below, and the number of observations selected in this way (N) must be equal to or greater than the minimum standard (N _min )

[Equation 6]

for any i, j in N, where N≥N _min

16 is a view for explaining a positioning method according to the present invention.

Referring to FIG. 16 , the figure (a) shows the distance from a point outside the straight line to the straight line, and the distance D can be expressed as a vector expression as in Equation 7 below.

[Equation 7]

In addition, the square of the distance D can be expressed as a quadratic equation for X, Y, and Z as shown in Equation 8 below.

[Equation 8]

Also, the sum of squared distances for all observations can be expressed as a mixed quadratic expression for X, Y, and Z as in Equation 9 below.

[Equation 9]

In addition, the case where the above Equation 9 becomes the minimum is the case where the partial derivatives with respect to X, Y, and Z become 0 as follows. Since the derivative of a quadratic equation becomes a linear equation, the result can be expressed as a linear equation for X, Y, and Z, and can be solved by applying a numerical method of linear algebra as shown in Equation 10 below.

[Equation 10]

However, since some observations may contain large errors due to time delay errors due to high-speed driving of the vehicle, sudden route changes, shocks (speed bumps) during driving, etc. can In this process, observation data that is homogeneous with other observations is called an inlier, and data that is separated from other observations is called an outlier. An outlier is an observation in which the distance from the estimated object position to the observation line is greater than the standard, and the object position is calculated again except for the outlier. When all N _min observations are inliers, the object position can be determined.

[Equation 11]

18 is a view for explaining a method of determining a location of a traffic light and determining arrangement information according to the present invention.

Referring to FIG. 18 , the arrangement estimating apparatus 130 may determine an object position with respect to at least one fixed object as a result of analysis of the observation data through the object position determiner 270 . When the identified fixed object corresponds to a traffic light, the arrangement estimating apparatus 130 may acquire arrangement information of the traffic light through additional observation.

That is, the arrangement estimating apparatus 130 may estimate the type of the corresponding traffic light and the number of lamps based on the observation data through the traffic light arrangement information determiner 290 . Here, the type of the traffic light may include a horizontal traffic light having a larger width and a vertical traffic light having a larger length based on the horizontal and vertical sizes of the traffic light. In addition, the number of lamps may be estimated according to the ratio of the width and length of the traffic light.

In addition, the traffic light arrangement information determining unit 290 may detect and match the lighting conditions for each lamp of the corresponding traffic light from the observation data. That is, when the lighting condition of the lamp of the traffic light is detected, the traffic light arrangement information determining unit 290 may determine which lamp is turned on in the corresponding lighting condition.

In addition, the traffic light arrangement information determining unit 290 may finally determine the lamp configuration and arrangement of the traffic light when matching with respect to lighting conditions for all lamps of the traffic light is completed. For example, in FIG. 18 , when N = 6 and all observations are valid, the arrangement estimator 130 may determine the location of the traffic light. However, since the observation of the lighting condition by the estimated number of lamps is not completed, the arrangement estimating apparatus 130 cannot determine the color of the third lamp.

Accordingly, the arrangement estimator 130 may repeat the operation until matching for all lamps is completed, and finally match the lighting condition of the third lamp in the ninth observation to finally form a traffic light (that is, the lamp configuration and placement) can be confirmed. Thereafter, the arrangement estimating apparatus 130 may detect replacement of traffic lights or change of arrangement through periodic observation.

Although the above has been described with reference to preferred embodiments of the present invention, those skilled in the art can variously modify and change the present invention within the scope without departing from the spirit and scope of the present invention as set forth in the claims below. You will understand that it can be done.

100: batch estimation system
110: vehicle 130: placement estimation device
150: database
210: data collection unit 230: data pre-processing unit
250: data analysis unit 270: object positioning unit
290: traffic light arrangement information determining unit
410: road facilities

Claims (8)

collecting observation data about a fixed object near a road in the driving process of each of the plurality of vehicles;
classifying the observation data according to a classification layer defined for each fixed object;
clustering the observation data based on a location within the same classification layer;
sequentially selecting the observation data according to observation conditions and observation characteristics;
analyzing the observation data by applying a numerical analysis model;
verifying observation reliability based on a result of the analysis;
determining an object position with respect to the at least one fixed object if the verification is passed;
estimating the type of the corresponding traffic light and the number of lamps based on the observation data when the fixed object corresponds to a traffic light;
detecting and matching the lighting conditions for each lamp of the corresponding traffic light from the observation data; and
When the matching is successful, determining the lamp configuration and arrangement of the corresponding traffic light;
The analyzing may include defining the positional coordinates of the fixed object and generating a vector expression for a distance from a direction vector based on the positional coordinates; converting the vector equation into a quadratic equation relating to the position coordinates and applying the quadratic equation to the observation data to generate a mixed quadratic equation relating to the position coordinates; and calculating the position coordinates based on the mixed quadratic formula,
The verifying step includes detecting and removing an outlier in which the distance from the observation data to the calculated location coordinate exceeds a verification criterion. method.
According to claim 1, wherein the collecting step
photographing a front image of the vehicle through at least one camera installed in each of the plurality of vehicles;
extracting at least one image including the fixed object from the front image; and
Multiple observation information comprising the step of receiving observation data including the at least one image, an observation number assigned to the image, and the location, driving direction, and time of the corresponding vehicle from the plurality of vehicles A method of estimating traffic light arrangement information.
The method of claim 1, wherein the selecting comprises:
If the degree of agreement for the classification layer is less than or equal to a threshold value, when the corresponding observation point corresponds to a specific time range, and when the weather at the corresponding observation point meets the specific condition, at least one observation condition is obtained from the observation data. removing;
removing a case in which a distance between observation points for the observation data is less than a preset threshold distance; and
and removing a case in which an angle between direction vectors with respect to the observation data is less than a preset critical angle.
delete delete The method of claim 1, wherein determining the position of the object comprises:
and determining as the location of the object when a predetermined number of observation data that have passed the verification are accumulated.
The method of claim 1, wherein the estimating comprises:
detecting a region of interest (ROI) for the corresponding traffic light from the observation data; and
and determining the type and the number of lamps, respectively, according to a comparison and a ratio between the width and length of the region of interest.
The method of claim 1, wherein determining the lamp configuration and arrangement comprises:
and determining the success of the matching when matching of the lighting conditions for all lamps of the corresponding traffic light is completed.

Images