KR102225321B1 - System and method for building road space information through linkage between image information and position information acquired from a plurality of image sensors - Google Patents

Abstract

A system and method for constructing road space information through linkage between image information acquired from a plurality of image sensors and location information are provided.
The system includes a receiver that receives image information acquired from a plurality of image sensors mounted on a mobile platform and location information acquired from the positioning sensor, combines and synthesizes the image information, and an object from the synthesized image information. The image is based on an object detector that detects and generates object information, and geometric information that defines a geometric relationship between the image information according to the position and posture information of the image information estimated from the position and posture of the plurality of image sensors. An object information analysis unit that analyzes a coupling relationship between the object information and the location information in information, a combination unit that generates first spatial information by combining the object information and the location information with the synthesized image information, and the plurality of A network connection/direction information analysis unit for generating second spatial information constituting a network related to the direction information and connection information between the photographing nodes based on image information photographed continuously moving between the photographing nodes of the image sensor, and the second And a database unit for generating and storing road space information by linking the first and second space information.

Description

System and method for building road space information through linkage between image information and position information acquired from a plurality of image sensors}

The present invention relates to a system and method for constructing road space information through linkage between image information acquired from a plurality of image sensors and location information.

Existing road space information and network construction has a disadvantage in that it is difficult to draw accurate locations of multiple lanes, road signs, signs, and facilities because it is made through drawing of aerial images or simple reading through field images.

In addition, when analyzing photographed data with a single camera and a flat camera, there is a limit to constructing road space information for the entire section of the road due to the limitation of the resolution or the limit of the viewing angle.

The main device for obtaining spatial information including position coordinates is LiDAR (Light Detection and Ranging), which can measure the three-dimensional position of a target object through a laser.

However, in the case of Lidar, it is expensive equipment, requires experts to handle data, and has disadvantages of being vulnerable to weather such as fog.

Accordingly, as a system that provides images around the road rather than a lidar sensor and a single camera, it is intended to generate an omnidirectional image through a plurality of image sensors, such as a plurality of fisheye cameras, and to provide a viewer service combined with location information. Technology is constantly being tried. However, a specific methodology and solution for constructing spatial information such as object location information and road networks displayed on images taken by multiple image sensors has not been presented.

This is not only very difficult to combine and analyze multiple wide-angle lens images, fish-eye lens images, omnidirectional image systems, and image information, but also, in particular, it is analyzed by linking image information with location information collected through a positioning sensor installed outside the camera. This is because no clear methodology has been given.

In addition, due to the lack of a linkage analysis method between image information and location information, there is a limit to creating a road network that is expressed in the form of lines and surfaces with continuity in space.

The technical problem to be achieved by the present invention is to propose a specific solution for constructing spatial information such as object location information and road network displayed on a plurality of images acquired from a plurality of image sensors, and also collected through a positioning sensor installed outside the camera. The present invention aims to provide an apparatus and method for constructing road space information that facilitates creation of a road network expressed in the form of lines and surfaces having continuity on the road space by linking and analyzing one location information and image information.

The object of the present invention is not limited to the objects mentioned above, and other objects not mentioned will be clearly understood by those skilled in the art from the following description.

According to an aspect of the present invention for achieving the above technical problem, a system for constructing road space information through linkage between image information acquired from a plurality of image sensors and location information includes image information acquired from a plurality of image sensors mounted on a mobile platform. And a receiving unit that receives the location information acquired from the positioning sensor and combines and synthesizes the image information, an object detection unit that detects an object from the synthesized image information to generate object information, and the location of the plurality of image sensors Object information analysis that analyzes a coupling relationship between the object information and the location information in the image information based on geometric information defining a geometric relationship between the position of the image information estimated from the posture and the image information according to the posture information A unit, a combination unit for generating first spatial information by combining the object information and the location information with the synthesized image information, and the photographing based on image information continuously moved between the photographing nodes of the plurality of image sensors. A network connection/direction information analysis unit that generates second spatial information constituting a network related to direction information and connection information between nodes, and a database that generates and stores road space information by linking the first and second spatial information And a drawing unit for generating a map composed of a predetermined coordinate model based on the road space information and storing the map in the database unit.

In another embodiment, the object information analysis unit analyzes a coupling relationship between the object information, the position information, and 3D geometric information related to the height of the object based on the geometric information, and the second generated by the combining unit One spatial information may further combine the 3D geometric information.

In another embodiment, when the object information analysis unit cannot calculate 3D geometric information related to the height of the object based on the geometric information, converts the image information into a 2D geometric model, and Geometric information that estimates the height of the object through pixel data analysis, or estimates the height of the object based on the height of the image sensor, and reflects the estimated height of the object to the 3D geometric information and the geometric information A part may be further included, and the first spatial information may be generated to combine the object information, the location information, and the 3D geometric information based on the reflected geometric information.

In another embodiment, the geometric information defines a geometric relationship between the image sensors according to an internal geometry calculated based on a geometric parameter defined for each image sensor and a position and posture of a mobile platform on which a plurality of image sensors are mounted. It can contain external geometry.

In another embodiment, the mobile platform implements a wheel sensor, a control area network (CAN) communication, which is installed in a portion where a rotational motion according to the movement of the mobile platform occurs, and transmits CAN information of the mobile platform to the mobile platform. In the case of having at least one of a CAN module and an inertial navigation sensor that is transmitted to the outside, rotation data, time data, or speed data of the CAN information, and time data of the wheel sensor are acquired, and a photograph taken by the image sensor When the photographing time data of the node and the positioning time data of the positioning sensor are inconsistent, together with the time data of the photographing node between adjacent positioning information measured from the positioning sensor, rotation and time data of the wheel sensor or the CAN It may further include a position correction/estimator for correcting the position information by calculating the position of the photographing node by using an interpolation or extrapolation model based on the speed and time data of the information, and reflecting it in the first spatial information.

Here, the position correction/estimator, when the location information of the photographing node exceeds a reference value from the movement trajectory information according to the movement path of the mobile platform based on the second spatial information, the location information of the exceeding photographing node It can be removed by filtering. In addition, the inertial navigation sensor may receive at least one inertial navigation data of acceleration, angular acceleration, and magnetic north of the mobile platform, and the position correction/estimator may correct or filter the location information based on the inertial navigation data.

In another embodiment, the difference between the location information between the photographing nodes photographed by the image sensor, the photographing direction of the image sensor photographing a specific object and the first angle between the specific object, the photographed image sensor and the adjacent image Calculate a second angle between the sensors and a third angle between the photographing direction and the north direction, calculate movement trajectory information of the mobile platform based on the difference, and based on the movement trajectory information, the first to third angles The posture information estimation unit may further include a posture information estimating unit for estimating the posture information of the image sensor, estimating posture information of the image information from the posture information of the image sensor, and reflecting the posture information of the image information to the geometric information. have.

In another embodiment, based on the existing road space information stored in the database unit, the geometric information belonging to the road space information, and the location information, the existing and new road space information is converted into an integrated geometric model, and the existing road space As a result of comparing information and the first spatial information belonging to the road space information by image matching, when there is a change in the object information, an update of updating the existing road space information with the road space information to change the map May contain more wealth.

According to an aspect of the present invention for achieving the above technical problem, a method for constructing road space information through linkage between image information acquired from a plurality of image sensors and location information includes image information acquired from a plurality of image sensors mounted on a mobile platform. Receiving the location information acquired from the and positioning sensor, combining and synthesizing the image information; generating object information by detecting an object from the synthesized image information; and the position and posture of the plurality of image sensors Analyzing a combination relationship between the object information and the location information in the image information based on geometric information defining a geometric relationship between the image information according to the position and posture information of the image information estimated from Generating first spatial information by combining the object information and the location information with the synthesized image information, and direction information between the photographing nodes based on image information photographed continuously moving between the photographing nodes of the plurality of image sensors. And generating second spatial information constituting a network related to connection information, generating and storing road space information by linking the first and second spatial information, and predetermined coordinates based on the road space information. And generating and storing a map composed of a model.

Details of other embodiments are included in the detailed description and drawings.

According to the present invention, a specific solution for constructing spatial information such as location information of an object and a network of roads displayed on a plurality of images acquired from a plurality of image sensors can be presented.

In addition, by linking and analyzing the location information collected through the positioning sensor installed outside the image sensor and the image information, it is possible to easily construct a road network that is expressed in a form having continuity on the road spatial information.

In addition, effects derived through configurations that can be grasped by those skilled in the art through the present specification are not excluded.

1 is a block diagram of an apparatus for constructing road space information through linkage between image information acquired from a plurality of image sensors and location information according to an embodiment of the present invention.
2 is a flowchart illustrating a method for constructing road space information through linkage between image information acquired from a plurality of image sensors and location information according to another embodiment of the present invention.
3 is a diagram illustrating image information acquired from a plurality of image sensors and a combination thereof.
4 is a diagram illustrating a process of recognizing object information by recognizing an object from image information.
5 is a diagram illustrating a process of generating road space information.
6 is a flowchart of a method for constructing road space information according to another embodiment of the present invention.
7 is a diagram illustrating a process of correcting road space information by combining external road space information with road space information.
8 is a flowchart of a method for constructing road space information according to another embodiment of the present invention.
9 is a diagram illustrating a process of estimating an object height through pixel data analysis in image information.
10 is a flowchart of a method for constructing road space information according to another embodiment of the present invention.
11 is a diagram for explaining a process of correcting location information of a photographing node.
12 is a flowchart of a method for constructing road space information according to another embodiment of the present invention.
13 is a diagram for describing a process of estimating posture information of image information.
14 is a flowchart of a method for constructing road space information according to another embodiment of the present invention.
15 is a diagram for describing a process of updating road space information.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings and the following description. However, the present invention is not limited to the embodiments described herein and may be embodied in other forms. Rather, the embodiments introduced herein are provided so that the disclosed contents may be thorough and complete, and the spirit of the present invention may be sufficiently conveyed to those skilled in the art. The same reference numbers throughout the specification indicate the same elements. Meanwhile, terms used in the present specification are for explaining embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used herein, "comprises" and/or "comprising" refers to the presence of one or more other components, steps, actions and/or elements in which the recited component, step, operation and/or element is Or does not preclude additions.

In addition, "sub" to "module" generally refer to components such as software (computer program) and hardware that can be logically separated. Therefore, the module in this embodiment refers not only to the module in the computer program, but also to the module in the hardware configuration. Therefore, the present embodiment is a computer program for making them function as modules (a program for executing each step in a computer, a program for making a computer function as each means, a program for realizing each function in a computer) ), also describes the system and method. However, for convenience of explanation, phrases equivalent to "save" and "save" are used, but these words are stored in a memory device or stored in a memory device when the embodiment is a computer program. It means to control like that. In addition, the "sub" to the module may correspond to the function one-to-one, but in the implementation, one module may be composed of one program, a plurality of modules may be composed of one program, and conversely, one module may be composed of a plurality of programs. do. Further, a plurality of modules may be executed by one computer, or one module may be executed by a plurality of computers by a computer in a distributed or parallel environment. In addition, another module may be included in one module. In addition, hereinafter, "connection" is adopted in the case of logical connection (data exchange, instruction, reference relationship between data, etc.) in addition to physical connection. The term "predetermined" means that it is determined before the target processing, as well as before the processing according to the present embodiment is started, even after the processing according to the present embodiment is started, and before the target processing, the It includes the meaning of what is determined according to the situation and state of the time, or the situation and state up to that time.

In addition, a system or device is configured by connecting a plurality of computers, hardware, devices, etc. by means of communication such as a network (including a one-to-one communication connection), and a single computer, hardware, device, etc. This includes cases where it is realized. The terms "device" and "system" are used as terms of mutual agreement. Of course, the "system" does not include anything but an artificial decision, a social "mechanism" (social system).

In addition, when processing by each unit or module or a plurality of processing within each unit or module, the target information is read and input from the storage device for each processing, and after the processing is performed, the processing result is displayed. It is to write to the storage device. Therefore, the description of the read input from the storage device before processing and the writing to the storage device after processing may be omitted in some cases. Further, the storage device herein may include a hard disk, a random access memory (RAM), an external storage medium, a storage device through a communication line, a register in a CPU (Central Processing Unit), and the like.

Hereinafter, a system for constructing road space information through linkage between image information acquired from a plurality of image sensors 104 and location information according to an embodiment of the present invention will be described with reference to FIG. 1.

1 is a block diagram of a system for constructing road space information through linkage between image information acquired from a plurality of image sensors and location information according to an embodiment of the present invention.

The system 100 for constructing road spatial information is equipped with a plurality of image sensors 104 on the mobile platform 102, and mobile mapping that recognizes and analyzes objects around the road by processing and analyzing data from the sensors. It can be a system. Such an object may be a facility such as an automobile road, a bicycle road, a footpath, or a road around a river, but does not exclude various types of facilities installed on the ground.

The system 100 includes a set of image sensors (104) and a positioning sensor 106 mounted on a mobile platform 102, as in FIG. 1. In addition, the system 100 may further include a sensor control module 108 for synchronizing photographing of the image sensor 104.

In addition, the system 100 includes a receiving unit 110, an object detection unit 112, an object information analysis unit 114, a geometric information unit 116, a position correction/filtering unit 118, a movement information unit 120, a combination unit ( 122), a posture information estimation unit 124, a network connection/direction information analysis unit 126, an external information acquisition unit 128, and a drawing unit 130. In addition, the system 100 may include a database unit 132, a display unit 134, a post-processing unit 136, and an update unit 142.

The image sensor set 104 is equipped with a plurality of sensors that are mounted on a mobile platform to capture an image of surrounding objects, such as terrain and features, and acquire image data and time information for an image at the time the image data was acquired, A wide area image sensor having a wider field of view than the planar image sensor 104 that generates a flat image, a surveying or non-surveying camera, and a stereo camera, are not limited thereto. The wide area image sensor generates a panoramic image, and may be, for example, a camera having a fisheye lens to generate a fisheye lens image, or a multi-camera system that outputs an omnidirectional image by multiple cameras using a plurality of cameras.

The positioning sensor 106 acquires position information including positioning data related to the object coordinates and positioning time data at the time the positioning information was acquired by detecting navigation information such as positioning coordinates of the object, position of the platform, attitude, and speed. It is a sensor. The positioning sensor 106 includes, for example, a position acquisition device for acquiring a moving position of a platform through a GPS system using an international satellite navigation system (GNSS), an inertial measurement unit (IMU), and inertial navigation. The device (Inertial Navigation System; INS) may be configured as a posture acquisition device that acquires the posture of the vehicle.

When the positioning sensor 106 uses GNSS, the location information basically includes positioning data and positioning time data, as well as the state and number of navigation satellite information, altitude information of the receiver, speed, positioning precision, and true north information. Can have as ancillary data. In addition, the location information may additionally include posture information of the image sensors 104.

The image sensor set 104 may be directly controlled from the positioning sensor 106 to synchronize photographing of objects around the road, or a sensor control module 108 connected by wire or wirelessly to the positioning sensor as shown in FIG. 1, that is, Shooting synchronization of the plurality of image sensors 104 constituting the image sensor set 104 may be implemented through the branch node (branch point).

In order to simultaneously utilize a plurality of wide-angle lenses or fisheye images constituting the image sensor set 104, the times of the plurality of image sensors 104 may be synchronized and used.

The image sensor 104 may be mounted on a fixed frame to perform photographing, and may be used to analyze the position and posture of an image on a mobile platform by defining relative geometric information between the image sensors 104. In addition, an omnidirectional image may be generated from multiple wide-angle or fish-eye lens images synchronized to be photographed and used for information construction.

In order to synchronize the time of the multiple image sensors 104, a simultaneous triggering technique for the image sensors 104 may be applied, and triggering may be requested from each camera at a certain point in time or at a distance. The triggering request may be controlled by simultaneously sending a signal to the image sensor 104 directly from an external computing device, or by separately installing an analog or digital device serving as a branch point.

The receiving unit 110 receives image information acquired from the plurality of image sensors 104 and position information acquired from the positioning sensor 106, and combines and synthesizes the image information.

The geometric information between the image information is basically determined based on the design drawing, but in order to secure accurate information, model variables may be estimated through the application of a calibration technology or an image matching technology between the image sensors 104. An omni-directional image or an ultra-wide-angle image can be artificially generated by projecting multiple images onto a virtual image plane based on geometric information between image information.

The geometric information defines the internal geometry calculated based on the geometric parameters defined for each image sensor 104 and the geometric relationship between the image sensors 104 according to the position and posture of the mobile platform on which the plurality of image sensors 104 are mounted. It can be composed of external geometries.

The internal geometry is an intrinsic value of the sensor itself, and is an error in observed data for each image sensor 104 due to a parameter maintained regardless of whether the platform or the like is moved.

The geometric parameter of the image sensor 104 may be at least one of a focal length, a main point position, a lens distortion parameter, and a sensor format size.

The external geometry is observed data calculated based on parameter values that change each time the image information of each image sensor 104 is acquired due to the position and posture of the platform being moved, that is, the position and posture of each image sensor 104 Is an error of, and in this case, it can be calculated through a predetermined equation or modeling.

The object detector 112 may detect and recognize object information related to an object around a road through machine learning based on the image information.

The object information may include attribute data including a type, a name of an object, metadata related to detailed information, a shape, a shape, and a texture in relation to a road facility.

The object may be a road facility that appears on the ground, such as lanes, road markings, crosswalks, stop lines, speed bumps, signs, and traffic lights.

The object detector 112 may detect candidate group data related to the object identified from the image information using a machine learning technique, and sequentially apply an additional machine learning model to the detected candidate group data to recognize the object information.

The object detector 112 may perform preprocessing to improve accuracy of object detection and recognition in image information prior to the machine learning technique. Pre-processing of image information includes, for example, image brightness adjustment, image color adjustment, image blurring, image sharpness, image texture analysis, and image due to differences in observed geometric characteristics. The sensor data may be moved, rotated, and/or scaled (transformed by at least one of Rigid-body, Affine, and Similartiy as 2D transformed data).

The object information analysis unit 114 includes object information in the image information based on geometric information defining a geometric relationship between image information according to the position and posture information of image information estimated from the position and posture of the plurality of image sensors 104. It analyzes and understands the coupling relationship between and location information. Accordingly, the object information analysis unit 114 calculates spatial information of an object based on geometric information. The geometric information is substantially the same as described in the receiving unit 110.

The object information analysis unit 114 may generate object space information by analyzing a coupling relationship between object information and location information and 3D geometric information related to the height of the object based on the geometric information.

In the case of an omnidirectional image generated through a plurality of images, the spatial information of an object photographed in the omnidirectional image may be recognized by inversely using the geometric information used to generate the omnidirectional image. For example, for convenience of calculation, a virtual geometric model can be substituted, and the virtual geometric model is trigonometric, polynomial, projective, affine, and perspective. ), a model including at least one of a homography geometry may be used.

In addition, in order to shorten the computing time during the process, information between image information based on geometric information or combination information between image information and location information may be indexed and utilized in a matrix or vector format.

The combining unit 122 includes object information on the synthesized image information. First spatial information may be generated by combining location information and 3D geometric information. The first spatial information may link not only the above-described information, but also geometric information to image information.

The network connection/direction information analysis unit 126 is based on the image information continuously moved between the photographing nodes of the plurality of image sensors 104 and the second spatial information constituting a network related to the connection information and direction information between the photographing nodes. Create

When the object information analysis unit 114 cannot calculate 3D geometric information related to the height of the object based on the geometric information, the geometric information unit 116 estimates the height of the object and calculates the estimated height of the object by 3 It is reflected in the dimensional geometric information and geometric information. The first spatial information generated by the combining unit 122 may be processed in a form in which object information, location information, and 3D geometric information are combined based on the reflected geometric information.

The geometric information unit 116 may convert the image information into a 2D geometric model, estimate the height of the object through pixel data analysis within the image information, or estimate the height of the object based on the height of the image sensor 104. A detailed description of pixel data analysis will be described later with reference to FIG. 9.

In addition to the above-described method of estimating the height of the object, when height information of the objects is limited, the height of the object may be registered and used as a specific height value from a datum or an index.

In addition, it is possible to simplify the geometry by assuming the roll and pitch information of the sensor as specific values.

In addition, each photographing direction information of the plurality of image sensors 104 may be defined as a specific value, and object space information may be calculated through a trigonometric function model.

On the other hand, the mobile platform 102 implements a wheel sensor and a control area network (CAN) communication that is installed in a portion where rotational motion occurs according to the movement of the platform 102 to transmit CAN information of the mobile platform 102 to the mobile platform. It may be provided with at least one of a CAN module and an inertial navigation sensor that transmits to the external device of the.

The movement information unit 120 may receive rotation data and time data from a wheel sensor, and may receive various vehicle data such as speed data and time data as vehicle CAN information from a CAN module, and a mobile platform ( 102) may receive inertial navigation data of at least one of acceleration, angular acceleration, and magnetic north. The movement information unit 120 may transmit the above-described data to the position correction/filtering unit 118.

The distance between the photographing nodes in the mobile platform 102 is calculated based on the movement information of the mobile platform 102, and may be controlled using at least one of, for example, wheel rotation information and positioning sensor information. For example, the information of the mobile platform 102 is recorded by mounting a recording sensor at a specific position of the rotating part of the wheel to record the number of revolutions or the number of pulse inputs in a certain time unit, and reverse the movement distance by using the circumference of the wheel. Calculate. In the case of the wheel circumference length, it can also be inversely calculated in conjunction with a positioning sensor to obtain more accurate information.

The position correction/filtering unit 118 acquires rotation data, time data of the wheel sensor, or speed data and time data of vehicle CAN information. The position correction/filtering unit 118 is the adjacent positioning information measured from the positioning sensor 106 when the photographing time data of the photographing node photographed by the image sensor 104 and the positioning time data of the positioning sensor 106 are inconsistent. It is possible to correct the position information by calculating the position of the photographing node by using an interpolation or extrapolation model based on the rotation and time data of the wheel sensor or the speed and time data of the vehicle CAN information, together with the time data of the photographing node between them. . In addition, the position correction/filtering unit 118 reflects the first spatial information generated by the combining unit 122. For interpolation or extrapolation models, an N-order polynomial can be applied.

In addition, the position correction/filtering unit 118 includes a reference value from the movement trajectory information according to the movement path of the mobile platform based on the second spatial information generated by the network connection/direction information analysis unit 126 of the location information of the viewing node. If it exceeds, the location information of the excess photographing node may be filtered and removed.

The reason why the position correction/filtering unit 118 corrects or filters the position information in this way is that there is no recording function of the posture information inside the image sensor 104, so that the posture information estimating unit 124 estimates the position information. In the case of using the information, it is because the error of the position information is transmitted as an error to the estimated value of the attitude information.

Correction and filtering of location information may be performed separately, and these may be combined and applied.

For example, a method of extracting an outlier or correcting a positioning error may be selected through the tendency of movement of the mobile platform 102.

The mobile platform 102 generally has straightness, and as described above, the distance between specific photographing nodes may be interpolated based on the straightness, and an N-order polynomial may be applied as an interpolation model. .

In another embodiment, the position information may be corrected or filtered by applying a statistical method in which an estimated error amount is inversely calculated when estimating posture information, and the estimated error amount is applied to estimation of a parameter value of a geometric model with a weight ratio.

As another embodiment, in addition to a GNSS-based position positioning sensor, a wheel sensor, an inertial navigation sensor, and vehicle CAN data may be additionally utilized. In the case of a wheel sensor, target nodes between reference nodes can be used as a constraint on the distance between each node when correcting or filtering.

As another embodiment, the position correction/filtering unit 118 may correct or filter position information based on at least one of inertial navigation data of acceleration, angular acceleration, and magnetic north of the mobile platform 102.

In addition, the position correction/filtering unit 118 inserts and records a synchronization signal at a separate branch point in order to solve the problem of delay in information transmission in recording the position information of the positioning sensor 106 through the sensor control module 108 can do.

When the frequency of the synchronization signal is insufficient, the position correction/filtering unit 118 may inversely estimate the position information of the time of interest through the time of the system and the time relationship with other sensors. Inverse estimation is basically performed by interpolation or extrapolation, and a polynomial model for time can be used as a model for mathematical calculation.

When the position information and the posture information are combined and used, the above-described information may be inversely estimated through repeated prediction-correction. When there is a lot of noise in the location information, etc., errors that may appear in the information may be reflected through a statistical technique, and the above-described information may be filtered or corrected.

The posture information estimating unit 124 may inversely estimate the posture information through a combination of location information between the photographing nodes when there is no recording function for the posture information inside the image sensor 104.

Specifically, referring to FIGS. 11 and 13, the attitude information estimating unit 124 includes a difference between location information between the photographing nodes 1 to n photographed by the image sensor 104, and an image sensor photographing a specific object ( The first angle (d) between the photographing direction of 104) and the specific object, the second angle (c) between the photographed image sensor 104 and the adjacent image sensor 104, and the third angle (b) between the photographing direction and the north direction. ) Is calculated, the movement trajectory information of the mobile platform 102 is calculated based on the difference, and the posture information of the image sensor 104 is calculated based on the movement trajectory information and the first to third angles (d to b). In addition to estimating, the posture information of the image information may be estimated from the posture information of the image sensor 104. The posture information estimating unit 124 reflects the posture information of the image information to the geometric information.

Meanwhile, the database unit 132 generates and stores road space information by linking the first and second space information.

The drawing unit 130 generates a map composed of a predetermined coordinate model based on the road space information and stores it in the database unit 132. Accordingly, the road space information model on the mapped coordinate model can be configured as a network between object space information using location information.

The external information acquisition unit 128 may receive external road space information having continuous data related to an area of the image information, and combine the external road space information with the road space information.

As the photographing nodes are discontinuous and involve errors, spatial information previously constructed such as aerial images, digital maps, and numerical models for correction as continuous data, that is, external road space information, is used as auxiliary data for road network construction. Can be.

In the case of using images taken from satellite or aerial, the resolution of the satellite, etc. is different from the image taken from the mobile platform 102 on the ground, so the statistical analysis of the pattern of changes in pixel values matches the same object appearing in different images. Can be used for

The updated part may be checked and corrected by back-projecting the structured external road space information or network onto the image information included in the first spatial information.

The update unit 142 converts the existing and new road space information into an integrated geometric model based on the existing road space information stored in the database unit 132 and the geometric information and location information belonging to the newly constructed road space information. The road space information and the first space information belonging to the road space information are compared by image matching. As a result, when there is a change in the object information, the updater 142 may update the existing road space information with the road space information to change the map.

The update unit 142 can simplify and analyze the geometry into two dimensions when the image-position combination information that can be secured due to the absence of three-dimensional spatial information such as three-dimensional geometric information exists only with the image position and direction value, which is a positioning sensor. It can be done in the same way as the technique used to map by combining the location information of (106).

The post-processing unit 136 provides image information to meet the request when a request for extracting an image of a user's desired method or a point of interest (POI) from the road space information stored in the database unit 132 is requested. An image information editing unit 138 for editing and a location information editing unit 140 for editing location information according to an image conversion method desired by the user may be provided.

Hereinafter, a method for constructing road space information through linkage between image information acquired from a plurality of image sensors 104 and location information according to embodiments of the present invention will be described with reference to FIGS. 1 to 7.

FIG. 2 is a flowchart of a method for constructing road space information through linkage between image information acquired from a plurality of image sensors and location information according to another embodiment of the present invention, and FIG. 3 is a flow chart showing image information acquired from a plurality of image sensors and It is a diagram showing a combination of these. 4 is a diagram illustrating a process of recognizing object information by recognizing an object from image information, and FIG. 5 is a diagram illustrating a process of generating road space information. 6 is a flowchart illustrating a method for constructing road space information according to another embodiment of the present invention, and FIG. 7 is a diagram illustrating a process of correcting road space information by combining external road space information with road space information.

First, the receiving unit 110 receives image information from the plurality of image sensors 104 and location information from the positioning sensor 106 (S205).

The receiving unit 110 receives front, rear, left and right image information captured by the omnidirectional image sensors 104 mounted on the mobile platform 102, as shown in FIG. 3.

Next, the receiving unit 110 combines and synthesizes the image information. The object detector 112 generates object information by detecting an object from the synthesized image information (S210).

The geometric information between the image information is basically determined based on the design drawing, but in order to secure accurate information, model variables may be estimated through the application of a calibration technology or an image matching technology between the image sensors 104. As shown in FIG. 3, the receiver 110 may artificially generate an omni-directional image or an ultra-wide-angle image by projecting multiple images onto a virtual image plane based on geometric information between image information.

The geometric information defines the internal geometry calculated based on the geometric parameters defined for each image sensor 104 and the geometric relationship between the image sensors 104 according to the position and posture of the mobile platform on which the plurality of image sensors 104 are mounted. It can be composed of external geometries.

The object detection unit 112 may detect and recognize object information related to an object around a road through machine learning based on front, left, and rear image information, as illustrated in FIG. 4, for example.

The object information may include attribute data including a type, a name of an object, metadata related to detailed information, a shape, a shape, and a texture in relation to a road facility.

The object may be a road facility that appears on the ground, such as lanes, road markings, crosswalks, stop lines, speed bumps, signs, and traffic lights.

The object information analysis unit 114 includes object information in the image information based on geometric information defining a geometric relationship between image information according to the position and posture information of image information estimated from the position and posture of the plurality of image sensors 104. The coupling relationship between the and the location information is analyzed, and the combining unit 122 generates first spatial information that combines the object information, the location information, and the 3D geometric information with the image information (S215).

In the case of an omnidirectional image generated through a plurality of images, the spatial information of an object photographed in the omnidirectional image may be recognized by inversely using the geometric information used to generate the omnidirectional image. For example, for convenience of calculation, a virtual geometric model can be substituted, and the virtual geometric model is trigonometric, polynomial, projective, affine, and perspective ), a model including at least one of a homography geometry may be used.

In addition, in order to shorten the computing time during the process, information between image information based on geometric information or combination information between image information and location information may be indexed and utilized in a matrix or vector format.

The combining unit 122 includes object information on the synthesized image information. First spatial information may be generated by combining location information and 3D geometric information. The first spatial information may link not only the above-described information, but also geometric information to image information.

Next, the network connection/direction information analysis unit 126 configures a network related to direction information and connection information between the photographing nodes based on the image information continuously moving and photographed between the photographing nodes of the plurality of image sensors 104. 2 Generate spatial information (S220).

Subsequently, the database unit 132 generates and stores road space information by linking the first and second space information (S225). 5 illustrates that each object is linked with road space information.

Subsequently, the drawing unit 130 generates a map composed of a predetermined coordinate model based on the road space information and stores it in the database unit 132 (S230).

Accordingly, as shown in FIG. 7, the road space information model on the mapped integrated coordinate model can be configured as a network between object space information using location information.

The process of FIG. 6 may be performed following the process of FIG. 2, which is a process of performing correction as continuous data as the photographing nodes are discontinuous and cause errors.

Referring to FIG. 6, the external information acquisition unit 128 receives external road space information having continuous data related to an area of the image information (S605).

The external information acquisition unit 128 combines the external road space information with the road space information (S610).

As shown in FIG. 7, the photographing nodes are discontinuous, and due to errors, spatial information previously constructed such as aerial images, digital maps, and numerical models for correction as continuous data, that is, external road space information is a road network. It can be used as an auxiliary material for construction.

In the case of using images taken from satellite or aerial, the resolution of the satellite, etc. is different from the image taken from the mobile platform 102 on the ground, so the statistical analysis of the pattern of changes in pixel values matches the same object appearing in different images. Can be used for

The updated part may be checked and corrected by back-projecting the structured external road space information or network onto the image information included in the first spatial information.

According to the present embodiment, a specific solution for constructing spatial information such as location information of an object and a network of roads displayed on a plurality of images acquired from a plurality of image sensors 104 can be presented. In addition, by linking and analyzing the location information collected through the positioning sensor installed outside the image sensor 104 and the image information, it is possible to easily construct a road network that is expressed in a form having continuity on the road spatial information. .

Hereinafter, a method for constructing road space information according to another embodiment of the present invention will be described with reference to FIGS. 1, 8 and 9.

8 is a flowchart illustrating a method for constructing road space information according to another embodiment of the present invention, and FIG. 9 is a diagram illustrating a process of estimating an object height through pixel data analysis in image information.

This embodiment is performed when the object information analysis unit 114 cannot calculate three-dimensional geometric information related to the height of an object based on the geometric information in the process of FIG. 2, and is performed by the geometric information unit 116. Some of the geometric information calculation methods will be described with reference to FIG. 8. However, other methods of the geometric information unit 116 are not excluded in this embodiment.

Referring to FIG. 8, when the object information analysis unit 114 cannot calculate 3D geometric information related to the height of the object based on the geometric information, the geometric information unit 116 converts the image information into a 2D geometric model. It converts (S805).

Next, the geometric information unit 116 estimates the height of the object 202 through pixel data analysis in the image information, or estimates the height of the object 202 based on the height of the image sensor 104 (S810).

9, by analyzing pixels in the image data for each horizontal and vertical pixel, the height of the image sensor 104, the angle h between the horizontal line and the highest point of the object from the image sensor 104, and the center focus of the image sensor 104 The angle (a) between the extended line from (f) and the horizontal line is calculated, and based on this, the height of the object is estimated in consideration of the number of pixels and vanishing points. Accordingly, 3D geometric information is collected.

Subsequently, the geometric information unit 116 estimates the 3D geometric information by reflecting it on the geometric information, and the combining unit 122 generates first spatial information accordingly (S815).

Hereinafter, a method for constructing road space information according to another embodiment of the present invention will be described with reference to FIGS. 1, 10 and 11.

10 is a flowchart illustrating a method for constructing road space information according to another embodiment of the present invention, and FIG. 11 is a diagram for explaining a process of correcting location information of a photographing node.

In the present embodiment, when the photographing node photographed by the image sensor 104 and the reception point of the positioning sensor 106 are inconsistent with each other in the process of FIG. 2, the position information of the photographing node is corrected with the positioning data of the photographing node, When the posture information estimating unit 124 uses the location information in estimating the posture information, the error of the location information is to minimize the error in the estimated value of the posture information.

Some of correction and filtering performed by the position correction/filtering unit 118 will be described with reference to FIG. 10. However, other methods of the position correction/filtering unit 118 in this embodiment are not excluded.

Referring to FIG. 10, the position correction/filtering unit 118 acquires rotation data of the wheel sensor, time data, or speed data of the CAN information, and the time data from the movement information unit 120 (S1005).

As shown in FIG. 11, the position correction/filtering unit 118 is in the case where the time data of the photographing nodes 1 to n photographed by the image sensor 104 and the time data of the positioning sensor 106 are inconsistent, Using the interpolation or extrapolation model based on the rotation and time data of the wheel sensor or the speed and time data of CAN information, together with the time data of the photographing node between adjacent positioning information measured from the positioning sensor 106, The position is calculated (S1010). For interpolation or extrapolation models, an N-order polynomial can be applied. Accordingly, the position correction/filtering unit 118 corrects the position information and reflects it in the first spatial information. This contributes to the generation of the first spatial information.

Next, the location correction/filtering unit 118 is the movement trajectory information according to the movement path of the mobile platform 102 based on the second spatial information generated by the network connection/direction information analysis unit 126 where the location information of the viewing node is It is determined whether the reference value is exceeded from (“Platform Movement Path” in FIG. 11) (S1015), and the location information of the excess photographing node is filtered and removed (S1020).

Hereinafter, a method for constructing road space information according to another embodiment of the present invention will be described with reference to FIGS. 1, 11, 12, and 13.

12 is a flowchart illustrating a method for constructing road space information according to another embodiment of the present invention, and FIG. 13 is a diagram for explaining a process of estimating posture information of image information.

This embodiment is applied to the case of inverse estimation of posture information through a combination of position information between photographing nodes when there is no recording function for posture information inside the image sensor 104 in the process of FIG. 2. Some of the posture information estimation methods performed by the posture information estimating unit 124 will be described with reference to FIG. 12. However, other methods of the geometric information unit 116 are not excluded in this embodiment.

11 to 13, the posture information estimating unit 124 includes a difference between positional information between the photographing nodes 1 to n photographed by the image sensor 104, and the image sensor 104 photographing a specific object. The first angle (d) between the photographing direction of and a specific object, the second angle (c) between the photographed image sensor 104 and the adjacent image sensor 104, and the third angle (b) between the photographing direction and the north direction. Calculate (S1205),

Next, the posture information estimating unit 124 calculates movement trajectory information ("platform movement path" in FIG. 11) of the mobile platform 102 based on the difference (S1210),

Subsequently, the posture information estimating unit 124 estimates the posture information of the image sensor 104 based on the movement trajectory information and the first to third angles d to b, and from the posture information of the image sensor 104 The posture information of the image information is estimated (S1215).

Next, the posture information estimating unit 124 reflects the posture information of the image information to the geometric information (S1220). This contributes to the generation of the first spatial information.

Hereinafter, a method for constructing road space information according to another embodiment of the present invention will be described with reference to FIGS. 1, 14 and 15.

14 is a flowchart illustrating a method for constructing road space information according to another embodiment of the present invention, and FIG. 15 is a diagram illustrating a process of updating road space information.

First, the update unit 142 converts the existing and new road space information into an integrated geometric model based on the existing road space information stored in the database unit 132 and the geometric information and location information belonging to the newly constructed road space information. (S1405).

Next, the updater 142 compares the existing road space information and the first space information belonging to the new road space information by image matching (S1410).

As a result, when there is a change in the object information (S1415), the updater 142 updates the existing road space information with new road space information to change the map as shown in FIG. 15.

If the image-location combination information that can be secured due to the absence of 3D spatial information exists only with the image position and direction value, the geometry can be simplified and analyzed in two dimensions, which is the technique used to map by combining the position information of the positioning sensor. The same can be done.

Components constituting the apparatus shown in Fig. 1 or steps according to the embodiments shown in Figs. 2 to 15 may be recorded on a computer-readable recording medium in the form of a program for realizing the function. Here, the computer-readable recording medium refers to a recording medium that can be read by a computer by storing information such as data or programs by electrical, magnetic, optical, mechanical, or chemical action. Examples of such recording media that can be separated from a computer include portable storage, flexible disks, magneto-optical disks, CD-ROMs, CD-R/Ws, DVDs, DATs, and memory cards. In addition, as recording media fixed to mobile devices and computers, there are solid state disks (SSDs), hard disks, and ROMs.

Further, in the above, even if all the constituent elements constituting the embodiments of the present invention are described as being combined and operated as one, the present invention is not necessarily limited to these embodiments. That is, within the scope of the object of the present invention, all the constituent elements may be selectively combined and operated in one or more. In addition, all of the components may be implemented as one independent hardware, but some or all of the components are selectively combined to provide a program module that performs some or all of the combined functions in one or a plurality of hardware. It may be implemented as a computer program having.

Although the present invention has been described in detail through exemplary embodiments above, a person of ordinary skill in the art to which the present invention pertains can make various modifications to the above-described embodiments without departing from the scope of the present invention. I will understand. Therefore, the scope of the present invention is limited to the described embodiments and should not be determined, and should be determined by all changes or modified forms derived from this claim and the concept of equality as well as the claims to be described later.

Claims (10)

In a system for constructing road space information for linkage between image information acquired from multiple image sensors and location information,
A receiver configured to receive image information acquired from a plurality of image sensors mounted on a mobile platform and position information acquired from a positioning sensor, and combine and synthesize the image information;
An object detector for generating object information by detecting an object from the synthesized image information;
Combining the object information and the location information with the image information based on geometric information defining a geometric relationship between the image information according to the position and posture information of the image information estimated from the position and posture of the plurality of image sensors An object information analysis unit that analyzes the relationship;
A combiner configured to generate first spatial information by combining the object information and the location information with the synthesized image information;
A network connection/direction information analysis unit for generating second spatial information constituting a network related to the direction information and connection information between the photographing nodes based on image information continuously moving and photographed between the photographing nodes of the plurality of image sensors;
A database unit for generating and storing road space information by linking the first and second space information; And
And a drawing unit for generating a map composed of a predetermined coordinate model based on the road space information and storing the map in the database unit,
The difference between the positional information between the photographing nodes photographed by the image sensor, a photographing direction of the image sensor photographing a specific object and a first angle between the specific object, a second angle between the photographed image sensor and an adjacent image sensor, Calculate a third angle between the photographing direction and the north direction, calculate movement trajectory information of the mobile platform based on the difference, and posture of the image sensor based on the movement trajectory information and the first to third angles In addition to estimating information, a system for constructing road space information including a posture information estimation unit that estimates posture information of the image information from posture information of the image sensor, and reflects the posture information of the image information to the geometric information. The method of claim 1,
The object information analysis unit analyzes a coupling relationship between the object information and the location information and 3D geometric information related to the height of the object based on the geometric information,
The first spatial information generated by the combining unit is a system for constructing road spatial information that further combines the 3D geometric information. The method of claim 1,
When the object information analysis unit cannot calculate 3D geometric information related to the height of the object based on the geometric information,
Converts the image information into a two-dimensional geometric model, estimates the height of the object through pixel data analysis within the image information, or estimates the height of the object based on the height of the image sensor, and Further comprising a geometric information unit for reflecting the height to the three-dimensional geometric information and the geometric information,
The first spatial information is a system for constructing road spatial information generated to combine the object information, the location information, and the 3D geometric information based on the reflected geometric information The method of claim 1,
The geometric information is a road including an internal geometry calculated based on a geometric parameter defined for each image sensor and an external geometry defining a geometric relationship between the image sensors according to a position and attitude of a mobile platform on which a plurality of image sensors are mounted. A system for building spatial information.
The method of claim 1,
A wheel sensor installed at a portion where the mobile platform is rotated according to the movement of the mobile platform, and a CAN module that implements CAN (Control Area Network) communication to transmit CAN information of the mobile platform to the outside of the mobile platform. And in the case of having at least one of an inertial navigation sensor,
When the rotation data of the wheel sensor, time data, or speed data of the CAN information, and the time data are acquired, and the photographing time data of the photographing node photographed by the image sensor and the positioning time data of the positioning sensor are inconsistent, Using an interpolation or extrapolation model based on the rotation and time data of the wheel sensor or speed and time data of the CAN information together with time data of the photographing node between adjacent positioning information measured from the positioning sensor A system for constructing road space information, further comprising a location correction/estimator that calculates a location, corrects the location information, and reflects the location information in the first space information. The method of claim 5,
When the position correction/estimator of the photographing node exceeds a reference value from the movement trajectory information according to the movement path of the mobile platform based on the second spatial information, the position correction/estimator filters the location information of the exceeding photographing node. A system for building space information on the road to be removed. The method of claim 6,
The inertial navigation sensor receives at least one inertial navigation data of acceleration, angular acceleration, and magnetic north of the mobile platform, and the position correction/estimator corrects or filters the location information based on the inertial navigation data. For the system. delete The method of claim 1,
Converts existing and new road space information into an integrated geometric model based on the existing road space information stored in the database unit, the geometric information belonging to the road space information, and the location information, and the existing road space information and the road space information Road space further comprising an update unit for changing the map by updating the existing road space information with the road space information when there is a change in the object information as a result of comparing the first spatial information belonging to the image matching System for building information. In the method for constructing road space information for linkage between image information acquired from a plurality of image sensors and location information,
Receiving image information acquired from a plurality of image sensors mounted on a mobile platform and position information acquired from a positioning sensor, combining the image information and synthesizing the image information;
Generating object information by detecting an object from the synthesized image information;
Combining the object information and the location information with the image information based on geometric information defining a geometric relationship between the image information according to the position and posture information of the image information estimated from the position and posture of the plurality of image sensors Analyzing the relationship;
Generating first spatial information by combining the object information and the location information with the synthesized image information;
Generating second spatial information constituting a network related to direction information and connection information between the photographing nodes based on image information photographed continuously moving between the photographing nodes of the plurality of image sensors;
Generating and storing road space information by linking the first and second space information; And
Generating and storing a map composed of a predetermined coordinate model based on the road space information,
The posture information of the image sensor includes a difference between location information between the photographing nodes photographed by the image sensor, a photographing direction of the image sensor photographing a specific object and a first angle between the specific object, and adjacent to the photographed image sensor. Calculate a second angle between image sensors and a third angle between the photographing direction and the north direction, calculate movement trajectory information of the mobile platform based on the difference, and calculate the movement trajectory information, the first to third angles. Is estimated on the basis of,
The posture information of the image information is estimated from the posture information of the image sensor,
A method for constructing road space information in which the posture information of the image information is reflected in the geometric information.

Images