WO2020192646A1

WO2020192646A1 - Camera calibration method, roadside sensing device, and smart transportation system

Info

Publication number: WO2020192646A1
Application number: PCT/CN2020/080834
Authority: WO
Inventors: 刘永卿; 顾超捷
Original assignee: 阿里巴巴集团控股有限公司
Priority date: 2019-03-28
Filing date: 2020-03-24
Publication date: 2020-10-01
Also published as: TW202036478A; CN111754580A

Abstract

Disclosed is a camera calibration method, comprising the steps: obtaining at least one image taken by a camera, said image comprising an image of a target object, the target object having a display device, said display device being suitable for displaying geographic location information of the target object; extracting from the obtained image the image location information of the target object in the image and the geographic location information displayed by the display device; and calibrating the camera according to the image location information and the geographic location information extracted from at least one image, so as to determine, according to the calibration result, the geographic location information of each object within the range of photo-capture of the camera. Also disclosed by the present invention are a roadside sensing device and smart transportation system which use the described calibration method.

Description

Camera calibration method, roadside sensing equipment and intelligent transportation system

This application claims the priority of a Chinese patent application filed on March 28, 2019 with the application number 201910244149.7 and the title of the invention "A camera calibration method, roadside sensing device and smart transportation system", the entire content of which is incorporated by reference In this application.

Technical field

The present invention relates to the field of vehicle assisted driving, in particular to the technical field of calibrating the camera of a roadside sensing device in assisted driving.

Background technique

With the development of vehicle networking V2X technology, a collaborative environment perception system has emerged. This system can comprehensively use the data of the vehicle and the surrounding environment to assist the driving of the vehicle and monitor the driving condition of the vehicle. This system can also be called an intelligent transportation system.

In the smart transportation system, a large number of roadside sensing devices need to be deployed on the road. These devices have cameras and can collect video data through the cameras to calculate some traffic scenarios, such as vehicle breakdowns and rear-end collisions. Before further processing the video and image data collected by the camera, the camera needs to be calibrated to determine the actual position and distance of the vehicle and other objects captured.

As the number of roadside sensing devices is large and they are deployed near the road, a safe and fast way is needed to calibrate the camera of the roadside sensing device.

One of the current calibration methods is the manual calibration method. In this method, you need to manually place some reference points on the road, and then use a ruler or a handheld positioning device to measure the actual physical distance between the calibration object and the sensing device, and take photos and record at the same time Find the pixel coordinates of the corresponding point manually. This method needs to be carried out in the closed road section or set up obstacles in order to successfully implement the calibration work. It has a certain impact on traffic and roads, and the recorded data needs to be sorted out for calibration calculations. The period is long, the error is large, and the calibration efficiency is not ideal.

How to safely and quickly complete the calibration of the camera in the roadside sensing device is one of the urgent problems in this field.

For this reason, a new camera calibration solution is needed, which can perform safe, fast and accurate calibration for the camera of the roadside sensing device in the intelligent transportation system, so that the calibration results can be used to accurately calculate the accuracy of each object in the camera coverage. Geographical location and distance information.

Summary of the invention

For this reason, the present invention provides a new camera calibration solution to try to solve or at least alleviate at least one of the above problems.

According to one aspect of the present invention, there is provided a method for calibrating a camera, including the steps of: acquiring at least one image taken by the camera, the image including an image of a target object, the target object has a display device, and the display device is suitable for To display the current geographic location information of the target object; extract the image location information of the target object in the image and the geographic location information displayed by the display device from the acquired image; and extract the image location information and geographic location information extracted from at least one image The location information calibrates the camera, so that based on the calibration result, the geographic location information of each object within the shooting range of the camera is determined according to the image taken by the camera.

Optionally, the calibration method according to the present invention further includes the steps of: acquiring geographic location information of the camera; and the step of calibrating the camera further includes: performing the calibration based on the image location information, the geographic location information of the target object, and the geographic location information of the camera. The camera is calibrated.

Optionally, in the calibration method according to the present invention, the image position information of the target object in the image includes the two-dimensional position information of the target object in the image, and the geographic position information of the target object includes the three-dimensional position information of the target object. The step of calibrating the camera includes establishing a mapping relationship between the two-dimensional position information and the three-dimensional position information of the target object.

Optionally, in the calibration method according to the present invention, the image location information of the target object in the image includes the pixel coordinate information of the target object in the image, and the geographic location information of the target object includes the world coordinate information of the target object and the geographic location of the camera. The position information includes the world coordinate information of the camera; and the step of calibrating the camera includes constructing a conversion matrix according to the pixel coordinate information and world coordinate information of the target object, and the world coordinate information of the camera, so that the conversion matrix is used to locate the camera. The pixel coordinates in the captured image are converted to world coordinates.

Optionally, in the calibration method according to the present invention, the step of constructing the conversion matrix includes: taking the world coordinates of the camera as the coordinate origin, processing the world coordinate information of the target object; and using the processed world coordinates of the target object Information and pixel coordinate information to construct the conversion matrix.

Optionally, in the calibration method according to the present invention, the step of constructing the conversion matrix includes: setting the height coordinate value in the world coordinate information of the target object to a fixed value.

Optionally, in the calibration method according to the present invention, the display device is adapted to display the current geographic location information of the target object in a graphical coding manner, and the step of extracting the current geographic location information of the target object includes identifying the image code in the image , And decode the recognized image code to obtain the current geographic location information of the target object.

Optionally, in the calibration method according to the present invention, the graphic code includes at least one or more of a two-dimensional code and a barcode.

Optionally, in the calibration method according to the present invention, the step of acquiring at least one image taken by the camera includes acquiring at least four images, wherein the at least four images include images of the target object located at four edge positions, respectively.

Optionally, in the calibration method according to the present invention, the target object has a positioning device suitable for providing geographic location information. The positioning device is arranged close to the display device and provides geographic location information for the display device as the current geographic location of the target object. Location information; and the step of extracting the image location information of the target object in the image from the acquired image includes: extracting the image location information of the display device in the image as the image location information of the target object in the image.

Optionally, the calibration method according to the present invention further includes the steps of: after the camera is calibrated, at least one image taken by the camera is newly acquired; the image position information of the target object in the image and the display device are extracted from the newly acquired image Display geographic location information; calculate the geographic location information of the target object based on the calibration result and based on the image location information of the target object in the image; and calibrate based on the difference between the calculated geographic location information and the acquired geographic location information The results are verified.

Optionally, in the calibration method according to the present invention, the target object is a vehicle, the camera is included in the roadside sensing device, the vehicle is driving on the road covered by the roadside sensing device, and the geographic location information of the camera includes roadside sensing The geographic location information of the device.

According to another aspect of the present invention, a roadside sensing device is provided, which is deployed at a road position. The roadside sensing device includes a camera, which is suitable for photographing various objects on the road; a positioning device, which is suitable for providing geographic location information of the roadside sensing device; and a computing unit, which is suitable for performing the method according to the present invention to control the camera Perform calibration.

According to another aspect of the present invention, a smart transportation system is provided, which includes the roadside sensing device according to the present invention, which is deployed at a road position; and a calibration vehicle, suitable for driving on the road. The calibration vehicle has a display device and is suitable for displaying the current geographic location information of the calibration vehicle.

Optionally, in the system according to the present invention, the calibration vehicle includes: a positioning device, which is arranged close to the display device, and is adapted to provide geographic location information of the calibration vehicle; and a computing unit, which is adapted to provide information provided by the positioning device The address location information is coded into image codes for display on the display device.

Optionally, in the system according to the present invention, the calibration vehicle further includes: a communication unit adapted to communicate with the roadside sensing device, and when receiving the calibration signal, instruct the computing unit to obtain current geographic location information from the positioning device , And encode as image encoding for display on the display device.

Optionally, the system according to the present invention further includes: a normal vehicle suitable for driving on a road, wherein the camera of the roadside sensing device captures an image containing the normal vehicle and is based on the calibration of the calculation unit in the roadside sensing device As a result, the geographic location information of the normal vehicle is determined based on the image location information of the normal vehicle in the image.

According to another aspect of the present invention, a computing device is also provided. The computing device includes at least one processor and a memory storing program instructions, wherein the program instructions are configured to be executed by the at least one processor and include instructions for executing the aforementioned auxiliary parking method.

According to yet another aspect of the present invention, there is also provided a readable storage medium storing program instructions, which when the program instructions are read and executed by a computing device, cause the computing device to execute the aforementioned auxiliary parking method.

According to the camera calibration solution of the present invention, a display device is deployed on the vehicle, and the display device displays the current geographic location information of the vehicle. In this way, in the image taken by the camera, the pixel coordinates of the vehicle in the image and the world coordinates based on the geographic location information can be obtained at the same time, so that the calibration result can be accurately calculated, regardless of the synchronization between the two. problem.

In addition, according to the camera calibration solution of the present invention, as long as the vehicle with the display device is driven on the road, the roadside sensing device can complete the calibration, so that the camera can be calibrated multiple times and the calibration result can be detected in a simple manner. , Which significantly improves the calibration efficiency.

Description of the drawings

In order to achieve the above and related purposes, this article describes certain illustrative aspects in conjunction with the following description and drawings. These aspects indicate various ways in which the principles disclosed herein can be practiced, and all aspects and their equivalents are intended to be Into the scope of the claimed subject matter. By reading the following detailed description in conjunction with the accompanying drawings, the above and other objectives, features and advantages of the present disclosure will become more apparent. Throughout this disclosure, the same reference numerals generally refer to the same parts or elements.

Fig. 1 shows a schematic diagram of a smart transportation system according to an embodiment of the present invention;

Figure 2 shows a schematic diagram of a roadside sensing device according to an embodiment of the present invention;

Fig. 3 shows a schematic diagram of a calibration vehicle according to an embodiment of the present invention;

Fig. 4 shows a schematic diagram of a camera calibration method according to an embodiment of the present invention; and

Fig. 5 shows a schematic diagram of a camera calibration method according to another embodiment of the present invention.

detailed description

Hereinafter, exemplary embodiments of the present disclosure will be described in more detail with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

Fig. 1 shows a schematic diagram of a smart transportation system 100 according to an embodiment of the present invention. As shown in FIG. 1, the smart transportation system 100 includes a vehicle 110 and a roadside sensing device 200. The vehicle 110 is traveling on the road 140. The road 140 includes a plurality of lanes 150. When the vehicle 110 is traveling on the road 140, it can switch between different lanes 150 according to the road conditions and driving goals.

The roadside sensing device 200 is deployed around the road and uses various sensors it has to collect various information within a predetermined range around the roadside sensing device 200, especially road data related to the road. As described below with reference to FIG. 2, the roadside sensing device 200 includes a camera 210. The camera 210 can shoot toward the road to shoot the road 140 within the coverage of the roadside sensing device 200 and various objects on the road 140. These objects include vehicles 110 driving on the road 140 and various roadside pedestrians 130 and the like. The position and shooting direction of the camera 210 are generally fixed after the roadside sensing device 200 is deployed on the road. According to an embodiment of the present invention, the roadside sensing device 200 may include a plurality of cameras 210 to take pictures in different directions.

The roadside sensing device 200 has a predetermined coverage area. According to the coverage and road conditions of each roadside sensing device 200, a sufficient number of roadside sensing devices 200 can be deployed on both sides of the road to achieve full coverage of the entire road. Of course, according to an embodiment, instead of realizing full coverage of the entire road, the roadside sensing device 200 can be deployed at the characteristic points (turns, intersections, bifurcations) of each road to obtain the characteristics of the road. The data is fine. The present invention is not limited to the specific number of roadside sensing devices 200 and the coverage of the road.

When deploying the roadside sensing device 200, first calculate the location of the sensing device 200 to be deployed according to the coverage area of a single roadside sensing device 200 and the condition of the road 140. The coverage area of the roadside sensing device 200 depends at least on the arrangement height of the sensing device 200 and the effective distance of the sensor in the sensing device 200 for sensing. The conditions of the road 140 include the length of the road, the number of lanes 150, the curvature and gradient of the road, and so on. Any method in the art can be used to calculate the deployment position of the sensing device 200.

After the deployment location is determined, the roadside sensing device 200 is deployed at the determined location. Since the data that the roadside sensing device 200 needs to sense includes motion data of a large number of objects, the clock of the roadside sensing device 200 should be synchronized, that is, the time of each sensing device 200 should be kept consistent with the time of the vehicle 110 and the cloud platform.

Subsequently, the location of each deployed roadside sensing device 200 is determined. Since the sensing device 200 needs to provide a driving assistance function for the vehicle 110 traveling at a high speed on the road 140, the position of the sensing device 200 must be highly accurate to serve as the absolute position of the sensing device. There may be many ways to calculate the high-precision absolute position of the sensing device 200. According to one embodiment, a global navigation satellite system (GNSS) can be used to determine a high-precision position.

A vehicle 110 entering the coverage area of a roadside sensing device 200 can communicate with the roadside sensing device 200. A typical communication method is the V2X communication method. Of course, mobile communication methods such as 5G, 4G, and 3G can be used to communicate with the roadside sensing device 200 through a mobile internet network provided by a mobile communication service provider. Considering that the speed of the vehicle is relatively fast and the time delay of communication is required to be as short as possible, the V2X communication mode is adopted in the general implementation of the present invention. However, any communication method that can meet the time delay requirement required by the present invention falls within the protection scope of the present invention.

The vehicle 110 may receive driving-related information related to the vehicle 110 and road data of the section of road in various ways. In one implementation, all vehicles 110 entering the coverage area of the roadside sensing device 200 can automatically receive these information and data. In another implementation, the vehicle 110 may send a request, and the roadside sensing device 200 sends driving-related information related to the vehicle 110 and road data of the road to the vehicle 110 in response to the request, so that the driver can base on these Information to control the driving behavior of the vehicle 110. The present invention is not limited to the specific manner in which the vehicle 110 receives driving-related information and road data of the section of road, and the manner in which all vehicles 110 can receive such information and data are within the protection scope of the present invention.

The vehicle 110 includes a vehicle normally running on the road 140 and a calibration vehicle 300 for calibrating the camera 210 in the roadside sensing device 200. Calibration refers to determining the relationship between the three-dimensional geometric position of a physical object and the corresponding point of the object in the image. By calibrating the camera 210, the relationship between the position of an object in the image captured by the camera and the actual geographic location of the object can be determined. For example, for the vehicle 110, after the camera 210 is calibrated, the actual geographic location of the vehicle 110 can be determined according to the position of the vehicle 110 in the image captured by the camera 210. In this way, in the event of a vehicle breakdown or a traffic accident such as a rear-end collision, the actual position and mutual distance of the vehicle objects can be determined.

As described below with reference to FIG. 3, the calibration vehicle 300 has a display device 310, and during driving on the road 140, the geographic location information of the calibration vehicle 300 is displayed on the display device 310 according to the calibration signal from the roadside sensing device 200. In order to allow the camera 210 to capture the content displayed on the display device 310, the display device 310 can be installed on the top front end of the calibration vehicle 300, and the displayed content is directed toward the camera, so that the calibration vehicle 300 enters the coverage of the roadside sensing device 200 When in the range, the content displayed on the display device 310 can be captured. Optionally, the display device 310 can also be installed at the top and rear of the calibration vehicle 300, so that when the calibration vehicle 300 leaves the coverage area of the roadside sensing device 200, the content displayed on the display device 310 can also be captured. The present invention is not limited to the installation position and quantity of the display device 310 on the calibration vehicle 300, and all the ways that the camera 210 can capture the content displayed on the display device 310 are within the protection scope of the present invention.

Fig. 2 shows a schematic diagram of a roadside sensing device 200 according to an embodiment of the present invention. As shown in FIG. 2, the roadside sensing device 200 includes a camera 210, a communication unit 220, a sensor group 230, a storage unit 240 and a calculation unit 250.

The roadside sensing device 200 communicates with each vehicle 110 entering its coverage area to provide driving-related information for the vehicle 110, send a calibration signal to the calibration vehicle 300, and receive vehicle driving information of the vehicle from the vehicle 110. At the same time, the roadside sensing device 200 also needs to communicate with the server and other surrounding roadside sensing devices 200. The communication unit 220 provides a communication function for the roadside sensing device 200. The communication unit 220 may adopt various communication methods, including but not limited to Ethernet, V2X, 5G, 4G and 3G mobile communication, etc., as long as these communication methods can complete data communication with the smallest possible time delay.

The camera 210 can photograph the road 140 within the coverage area of the roadside sensing device 200 and various objects on the road 140. These objects include vehicles 110 traveling on the road 140 and various roadside pedestrians. The position and shooting direction of the camera 210 are generally fixed after the roadside sensing device 200 is deployed on the road. According to an embodiment of the present invention, the roadside sensing device 200 may include a plurality of cameras 210 to take pictures in different directions. The shooting range of the camera 210 covers the entire road 140, and generates a video stream and stores the video stream in the storage unit 240 for subsequent processing. For example, when the camera 210 is calibrated, the video stream between the time when the roadside sensing device 200 issues the start calibration command and the time when the calibration ends can be specially stored for subsequent camera calibration processing described with reference to FIGS. 4 and 5 .

The sensor group 230 includes various sensors other than cameras, for example, radar sensors such as millimeter wave radar 232 and lidar 234, and image sensors such as infrared probe 236. For the same object, various sensors can obtain different attributes of the object. For example, a radar sensor can measure the speed and acceleration of the object, while an image sensor can obtain the shape and relative angle of the object.

The sensor group 230 uses each sensor to analyze the static conditions of the road in the coverage area (lane lines, guardrails, isolation belts, roadside parking spaces, road slope and inclination, road water and snow, etc.) and dynamic conditions (driving vehicles 110, Pedestrians and throwing objects) collect and sense, and store the data collected and sensed by each sensor in the storage unit 240.

The calculation unit 250 may fuse the data sensed by the sensors to form road data of the section of road, and also store the road data in the storage unit 240. In addition, the calculation unit 250 may also continue to perform data analysis on the basis of the road data, identify one or more of the vehicles and vehicle motion information, and further determine driving-related information for the vehicle 110. These data and information can all be stored in the storage unit 240.

The present invention is not limited to a specific way of fusing data from various sensors to form road data. As long as the road data contains static and dynamic information of various objects within a predetermined range of the road position, this method falls within the protection scope of the present invention.

The calculation unit 250 may also process the image of the calibration vehicle 300 captured by the camera 210, for example, process the calibration video stored in the storage unit 240 to implement the camera calibration method described below with reference to FIGS. 4 and 5.

Optionally, the roadside sensing device 200 further includes a positioning device 260. The positioning device 260 provides geographic location information of the roadside sensing device 200. Since the sensing device 200 needs to provide a driving assistance function for the vehicle 110 traveling at a high speed on the road 140, the position of the sensing device 200 must be highly accurate. The positioning device 260 can be implemented in multiple ways. According to one embodiment, the positioning device 260 may be implemented as a high-precision GPS device that uses a global navigation satellite system (GNSS) to determine a high-precision position. According to an embodiment of the present invention, considering that the camera 210 and the roadside sensing device 200 are very close, the geographic location information of the camera 210 may be set as the geographic location information provided by the positioning device 260. The present invention is not limited to this, and a positioning device can also be embedded in the camera 210 to provide geographic location information of the camera 210.

FIG. 3 shows a schematic diagram of a calibration vehicle 300 according to an embodiment of the present invention. As shown in FIG. 3, the calibration vehicle 300 includes a display device 310, a communication unit 320, a calculation unit 330 and a positioning device 340.

As described above with reference to Figures 1 and 2, the calibration vehicle 300, like other vehicles, can communicate with the roadside sensing device 200 when driving within the road covered by the roadside sensing device 200, so as to sense the device from the roadside 200 obtains auxiliary information about roads and driving, such as roadblocks ahead and vehicle avoidance, and can also send information such as the running status of the vehicle 300 to the roadside sensing device 200 to facilitate the roadside sensing device 200 to process road data. In addition, the calibration vehicle 300 may also receive instructions for starting and ending calibration from the roadside sensing device 200, so as to allow the calibration vehicle 300 to enter the calibration state and exit the calibration state.

The communication unit 320 provides a communication function for the calibration vehicle 300. The communication unit 320 may adopt various communication methods, including but not limited to Ethernet, V2X, 5G, 4G and 3G mobile communication, etc., as long as these communication methods can complete data communication with the smallest possible time delay.

The display device 310 is arranged on the calibration vehicle 300 so that when the calibration vehicle 300 enters the coverage area of the roadside sensing device 200, the content displayed on the display device 310 can be captured by the camera 210 for subsequent processing. According to an embodiment, the display device 310 may be installed on the top front end of the calibration vehicle 300 and direct the displayed content toward the camera. According to another embodiment, the display device 310 may also be installed on the top and rear ends of the calibration vehicle 300. In this way, when the calibration vehicle 300 enters or leaves the coverage area of the roadside sensing device 200, the content displayed on the display device 310 can be captured. The present invention is not limited to the installation position and quantity of the display device 310 on the calibration vehicle 300, and all the ways that the camera 210 can capture the content displayed on the display device 310 are within the protection scope of the present invention.

The display device 310 displays the current geographic location information of the calibration vehicle 300, so that when the camera 210 captures the content displayed on the display device 310, the current geographic location information of the calibration vehicle 300 can be obtained from the image. The display device 310 can display geographic location information in multiple ways.

According to an embodiment, the display device 310 can display geographic location information in a digital or text manner, and the sensing unit 200 can obtain digital content through text recognition. For example, the display device 310 may display multiple rows of numbers, and each row of numbers corresponds to a coordinate value in one direction. The sensing unit 200 performs word recognition line by line to obtain the coordinate value of the geographic location in each direction.

According to another embodiment, the display device 310 may display geographic location information in a graphical manner. That is, the geographic location information can be coded graphically, and the coded graphics can be displayed by the display device 310. After the camera 210 captures an image containing the graphic, the roadside sensing device 200 decodes the graphic to extract the geographic location information encoded therein. The advantage of using the graphic coding method is that, generally speaking, graphic coding has a certain error tolerance rate. Even if the camera 210 cannot capture a very clear coded graphic, the geographic location information can be correctly decoded from the captured graphic.

Graphic coding methods can include two-dimensional codes, barcodes, and so on. There are many ways of encoding data into two-dimensional codes and barcodes and decoding them in this field, which will not be repeated here. The present invention is not limited to the specific form of graphics encoding, and all the ways in which information can be encoded into graphics and the graphics can be decoded to obtain information are within the protection scope of the present invention.

Similar to the positioning device 260 in the roadside sensing device 200, the calibration vehicle 300 also includes the positioning device 340. The positioning device 340 provides geographic location information of the calibration vehicle 300. According to the requirements of calibration accuracy, the position of the positioning device 340 must be highly accurate. The positioning device 340 can be implemented in multiple ways. According to one embodiment, the positioning device 340 may be implemented as a high-precision GPS device that uses a global navigation satellite system (GNSS) to determine a high-precision position.

The volume of the calibration vehicle 300 is generally much larger than the size of the positioning device 340. Therefore, in the image captured by the camera 210, the calibration vehicle 300 generally occupies an area in the image. The display device 310 displays the geographic location information of the positioning device 340 provided by the positioning device 340. When the pixel coordinates of a point in the captured image area occupied by the calibration vehicle 300 are selected to correspond with the geographic location coordinates, errors will occur. To this end, according to an embodiment, the positioning device 340 can be arranged close to the display device 310, and the pixel coordinates of the display device 310 in the image taken by the camera 210 are selected as the pixel coordinates of the calibration vehicle 300, thereby reducing the Error. According to a further embodiment, the positioning device 340 is arranged close to the center of the display area of the display device 310, and the pixel coordinates of the center point of the image area in the image taken by the camera 210 of the display content of the display device 310 are selected as the calibration vehicle 300 The pixel coordinates can further improve the degree of matching between geographic location information and image location information.

The calibration vehicle 300 also includes a calculation unit 330. The calculation unit 330 controls the calibration operation of the calibration vehicle 300. According to an embodiment of the present invention, when the calibration vehicle 300 enters the road area covered by the roadside sensing device 200, the calculation unit 330 receives the start calibration instruction from the roadside sensing device 200 via the communication unit 320, and then The positioning device 340 obtains the current geographic location information, encodes the geographic location information (graphic coding such as a QR code, or directly organizes the content in text mode), and sends the encoded content to the display device 310 for It is displayed by the display device 310 in its display area. The camera 210 of the roadside sensing device 200 photographs the calibration vehicle 300 with display content, and when the calibration vehicle 300 leaves the shooting range of the camera 210, an end calibration instruction can be sent to the calibration vehicle 300 so that the computing unit 300 can stop the display device. The content corresponding to the current geographic location information is displayed in 310.

The calibration vehicle 300 can pass lane by lane or back and forth through the road area covered by the sensing device 200, so that the camera 210 can capture multiple calibration videos and store them in the storage unit 240, so that the computing unit 250 can execute the camera described below with reference to FIGS. 4 and 5 The calibration method is used to calibrate the camera 210.

FIG. 4 shows a schematic diagram of a camera calibration method 400 according to an embodiment of the present invention. The camera calibration method 400 is suitable to be executed in the roadside sensing device 200 shown in FIG. 2, particularly in the calculation unit 250 of the roadside sensing device 200, so as to calibrate the camera 210.

As shown in FIG. 4, the method 400 starts at step S410. In step S410, at least one image taken by the camera 210 is acquired. The image frame may be cut from the calibration video stored in the storage unit 240 as the image to be acquired in step S410. The image should include the image of the calibration vehicle 300, and the current geographic location information of the calibration vehicle 300 should be displayed in the display device 310 of the calibration vehicle 300.

Optionally, in order to make the calibration result more accurate, multiple images may be acquired to perform calibration based on the content of the multiple images. According to an embodiment, at least four image frames or at least 6-8 image frames of the calibration vehicle 300 respectively located in the four corners of the camera shooting range can be intercepted from the calibration video. Optionally, more images can be acquired, and the calibration vehicle 300 can be more evenly distributed in each position of the image in these images. The present invention is not limited to the specific number of images to be acquired and the specific position of the calibration vehicle 300 in the image, as long as sufficient geographic location and image position information can be obtained according to the number of images and the position of the calibration vehicle 300 in the image, so as to When the camera 210 is calibrated, the number of such images and the position of the calibrated vehicle 300 are both within the protection scope of the present invention.

Subsequently, in step S420, for each image acquired in step S410, the image location information of the calibration vehicle 300 in the image and the geographic location information displayed by the display device 310 are extracted. There are many ways to identify the calibration vehicle 300, the display device 310, and the content displayed by the display device in the image captured by the camera 210. According to an embodiment, various deep learning methods including convolutional neural network methods can be used for image recognition. Other image processing methods that extract image objects based on image features can also be used. The present invention is not limited to the specific method of image recognition, and all the methods that can identify the calibration vehicle 300, the display device 310 and the displayed content in the image are within the protection scope of the present invention.

The identified object such as the calibration vehicle 300 occupies an image area in the image. According to an embodiment of the present invention, the image position information of the object is the two-dimensional position information of the object on the plane where the image is located. According to an embodiment of the present invention, the pixel coordinate information of the image area of the calibration vehicle 300 can be obtained as the image position information of the object. Further, the origin of the pixel coordinates of the image can be set to the upper left corner of the image, and the two-dimensional pixel value of the image region relative to the origin can be used as the pixel coordinate information of the image region of the calibration vehicle 300. According to a further embodiment, when the positioning device 340 in the calibration vehicle 300 is arranged close to the center of the display area, the pixel coordinate information of the display device 310 can be obtained, and further, the image area corresponding to the content displayed by the display device 310 can be obtained. The pixel coordinate information of the center point is used as the pixel coordinate information of the calibration vehicle 300. At this time, this position is closest to the physical address of the positioning device 340. Of course, when the positioning device 340 is arranged in another location, the pixel coordinate information of the position closest to the positioning device 340 in the calibration vehicle 300 can be selected as the pixel coordinate information of the calibration vehicle 300.

In step S420, after the display content of the display device 310 is recognized in the image captured by the camera 210, content recognition is performed on the display content to obtain the geographic location information of the calibration vehicle 300. As described above with reference to FIGS. 1 and 3, the display content can display geographic location information in multiple ways. Therefore, in step S420, it is also necessary to adopt multiple corresponding methods to obtain geographic location information from the displayed content. According to an embodiment, when the display content displays geographic location information in, for example, text, text recognition (OCR) is performed on the displayed content to obtain geographic location information. According to another embodiment, when the display content presents the geographic location information of the graphic code, the graphic in the display content is decoded in a decoding manner corresponding to the graphic code to obtain the geographic location information.

According to one embodiment, the geographic location information is three-dimensional location information, and a unique location point is determined for each location. Therefore, according to one embodiment, the geographic location information is world coordinate information, and the world coordinate defines a coordinate system with a unique coordinate value for each location in the world. Examples of these include but are not limited to the Global Positioning System (GPS), Beidou system and The world coordinate value defined by the Galileo system. The present invention is not limited to a specific coordinate system, and all coordinate systems that can provide unique world coordinate information for a calibration vehicle are within the protection scope of the present invention.

After determining the image location information and geographic location information of the calibrated vehicle 300 in step S420 for each selected image, in step S430, the camera 210 is calibrated according to the extracted image location information and geographic location information so as to be based on the calibration As a result, the geographic location information of each object in the image taken by the camera is determined.

According to an embodiment of the present invention, the calibration process is to find the mapping relationship between image location information and geographic location information. When the image location information is two-dimensional information and geographic location information is three-dimensional information, the purpose of the calibration process is In order to find the mapping relationship between two-dimensional and three-dimensional information.

In order to facilitate calibration, the geographic location information of the camera 210 may also be used in the calibration process. To this end, the method 400 further includes a step of obtaining the geographic location information of the camera 210 in advance. The camera 210 uses the same world coordinate system as the calibration vehicle 300 to characterize its position. As described above with reference to FIGS. 1 and 2, the positioning device 260 of the roadside sensing device 200 can be used to obtain the geographic location of the camera 210.

As described above in step S420, according to an embodiment, the image position information and geographic location information of the calibration vehicle 300 in each image constitute a pair of pixel coordinate information and world coordinate information. In step S430, a conversion matrix is constructed according to a plurality of pixel coordinate information and world coordinate information pairs, and the world coordinate pairs of the camera 210. The conversion matrix can convert pixel coordinates into world coordinates with minimal errors.

There are many ways to construct the conversion matrix. Constructing the conversion matrix becomes searching for a conversion matrix, which represents the mapping relationship from the plane formed by the calibration vehicle 300 on the road to the plane corresponding to the image taken by the camera 210. According to one embodiment, the transformation matrix T is a 3×3 matrix, and the pixel coordinates A and world coordinates W of the calibration vehicle 300 and the geographic coordinates B of the camera 210 are a 1×3 array. Constructing the transformation matrix T becomes, in the case of known multiple groups (A, W, B), to find such a matrix T, which satisfies:

A×T+B=W

And minimize the overall error. There may be multiple calculation methods to calculate such T. The present invention is not limited to specific calculation methods, and all the ways in which the conversion matrix T can be calculated are within the protection scope of the present invention.

According to an embodiment, the origin of the geographic location information can be set at the geographic location where the camera 210 is located, that is, the location of the camera 210 is set as the origin of world coordinates. In this way, the world coordinate value of the calibration vehicle 300 can be processed according to the relative position of the calibration vehicle 300 and the camera 210, that is, the processing of W'=W-B is performed first, and then according to the conditions:

A×T=W’

To construct the conversion matrix T, which can simplify the process of solving the conversion matrix T.

According to another embodiment, considering that the road segments within the coverage of the roadside sensing device 200 are basically roads in the same plane, the height value in the world coordinate value W of the calibration vehicle 300 can be set to a fixed The value, for example, is fixed to 0, so that the process of solving the conversion matrix T can be further simplified.

After the conversion matrix T is obtained in step S430 and the calibration of the camera 210 is completed, the calibration result, for example, the conversion matrix T, can be used for subsequent processing. For example, when the camera 210 captures another vehicle 110 that is driving normally, the geographic location of the vehicle 110 can be determined according to the pixel coordinate position of the vehicle 110 in the captured image, so that the vehicle 110 can be determined when an accident occurs. The physical location of the 110 accident site. According to another implementation manner, for example, the relative physical distance between the vehicle and other vehicles or obstacles can be accurately determined to remind the vehicle 110 to assist the driving of the vehicle 110.

According to the calibration method described in FIG. 4, the calibration of the camera 210 in the roadside sensing device 200 can be completed by passing the calibration vehicle 300 through the area covered by the roadside sensing device 200 several times. This method is very simple and will not affect the driving of other vehicles on the road. For this reason, when the shooting angle of the camera 210 changes due to various reasons (for example, when a typhoon or the roadside sensing device 200 is hit by a vehicle and rearranged), the camera 210 can be easily recalibrated.

The camera calibration method 400 described in FIG. 4 is described by taking the specific form of the calibration vehicle 300 running on the road as an example. It should be noted that the present invention is not limited to the specific form of the calibration vehicle 300, and all have display devices. All objects that display their geographic location information can be used to calibrate the camera without exceeding the protection scope of the present invention.

FIG. 5 shows a schematic diagram of a camera calibration method 500 according to another embodiment of the present invention. The camera calibration method 500 shown in FIG. 5 is a further development of the camera calibration method 400 shown in FIG. 4, so the same or similar steps as those in the method 400 shown in FIG. 4 can be represented by the same or similar reference numerals, and represent Basically the same or similar treatments will not be repeated here.

As shown in FIG. 5, after the camera 210 is calibrated and the calibration result is obtained in step S430 (for example, according to an embodiment of the present invention, a conversion matrix is obtained). In steps S510-S540, the calibration result is verified. Specifically, in step S510, an image including the calibration vehicle 300 is acquired through the camera 210. As described above in step S410, in the image, the display device 310 of the calibration vehicle 300 displays the current position information of the calibration vehicle.

Subsequently, in step S520, the image location information of the calibration vehicle 300 in the image and the geographic location information displayed by the display device 310 are extracted from the image obtained in step S510. In step S520, the same method as in step S520 can be adopted to obtain the image location information and geographic location information of the calibration vehicle 300. For example, the pixel coordinate information at the center position of the display content can be obtained as the image position information of the calibration vehicle, and the geographic position information displayed in the graphical encoding manner can be decoded to obtain the corresponding world coordinate information.

In step S530, the geographic location information of the calibration vehicle 300 is calculated for the image location information of the calibration vehicle 300 obtained in step S520 according to the calibration result obtained in step S430. According to one embodiment, the calibration result is a conversion matrix, and the image position information of the calibration vehicle 300 obtained in step S530 is the pixel coordinate information of the calibration vehicle 300 in the image. The conversion matrix and pixel coordinates can be used to calculate the world coordinate information. Then, in step S540, the world coordinate information calculated in step S530 is compared with the world coordinate information obtained in step S520 to determine the difference between the two. There are various ways to calculate the difference between two coordinate values, for example, the distance between two coordinate points or the mean square value of several coordinate points can be calculated. If it is determined in step S540 that the difference between the geographic location information calculated in step S530 and the geographic location information obtained in step S520 exceeds the predetermined threshold, it can be considered that there is a problem with the previous calibration result, and you can return to step S410 to restart the calibration . It is also possible to perform the processing in steps S510 to S540 after a certain period of time after the calibration result is obtained, so as to restart the calibration when it is determined in step S540 that the difference exceeds a predetermined threshold.

According to the calibration scheme of the present invention, the current location information of the calibration vehicle can be displayed on the calibration vehicle. In this way, when the camera shoots an image containing the calibration vehicle, the image location information and geographic location information of the calibration vehicle can be simultaneously extracted from the image. Position information, and the calibration result can be calculated based on these two kinds of information. In this way, geographic location information and image location information at the same time can be obtained, which reduces the problem of inconsistency between the two caused by the delay of communication transmission, and improves the accuracy of calibration.

In addition, according to the calibration solution of the present invention, a calibration vehicle with a display device and a positioning device can be used to pass through the road covered by the camera of the roadside sensing device at a normal speed to complete the calibration. This is necessary for re-calibration from time to time, or compare For busy roads, traffic problems caused by calibration are significantly reduced.

It should be understood that, in order to simplify the present disclosure and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or In its description. However, the disclosed method should not be interpreted as reflecting the intention that the claimed invention requires more features than those explicitly stated in each claim. More precisely, as reflected in the following claims, the inventive aspect lies in less than all the features of a single embodiment disclosed previously. Therefore, the claims following the specific embodiment are thus explicitly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present invention.

Those skilled in the art should understand that the modules or units or components of the device in the example disclosed herein can be arranged in the device as described in this embodiment, or alternatively can be positioned differently from the device in this example In one or more devices. The modules in the foregoing examples can be combined into one module or further divided into multiple sub-modules.

Those skilled in the art can understand that it is possible to adaptively change the modules in the device in the embodiment and set them in one or more devices different from the embodiment. The modules or units or components in the embodiments can be combined into one module or unit or component, and in addition, they can be divided into multiple sub-modules or sub-units or sub-components. Except that at least some of such features and/or processes or units are mutually exclusive, any combination can be used to compare all features disclosed in this specification (including the accompanying claims, abstract and drawings) and any method or methods disclosed in this manner or All the processes or units of the equipment are combined. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

In addition, those skilled in the art can understand that although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments means that they are within the scope of the present invention. Within and form different embodiments. For example, in the following claims, any one of the claimed embodiments can be used in any combination.

In addition, some of the embodiments are described herein as methods or combinations of method elements that can be implemented by a processor of a computer system or by other devices that perform the described functions. Therefore, a processor with the necessary instructions for implementing the method or method element forms a device for implementing the method or method element. In addition, the elements described herein of the device embodiments are examples of devices for implementing functions performed by the elements for the purpose of implementing the invention.

As used herein, unless otherwise specified, the use of ordinal numbers "first", "second", "third", etc. to describe ordinary objects merely refers to different instances of similar objects, and is not intended to imply such The described objects must have a given order in terms of time, space, order, or in any other way.

Although the present invention has been described in terms of a limited number of embodiments, benefiting from the above description, those skilled in the art understand that other embodiments can be envisaged within the scope of the invention thus described. In addition, it should be noted that the language used in this specification is mainly selected for the purpose of readability and teaching, not for explaining or limiting the subject of the present invention. Therefore, without departing from the scope and spirit of the appended claims, many modifications and alterations are obvious to those of ordinary skill in the art. For the scope of the present invention, the disclosure of the present invention is illustrative rather than restrictive, and the scope of the present invention is defined by the appended claims.

Claims

A method for calibrating a camera includes the steps:

Acquiring at least one image taken by the camera, the image including an image of a target object, the target object has a display device, and the display device is adapted to display geographic location information of the target object;

Extracting the image location information of the target object in the image and the geographic location information displayed by the display device from the acquired image; and

The camera is calibrated according to the image location information and geographic location information extracted from the at least one image, so that based on the calibration result, the camera is determined to be within the shooting range of the camera according to the image taken by the camera. The geographic location information of the object.
The method according to claim 1, wherein the display device is adapted to display the current geographic location information of the target object in a graphical coding manner, and

The step of extracting the current geographic location information of the target object includes identifying an image code in the image, and decoding the identified image code to obtain the current geographic location information of the target object.
The method according to claim 2, wherein the graphic code includes at least one or more of a two-dimensional code and a barcode.
The method according to any one of claims 1 to 3, wherein the step of obtaining at least one image taken by the camera comprises: obtaining at least four images, wherein the at least four images include the target objects, respectively Images located at the four edge positions.
The method according to any one of claims 1 to 4, wherein the target object has a positioning device suitable for providing geographic location information, and the positioning device is arranged close to the display device and provides the display device with the Geographic location information as the current geographic location information of the target object; and

The step of extracting the image position information of the target object in the image from the acquired image includes: extracting the image position information of the display device in the image as the target object in the image Image location information in.
The method according to any one of claims 1-5, further comprising the steps:

Acquiring geographic location information of the camera; and

The step of calibrating the camera further includes: calibrating the camera according to the image location information, the geographic location information of the target object, and the geographic location information of the camera.
The method according to claim 6, wherein the image position information of the target object in the image includes the two-dimensional position information of the target object in the image, and the geographic position information of the target object includes the 3D position information of the target object,

The step of calibrating the camera includes establishing a mapping relationship between the two-dimensional position information and the three-dimensional position information of the target object.
The method according to claim 7, wherein the image position information of the target object in the image includes the pixel coordinate information of the target object in the image, and the geographic location information of the target object includes the target object World coordinate information of the camera, where the geographic location information of the camera includes the world coordinate information of the camera; and

The step of calibrating the camera includes constructing a conversion matrix according to the pixel coordinate information of the target object, the world coordinate information of the target object, and the world coordinate information of the camera, so that the conversion matrix is used to convert the The pixel coordinates in the image captured by the camera are converted to world coordinates.
The method according to claim 8, wherein the step of constructing a conversion matrix comprises:

Processing the world coordinate information of the target object with the world coordinates of the camera as the coordinate origin; and

The conversion matrix is constructed using the processed world coordinate information of the target object and the pixel coordinate information of the target object.
The method according to claim 8 or 9, wherein the step of constructing a conversion matrix comprises:

The height coordinate value in the world coordinate information of the target object is set to a fixed value.
The method according to any one of claims 1-10, after calibrating the camera, further comprising:

Newly acquiring at least one image taken by the camera;

Extracting the image location information of the target object in the image and the geographic location information displayed by the display device from the newly acquired image;

Based on the calibration result, so as to calculate the geographic location information of the target object according to the image location information of the target object in the image; and

The calibration result is verified according to the difference between the calculated geographic location information and the acquired geographic location information.
The method according to any one of claims 1-11, wherein the target object is a vehicle, the camera is included in a roadside sensing device, and the vehicle is driving on a road covered by the roadside sensing device, And the geographic location information of the camera includes geographic location information of the roadside sensing device.
A roadside sensing device, deployed at a road location, including:

A camera, suitable for shooting various objects on the road;

A positioning device adapted to provide geographic location information of the roadside sensing device; and

The calculation unit is adapted to execute the method according to any one of claims 1-12, so as to calibrate the camera.
The roadside sensing device according to claim 13, wherein the calculation unit is further adapted to determine the geographic location information of each object photographed by the camera according to the calibration result.
A smart transportation system, including:

The roadside sensing device according to claim 13 or 14, deployed at a road position; and

The calibration vehicle is adapted to travel on the road, the calibration vehicle has a display device, and the calibration vehicle is adapted to display current geographic location information of the calibration vehicle.
The system of claim 15, wherein the calibration vehicle comprises:

A positioning device arranged close to the display device and suitable for providing geographic location information of the calibration vehicle; and

The calculation unit is adapted to encode the address location information provided by the positioning device into an image code for display on the display device.
The system of claim 16, wherein the calibration vehicle further comprises:

The communication unit is adapted to communicate with the roadside sensing device, and when a calibration signal is received, instruct the calculation unit to obtain the current geographic location information from the positioning device, and encode it as an image code for the display device上display.
The system according to any one of claims 15-17, further comprising:

A normal vehicle, suitable for driving on the road,

The camera of the roadside sensing device captures an image containing the normal vehicle, and based on the calibration result of the calculation unit in the roadside sensing device, the image location information of the normal vehicle in the image is obtained Determine the geographic location information of the normal vehicle.
A computing device including:

At least one processor; and

A memory storing program instructions, wherein the program instructions are configured to be executed by the at least one processor, and the program instructions include instructions for executing the method according to any one of claims 1-12 .