WO2018177159A1 - 运动物体的位置确定方法及系统 - Google Patents

运动物体的位置确定方法及系统 Download PDF

Info

Publication number
WO2018177159A1
WO2018177159A1 PCT/CN2018/079596 CN2018079596W WO2018177159A1 WO 2018177159 A1 WO2018177159 A1 WO 2018177159A1 CN 2018079596 W CN2018079596 W CN 2018079596W WO 2018177159 A1 WO2018177159 A1 WO 2018177159A1
Authority
WO
WIPO (PCT)
Prior art keywords
moving object
unit time
virtual
relative distance
look
Prior art date
Application number
PCT/CN2018/079596
Other languages
English (en)
French (fr)
Inventor
周煜远
何彬
赵来刚
Original Assignee
上海蔚来汽车有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 上海蔚来汽车有限公司 filed Critical 上海蔚来汽车有限公司
Publication of WO2018177159A1 publication Critical patent/WO2018177159A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
    • G06V20/586—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads of parking space
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30248—Vehicle exterior or interior
    • G06T2207/30252—Vehicle exterior; Vicinity of vehicle
    • G06T2207/30261—Obstacle
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30248—Vehicle exterior or interior
    • G06T2207/30252—Vehicle exterior; Vicinity of vehicle
    • G06T2207/30264—Parking

Definitions

  • the present invention relates to a positioning ranging technique, and more particularly to a method and system for determining a position of a moving object.
  • the rest of the automatic parking products are mainly based on traditional machine vision to locate the parking space.
  • the parking space recognition rate does not exceed 40%, and obstacles in the parking spaces.
  • the object is not treated and is prone to cause a reverse accident, so it is rarely used.
  • the present invention has been made to overcome the above disadvantages, and the technical solution adopted is as follows.
  • a method for determining a position of a moving object comprising: step a, acquiring a look-around image around the moving object for each unit time; and step b, determining a virtual target by using a previously trained recognition model based on the look-around image a region; step c, for the virtual target region, determining imaging position information of each virtual key point by using the recognition model; step d, based on the imaging coordinate system of the look-around image and the body coordinate system of the moving object a mapping relationship between the actual target area corresponding to the virtual target area and the moving object according to the imaging position information of each virtual key point; and step e, for each unit time, use Correcting the current unit time for the difference between the relative distance obtained by the steps a to the step d and the distance moved by the moving object in the previous unit time for the previous unit time of the current unit time The relative distance obtained by the step a to the step d to obtain the current Relative distance corrected bit time.
  • the method further includes the step of training the deep learning model using the sample data to obtain the recognition model for the virtual target area.
  • the step b includes: step b1, adaptively selecting a plurality of candidate regions of different sizes in the look-around image; and step b2, using the identification a model to calculate a probability value of each of the candidate regions to become the virtual target region; and a step b3, at least one probability cluster formed based on respective probability values of the plurality of candidate regions having different sizes, from the plurality of One candidate region is selected as the virtual target region among the candidate regions.
  • the step d includes: step d1, calculating, according to the imaging position information of the respective virtual key points, the respective virtual key points in the look-around image Virtual three-dimensional angle information in the imaging coordinate system; step d2, based on the mapping relationship between the imaging coordinate system of the look-around image and the body coordinate system of the moving object, calculate and virtual key according to the virtual three-dimensional angle information Point corresponding to the actual three-dimensional angle information of each actual key point of the actual target area in the body coordinate system of the moving object; and step d3, calculating based on the actual three-dimensional angle information and the height of the moving object a relative distance between each of the actual key points and the moving object.
  • the difference between the distances regarding the previous unit time and the relative distance calculated for the current unit time are performed. Kalman filtering to obtain a corrected relative distance for the current unit time.
  • a position determining device system for a moving object includes: a first module that acquires a look-around image around the moving object for each unit time; and a second module that uses a previously trained recognition model based on the look-around image, Determining a virtual target area; a third module, for which the imaging position information of each virtual key point is determined by using the recognition model; a fourth module, an imaging coordinate system based on the look-around image and the motion a mapping relationship between the object coordinate systems of the objects, calculating, according to the imaging position information of the respective virtual key points, a relative distance between the actual target area corresponding to the virtual target area and the moving object; and a fifth module, For each unit time, the difference between the relative distance obtained by the first module to the fourth module for the previous unit time of the current unit time and the distance moved by the moving object in the previous unit time is used. Modifying the first module to the fourth module for the current unit time The relative distance, a relative distance to obtain a corrected for the current unit of time.
  • the method further includes: training the deep learning model using the sample data to obtain a module for the recognition model of the virtual target area.
  • the second module includes: adaptively selecting a plurality of units of different size candidate regions in the look-around image; calculating by using the recognition model Each of the candidate regions becomes a unit of probability values of the virtual target region; and at least one probability cluster formed based on respective probability values of the plurality of candidate regions having different sizes, and selecting from the plurality of candidate regions A candidate area is used as a unit of the virtual target area.
  • the fourth module includes: calculating imaging coordinates of the respective virtual key points in the look-around image according to imaging position information of the respective virtual key points a unit of virtual three-dimensional angle information; a mapping relationship between the imaging coordinate system of the look-around image and a body coordinate system of the moving object, and calculating, corresponding to each virtual key point, according to the virtual three-dimensional angle information Means for determining actual three-dimensional angle information of each actual key point of the actual target area in the body coordinate system of the moving object; and calculating the actual actuality based on the actual three-dimensional angle information and the height of the moving object A unit of relative distance between the key point and the moving object.
  • the fifth module performs Kalman on the difference between the distances regarding the previous unit time and the relative distance calculated for the current unit time Filtering to obtain a corrected relative distance for the current unit time.
  • a moving object position determining program for causing a computer to execute a position determining method of the moving object.
  • a moving object position determining program for causing a computer to implement the function of the position determining system of the moving object.
  • a computer readable recording medium on which a moving object position determining program for causing a computer to execute a position determining method of the moving object is recorded.
  • a computer readable recording medium on which a moving object position determining program for causing a computer to implement the function of a position determining system of the moving object is recorded.
  • the invention has the beneficial effects that: 1) through the use of machine vision deep learning combined with unique projective geometry and imaging principles, accurate recognition of the target area around the moving object and realization of motion are achieved.
  • FIG. 1 is a schematic diagram of a depth learning model in accordance with one example of the present invention.
  • FIG. 2 is a flow chart of a method of determining a position of a moving object according to an example of the present invention
  • FIG 3 is a detailed flow chart of step b in the flow chart shown in Figure 2;
  • Figure 4 is a detailed flow chart of step d in the flow chart shown in Figure 2;
  • Figure 5 is a block diagram of a position determining system of a moving object in accordance with one example of the present invention.
  • ordinal numbers may be used as the adjectives of the elements (ie, any nouns in the application).
  • the use of ordinal does not imply or create any particular ordering of the elements, and does not limit any element to only a single element unless explicitly disclosed, such as by the use of the terms “before”, “after”, “single”, and others Such a term. Instead, the use of ordinals will distinguish between features.
  • the first element is different from the second element, and the first element may encompass more than one element and be in (or before) the second element in the ordering of the elements.
  • the method and system for determining the position of a moving object according to the present invention combines the breakthrough of depth learning in the field of image recognition and positions the moving object by machine vision. Therefore, the method and system for determining the position of a moving object according to the present invention is applicable to various scenes of various moving objects, including but not limited to vehicles, airplanes, etc., and the various scenes include It is not limited to common scenes such as automatic parking and parking space detection.
  • the position determining method and system of the vehicle as an example of the present invention Before applying the position determining method and system of the vehicle as an example of the present invention to the vehicle to be positioned, it is necessary to train an identification model in advance for the actual parking space image.
  • the above recognition model for example, it can be obtained by training the LeNet-5 deep learning model as shown in FIG. 1.
  • the LeNet-5 deep learning model shown in FIG. 1 is taken as an example to explain in detail how to train the model.
  • a large number of samples need to be collected, for example, collecting various types of reversing videos in various reversing environments, and in addition, crawling various parking space pictures through the Internet, in addition, through various angles and distances of the crawled by matlab
  • the parking space picture is simulated.
  • the number of samples is further expanded.
  • the corresponding parking space information is extracted and marked for all the samples, and the parking space information includes, but is not limited to, the type of the parking space, whether there is an interference object in the parking space, and the like.
  • the LeNet-5 deep learning model is trained using all samples labeled with parking space information to obtain individual model parameters for the recognition model of the subsequent parking space area identification.
  • the LeNet-5 deep learning model is divided into seven layers, no input, and each layer contains trainable parameters (connection weights).
  • the input image is 32 x 32 in size.
  • the first layer C1 is a convolution layer composed of six 28 ⁇ 28 size feature maps for forming a feature map of the parking space;
  • the second layer S2 is a lower sampling layer, which is composed of six 14 ⁇ 14 size feature maps. It is used to downsample the feature map of the parking space by using the image local correlation to preserve the useful information while reducing the amount of data processing;
  • the third layer C3 is a convolution layer composed of 16 10 ⁇ 10 size feature maps.
  • the fourth layer S4 is a downsampling layer, similar to the second layer, and is composed of 16 5 ⁇ 5 size feature maps
  • Layer C5 is a convolutional layer, consisting of 120 feature maps, fully connected with the fourth layer S4 to extract global features
  • the sixth layer F6 is a fully connected layer, composed of 84 units, and the entire fifth layer C5 The connection corresponds to the coding of the last layer
  • the seventh layer is an output layer, which is composed of an Euclidean Radial Basis Function unit for outputting the positioning information of the feature map of the parking space, such as shown in FIG.
  • the LeNet-5 deep learning model of FIG. 1 is trained by using a large number of samples to obtain an identification model for the identification of the actual parking space frame for later use.
  • the deep learning model involved in the present invention is not limited thereto, as long as it is possible to use a large amount of sample data for training for a parking space.
  • the model for recognizing an image can function as a deep learning model involved in the present invention.
  • FIG. 2 is a flow chart of a method of determining a position of a moving object (in this example, a vehicle) according to an example of the present invention.
  • the look-around image around the vehicle is acquired for each unit time using, for example, an image sensor installed in the in-vehicle vision system of the vehicle (step a).
  • the above unit time may be one frame interval, or may be a plurality of frame intervals, or may be other fixed intervals according to a vehicle body deformation or the like.
  • the image sensor or the like transmits the look-around image to, for example, an electronic control unit (hereinafter referred to as an ECU) of the vehicle.
  • the ECU recognizes the look-around image by using the above-described recognition model that has been trained in advance, and determines an area in which the parking space to be parked is framed on the image, that is, a virtual target area (step b).
  • Figure 3 is a detailed flow chart of step b in the flow chart shown in Figure 2.
  • a plurality of candidate regions of different sizes are adaptively selected in the look-around image by the following formula (1) (step b1) ):
  • z is the distance of the pixel point on a certain edge of the selected frame from the corresponding center line
  • g(z) is the side length of the side of the strip.
  • a probability distribution map for the look-around image is formed according to the positions of the respective frames and the probability values of each frame, and the distribution is distributed in the probability distribution map.
  • one frame is selected from the plurality of frames as the virtual target region (step b3).
  • the probability distribution map of the look-around image is only distributed with one probability cluster.
  • the box with the highest probability value in the probability cluster can be selected as the virtual target region.
  • the probability distribution map of the look-around image has two or more probability clusters. In this case, the box with the highest probability value among all the probability clusters may be selected as the virtual target area.
  • the probability distribution map of the look-around image has two or more probability clusters. In this case, the box with the highest probability value among the probability clusters closest to the vehicle can be selected as the virtual target region.
  • step c After determining the area of the parking space to be parked on the image, that is, the virtual target area, returning to FIG. 2, step c is performed, that is, for the virtual target area, the imaging position of each virtual key point is determined by using the above recognition model. information.
  • each of the above virtual key points may be four outer vertices and four inner vertices of the parking space frame, but is not limited thereto, and may also be several points selected on each side of the parking space frame.
  • the imaging position information may be coordinate values of four outer vertices and four inner vertices of the parking space frame, but is not limited thereto, and may also be several points selected on each side of the parking space frame.
  • the coordinates can also be the outer 4 vertices of the parking space frame and the parking line width, and so on.
  • the ECU calculates the virtual target area based on the mapping position relationship between the imaging coordinate system of the look-around image and the body coordinate system of the vehicle itself, based on the imaging position information of each virtual key point determined in the above step c.
  • the relative distance between the corresponding actual target area and the vehicle step d).
  • Figure 4 is a detailed flow chart of step d in the flow chart shown in Figure 2.
  • step d1 virtual three-dimensional angle information of each virtual key point in the imaging coordinate system of the look-around image is calculated according to the imaging position information.
  • the optical axis angle of the vehicle vision system of the vehicle is calibrated in advance, the imaging coordinate system corresponding to the optical axis is determined, and the connection between the virtual key points and the vehicle vision system of the vehicle and the imaging coordinate system are calculated by the imaging principle.
  • the projection angle formed by each of the axes obtains virtual three-dimensional angle information of each virtual key point in the imaging coordinate system of the look-around image.
  • step d2 based on the mapping relationship between the imaging coordinate system of the look-around image defined by the following formula (2) and the body coordinate system of the vehicle itself (that is, the transformation matrix [R T] obtained by the following formula (2) And transforming the virtual three-dimensional angle information of each virtual key point in the imaging coordinate system of the look-around image into actual numbers of the actual key points corresponding to the respective virtual key points for determining the actual target area in the body coordinate system of the moving object 3D angle information
  • Z c is a scaling factor, and the value is any number greater than or equal to 0;
  • u, v is the pixel coordinate of any point on the ring image;
  • f/a is the number of pixels in the long dimension of the ring image, f/b Is the number of pixels in the short dimension direction of the look-around image;
  • u 0 and v 0 are the pixel coordinates of the point located at the center of the look-around image, and the values are equal to 1/2*f/a, 1/2*f/b, respectively;
  • R is from the look-around
  • the imaging coordinate of the image is to the 3 ⁇ 3 Rodrigues rotation matrix of the vehicle's own body coordinate system;
  • T is a 3 ⁇ 1 translation matrix, where the origin of the imaging coordinate system of the viewing image and the origin of the vehicle's own body coordinate system When coincident, T is a 3 ⁇ 1 translation matrix whose value is 0 for each component;
  • X w , Y w , Z w
  • step d3 based on the actual three-dimensional angle information calculated in step d2 and the height of the vehicle body, the relative distance between each actual key point and the vehicle body is calculated by the triangle principle (step d3).
  • the relative distance between the vehicle and the target parking space frame can be calculated for each unit time.
  • the chassis sensor may dynamically adjust the height of the vehicle body, and therefore, in order to ensure the robustness of the calculation of the relative distance for each unit time, It is necessary to correct the value of the relative distance corresponding to each unit time by the value of the relative distance corresponding to the previous unit time.
  • the ECU calculates the relative distance of the current unit time by using the above steps a to b and uses it as an observation value.
  • the ECU is on the wheel speed of the vehicle.
  • the sensor controls to obtain the distance that the vehicle moves in the previous unit time of the current unit time by the wheel speed sensor, and then the ECU subtracts the relative distance of the previous unit time that has been calculated through the above steps a to step d Going through the moving distance obtained by the wheel speed sensor and using the obtained result as a predicted value, the ECU then passes the above observation value and the above predicted value through a set of Kalman filters, and finally obtains the current unit represented by the following formula (3).
  • the corrected relative distance of time step e:
  • F(x, y) represents the result of the calculated corrected relative distance
  • x, y represent the above observation value
  • k represents the number of filters
  • w k represents the above-mentioned group of Kalman filters.
  • the weight of the kth Kalman filter in the device, g k (x, y) represents the relative distance calculated by the kth Kalman filter.
  • steps of the above-described method for determining the position of a moving object according to the present invention may be performed on hardware by a logic circuit formed on an integrated circuit (IC chip), or a CPU (Central Processing Unit) may be used. ) Executed in software, it can also be executed by a combination of software and hardware.
  • IC chip integrated circuit
  • CPU Central Processing Unit
  • the present invention When the position determining method of the moving object according to the present invention is applied to the parking space detecting scene of the vehicle as described above, since the present invention combines the breakthrough of depth learning in the field of picture recognition, the vehicle position around the vehicle body is accurately positioned by the vehicle vision system. And identification, and effective identification of obstacles in the parking space, combined with a unique projective geometry method, vertical bird map conversion of available parking spaces, calculation of two-dimensional parking space coordinates, thereby dynamic closed-loop realization of automatic parking and other vehicle-assisted driving.
  • the position determining method of the moving object according to the present invention realizes the indiscriminate parking space recognition and positioning effect in various environments, and can be applied to various types of vehicle assisted driving, and the corresponding frame can be used to detect other objects common around the vehicle body. And information such as pedestrians, therefore, provide new protection for vehicle safety.
  • the driver of the vehicle is not required to drive back and forth and detect the parking space as compared with the millimeter wave radar, but the parking space can be detected in real time at any position. There are no requirements on both sides of the parking space.
  • the parking space recognition rate is finally 99%, and some parking spaces are difficult to recognize by the human eye due to blurring, but the vision-based machine learning can also work well, and in addition, obstacle recognition
  • the rate is 100%, thus ensuring the safety of automatic parking.
  • the recognition error is -2cm to 2cm, which also satisfies the requirements of the automatic parking model.
  • FIG. Figure 5 is a block diagram of a position determining system of a moving object (in this example, a vehicle) in accordance with one example of the present invention.
  • the position determining system of the moving object includes a first module 101, a second module 102, a third module 103, a fourth module 104, and a fifth module 105.
  • the first module 101 acquires a look-around image around the vehicle for each unit time.
  • the first module may be an image sensor or the like installed in a vehicle vision system of a vehicle.
  • the above unit time may be one frame interval, or may be a plurality of frame intervals, or may be other fixed intervals according to a vehicle body deformation or the like.
  • the second module 102 determines the virtual target area based on the look-around image acquired by the first module 101 and using the previously trained recognition model.
  • the above recognition model for example, it can be obtained by training the LeNet-5 deep learning model as shown in FIG. 1.
  • the second module may be a function module in the ECU of the vehicle, and the image sensor or the like transmits the look-around image to the ECU after obtaining a look-around image around the vehicle for each unit time, for example, using the image sensor or the like described above.
  • the ECU recognizes the look-around image by using the above-described recognition model and determines an area in which the parking space to be parked is framed on the image, that is, the virtual target area.
  • the second module 102 includes: adaptively selecting, in the foregoing look-around image, a plurality of units of different size candidate regions, and the unit adaptively selects a plurality of candidate regions with different sizes by the following formula (4) ( For example, several boxes with different side lengths):
  • z is the distance of the pixel point on a certain edge of the selected frame from the corresponding center line
  • g(z) is the side length of the side of the strip.
  • the second module 102 further includes: a unit for calculating a probability value that each candidate region becomes a virtual target region by using the above-mentioned recognition model, by which a probability distribution map regarding the look-around image is obtained, in the probability distribution map
  • the second module 102 further includes: selecting at least one probability cluster based on the probability values of the plurality of candidate regions having different sizes, and selecting one candidate region from the plurality of candidate regions as the unit of the virtual target region.
  • the probability distribution map of the look-around image is only distributed with one probability cluster. In this case, the box with the highest probability value in the probability cluster can be selected as the virtual target region.
  • the probability distribution map of the look-around image has two or more probability clusters. In this case, the box with the highest probability value among all the probability clusters may be selected as the virtual target area.
  • the probability distribution map of the look-around image has two or more probability clusters. In this case, the box with the highest probability value among the probability clusters closest to the vehicle can be selected as the virtual target region.
  • the third module 103 determines the imaging position information of its respective virtual key points using the above-described recognition model for the virtual target area determined by the second module 102.
  • the third module may be a functional module in the ECU of the vehicle.
  • each of the above virtual key points may be four outer vertices and four inner vertices of the parking space frame, but is not limited thereto, and may also be several points selected on each side of the parking space frame.
  • the imaging position information may be coordinate values of four outer vertices and four inner vertices of the parking space frame, but is not limited thereto, and may also be several points selected on each side of the parking space frame.
  • the coordinates can also be the outer 4 vertices of the parking space frame and the parking line width, and so on.
  • the fourth module 104 calculates the actual target area corresponding to the virtual target area and the vehicle according to the imaging position information of each virtual key point based on the mapping relationship between the imaging coordinate system of the look-around image and the body coordinate system of the vehicle itself. relative distance.
  • the fourth module 104 can be a functional module in the ECU of the vehicle.
  • the fourth module 104 includes: a unit for calculating virtual three-dimensional angle information of each virtual key point in an imaging coordinate system of the look-around image according to imaging position information of each virtual key point. Specifically, the unit determines the imaging coordinate system corresponding to the optical axis by pre-calibrating the optical axis angle of the vehicle vision system of the vehicle, and calculates the connection between each virtual key point and the vehicle vision system of the vehicle by means of the imaging principle. The projection angle formed with each axis in the imaging coordinate system, thereby obtaining virtual three-dimensional angle information of each virtual key point in the imaging coordinate system of the look-around image.
  • the fourth module 104 further includes: a mapping relationship between the imaging coordinate system based on the look-around image defined by the following formula (5) and the body coordinate system of the vehicle itself (that is, obtained by the following formula (5))
  • the transformation matrix [R T]) calculates, according to the virtual three-dimensional angle information, a unit corresponding to each virtual key point for determining actual three-dimensional angle information of each actual key point of the actual target area in the vehicle body's own body coordinate system.
  • Z c is a scaling factor, and the value is any number greater than or equal to 0;
  • u, v is the pixel coordinate of any point on the ring image;
  • f/a is the number of pixels in the long dimension of the ring image, f/b Is the number of pixels in the short dimension direction of the look-around image;
  • u 0 and v 0 are the pixel coordinates of the point located at the center of the look-around image, and the values are equal to 1/2*f/a, 1/2*f/b, respectively;
  • R is from the look-around
  • the imaging coordinate of the image is to the 3 ⁇ 3 Rodrigues rotation matrix of the vehicle's own body coordinate system;
  • T is a 3 ⁇ 1 translation matrix, where the origin of the imaging coordinate system of the viewing image and the origin of the vehicle's own body coordinate system When coincident, T is a 3 ⁇ 1 translation matrix whose value is 0 for each component;
  • X w , Y w , Z w
  • the fourth module 104 further includes: a unit that calculates a relative distance between each actual key point and the vehicle body based on the actual three-dimensional angle information and the height of the vehicle body.
  • the relative distance between the vehicle and the target parking space frame can be calculated for each unit time.
  • the chassis sensor may dynamically adjust the height of the vehicle body, and therefore, in order to ensure the robustness of the calculation of the relative distance for each unit time, It is necessary to correct the value of the relative distance corresponding to each unit time by the value of the relative distance corresponding to the previous unit time. Therefore, the fifth module 105 is provided in the position determining system shown in FIG. 5, and the fifth module 105 uses the first module 101 to the above for the unit time of the current unit time for each unit time.
  • the fifth module 105 may be a functional module in the ECU of the vehicle, and the distance that the vehicle moves in the previous unit time can be obtained by the wheel speed sensor of the vehicle. More specifically, the fifth module 105 performs Kalman filtering on the difference between the above-described distances with respect to the previous unit time and the relative distance calculated for the above-described current unit time to obtain the above-mentioned current unit time represented by the following formula (6).
  • the relative distance after the correction is a functional module in the ECU of the vehicle, and the distance that the vehicle moves in the previous unit time can be obtained by the wheel speed sensor of the vehicle. More specifically, the fifth module 105 performs Kalman filtering on the difference between the above-described distances with respect to the previous unit time and the relative distance calculated for the above-described current unit time to obtain the above-mentioned current unit time represented by the following formula (6).
  • the relative distance after the correction is a functional module in the ECU of the vehicle, and the distance that the vehicle moves in the previous unit time can
  • F(x, y) represents the result of the calculated corrected relative distance
  • x, y represent the above observation value
  • k represents the number of filters
  • w k represents the above-mentioned group of Kalman filters.
  • the weight of the kth Kalman filter in the device, g k (x, y) represents the relative distance calculated by the kth Kalman filter.
  • the position determining system of the moving object further includes means for training the deep learning model using the sample data to obtain the above-described recognition model for the virtual target area.
  • each module of the position determining system of the moving object described above may be implemented in hardware through a logic circuit formed on an integrated circuit (IC chip), or may be implemented in software using a CPU (Central Processing Unit). It can also be realized by a combination of software and hardware.
  • IC chip integrated circuit
  • CPU Central Processing Unit
  • the method and system for determining the position of a moving object realizes accurate recognition of a target area around a moving object by using deep learning of machine vision and combining unique projective geometry and imaging principles, and realizing relative of the moving object.
  • Accurate positioning in the target area by considering the characteristics of the moving object deformation and so on, the relative distance between the moving object and the target area is determined in a closed loop, so that it can be more robust in various application scenarios; Instead, it relies on the image acquisition device and the closed-loop calculation method. Therefore, the position of the moving object relative to the target region can be determined in real time at any position, and it is not required that obstacles for positioning ranging must exist on both sides of the moving object.
  • the present invention is not limited to these embodiments, and the present invention may be embodied in the following manner: for performing the position of the moving object described above.
  • a disk for example, a magnetic disk, an optical disk, or the like
  • a card for example, a memory card, an optical card, or the like
  • a semiconductor memory for example, a ROM, a nonvolatile memory, or the like
  • a tape can be used as the recording medium.
  • Various types of recording media such as tapes, cassette tapes, and the like.
  • a computer program that causes the computer to execute the position determining method of the moving object in the above-described embodiment or a computer program that causes the computer to implement the function of the position determining system of the moving object in the above-described embodiment, thereby circulating The cost can be reduced, and the portability and versatility can be improved.
  • the above-mentioned recording medium is loaded on a computer, and a computer program recorded on the recording medium is read by the computer and stored in the memory, and the processor (CPU: Central Processing Unit), MPU: Micro Processing The unit (micro processing unit) reads out the computer program from the memory and executes it, whereby the position determining method of the moving object in the above embodiment can be executed and the function of the position determining system of the moving object in the above embodiment can be realized.
  • CPU Central Processing Unit

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Image Analysis (AREA)

Abstract

本发明提供了一种运动物体的位置确定方法，包括：步骤a，针对每一个单位时间，获取运动物体周围的环视图像；步骤b，基于环视图像，利用事先训练好的识别模型，来确定虚拟目标区域；步骤c，针对虚拟目标区域，利用所述识别模型，确定其各个虚拟关键点的成像位置信息；步骤d，基于环视图像的成像坐标系与运动物体的主体坐标系之间的映射关系，根据各个虚拟关键点的成像位置信息来计算虚拟目标区域所对应的实际目标区域与运动物体的相对距离；以及步骤e，针对每一个单位时间，使用针对当前单位时间的前一个单位时间的通过步骤a～步骤d得到的相对距离与运动物体在前一个单位时间内移动的距离之差来修正针对当前单位时间的通过步骤a～步骤d得到的相对距离，以获得针对当前单位时间的修正后的相对距离。

Description

运动物体的位置确定方法及系统 技术领域
本发明涉及定位测距技术，更具体地涉及运动物体的位置确定方法及系统。
背景技术
在对诸如车辆、飞机等运动物体进行定位以测量其与目标区域的相对距离时，主要是借助于雷达。
以汽车辅助驾驶中的自动泊车场景为例，自动泊车向来是难点，其中最重要的是对车位准确的检测定位。
目前，量产的自动泊车产品大部分是基于毫米波雷达来对车位进行探测，受限于雷达的特性，车位两侧需要有障碍物，车辆通过来回探测来猜测车位的信息，因此，倒车体验很差。
其余的自动泊车产品主要是基于传统机器视觉来定位车位，可是，由于车位复杂多样，加上光照、角度等各种因素的影响，车位识别率不超过40％，并且，对车位内的障碍物没有处理，容易造成倒车事故，因此很少采用。
当然也有两者结合的产品，可是，由于各自缺陷导致未能很好地互补，再加上成本和安装校准等各方面的问题，也很难得到很好的应用。
发明内容
本发明是为了克服上述缺点而完成的，所采用的技术方案如下。
一种运动物体的位置确定方法，包括：步骤a，针对每一个单位时间，获取所述运动物体周围的环视图像；步骤b，基于所述环视图像，利用事先训练好的识别模型，确定虚拟目标区域；步骤c，针对所述虚拟目标区域，利用所述识别模型，确定其各个虚拟关键点的成像位置信息；步骤d，基于所述环视图像的成像坐标系与所述运动物体的主体坐标系之间的映射关系，根据所述各个虚拟关键点的成像位置信息来计算所述虚拟目标区域所对应的实际目标区域与所述运动物体的相对距离；以及步骤e，针对每一个单位时间，使用针对当前单位时间的前一个单位时间的通过所述步骤a～所述步骤d得到的相对距离与所述运动物体在所述前一个单位时间内移动的距离之差来修正针对所述当前单位时间的通过所述步骤a～所述步骤d得到的相对距离，以获得针对所述当前单位时间的修正后的相对距离。
进一步地，在根据本发明的运动物体的位置确定方法中，还包括：使用样本数据对深度学习模型进行训练以获得用于所述虚拟目标区域的所述识别模型的步骤。
进一步地，在根据本发明的运动物体的位置确定方法中，所述步骤b包括：步骤b1，在所述环视图像中自适应地选取若干个大小不同的候选区域；步骤b2，利用所述识别模型来计算每一个所述候选区域成为所述虚拟目标区域的概率值；以及步骤b3，基于所述若干个大小不同的候选区域各自的概率值所形成的至少一个概率簇，从所述若干个候选区域中选择一个候选区域来作为所述虚拟目标区域。
进一步地，在根据本发明的运动物体的位置确定方法中，所述步骤d包括：步骤d1，根据所述各个虚拟关键点的成像位置信息来计算所述各个虚拟关键点在所述环视图像的成像坐标系下的虚拟三维 角度信息；步骤d2，基于所述环视图像的成像坐标系与所述运动物体的主体坐标系之间的映射关系，根据所述虚拟三维角度信息来计算与各个虚拟关键点对应的用于确定实际目标区域的各个实际关键点在所述运动物体的主体坐标系下的实际三维角度信息；以及步骤d3，基于所述实际三维角度信息和所述运动物体的高度来计算所述各个实际关键点与所述运动物体的相对距离。
进一步地，在根据本发明的运动物体的位置确定方法中，在所述步骤e中，对关于所述前一个单位时间的所述距离之差和针对所述当前单位时间计算出的相对距离进行卡尔曼滤波以获得针对所述当前单位时间的修正后的相对距离。
一种运动物体的位置确定装置系统，包括：第一模块，针对每一个单位时间，获取所述运动物体周围的环视图像；第二模块，基于所述环视图像，利用事先训练好的识别模型，确定虚拟目标区域；第三模块，针对所述虚拟目标区域，利用所述识别模型，确定其各个虚拟关键点的成像位置信息；第四模块，基于所述环视图像的成像坐标系与所述运动物体的主体坐标系之间的映射关系，根据所述各个虚拟关键点的成像位置信息来计算所述虚拟目标区域所对应的实际目标区域与所述运动物体的相对距离；以及第五模块，针对每一个单位时间，使用针对当前单位时间的前一个单位时间的通过所述第一模块～所述第四模块得到的相对距离与所述运动物体在所述前一个单位时间内移动的距离之差来修正针对所述当前单位时间的通过所述第一模块～所述第四模块得到的相对距离，以获得针对所述当前单位时间的修正后的相对距离。
进一步地，在根据本发明的运动物体的位置确定系统中，还包括：使用样本数据对深度学习模型进行训练以获得用于所述虚拟目标区域的所述识别模型的模块。
进一步地，在根据本发明的运动物体的位置确定系统中，所述第二模块包括：在所述环视图像中自适应地选取若干个大小不同的候选区域的单元；利用所述识别模型来计算每一个所述候选区域成为所述虚拟目标区域的概率值的单元；以及基于所述若干个大小不同的候选区域各自的概率值所形成的至少一个概率簇，从所述若干个候选区域中选择一个候选区域来作为所述虚拟目标区域的单元。
进一步地，在根据本发明的运动物体的位置确定系统中，所述第四模块包括：根据所述各个虚拟关键点的成像位置信息来计算所述各个虚拟关键点在所述环视图像的成像坐标系下的虚拟三维角度信息的单元；基于所述环视图像的成像坐标系与所述运动物体的主体坐标系之间的映射关系，根据所述虚拟三维角度信息来计算与各个虚拟关键点对应的用于确定实际目标区域的各个实际关键点在所述运动物体的主体坐标系下的实际三维角度信息的单元；以及基于所述实际三维角度信息和所述运动物体的高度来计算所述各个实际关键点与所述运动物体的相对距离的单元。
进一步地，在根据本发明的运动物体的位置确定系统中，所述第五模块对关于所述前一个单位时间的所述距离之差和针对所述当前单位时间计算出的相对距离进行卡尔曼滤波以获得针对所述当前单位时间的修正后的相对距离。
一种运动物体位置确定程序，用于使计算机执行所述运动物体的位置确定方法。
一种运动物体位置确定程序，用于使计算机实现所述运动物体的位置确定系统的功能。
一种计算机可读取的记录介质，记录有用于使计算机执行所述运动物体的位置确定方法的运动物体位置确定程序。
一种计算机可读取的记录介质，记录有用于使计算机实现所述运动物体的位置确定系统的功能的运动物体位置确定程序。
相对于现有技术，本发明的有益效果是，1)通过采用机器视觉的深度学习并结合独特的射影几何和成像原理，从而实现了对运动物体周边的目标区域的准确识别、以及实现了运动物体相对于目标区域的准确定位；2)通过考虑运动物体变形等特点而闭环地确定运动物体与目标区域的相对距离，从而能够在各类应用场景下具有更强的鲁棒性；3)由于不使用雷达等设备而是代之以依靠图像获取装置和闭环计算方法，因此，能够在任何位置实时确定运动物体相对于目标区域的位置，不要求运动物体两侧必须存在用于定位测距的障碍物。
附图说明
图1是根据本发明的一个示例的深度学习模型的原理图；
图2是根据本发明的一个示例的运动物体的位置确定方法的流程图；
图3是图2所示出的流程图中的步骤b的细节流程图；
图4是图2所示出的流程图中的步骤d的细节流程图；
图5是根据本发明的一个示例的运动物体的位置确定系统的框图。
具体实施方式
以下将结合附图对本发明涉及的运动物体的位置确定方法及系统作进一步的详细描述。需要注意的是，以下的具体实施方式是示例性而非限制的，其旨在提供对本发明的基本了解，并不旨在确认本发明的关键或决定性的要素或限定所要保护的范围。
此外，遍及本申请，可以使用序数(例如，第一、第二、第三等)作为要素(即，本申请中的任何名词)的形容词。序数的使用不暗示或创建要素的任何特定排序，也不将任何要素限于仅单个要素，除非明确公开，诸如通过使用术语“在……之前”、“在……之后”、“单个”和其它这样的术语。相反，序数的使用将区分要素。作为示例，第一要素与第二要素不同，并且第一要素可以涵盖多于一个要素且在要素的排序中处于第二要素之后(或之前)。
根据本发明的运动物体的位置确定方法及系统结合深度学习在图像识别领域的突破并且通过机器视觉来对运动物体进行定位。因此，根据本发明的运动物体的位置确定方法及系统适用于各种运动物体的各种场景，其中，所述各种运动物体包括但不限于车辆、飞机等，而所述各种场景包括但不限于自动泊车、车位检测等常用场景。
下面以车辆的车位检测场景为例来详细地说明根据本发明的运动物体的位置确定方法及系统。
在将作为本发明的一个示例的车辆的位置确定方法及系统应用于待定位的车辆之前，需要事先训练好一个识别模型以用于实际车位图像。作为上述识别模型，例如，可以通过对如图1所示的LeNet-5深度学习模型进行训练来获得。下面以图1中示出的LeNet-5深度学习模型为例来详细地说明如何训练模型。
具体地，首先，需要采集大量样本，例如，搜集在各种倒车环境下的各类倒车视频，此外通过互联网爬取各种车位图片，此外，还通过matlab对所爬取的各个视角、距离的车位图片进行仿真。接着，通过对所搜集的各类倒车视频的每一帧的图像、所爬取的各种车位图片、以及仿真后的车位图片进行缩放、旋转、平移等操作，从而进一步扩大样本的数量。然后，对所有样本提取出对应的车位信息并进行标注，所述车位信息包括但不限于车位的类型、车位内是否有干扰物 体等。最后，利用标注有车位信息的所有样本对LeNet-5深度学习模型进行训练以获得适合后续的车位区域识别的识别模型的各个模型参数。
如图1所示，LeNet-5深度学习模型共分为七层，不包含输入，每层都包含可训练参数(连接权重)。输入图像为32×32的大小。第一层C1为卷积层，由6个28×28大小的特征图构成，用于形成车位的特征图谱；第二层S2为下抽样层，由6个14×14大小的特征图构成，用于利用图像局部相关性来对车位的特征图谱进行下抽样，以便在减少数据处理量的同时保留有用信息；第三层C3为卷积层，由16个10×10大小的特征图构成，用于对车位的特征图谱再次进行卷积操作以用于提取多种组合特征；第四层S4为下抽样层，与第二层类似，由16个5×5大小的特征图构成；第五层C5为卷积层，由120个特征图构成，与第四层S4进行全连接以便于提取全局特征；第六层F6为全连接层，由84个单元构成，与第五层C5进行全连接，对应最后一层的编码；第七层为输出层，由欧式径向基函数(Euclidean Radial Basis Function)单元组成，用于输出车位的特征图谱的定位信息，诸如图1中所示出的车位框状的输入图像的外部4个顶点和内部4个顶点的坐标、以及虚拟车位线宽度等。
通过利用大量样本来对图1这样的LeNet-5深度学习模型进行训练，从而获得用于实际车位框的识别的识别模型以备后续使用。
需要注意的是，虽然在以上将LeNet-5深度学习模型作为示例进行了说明，但是，本发明中所涉及的深度学习模型不限于此，只要是能够使用大量样本数据来进行训练以用于车位图像的识别的模型，都可以作为本发明中所涉及的深度学习模型来发挥作用。
接下来，参照图2至图4来说明如何使用上述获得的识别模型来确定运动物体(在本示例中为车辆)相对于目标区域(在本示例中 为车位框)的位置。
图2是根据本发明的一个示例的运动物体(在本示例中为车辆)的位置确定方法的流程图。
在本示例中，当车辆的用户进行倒车而开启环视功能时，使用例如安装于车辆的车载视觉系统中的图像传感器等针对每一个单位时间来获取车辆周围的环视图像(步骤a)。
需要注意的是，上述单位时间可以是1帧间隔，也可以是数帧间隔，还可以是根据车体变形等预先设定的其它固定的一段间隔时间。
在本示例中，在例如使用上述图像传感器等针对每一个单位时间获得车辆周围的环视图像之后，上述图像传感器等将该环视图像传送至例如车辆的电子控制单元(Electronic Control Unit，以下简称ECU)，由该ECU利用事先训练好的上述识别模型来对上述环视图像进行识别并且确定即将停车的车位框在图像上的区域、即虚拟目标区域(步骤b)。
图3是图2所示出的流程图中的步骤b的细节流程图。
具体地，在获得车辆周围的环视图像之后，在该环视图像中通过下述公式(1)自适应地选取若干个大小不同的候选区域(例如，若干个边长大小不同的框)(步骤b1)：
Figure PCTCN2018079596-appb-000001
其中，z为处于所选取的框的某条边上的像素点离相应中线的距离，g(z)为该条边的边长。
然后，针对上述若干个边长大小不同的框中的每一个，将框及其内部的图像放入事先训练好的上述识别模型来计算该框与能够停车的车位框的匹配程度、即成为上述虚拟目标区域的概率值(步骤b2)。在对上述若干个边长大小不同的框均计算完该概率值之后，将根据各个框的位置以及每个框的概率值来形成关于上述环视图像的概率分布图，在该概率分布图中分布有至少一个概率簇，每个概率簇中存在一个概率最大值，其代表上述环视图像中该概率簇所处的区域存在成为车位框的可能性最大的一个框。
接下来，基于上述若干个边长大小不同的框的概率值所形成的上述至少一个概率簇，从上述若干个框中选择一个框来作为上述虚拟目标区域(步骤b3)。在一个示例中，环视图像的概率分布图仅分布有一个概率簇，此时，可以选择该概率簇中概率值最大的那个框来作为上述虚拟目标区域。在另一个示例中，环视图像的概率分布图分布有两个或两个以上的概率簇，此时，可以选择所有概率簇中概率值最大的那个框来作为上述虚拟目标区域。在又一个示例中，环视图像的概率分布图分布有两个或两个以上的概率簇，此时，可以选择离车辆最近的那个概率簇中概率值最大的那个框来作为上述虚拟目标区域。
在确定即将停车的车位框在图像上的区域、即虚拟目标区域之后，返回至图2，执行步骤c，即，针对该虚拟目标区域，利用上述识别模型，确定其各个虚拟关键点的成像位置信息。
需要注意的是，上述各个虚拟关键点可以是车位框的外部4个顶点和内部4个顶点，但是不限于此，也可以是在车位框的各条边上选取的若干点。此外，需要注意的是，上述成像位置信息可以是车位框的外部4个顶点和内部4个顶点的坐标值，但是不限于此，也可以是在车位框的各条边上选取的若干点的坐标，还可以是仅车位框的外部4个顶点和车位线宽度，等等。
然后，在上述示例中，ECU基于环视图像的成像坐标系与车辆自身的主体坐标系之间的映射关系，根据在上述步骤c中确定的各个虚拟关键点的成像位置信息来计算虚拟目标区域所对应的实际目标区域与车辆的相对距离(步骤d)。
图4是图2所示出的流程图中的步骤d的细节流程图。
具体地，在确定出虚拟目标区域的各个虚拟关键点的成像位置信息之后，根据所述成像位置信息来计算各个虚拟关键点在环视图像的成像坐标系下的虚拟三维角度信息(步骤d1)。具体地，事先标定好车辆的车载视觉系统的光轴角度，确定对应于该光轴的成像坐标系，通过成像原理计算出各个虚拟关键点和车辆的车载视觉系统的连线与成像坐标系中的各个轴形成的投影夹角，从而获得各个虚拟关键点在环视图像的成像坐标系下的虚拟三维角度信息。
接下来，基于由下述公式(2)限定的环视图像的成像坐标系与车辆自身的主体坐标系之间的映射关系(即，通过下述公式(2)求取的变换矩阵[R T])，将各个虚拟关键点在环视图像的成像坐标系下的虚拟三维角度信息变换成与各个虚拟关键点对应的用于确定实际目标区域的各个实际关键点在运动物体的主体坐标系下的实际三维角度信息(步骤d2)：
Figure PCTCN2018079596-appb-000002
其中，Z c是缩放因子，取值为大于或等于0的任意一个数；u、v是环视图像上的任意点的像素坐标；f/a是环视图像长尺寸方向的像素数目，f/b是环视图像短尺寸方向的像素数目；u 0、v 0是位于环视图像中心的点的像素坐标，取值分别等于1/2*f/a、1/2*f/b；R是从环视 图像的成像坐标系到车辆自身的主体坐标系的3×3的Rodrigues旋转矩阵；T是3×1的平移矩阵，其中，在环视图像的成像坐标系的原点与车辆自身的主体坐标系的原点重合时，T是每个分量的取值为0的3×1的平移矩阵；X w、Y w、Z w是车辆自身的主体坐标系中与u、v对应的点的坐标。
然后，基于在步骤d2中计算出的实际三维角度信息和车身的高度，通过三角形原理来计算各个实际关键点与车身的相对距离(步骤d3)。
通过图2的步骤a～步骤d，能够针对每一个单位时间计算出车辆与目标车位框的相对距离。
然而，考虑到倒车过程中车体载重状态可能会发生变化，与此伴随地，底盘传感器可能会动态调整车身的高度，因此，为了确保针对每一个单位时间的相对距离的计算的鲁棒性，需要对每一个单位时间所对应的相对距离的值用其前一个单位时间所对应的相对距离的值进行修正。具体地，在上述示例中，对于某一个单位时间而言，ECU通过上述步骤a～上述步骤d计算出当前单位时间的相对距离并将其作为观测值，另一方面，ECU对车辆的轮速传感器进行控制以通过该轮速传感器获得车辆在当前单位时间的前一个单位时间内移动的距离，然后ECU用已经通过上述步骤a～上述步骤d计算完毕的所述前一个单位时间的相对距离减去通过轮速传感器获得的移动距离并将所得结果作为预测值，之后，ECU将上述观测值和上述预测值通过一组卡尔曼滤波器，最终得到由下述公式(3)表示的针对当前单位时间的修正后的相对距离(步骤e)：
Figure PCTCN2018079596-appb-000003
其中，F(x,y)表示所计算的修正后的相对距离的结果，x、y分别表示 上述观测值、上述预测值，k表示滤波器的个数，w k表示上述一组卡尔曼滤波器中的第k个卡尔曼滤波器的权重，g k(x,y)表示由第k个卡尔曼滤波器计算出的相对距离。
需要注意的是，上述根据本发明的运动物体的位置确定方法的各步骤可以通过形成在集成电路(IC芯片)上的逻辑电路在硬件上执行，也可以使用CPU(Central Processing Unit：中央处理单元)在软件上执行，还可以通过软硬件结合的方式来执行。
当如上所述将根据本发明的运动物体的位置确定方法应用于车辆的车位检测场景时，由于本发明结合深度学习在图片识别领域的突破，通过车载视觉系统对车身周边的车位进行准确的定位和识别，并且对车位内障碍物进行有效的判别，结合独特的射影几何方法，对可用车位进行垂直鸟览图转化，计算二维的车位坐标，从而动态闭环实现自动泊车等汽车辅助驾驶。
根据本发明的运动物体的位置确定方法实现了在各种环境下的无差别的车位识别定位效果，可应用于各类车辆辅助驾驶中，而且，可利用对应框架来检测车身周围常见的其它物体和行人等信息，因此，对车辆安全提供新的保障。
根据本发明的运动物体的位置确定方法，与毫米波雷达相比，不需要车辆的驾驶员来回开车和检测车位，而是在任何位置都能够实时检测出车位。对车位两侧也没有要求。
根据本发明的运动物体的位置确定方法，车位识别率最终为99％，部分车位因模糊而导致人眼很难辨识，但是，基于视觉的机器学习也能很好得工作，此外，障碍物识别率为100％，从而保障了自动泊车的安全性，识别的误差为-2cm至2cm，也满足了自动泊车模型的要求。
最后，参照图5来说明根据本发明的运动物体的位置确定系统。图5是根据本发明的一个示例的运动物体(在本示例中为车辆)的位置确定系统的框图。
如图5所示，运动物体(在本示例中为车辆)的位置确定系统包括：第一模块101、第二模块102、第三模块103、第四模块104、以及第五模块105。
第一模块101针对每一个单位时间获取车辆周围的环视图像。
在本示例中，第一模块可以是安装于车辆的车载视觉系统中的图像传感器等。
需要注意的是，上述单位时间可以是1帧间隔，也可以是数帧间隔，还可以是根据车体变形等预先设定的其它固定的一段间隔时间。
第二模块102基于由第一模块101获取的环视图像并利用事先训练好的识别模型来确定虚拟目标区域。
关于上述识别模型，例如，可以通过对如图1所示的LeNet-5深度学习模型进行训练来获得。
在本示例中，第二模块可以是车辆的ECU中的功能模块，在例如使用上述图像传感器等针对每一个单位时间获得车辆周围的环视图像之后，上述图像传感器等将该环视图像传送至该ECU，由该ECU利用上述识别模型来对上述环视图像进行识别并且确定即将停车的车位框在图像上的区域、即虚拟目标区域。
具体地，第二模块102包括：在上述环视图像中自适应地选取若干个大小不同的候选区域的单元，所述单元通过下述公式(4)自适应地选取若干个大小不同的候选区域(例如，若干个边长大小不同 的框)：
Figure PCTCN2018079596-appb-000004
其中，z为处于所选取的框的某条边上的像素点离相应中线的距离，g(z)为该条边的边长。
此外，第二模块102还包括：利用上述识别模型来计算每一个候选区域成为虚拟目标区域的概率值的单元，通过所述单元，将获得关于上述环视图像的概率分布图，在该概率分布图中分布有至少一个概率簇，每个概率簇中存在一个概率最大值，其代表上述环视图像中该概率簇所处的区域存在成为车位框的可能性最大的一个框。
此外，第二模块102还包括：基于上述若干个大小不同的候选区域各自的概率值所形成的至少一个概率簇，从上述若干个候选区域中选择一个候选区域来作为上述虚拟目标区域的单元。在一个示例中，环视图像的概率分布图仅分布有一个概率簇，此时，可以选择该概率簇中概率值最大的那个框来作为上述虚拟目标区域。在另一个示例中，环视图像的概率分布图分布有两个或两个以上的概率簇，此时，可以选择所有概率簇中概率值最大的那个框来作为上述虚拟目标区域。在又一个示例中，环视图像的概率分布图分布有两个或两个以上的概率簇，此时，可以选择离车辆最近的那个概率簇中概率值最大的那个框来作为上述虚拟目标区域。
第三模块103针对由第二模块102确定的虚拟目标区域利用上述识别模型来确定其各个虚拟关键点的成像位置信息。
在本示例中，第三模块可以是车辆的ECU中的功能模块。
需要注意的是，上述各个虚拟关键点可以是车位框的外部4个 顶点和内部4个顶点，但是不限于此，也可以是在车位框的各条边上选取的若干点。此外，需要注意的是，上述成像位置信息可以是车位框的外部4个顶点和内部4个顶点的坐标值，但是不限于此，也可以是在车位框的各条边上选取的若干点的坐标，还可以是仅车位框的外部4个顶点和车位线宽度，等等。
第四模块104基于上述环视图像的成像坐标系与车辆自身的主体坐标系之间的映射关系，根据上述各个虚拟关键点的成像位置信息来计算上述虚拟目标区域所对应的实际目标区域与车辆的相对距离。
在本示例中，第四模块104可以是车辆的ECU中的功能模块。
具体地，第四模块104包括：根据各个虚拟关键点的成像位置信息来计算所述各个虚拟关键点在环视图像的成像坐标系下的虚拟三维角度信息的单元。具体地，所述单元通过事先标定好车辆的车载视觉系统的光轴角度，确定对应于该光轴的成像坐标系，借助于成像原理计算出各个虚拟关键点和车辆的车载视觉系统的连线与成像坐标系中的各个轴形成的投影夹角，从而获得各个虚拟关键点在环视图像的成像坐标系下的虚拟三维角度信息。
此外，第四模块104还包括：基于由下述公式(5)限定的环视图像的成像坐标系与车辆自身的主体坐标系之间的映射关系(即，通过下述公式(5)求取的变换矩阵[R T])，根据上述虚拟三维角度信息来计算与各个虚拟关键点对应的用于确定实际目标区域的各个实际关键点在车辆自身的主体坐标系下的实际三维角度信息的单元。
Figure PCTCN2018079596-appb-000005
其中，Z c是缩放因子，取值为大于或等于0的任意一个数；u、v是环 视图像上的任意点的像素坐标；f/a是环视图像长尺寸方向的像素数目，f/b是环视图像短尺寸方向的像素数目；u 0、v 0是位于环视图像中心的点的像素坐标，取值分别等于1/2*f/a、1/2*f/b；R是从环视图像的成像坐标系到车辆自身的主体坐标系的3×3的Rodrigues旋转矩阵；T是3×1的平移矩阵，其中，在环视图像的成像坐标系的原点与车辆自身的主体坐标系的原点重合时，T是每个分量的取值为0的3×1的平移矩阵；X w、Y w、Z w是车辆自身的主体坐标系中与u、v对应的点的坐标。
此外，第四模块104还包括：基于实际三维角度信息和车身的高度来计算各个实际关键点与车身的相对距离的单元。
通过上述第一模块101～第四模块104，能够针对每一个单位时间计算出车辆与目标车位框的相对距离。
然而，考虑到倒车过程中车体载重状态可能会发生变化，与此伴随地，底盘传感器可能会动态调整车身的高度，因此，为了确保针对每一个单位时间的相对距离的计算的鲁棒性，需要对每一个单位时间所对应的相对距离的值用其前一个单位时间所对应的相对距离的值进行修正。因此，在图5所示的位置确定系统中设置有第五模块105，所述第五模块105针对每一个单位时间使用针对当前单位时间的前一个单位时间的通过上述第一模块101～上述第四模块104得到的相对距离与车辆在上述前一个单位时间内移动的距离之差来修正针对上述当前单位时间的通过上述第一模块101～上述第四模块104得到的相对距离，以获得针对上述当前单位时间的修正后的相对距离。其中，第五模块105可以是车辆的ECU中的功能模块，而车辆在前一个单位时间内移动的距离可以通过车辆的轮速传感器来获得。更具体地，第五模块105对关于前一个单位时间的上述距离之差和针对上述当前单位时间计算出的相对距离进行卡尔曼滤波以获得由下述公式(6) 表示的针对上述当前单位时间的修正后的相对距离。
Figure PCTCN2018079596-appb-000006
其中，F(x,y)表示所计算的修正后的相对距离的结果，x、y分别表示上述观测值、上述预测值，k表示滤波器的个数，w k表示上述一组卡尔曼滤波器中的第k个卡尔曼滤波器的权重，g k(x,y)表示由第k个卡尔曼滤波器计算出的相对距离。
除了上述第一模块101至第五模块105以外，根据本发明的运动物体的位置确定系统还包括：使用样本数据对深度学习模型进行训练以获得用于虚拟目标区域的上述识别模型的模块。
此外，上述的运动物体的位置确定系统的各模块可以通过形成在集成电路(IC芯片)上的逻辑电路在硬件上实现，也可以使用CPU(Central Processing Unit：中央处理单元)在软件上实现，还可以通过软硬件结合的方式来实现。
根据本发明的运动物体的位置确定方法及系统，通过采用机器视觉的深度学习并结合独特的射影几何和成像原理，从而实现了对运动物体周边的目标区域的准确识别、以及实现了运动物体相对于目标区域的准确定位；通过考虑运动物体变形等特点而闭环地确定运动物体与目标区域的相对距离，从而能够在各类应用场景下具有更强的鲁棒性；由于不使用雷达等设备而是代之以依靠图像获取装置和闭环计算方法，因此，能够在任何位置实时确定运动物体相对于目标区域的位置，不要求运动物体两侧必须存在用于定位测距的障碍物。
虽然在此之前以运动物体的位置确定方法及系统的实施方式为中心进行了说明，但是本发明不限定于这些实施方式，也可以将本发明实施为以下方式：用于执行上述运动物体的位置确定方法的计算机程序的方式或者用于实现上述运动物体的位置确定系统的功能的 计算机程序的方式或者记录有该计算机程序的计算机可读取的记录介质的方式。
在此，作为记录介质，能采用盘类(例如，磁盘、光盘等)、卡类(例如，存储卡、光卡等)、半导体存储器类(例如，ROM、非易失性存储器等)、带类(例如，磁带、盒式磁带等)等各种方式的记录介质。
通过在这些记录介质中记录使计算机执行上述实施方式中的运动物体的位置确定方法的计算机程序或使计算机实现上述实施方式中的运动物体的位置确定系统的功能的计算机程序并使其流通，从而能使成本的低廉化以及可携带性、通用性提高。
而且，在计算机上装载上述记录介质，由计算机读出在记录介质中记录的计算机程序并储存在存储器中，计算机所具备的处理器(CPU：Central Processing Unit(中央处理单元)、MPU：Micro Processing Unit(微处理单元))从存储器读出该计算机程序并执行，由此，能执行上述实施方式中的运动物体的位置确定方法并能实现上述实施方式中的运动物体的位置确定系统的功能。
本领域普通技术人员应当了解，本发明不限定于上述的实施方式，本发明可以在不偏离其主旨与范围内以许多其它的形式实施。因此，所展示的示例与实施方式被视为示意性的而非限制性的，在不脱离如所附各权利要求所定义的本发明精神及范围的情况下，本发明可能涵盖各种的修改与替换。

Claims (10)

  1. 一种运动物体的位置确定方法，其特征在于，包括：
    步骤a，针对每一个单位时间，获取所述运动物体周围的环视图像；
    步骤b，基于所述环视图像，利用事先训练好的识别模型，确定虚拟目标区域；
    步骤c，针对所述虚拟目标区域，利用所述识别模型，确定其各个虚拟关键点的成像位置信息；
    步骤d，基于所述环视图像的成像坐标系与所述运动物体的主体坐标系之间的映射关系，根据所述各个虚拟关键点的成像位置信息来计算所述虚拟目标区域所对应的实际目标区域与所述运动物体的相对距离；以及
    步骤e，针对每一个单位时间，使用针对当前单位时间的前一个单位时间的通过所述步骤a～所述步骤d得到的相对距离与所述运动物体在所述前一个单位时间内移动的距离之差来修正针对所述当前单位时间的通过所述步骤a～所述步骤d得到的相对距离，以获得针对所述当前单位时间的修正后的相对距离。
  2. 根据权利要求1所述的运动物体的位置确定方法，其特征在于，还包括：
    使用样本数据对深度学习模型进行训练以获得用于所述虚拟 目标区域的所述识别模型的步骤。
  3. 根据权利要求1或2所述的运动物体的位置确定方法，其特征在于，所述步骤b包括：
    步骤b1，在所述环视图像中自适应地选取若干个大小不同的候选区域；
    步骤b2，利用所述识别模型来计算每一个所述候选区域成为所述虚拟目标区域的概率值；以及
    步骤b3，基于所述若干个大小不同的候选区域各自的概率值所形成的至少一个概率簇，从所述若干个候选区域中选择一个候选区域来作为所述虚拟目标区域。
  4. 根据权利要求1或2所述的运动物体的位置确定方法，其特征在于，所述步骤d包括：
    步骤d1，根据所述各个虚拟关键点的成像位置信息来计算所述各个虚拟关键点在所述环视图像的成像坐标系下的虚拟三维角度信息；
    步骤d2，基于所述环视图像的成像坐标系与所述运动物体的主体坐标系之间的映射关系，根据所述虚拟三维角度信息来计算与各个虚拟关键点对应的用于确定实际目标区域的各个实际关键点在所述运动物体的主体坐标系下的实际三维角度信息；以及
    步骤d3，基于所述实际三维角度信息和所述运动物体的高度来计算所述各个实际关键点与所述运动物体的相对距离。
  5. 根据权利要求1或2所述的运动物体的位置确定方法，其特征在于，
    在所述步骤e中，对关于所述前一个单位时间的所述距离之差和针对所述当前单位时间计算出的相对距离进行卡尔曼滤波以获得针对所述当前单位时间的修正后的相对距离。
  6. 一种运动物体的位置确定系统，其特征在于，包括：
    第一模块，针对每一个单位时间，获取所述运动物体周围的环视图像；
    第二模块，基于所述环视图像，利用事先训练好的识别模型，确定虚拟目标区域；
    第三模块，针对所述虚拟目标区域，利用所述识别模型，确定其各个虚拟关键点的成像位置信息；
    第四模块，基于所述环视图像的成像坐标系与所述运动物体的主体坐标系之间的映射关系，根据所述各个虚拟关键点的成像位置信息来计算所述虚拟目标区域所对应的实际目标区域与所述运动物体的相对距离；以及
    第五模块，针对每一个单位时间，使用针对当前单位时间的前一个单位时间的通过所述第一模块～所述第四模块得到的相对距离与所述运动物体在所述前一个单位时间内移动的距离之差来修正针对所述当前单位时间的通过所述第一模块～所述第四模块得到的相对距离，以获得针对所述当前单位时间的修正后的相对距离。
  7. 根据权利要求6所述的运动物体的位置确定系统，其特征在于，还包括：
    使用样本数据对深度学习模型进行训练以获得用于所述虚拟目标区域的所述识别模型的模块。
  8. 根据权利要求6或7所述的运动物体的位置确定系统，其特征在于，所述第二模块包括：
    在所述环视图像中自适应地选取若干个大小不同的候选区域的单元；
    利用所述识别模型来计算每一个所述候选区域成为所述虚拟目标区域的概率值的单元；以及
    基于所述若干个大小不同的候选区域各自的概率值所形成的至少一个概率簇，从所述若干个候选区域中选择一个候选区域来作为所述虚拟目标区域的单元。
  9. 根据权利要求6或7所述的运动物体的位置确定系统，其特征在于，所述第四模块包括：
    根据所述各个虚拟关键点的成像位置信息来计算所述各个虚拟关键点在所述环视图像的成像坐标系下的虚拟三维角度信息的单元；
    基于所述环视图像的成像坐标系与所述运动物体的主体坐标系之间的映射关系，根据所述虚拟三维角度信息来计算与各个虚拟关键点对应的用于确定实际目标区域的各个实际关键点在所述运动物体的主体坐标系下的实际三维角度信息的单元；以及
    基于所述实际三维角度信息和所述运动物体的高度来计算所述各个实际关键点与所述运动物体的相对距离的单元。
  10. 根据权利要求6或7所述的运动物体的位置确定系统，其特征在于，
    所述第五模块对关于所述前一个单位时间的所述距离之差和 针对所述当前单位时间计算出的相对距离进行卡尔曼滤波以获得针对所述当前单位时间的修正后的相对距离。
PCT/CN2018/079596 2017-04-01 2018-03-20 运动物体的位置确定方法及系统 WO2018177159A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201710212439.4 2017-04-01
CN201710212439.4A CN106952308B (zh) 2017-04-01 2017-04-01 运动物体的位置确定方法及系统

Publications (1)

Publication Number Publication Date
WO2018177159A1 true WO2018177159A1 (zh) 2018-10-04

Family

ID=59475157

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/079596 WO2018177159A1 (zh) 2017-04-01 2018-03-20 运动物体的位置确定方法及系统

Country Status (2)

Country Link
CN (1) CN106952308B (zh)
WO (1) WO2018177159A1 (zh)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111179332A (zh) * 2018-11-09 2020-05-19 北京市商汤科技开发有限公司 图像处理方法及装置、电子设备及存储介质
CN111623776A (zh) * 2020-06-08 2020-09-04 昆山星际舟智能科技有限公司 使用近红外视觉传感器和陀螺仪进行目标测距的方法
CN113053131A (zh) * 2019-12-26 2021-06-29 北京新能源汽车股份有限公司 一种空闲车位识别方法、装置及车辆
CN113191329A (zh) * 2021-05-26 2021-07-30 超级视线科技有限公司 一种基于单目视觉图片的车辆泊位匹配方法及系统
CN113284120A (zh) * 2021-05-31 2021-08-20 北京经纬恒润科技股份有限公司 限高高度测量方法及装置

Families Citing this family (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106952308B (zh) * 2017-04-01 2020-02-28 上海蔚来汽车有限公司 运动物体的位置确定方法及系统
CN109697860A (zh) * 2017-10-20 2019-04-30 上海欧菲智能车联科技有限公司 车位检测和跟踪系统及方法及车辆
CN109927731B (zh) * 2017-12-15 2020-09-18 蔚来(安徽)控股有限公司 驾驶员放手检测方法、装置、控制器及存储介质
CN108573226B (zh) * 2018-04-08 2021-10-08 浙江大学 基于级联姿势回归的果蝇幼虫体节关键点定位方法
DE102018209607A1 (de) * 2018-06-14 2019-12-19 Volkswagen Aktiengesellschaft Verfahren und Vorrichtung zum Bestimmen einer Position eines Kraftfahrzeugs
CN109086708A (zh) * 2018-07-25 2018-12-25 深圳大学 一种基于深度学习的停车位检测方法及系统
JP6915605B2 (ja) * 2018-11-29 2021-08-04 オムロン株式会社 画像生成装置、ロボット訓練システム、画像生成方法、及び画像生成プログラム
CN109614914A (zh) * 2018-12-05 2019-04-12 北京纵目安驰智能科技有限公司 车位顶点定位方法、装置和存储介质
CN109872366B (zh) * 2019-02-25 2021-03-12 清华大学 一种物体的三维位置检测方法和装置
CN113643355B (zh) * 2020-04-24 2024-03-29 广州汽车集团股份有限公司 一种目标车辆位置和朝向的检测方法、系统及存储介质

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090161911A1 (en) * 2007-12-21 2009-06-25 Ming-Yu Shih Moving Object Detection Apparatus And Method
US20140118716A1 (en) * 2012-10-31 2014-05-01 Raytheon Company Video and lidar target detection and tracking system and method for segmenting moving targets
CN105818763A (zh) * 2016-03-09 2016-08-03 乐卡汽车智能科技（北京）有限公司 一种确定车辆周围物体距离的方法、装置及系统
CN106153000A (zh) * 2016-06-17 2016-11-23 合肥工业大学 一种前方车辆距离检测方法
CN106295459A (zh) * 2015-05-11 2017-01-04 青岛若贝电子有限公司 基于机器视觉和级联分类器的车辆检测和预警方法
CN106503653A (zh) * 2016-10-21 2017-03-15 深圳地平线机器人科技有限公司 区域标注方法、装置和电子设备
CN106952308A (zh) * 2017-04-01 2017-07-14 上海蔚来汽车有限公司 运动物体的位置确定方法及系统

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103000067B (zh) * 2012-12-28 2014-12-17 苏州苏迪智能系统有限公司 直角转弯检测系统及其检测方法
CN103065520B (zh) * 2012-12-28 2015-04-01 苏州苏迪智能系统有限公司 倒车入库检测系统及其检测方法
CN103600707B (zh) * 2013-11-06 2016-08-17 同济大学 一种智能泊车系统的泊车位检测装置及方法

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090161911A1 (en) * 2007-12-21 2009-06-25 Ming-Yu Shih Moving Object Detection Apparatus And Method
US20140118716A1 (en) * 2012-10-31 2014-05-01 Raytheon Company Video and lidar target detection and tracking system and method for segmenting moving targets
CN106295459A (zh) * 2015-05-11 2017-01-04 青岛若贝电子有限公司 基于机器视觉和级联分类器的车辆检测和预警方法
CN105818763A (zh) * 2016-03-09 2016-08-03 乐卡汽车智能科技（北京）有限公司 一种确定车辆周围物体距离的方法、装置及系统
CN106153000A (zh) * 2016-06-17 2016-11-23 合肥工业大学 一种前方车辆距离检测方法
CN106503653A (zh) * 2016-10-21 2017-03-15 深圳地平线机器人科技有限公司 区域标注方法、装置和电子设备
CN106952308A (zh) * 2017-04-01 2017-07-14 上海蔚来汽车有限公司 运动物体的位置确定方法及系统

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111179332A (zh) * 2018-11-09 2020-05-19 北京市商汤科技开发有限公司 图像处理方法及装置、电子设备及存储介质
CN111179332B (zh) * 2018-11-09 2023-12-19 北京市商汤科技开发有限公司 图像处理方法及装置、电子设备及存储介质
CN113053131A (zh) * 2019-12-26 2021-06-29 北京新能源汽车股份有限公司 一种空闲车位识别方法、装置及车辆
CN113053131B (zh) * 2019-12-26 2022-11-01 北京新能源汽车股份有限公司 一种空闲车位识别方法、装置及车辆
CN111623776A (zh) * 2020-06-08 2020-09-04 昆山星际舟智能科技有限公司 使用近红外视觉传感器和陀螺仪进行目标测距的方法
CN111623776B (zh) * 2020-06-08 2022-12-02 昆山星际舟智能科技有限公司 使用近红外视觉传感器和陀螺仪进行目标测距的方法
CN113191329A (zh) * 2021-05-26 2021-07-30 超级视线科技有限公司 一种基于单目视觉图片的车辆泊位匹配方法及系统
CN113284120A (zh) * 2021-05-31 2021-08-20 北京经纬恒润科技股份有限公司 限高高度测量方法及装置
CN113284120B (zh) * 2021-05-31 2024-03-08 北京经纬恒润科技股份有限公司 限高高度测量方法及装置

Also Published As

Publication number Publication date
CN106952308A (zh) 2017-07-14
CN106952308B (zh) 2020-02-28

Similar Documents

Publication Publication Date Title
WO2018177159A1 (zh) 运动物体的位置确定方法及系统
CN112292711B (zh) 关联lidar数据和图像数据
US10885659B2 (en) Object pose estimating method and apparatus
US10275649B2 (en) Apparatus of recognizing position of mobile robot using direct tracking and method thereof
US10133279B2 (en) Apparatus of updating key frame of mobile robot and method thereof
US11064178B2 (en) Deep virtual stereo odometry
US9420265B2 (en) Tracking poses of 3D camera using points and planes
US10402724B2 (en) Method for acquiring a pseudo-3D box from a 2D bounding box by regression analysis and learning device and testing device using the same
CN109872366B (zh) 一种物体的三维位置检测方法和装置
US11138742B2 (en) Event-based feature tracking
EP3566172A1 (en) Systems and methods for lane-marker detection
US11887336B2 (en) Method for estimating a relative position of an object in the surroundings of a vehicle and electronic control unit for a vehicle and vehicle
US11062475B2 (en) Location estimating apparatus and method, learning apparatus and method, and computer program products
US20200226392A1 (en) Computer vision-based thin object detection
US11704825B2 (en) Method for acquiring distance from moving body to at least one object located in any direction of moving body by utilizing camera-view depth map and image processing device using the same
CN114972492A (zh) 一种基于鸟瞰图的位姿确定方法、设备和计算机存储介质
CN114648639B (zh) 一种目标车辆的检测方法、系统及装置
US20220335732A1 (en) Method and system for recognizing surrounding driving environment based on svm original image
Al-Harasis et al. On the design and implementation of a dual fisheye camera-based surveillance vision system
US20220245860A1 (en) Annotation of two-dimensional images
WO2021114775A1 (en) Object detection method, object detection device, terminal device, and medium
CN112801077B (zh) 用于自动驾驶车辆的slam初始化的方法及相关装置
WO2022130618A1 (ja) 位置・姿勢推定装置、位置・姿勢推定方法、及びプログラム
WO2022198603A1 (en) Real-time simultaneous localization and mapping using an event camera
Sayem Vision-Aided Navigation for Autonomous Vehicles Using Tracked Feature Points

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18776205

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 24.01.2020)

122 Ep: pct application non-entry in european phase

Ref document number: 18776205

Country of ref document: EP

Kind code of ref document: A1