CN116152301B - Target speed estimation method, device, equipment and medium - Google Patents
Target speed estimation method, device, equipment and medium Download PDFInfo
- Publication number
- CN116152301B CN116152301B CN202310443696.4A CN202310443696A CN116152301B CN 116152301 B CN116152301 B CN 116152301B CN 202310443696 A CN202310443696 A CN 202310443696A CN 116152301 B CN116152301 B CN 116152301B
- Authority
- CN
- China
- Prior art keywords
- frame image
- target
- determining
- current frame
- pixel
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 59
- 230000003287 optical effect Effects 0.000 claims abstract description 85
- 239000011159 matrix material Substances 0.000 claims abstract description 48
- 230000009466 transformation Effects 0.000 claims abstract description 43
- 230000001960 triggered effect Effects 0.000 claims abstract description 38
- 238000005259 measurement Methods 0.000 claims abstract description 31
- 238000001914 filtration Methods 0.000 claims abstract description 22
- 230000033001 locomotion Effects 0.000 claims description 41
- 230000008859 change Effects 0.000 claims description 24
- 238000004590 computer program Methods 0.000 claims description 15
- 238000001514 detection method Methods 0.000 claims description 12
- 238000004422 calculation algorithm Methods 0.000 claims description 11
- 238000003384 imaging method Methods 0.000 claims description 5
- 238000005516 engineering process Methods 0.000 claims description 3
- 238000012545 processing Methods 0.000 description 20
- 230000008569 process Effects 0.000 description 12
- 238000004364 calculation method Methods 0.000 description 11
- 238000004891 communication Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 238000009499 grossing Methods 0.000 description 5
- 230000009471 action Effects 0.000 description 3
- 238000013473 artificial intelligence Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000010191 image analysis Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000004043 responsiveness Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01P—MEASURING LINEAR OR ANGULAR SPEED, ACCELERATION, DECELERATION, OR SHOCK; INDICATING PRESENCE, ABSENCE, OR DIRECTION, OF MOVEMENT
- G01P3/00—Measuring linear or angular speed; Measuring differences of linear or angular speeds
- G01P3/64—Devices characterised by the determination of the time taken to traverse a fixed distance
- G01P3/68—Devices characterised by the determination of the time taken to traverse a fixed distance using optical means, i.e. using infrared, visible, or ultraviolet light
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/269—Analysis of motion using gradient-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/277—Analysis of motion involving stochastic approaches, e.g. using Kalman filters
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20076—Probabilistic image processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30248—Vehicle exterior or interior
- G06T2207/30252—Vehicle exterior; Vicinity of vehicle
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
Abstract
The application discloses a speed estimation method, a device, equipment and a medium of a target, which relate to the technical field of computer vision and comprise the following steps: when the target speed measurement task is triggered under a first preset period time node, a first pixel coordinate and a second pixel coordinate corresponding to a pixel point of a target object in a current frame image and a previous frame image respectively are obtained; the task scheduler is used for controlling the target speed measurement task to trigger after the target optical flow estimation task is completed; determining a homography transformation matrix between the previous frame image and the current frame image based on the first pixel coordinate and the second pixel coordinate; determining the actual longitudinal distance between the target object and the image sensor by using the current frame image, and determining the moving speed of the target object by using the homography transformation matrix and the actual longitudinal distance; and filtering the moving speed by using a preset filter to obtain the filtered moving speed. The method and the device can accurately estimate the target speed and maintain the smoothness of speed estimation.
Description
Technical Field
The present invention relates to the field of computer vision, and in particular, to a method, apparatus, device, and medium for estimating a target speed.
Background
Currently, the amount of the smart camera is increasing on the car with the smart driving function, the smart camera is classified into monocular, binocular and trinocular cameras, the monocular camera is widely adopted because of its low price and small volume, and the challenge is in the estimation of the physical distance and speed of the target based on the smart driving system of the monocular camera.
At present, the distance estimation of a monocular camera to a target in the industry generally adopts the principle of small-hole imaging, and the method has enough stability in a certain distance range. The estimation of the target motion speed mostly adopts the calculation of the position difference of the target between the front frame and the rear frame, and the speed is calculated by using the kinematics principle or is obtained by filtering by using a traditional distance filter. Both of these methods have significant problems: the method for calculating the speed by utilizing the position difference between two frames can lose speed information in the initial period of the target, and the speed value obtained by directly utilizing the position difference calculation is not smooth enough, and the jitter is larger; the velocity filtered with the distance filter also has problems with loss of target initial frame rate information and has hysteresis, as the target moving velocity changes, the filter requires more computation cycles to recognize such changes.
In summary, how to accurately estimate the moving speed of the target, and maintain the smoothness of the speed estimation, without losing the initial frame speed information is a problem to be solved at present.
Disclosure of Invention
In view of the above, an object of the present invention is to provide a method, apparatus, device, and medium for estimating a speed of a target, which can accurately estimate a moving speed of the target, maintain smoothness of the speed estimation, and not lose initial frame speed information. The specific scheme is as follows:
in a first aspect, the present application discloses a method for estimating a speed of a target, including:
when the target speed measurement task is triggered under a first preset period time node, a first pixel coordinate and a second pixel coordinate corresponding to a pixel point of a target object in a current frame image and a previous frame image respectively are obtained; the task scheduler is used for controlling the target speed measurement task to trigger after the target optical flow estimation task is completed; after the target optical flow estimation task is triggered under a second preset period time node, the target optical flow estimation task is used for calculating the first pixel coordinate and the second pixel coordinate;
determining a homography transformation matrix between the previous frame image and the current frame image based on the first pixel coordinates and the second pixel coordinates;
Determining an actual longitudinal distance between the target object and an image sensor by using the current frame image, and determining a moving speed of the target object by using the homography transformation matrix and the actual longitudinal distance;
and filtering the moving speed by using a preset filter to obtain the filtered moving speed.
Optionally, the determining, by using the current frame image, an actual longitudinal distance between the target object and the image sensor includes:
acquiring a bounding box corresponding to a target object, which is obtained by performing target detection on the current frame image by an image sensor, and determining the central point position coordinates, the height information and the width information of the bounding box;
and determining the actual longitudinal distance between the target object and the image sensor by using a small-hole imaging projection technology based on the central point position coordinates, the height information and the width information.
Optionally, before the first pixel coordinates and the second pixel coordinates corresponding to the current frame image and the previous frame image respectively, the acquiring the pixel point of the target object further includes:
after the target optical flow estimation task is triggered, acquiring bounding boxes corresponding to target objects, which are obtained by respectively carrying out target detection on a current frame image and a previous frame image by an image sensor;
And determining a first pixel coordinate and a second pixel coordinate corresponding to each pixel point in the bounding box in the current frame image and the previous frame image respectively.
Optionally, the determining the first pixel coordinate and the second pixel coordinate of each pixel point in the bounding box, which correspond to the current frame image and the previous frame image respectively, includes:
determining optical flow coordinates of each pixel point in the bounding box in the current frame image by using a preset optical flow calculation algorithm, and determining first pixel coordinates of each pixel point in the current frame image;
and determining a second pixel coordinate of each pixel point in the bounding box in the previous frame image based on the optical flow coordinate and the first pixel coordinate.
Optionally, the determining a homography transformation matrix between the previous frame image and the current frame image based on the first pixel coordinates and the second pixel coordinates includes:
determining a scale change rate, a horizontal movement amount and a vertical movement amount of the bounding box between the current frame image and the previous frame image based on the first pixel coordinates and the second pixel coordinates;
and obtaining a homography transformation matrix between the previous frame image and the current frame image based on the scale change rate, the horizontal movement amount and the vertical movement amount.
Optionally, the speed estimation method of the target further includes:
determining a first width value of a bounding box in the current frame image, and determining a second width value of the bounding box in the previous frame image;
a ratio of the first width value to the second width value is determined as the rate of change of dimension.
Optionally, the determining the moving speed of the target object by using the homography transformation matrix and the actual longitudinal distance includes:
determining a time difference between the current frame image and the previous frame image, and determining horizontal pixel coordinates of surrounding frame anchor points in the current frame image;
determining the longitudinal movement speed of the target object by using the scale change rate, the actual longitudinal distance and the time difference;
and determining the transverse moving speed of the target object by using the horizontal moving amount, the actual longitudinal distance, the time difference, the horizontal pixel coordinates and the longitudinal moving speed.
In a second aspect, the present application discloses a speed estimation device of a target, including:
the pixel coordinate acquisition module is used for acquiring a first pixel coordinate and a second pixel coordinate respectively corresponding to a pixel point of a target object in a current frame image and a previous frame image after a target speed measurement task is triggered under a first preset period time node; the task scheduler is used for controlling the target speed measurement task to trigger after the target optical flow estimation task is completed; after the target optical flow estimation task is triggered under a second preset period time node, the target optical flow estimation task is used for calculating the first pixel coordinate and the second pixel coordinate;
The matrix determining module is used for determining a homography transformation matrix between the previous frame image and the current frame image based on the first pixel coordinates and the second pixel coordinates;
the speed determining module is used for determining the actual longitudinal distance between the target object and the image sensor by using the current frame image, and determining the moving speed of the target object by using the homography transformation matrix and the actual longitudinal distance;
and the filtering module is used for filtering the moving speed by using a preset filter so as to obtain the filtered moving speed.
In a third aspect, the present application discloses an electronic device comprising:
a memory for storing a computer program;
a processor for executing the computer program to achieve the steps of the velocity estimation method of the object disclosed above.
In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the steps of the velocity estimation method of the object disclosed previously.
As can be seen, after the target speed measurement task is triggered under the first preset period time node, the method acquires the first pixel coordinate and the second pixel coordinate of the pixel point of the target object, which correspond to the current frame image and the previous frame image respectively; the task scheduler is used for controlling the target speed measurement task to trigger after the target optical flow estimation task is completed; after the target optical flow estimation task is triggered under a second preset period time node, the target optical flow estimation task is used for calculating the first pixel coordinate and the second pixel coordinate; determining a homography transformation matrix between the previous frame image and the current frame image based on the first pixel coordinates and the second pixel coordinates; determining an actual longitudinal distance between the target object and an image sensor by using the current frame image, and determining a moving speed of the target object by using the homography transformation matrix and the actual longitudinal distance; and filtering the moving speed by using a preset filter to obtain the filtered moving speed. Therefore, after the target speed measurement task is triggered, the corresponding target optical flow estimation task is completed, that is, the first pixel coordinate and the second pixel coordinate corresponding to the pixel point of the target object in the current frame image and the previous frame image are obtained through the target optical flow estimation task processing, in the process, the target speed measurement task and the target optical flow estimation task are controlled to be triggered under the corresponding cycle time nodes respectively through the task scheduler, and the target speed measurement task and the target optical flow estimation task are two independent cycle tasks. Further, a homography transformation matrix between the previous frame image and the current frame image is determined based on the obtained first pixel coordinates and the second pixel coordinates, and the actual longitudinal distance between the target object and the image sensor is determined by utilizing the current frame image, so that the moving speed of the target object is determined by utilizing the homography transformation matrix and the actual longitudinal distance, and accurate estimation of the moving speed of the target is realized. And finally, filtering the moving speed by using a preset filter to obtain the moving speed after the smoothing treatment, so that the smoothness of speed estimation is maintained.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present invention, and that other drawings may be obtained according to the provided drawings without inventive effort to a person skilled in the art.
FIG. 1 is a flow chart of a method for estimating the velocity of a target disclosed in the present application;
FIG. 2 is a flow chart of a method for estimating speed of a specific object disclosed in the present application;
FIG. 3 is a timing diagram of a particular task schedule disclosed herein;
FIG. 4 is a schematic diagram of a speed estimation device for a target disclosed in the present application;
fig. 5 is a block diagram of an electronic device disclosed in the present application.
Detailed Description
The following description of the technical solutions in the embodiments of the present application will be made clearly and completely with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Currently, the estimation of the motion speed of the target mostly adopts the calculation of the position difference of the target between the front frame and the rear frame, the calculation of the speed by using the kinematics principle or the filtering of the traditional distance filter to obtain the speed. Both of these methods have significant problems: the method for calculating the speed by utilizing the position difference between two frames can lose speed information in the initial period of the target, and the speed value obtained by directly utilizing the position difference calculation is not smooth enough, and the jitter is larger; the velocity filtered with the distance filter also has problems with loss of target initial frame rate information and has hysteresis, as the target moving velocity changes, the filter requires more computation cycles to recognize such changes. Therefore, the embodiment of the application discloses a speed estimation method, a device, equipment and a medium of a target, which can accurately estimate the moving speed of the target, keep the smoothness of speed estimation and do not lose initial frame speed information.
Referring to fig. 1, an embodiment of the present application discloses a method for estimating a speed of a target, including:
step S11: when the target speed measurement task is triggered under a first preset period time node, a first pixel coordinate and a second pixel coordinate corresponding to a pixel point of a target object in a current frame image and a previous frame image respectively are obtained; the task scheduler is used for controlling the target speed measurement task to trigger after the target optical flow estimation task is completed; and after the target optical flow estimation task is triggered under a second preset period time node, the target optical flow estimation task is used for calculating the first pixel coordinate and the second pixel coordinate.
In this embodiment, when the target speed measurement task is triggered at the first preset period time node and is used for performing speed estimation on the target in the current frame image, the corresponding target optical flow estimation task is completed, that is, the first pixel coordinate and the second pixel coordinate corresponding to the pixel point of the target object in the current frame image and the previous frame image are obtained through the processing of the target optical flow estimation task, so that the first pixel coordinate and the second pixel coordinate are obtained. In addition, the task scheduler controls the triggering of the tasks, so that good real-time responsiveness can be ensured.
In the above process, the task scheduler controls the target speed measurement task and the target optical flow estimation task to be triggered under the first preset period time node and the second preset period time node respectively, wherein the target speed measurement task and the target optical flow estimation task are two independent period tasks, and the time periods of the two tasks are inconsistent. Because the task is triggered under the preset period time node, the corresponding speed can be estimated even in the initial period, namely, the speed estimation of the target motion can be completed from the initial period, and the initial frame speed information is not lost. Fig. 2 is a specific task scheduling timing diagram, referring to fig. 2, where task a is a target speed measurement task, and task c is a target optical flow estimation task, it can be seen from the figure that when task a is triggered, that is, when a new frame of image is received to perform distance speed estimation on a target in task, task c has processed a current frame of image and a previous frame of image, and calculated an optical flow, that is, the scheduling period of the target optical flow estimation task is twice or more than that of the target speed measurement task, and when a task scheduler ensures that each frame of image is acquired and speed estimation is performed on the target, optical flow change between two continuous frames can be obtained from the optical flow task. For example, when frame2 in the figure is subjected to distance velocity processing, optical flow information from frame1 to frame2 is present, and when frame6 is subjected to distance velocity processing, optical flow information from frame5 to frame6 is present. Thus, one key dependency point of the present application is to obtain the optical flow changes of the objects, and assuming that the objects on the frame t picture are being processed, the optical flow change rate of these objects can be obtained from the difference between the frame t-1 and the frame t picture, and it can be seen from fig. 2 that the task scheduler ensures that the optical flow changes from frame t-1 to frame t are always present.
Step S12: and determining a homography transformation matrix between the previous frame image and the current frame image based on the first pixel coordinates and the second pixel coordinates.
In this embodiment, the homography transformation matrix between the previous frame image and the current frame image is determined based on the first pixel coordinates and the second pixel coordinates corresponding to the pixel points of the target object in the current frame image and the previous frame image, respectively.
Step S13: and determining the actual longitudinal distance between the target object and the image sensor by using the current frame image, and determining the moving speed of the target object by using the homography transformation matrix and the actual longitudinal distance.
In this embodiment, the image acquisition is based on the image sensor, so that the actual longitudinal distance between the target object and the image sensor in the current frame image is determined, and the movement speed of the target object is determined by using the homography transformation matrix and the actual longitudinal distance, so as to realize accurate estimation of the movement speed of the target. The image sensor may specifically be a monocular camera, and the actual longitudinal distance is a distance between the monocular camera and the target object in the real world.
Further, the determining the actual longitudinal distance between the target object and the image sensor in the current frame image includes: acquiring a bounding box corresponding to a target object, which is obtained by performing target detection on the current frame image by an image sensor, and determining the central point position coordinates, the height information and the width information of the bounding box; and determining the actual longitudinal distance between the target object and the image sensor by using a small-hole imaging projection technology based on the central point position coordinates, the height information and the width information. It will be appreciated that the detection result of the image sensor on the target is given in the form of a bounding box (i.e. a bounding box), and the detection process may be specifically performed in the task b image processing task in fig. 2. According to the center point position coordinates and the high width information of the bounding box, the actual longitudinal distance of the target in the physical space can be calculated by utilizing the principle of small hole imaging projection dxAnd actual lateral distancedy。
Step S14: and filtering the moving speed by using a preset filter to obtain the filtered moving speed.
In this embodiment, the moving speed is filtered by using a preset filter to obtain a moving speed after the smoothing processing, so that the smoothness of speed estimation is maintained; the preset filter may specifically be a kalman filter. In addition, instead of performing the filtering process on the velocity, the obtained actual longitudinal distance and the obtained actual lateral distance may also be subjected to the filtering process to obtain a smoothed distance value.
As can be seen, after the target speed measurement task is triggered under the first preset period time node, the method acquires the first pixel coordinate and the second pixel coordinate of the pixel point of the target object, which correspond to the current frame image and the previous frame image respectively; the task scheduler is used for controlling the target speed measurement task to trigger after the target optical flow estimation task is completed; after the target optical flow estimation task is triggered under a second preset period time node, the target optical flow estimation task is used for calculating the first pixel coordinate and the second pixel coordinate; determining a homography transformation matrix between the previous frame image and the current frame image based on the first pixel coordinates and the second pixel coordinates; determining an actual longitudinal distance between the target object and an image sensor by using the current frame image, and determining a moving speed of the target object by using the homography transformation matrix and the actual longitudinal distance; and filtering the moving speed by using a preset filter to obtain the filtered moving speed. Therefore, after the visual speed measuring task is triggered, the corresponding target optical flow estimation task is completed, that is, the first pixel coordinate and the second pixel coordinate corresponding to the pixel point of the target object in the current frame image and the previous frame image are obtained through the processing of the target optical flow estimation task, in the process, the target speed measuring task and the target optical flow estimation task are controlled to be triggered under the corresponding cycle time nodes respectively through the task scheduler, and the target speed measuring task and the target optical flow estimation task are two independent cycle tasks. Further, a homography transformation matrix between the previous frame image and the current frame image is determined based on the obtained first pixel coordinates and the second pixel coordinates, and the actual longitudinal distance between the target object and the image sensor is determined by utilizing the current frame image, so that the moving speed of the target object is determined by utilizing the homography transformation matrix and the actual longitudinal distance, and accurate estimation of the moving speed of the target is realized. And finally, filtering the moving speed by using a preset filter to obtain the moving speed after the smoothing treatment, so that the smoothness of speed estimation is maintained.
Referring to fig. 3, an embodiment of the present application discloses a specific target speed estimation method, and in comparison with the previous embodiment, the present embodiment further describes and optimizes a technical solution. The method specifically comprises the following steps:
step S21: when the target speed measurement task is triggered under a first preset period time node, a first pixel coordinate and a second pixel coordinate corresponding to a pixel point of a target object in a current frame image and a previous frame image respectively are obtained; the task scheduler is used for controlling the target speed measurement task to trigger after the target optical flow estimation task is completed; and after the target optical flow estimation task is triggered under a second preset period time node, the target optical flow estimation task is used for calculating the first pixel coordinate and the second pixel coordinate.
In this embodiment, before the first pixel coordinates and the second pixel coordinates corresponding to the current frame image and the previous frame image, the acquiring pixel point of the target object further includes: after the target optical flow estimation task is triggered, acquiring bounding boxes corresponding to target objects, which are obtained by respectively carrying out target detection on a current frame image and a previous frame image by an image sensor; and determining a first pixel coordinate and a second pixel coordinate corresponding to each pixel point in the bounding box in the current frame image and the previous frame image respectively. It should be noted that, the object speed measurement task needs to rely on the optical flow of the object as necessary input information, the optical flow estimation is used as a classical traditional algorithm for motion image analysis in the computer vision industry, and currently is already supported by most intelligent driving chips, without needing deep knowledge and technical realization capability of developers on the essence of the optical flow algorithm, only the related technical support algorithm provided by a chip manufacturer needs to be called, so that an optical flow calculation result of an image can be obtained, and the popular meaning is that each pixel point of the optical flow calculation result has corresponding motion when the object moves in the position of a picture. Therefore, after the target optical flow estimation task is triggered, acquiring bounding boxes corresponding to the target objects, which are obtained by respectively carrying out target detection on the current frame image and the previous frame image by the image sensor in the image processing task, and respectively carrying out optical flow estimation to determine first pixel coordinates and second pixel coordinates respectively corresponding to each pixel point in the bounding boxes in the current frame image and the previous frame image.
Further, when determining the first pixel coordinates and the second pixel coordinates corresponding to each pixel point in the bounding box in the current frame image and the previous frame image respectively, the method specifically includes: determining optical flow coordinates of each pixel point in the bounding box in the current frame image by using a preset optical flow calculation algorithm, and determining first pixel coordinates of each pixel point in the current frame image; and determining a second pixel coordinate of each pixel point in the bounding box in the previous frame image based on the optical flow coordinate and the first pixel coordinate. Firstly, determining the optical flow coordinates of each pixel point in the bounding box in the current frame image by using a preset optical flow calculation algorithmdu，dv) Determining the first pixel coordinates of each pixel point in the current frame imageU t ，V t ). Wherein, it should be pointed out that the original representation form of the optical flow is similar to a picture with the same size, and each point in the picture is a vector #du，dv) The moving speeds of the pixel points in the horizontal U direction and the vertical V direction are respectively shown, and each pixel point exists within a given surrounding frame rangedu，dv) Information; in addition, in the case of the optical fiber,U t to enclose the horizontal coordinates of a pixel point within the frame in the current frame image, V t The vertical coordinates of the pixel point in the current frame image. And then according to lightDetermining second pixel coordinates of each pixel point in the bounding box in the previous frame image according to the flow coordinates and the first pixel coordinatesU t-1 ，V t-1 ) The relation is as follows:
U t =U t-1 +du；
V t =V t-1 +dv；
wherein,,U t-1 to enclose the horizontal coordinates of a pixel point within the frame in the previous frame image,V t-1 is the vertical coordinate of the pixel point in the previous frame of image.
Step S22: and determining the scale change rate, the horizontal movement amount and the vertical movement amount of the bounding box between the current frame image and the previous frame image based on the first pixel coordinates and the second pixel coordinates, and obtaining a homography transformation matrix between the previous frame image and the current frame image based on the scale change rate, the horizontal movement amount and the vertical movement amount.
In this embodiment, after the first pixel coordinate and the second pixel coordinate are determined by using the optical flow information of the points in the bounding box, the scale change rate, the horizontal movement amount and the vertical movement amount between the current frame image and the previous frame image can be determined, and the homography transformation matrix between the previous frame image and the current frame image is obtained based on the scale change rate, the horizontal movement amount and the vertical movement amount.
The calculation equation is as follows:
wherein,,s，T x ，T y that is, elements of the homography transformation matrix H of three degrees of freedom respectively represent a scale change rate, a horizontal movement amount and a vertical movement amount of the bounding box between the current frame image and the previous frame image, and the matrix H is:
it should be noted that when solving the elements of the homography transformation matrix, the first pixel coordinates and the second pixel coordinates corresponding to the plurality of pixel points can be given, that is, the problem is transformed into a given batch of #du，dv) And%U t ，V t ) Data and solving the above equation to obtain homography transformation matrixs，T x ，T y An element. This is an overdetermined problem, which requires solving equations by least squares or other algorithms, thereby obtainings，T x ，T y Is a value of (2).
In calculating the scale change rate, specifically, it may include: determining a first width value of a bounding box in the current frame image, and determining a second width value of the bounding box in the previous frame image; a ratio of the first width value to the second width value is determined as the rate of change of dimension. It will be appreciated that for two bounding boxes of a given two frames, the rate of change of scale is equal to the ratio of the widths of the bounding boxes. That is, a first width value and a second width value of the bounding box in the current frame image and the previous frame image respectively are determined, and the ratio of the two values is the scale change rate.
Step S23: and determining the actual longitudinal distance between the target object and the image sensor by using the current frame image, determining the time difference between the current frame image and the previous frame image, and determining the horizontal pixel coordinates of surrounding frame anchor points in the current frame image.
In this embodiment, the actual longitudinal distance between the target object and the image sensor in the current frame image is determineddxTime difference between current frame image and previous frame image
And horizontal pixel coordinates of bounding box anchor in the current frame image +.>
。
Step S24: and determining the longitudinal movement speed of the target object by using the scale change rate, the actual longitudinal distance and the time difference, and determining the transverse movement speed of the target object by using the horizontal movement amount, the actual longitudinal distance, the time difference, the horizontal pixel coordinates and the longitudinal movement speed.
In this embodiment, in obtaining the homography transformation matrixs，T x ，T y And after the elements, determining the moving speed of the target according to the actual longitudinal distance between the target object and the image sensor in the current frame image, the time difference between the current frame image and the previous frame image and the horizontal pixel coordinates of the surrounding frame anchor point in the current frame image, wherein the moving speed comprises the longitudinal moving speed and the transverse moving speed.
In particular, the rate of change of dimension is usedsActual longitudinal distancedxSum and time difference
Determining the longitudinal movement speed of the target objectv x The calculation formula is as follows:
by means of horizontal movementT x Actual longitudinal distancedxTime difference
Horizontal pixel coordinates->
And a longitudinal movement speedv x Determining the lateral movement speed of the target objectv y The calculation formula is as follows:
step S25: and filtering the transverse moving speed and the longitudinal moving speed by using a preset filter to obtain the filtered transverse moving speed and the filtered longitudinal moving speed.
In the embodiment, the transverse and longitudinal distance and the transverse and longitudinal moving speed of the target are calculated through the process, and a vector is formeddx，dy，v x ，v y ) And inputting the vector into a traditional preset filter algorithm and the like to obtain the smoothed distance and speed.
For more specific processing in step S21, reference may be made to the corresponding content disclosed in the foregoing embodiment, and no further description is given here.
Therefore, after the target optical flow estimation task is triggered, a bounding box corresponding to the target object, which is obtained by respectively performing target detection on the current frame image and the previous frame image by the image sensor in the image processing task, is acquired, and optical flow estimation is performed respectively, so as to determine a first pixel coordinate and a second pixel coordinate corresponding to each pixel point in the bounding box in the current frame image and the previous frame image respectively. And determining the scale change rate, the horizontal movement amount and the vertical movement amount between the current frame image and the previous frame image based on the first pixel coordinates and the second pixel coordinates, and further obtaining a homography transformation matrix. And finally, determining the transverse and longitudinal movement speed of the target object by utilizing parameters such as a homography transformation matrix, an actual longitudinal distance and the like, and performing smoothing filtering treatment. In this way, the moving speed of the target is directly calculated by using the correlation principle of computer vision based on the image sensor. Compared with the traditional speed measuring method, the method has the characteristics of smoother calculation result, better real-time response and no loss of initial frame speed information.
Referring to fig. 4, an embodiment of the present application discloses a speed estimation device of a target, including:
the pixel coordinate acquiring module 11 is configured to acquire a first pixel coordinate and a second pixel coordinate corresponding to a pixel point of a target object in a current frame image and a previous frame image respectively after the target speed measurement task is triggered at a first preset period time node; the task scheduler is used for controlling the target speed measurement task to trigger after the target optical flow estimation task is completed; after the target optical flow estimation task is triggered under a second preset period time node, the target optical flow estimation task is used for calculating the first pixel coordinate and the second pixel coordinate;
a matrix determining module 12, configured to determine a homography transformation matrix between the previous frame image and the current frame image based on the first pixel coordinates and the second pixel coordinates;
a speed determining module 13, configured to determine an actual longitudinal distance between the target object and the image sensor using the current frame image, and determine a moving speed of the target object using the homography transformation matrix and the actual longitudinal distance;
and the filtering module 14 is configured to perform filtering processing on the moving speed by using a preset filter, so as to obtain a filtered moving speed.
As can be seen, after the target speed measurement task is triggered under the first preset period time node, the method acquires the first pixel coordinate and the second pixel coordinate of the pixel point of the target object, which correspond to the current frame image and the previous frame image respectively; the task scheduler is used for controlling the target speed measurement task to trigger after the target optical flow estimation task is completed; after the target optical flow estimation task is triggered under a second preset period time node, the target optical flow estimation task is used for calculating the first pixel coordinate and the second pixel coordinate; determining a homography transformation matrix between the previous frame image and the current frame image based on the first pixel coordinates and the second pixel coordinates; determining an actual longitudinal distance between the target object and an image sensor by using the current frame image, and determining a moving speed of the target object by using the homography transformation matrix and the actual longitudinal distance; and filtering the moving speed by using a preset filter to obtain the filtered moving speed. Therefore, after the visual speed measuring task is triggered, the corresponding target optical flow estimation task is completed, that is, the first pixel coordinate and the second pixel coordinate corresponding to the pixel point of the target object in the current frame image and the previous frame image are obtained through the processing of the target optical flow estimation task, in the process, the target speed measuring task and the target optical flow estimation task are controlled to be triggered under the corresponding cycle time nodes respectively through the task scheduler, and the target speed measuring task and the target optical flow estimation task are two independent cycle tasks. Further, a homography transformation matrix between the previous frame image and the current frame image is determined based on the obtained first pixel coordinates and the second pixel coordinates, and the actual longitudinal distance between the target object and the image sensor is determined by utilizing the current frame image, so that the moving speed of the target object is determined by utilizing the homography transformation matrix and the actual longitudinal distance, and accurate estimation of the moving speed of the target is realized. And finally, filtering the moving speed by using a preset filter to obtain the moving speed after the smoothing treatment, so that the smoothness of speed estimation is maintained.
Fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present application. Specifically, the method comprises the following steps: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input output interface 25, and a communication bus 26. Wherein the memory 22 is adapted to store a computer program to be loaded and executed by the processor 21 for implementing the relevant steps of the method for estimating the speed of a target to be executed by an electronic device as disclosed in any of the previous embodiments.
In this embodiment, the power supply 23 is configured to provide an operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and an external device, and the communication protocol to be followed is any communication protocol applicable to the technical solution of the present application, which is not specifically limited herein; the input/output interface 25 is used for acquiring external input data or outputting external output data, and the specific interface type thereof may be selected according to the specific application requirement, which is not limited herein.
Processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 may be implemented in at least one hardware form of DSP (Digital Signal Processing ), FPGA (Field-Programmable Gate Array, field programmable gate array), PLA (Programmable Logic Array ). The processor 21 may also comprise a main processor, which is a processor for processing data in an awake state, also called CPU (Central Processing Unit ); a coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 21 may integrate a GPU (Graphics Processing Unit, image processor) for rendering and drawing of content required to be displayed by the display screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence ) processor for processing computing operations related to machine learning.
The memory 22 may be a carrier for storing resources, such as a read-only memory, a random access memory, a magnetic disk, or an optical disk, and the resources stored thereon include an operating system 221, a computer program 222, and data 223, and the storage may be temporary storage or permanent storage.
The operating system 221 is used for managing and controlling various hardware devices on the electronic device 20 and the computer program 222, so as to implement the operation and processing of the processor 21 on the mass data 223 in the memory 22, which may be Windows, unix, linux. The computer program 222 may further include a computer program that can be used to perform other specific tasks in addition to the computer program that can be used to perform the speed estimation method of the object performed by the electronic device 20 as disclosed in any of the previous embodiments. The data 223 may include, in addition to data received by the electronic device and transmitted by the external device, data collected by the input/output interface 25 itself, and so on.
Further, the embodiments of the present application also disclose a computer readable storage medium, in which a computer program is stored, which when loaded and executed by a processor, implements the method steps disclosed in any of the foregoing embodiments and executed in the speed estimation process of the target.
In this specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, so that the same or similar parts between the embodiments are referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative elements and steps are described above generally in terms of functionality in order to clearly illustrate the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software modules may be disposed in Random Access Memory (RAM), memory, read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
Finally, it is further noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The above description of the speed estimation method, apparatus, device and storage medium for an object provided by the present invention has been provided in detail, and specific examples are applied herein to illustrate the principles and embodiments of the present invention, and the above examples are only used to help understand the method and core idea of the present invention; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope in accordance with the ideas of the present invention, the present description should not be construed as limiting the present invention in view of the above.
Claims (9)
1. A method of estimating a velocity of a target, comprising:
when the target speed measurement task is triggered under a first preset period time node, a first pixel coordinate and a second pixel coordinate corresponding to a pixel point of a target object in a current frame image and a previous frame image respectively are obtained; the task scheduler is used for controlling the target speed measurement task to trigger after the target optical flow estimation task is completed; after the target optical flow estimation task is triggered under a second preset period time node, the target optical flow estimation task is used for calculating the first pixel coordinate and the second pixel coordinate;
determining a homography transformation matrix between the previous frame image and the current frame image based on the first pixel coordinates and the second pixel coordinates;
determining an actual longitudinal distance between the target object and an image sensor by using the current frame image, and determining a moving speed of the target object by using the homography transformation matrix and the actual longitudinal distance;
filtering the moving speed by using a preset filter to obtain a filtered moving speed;
after the target optical flow estimation task is triggered, acquiring bounding boxes corresponding to target objects, which are obtained by respectively carrying out target detection on a current frame image and a previous frame image by an image sensor;
Correspondingly, the determining the homography transformation matrix between the previous frame image and the current frame image based on the first pixel coordinate and the second pixel coordinate includes:
determining a scale change rate, a horizontal movement amount and a vertical movement amount of the bounding box between the current frame image and the previous frame image based on the first pixel coordinates and the second pixel coordinates;
and obtaining a homography transformation matrix between the previous frame image and the current frame image based on the scale change rate, the horizontal movement amount and the vertical movement amount.
2. The method of claim 1, wherein determining an actual longitudinal distance between the target object and an image sensor using the current frame image comprises:
acquiring a bounding box corresponding to a target object, which is obtained by performing target detection on the current frame image by an image sensor, and determining the central point position coordinates, the height information and the width information of the bounding box;
and determining the actual longitudinal distance between the target object and the image sensor by using a small-hole imaging projection technology based on the central point position coordinates, the height information and the width information.
3. The method for estimating a velocity of a target according to claim 1, wherein the acquiring the pixel point of the target object further includes, before the first pixel coordinate and the second pixel coordinate corresponding to the current frame image and the previous frame image, respectively:
after the target optical flow estimation task is triggered, acquiring bounding boxes corresponding to target objects, which are obtained by respectively carrying out target detection on a current frame image and a previous frame image by an image sensor;
and determining a first pixel coordinate and a second pixel coordinate corresponding to each pixel point in the bounding box in the current frame image and the previous frame image respectively.
4. The method for estimating a velocity of a target according to claim 3, wherein determining the first pixel coordinate and the second pixel coordinate of each pixel point in the bounding box, which correspond to the current frame image and the previous frame image respectively, includes:
determining optical flow coordinates of each pixel point in the bounding box in the current frame image by using a preset optical flow calculation algorithm, and determining first pixel coordinates of each pixel point in the current frame image;
and determining a second pixel coordinate of each pixel point in the bounding box in the previous frame image based on the optical flow coordinate and the first pixel coordinate.
5. The method for estimating the velocity of the object according to claim 1, further comprising:
determining a first width value of a bounding box in the current frame image, and determining a second width value of the bounding box in the previous frame image;
a ratio of the first width value to the second width value is determined as the rate of change of dimension.
6. The method of speed estimation of a target according to claim 1, wherein the determining the moving speed of the target object using the homography transformation matrix and the actual longitudinal distance includes:
determining a time difference between the current frame image and the previous frame image, and determining horizontal pixel coordinates of surrounding frame anchor points in the current frame image;
determining the longitudinal movement speed of the target object by using the scale change rate, the actual longitudinal distance and the time difference;
and determining the transverse moving speed of the target object by using the horizontal moving amount, the actual longitudinal distance, the time difference, the horizontal pixel coordinates and the longitudinal moving speed.
7. A speed estimation device of an object, comprising:
the pixel coordinate acquisition module is used for acquiring a first pixel coordinate and a second pixel coordinate respectively corresponding to a pixel point of a target object in a current frame image and a previous frame image after a target speed measurement task is triggered under a first preset period time node; the task scheduler is used for controlling the target speed measurement task to trigger after the target optical flow estimation task is completed; after the target optical flow estimation task is triggered under a second preset period time node, the target optical flow estimation task is used for calculating the first pixel coordinate and the second pixel coordinate;
The matrix determining module is used for determining a homography transformation matrix between the previous frame image and the current frame image based on the first pixel coordinates and the second pixel coordinates;
the speed determining module is used for determining the actual longitudinal distance between the target object and the image sensor by using the current frame image, and determining the moving speed of the target object by using the homography transformation matrix and the actual longitudinal distance;
the filtering module is used for filtering the moving speed by using a preset filter so as to obtain a filtered moving speed;
wherein the device is further for: after the target optical flow estimation task is triggered, acquiring bounding boxes corresponding to target objects, which are obtained by respectively carrying out target detection on a current frame image and a previous frame image by an image sensor;
correspondingly, the matrix determining module is used for: determining a scale change rate, a horizontal movement amount and a vertical movement amount of the bounding box between the current frame image and the previous frame image based on the first pixel coordinates and the second pixel coordinates; and obtaining a homography transformation matrix between the previous frame image and the current frame image based on the scale change rate, the horizontal movement amount and the vertical movement amount.
8. An electronic device, comprising:
a memory for storing a computer program;
processor for executing the computer program to carry out the steps of the method of speed estimation of an object according to any one of claims 1 to 6.
9. A computer-readable storage medium storing a computer program; wherein the computer program when executed by a processor realizes the steps of the speed estimation method of an object according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310443696.4A CN116152301B (en) | 2023-04-24 | 2023-04-24 | Target speed estimation method, device, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310443696.4A CN116152301B (en) | 2023-04-24 | 2023-04-24 | Target speed estimation method, device, equipment and medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116152301A CN116152301A (en) | 2023-05-23 |
| CN116152301B true CN116152301B (en) | 2023-07-14 |
Family
ID=86354760
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310443696.4A Active CN116152301B (en) | 2023-04-24 | 2023-04-24 | Target speed estimation method, device, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116152301B (en) |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110062272A (en) * | 2019-04-30 | 2019-07-26 | 腾讯科技（深圳）有限公司 | A kind of video data handling procedure and relevant apparatus |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111275743B (en) * | 2020-01-20 | 2024-03-12 | 深圳奇迹智慧网络有限公司 | Target tracking method, device, computer readable storage medium and computer equipment |
| CN115104125A (en) * | 2020-02-19 | 2022-09-23 | 华为技术有限公司 | Optical flow acquisition method and device |
| CN111897429A (en) * | 2020-07-30 | 2020-11-06 | 腾讯科技（深圳）有限公司 | Image display method, image display device, computer equipment and storage medium |
| CN112001949B (en) * | 2020-08-13 | 2023-12-05 | 地平线(上海)人工智能技术有限公司 | Method, device, readable storage medium and equipment for determining target point moving speed |
| CN113470374B (en) * | 2021-06-30 | 2022-09-16 | 平安科技（深圳）有限公司 | Vehicle overspeed monitoring method and device, computer equipment and storage medium |
| CN115760912A (en) * | 2022-11-03 | 2023-03-07 | 北京灵汐科技有限公司 | Moving object tracking method, device, equipment and computer readable storage medium |
-
2023
- 2023-04-24 CN CN202310443696.4A patent/CN116152301B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110062272A (en) * | 2019-04-30 | 2019-07-26 | 腾讯科技（深圳）有限公司 | A kind of video data handling procedure and relevant apparatus |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116152301A (en) | 2023-05-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11668571B2 (en) | Simultaneous localization and mapping (SLAM) using dual event cameras | |
| JP7369847B2 (en) | Data processing methods and devices, electronic devices, storage media, computer programs, and self-driving vehicles for self-driving vehicles | |
| JP6198230B2 (en) | Head posture tracking using depth camera | |
| CN111665842B (en) | Indoor SLAM mapping method and system based on semantic information fusion | |
| WO2022241874A1 (en) | Infrared thermal imaging monocular vision ranging method and related assembly | |
| CN109308469B (en) | Method and apparatus for generating information | |
| CN113377888B (en) | Method for training object detection model and detection object | |
| CN113286194A (en) | Video processing method and device, electronic equipment and readable storage medium | |
| JP2017223648A (en) | Reducing power consumption for time-of-flight depth imaging | |
| WO2018205803A1 (en) | Pose estimation method and apparatus | |
| CN105953796A (en) | Stable motion tracking method and stable motion tracking device based on integration of simple camera and IMU (inertial measurement unit) of smart cellphone | |
| US20190301871A1 (en) | Direct Sparse Visual-Inertial Odometry Using Dynamic Marginalization | |
| CN109883423A (en) | Localization method, system, equipment and storage medium based on Kalman filtering | |
| CN108665541A (en) | A kind of ground drawing generating method and device and robot based on laser sensor | |
| CN114399539B (en) | Method, apparatus and storage medium for detecting moving object | |
| CN109040525B (en) | Image processing method, image processing device, computer readable medium and electronic equipment | |
| US11494961B2 (en) | Sticker generating method and apparatus, and medium and electronic device | |
| CN113610702B (en) | Picture construction method and device, electronic equipment and storage medium | |
| CN115088244A (en) | Depth sensor activation for localization based on data from monocular camera | |
| US12113954B2 (en) | Processing of signals using a recurrent state estimator | |
| CN116152301B (en) | Target speed estimation method, device, equipment and medium | |
| KR20240056757A (en) | Efficient orientation tracking using future orientation prediction | |
| CN114494574A (en) | Deep learning monocular three-dimensional reconstruction method and system based on multi-loss function constraint | |
| KR20130124837A (en) | Method and apparatus for removing motion blur of image | |
| EP4344227A1 (en) | Video frame interpolation method and apparatus, and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |