CN114898281A - Basketball hoop abnormity detection method and system based on computer vision - Google Patents
Basketball hoop abnormity detection method and system based on computer vision Download PDFInfo
- Publication number
- CN114898281A CN114898281A CN202210818930.2A CN202210818930A CN114898281A CN 114898281 A CN114898281 A CN 114898281A CN 202210818930 A CN202210818930 A CN 202210818930A CN 114898281 A CN114898281 A CN 114898281A
- Authority
- CN
- China
- Prior art keywords
- basketball
- target
- frame
- hoop
- rim
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/41—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items
- G06V20/42—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items of sport video content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/70—Labelling scene content, e.g. deriving syntactic or semantic representations
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Artificial Intelligence (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Image Analysis (AREA)
Abstract
The invention relates to a basketball rim abnormity detection method and system based on computer vision, and relates to the technical field of basketball rim abnormity detection. The method comprises the following steps: obtaining morphological indexes corresponding to the target basketball rims according to pixel point differences between the target basketball rims and the corresponding standard circles; calculating the minimum circumscribed rectangle of each target basketball frame corresponding to the last frame of basketball court panoramic image; obtaining a bending degree index corresponding to each target basketball hoop according to the difference between the minimum external matrix shape of each target basketball hoop and the corresponding standard minimum external moment; inputting each frame of basketball court panoramic image into a key point detection network to obtain a key point thermodynamic diagram corresponding to each frame of basketball court panoramic image; obtaining a heat index corresponding to each target basketball hoop according to the key point thermodynamic diagram; and obtaining the abnormal degree of each target basketball hoop according to the form index, the bending degree index and the heat index corresponding to each target basketball hoop. The method and the device can improve the accuracy of basketball hoop abnormity detection.
Description
Technical Field
The invention relates to the technical field of basketball rim abnormity detection, in particular to a basketball rim abnormity detection method and system based on computer vision.
Background
Basketball is one of the most popular sports, and not only can the reaction ability, decision-making ability and observation ability of people be improved, but also the team cooperation ability can be enhanced. Shooting is a key technology in the basketball game and is the only scoring means in the basketball game; therefore, when the basketball hoop has abnormal problems such as deformation of the basketball hoop or bending of the basketball hoop due to the external factors and the self factors, the goal rate of people in the process of playing the basketball can be influenced, and the exercise and competition effects of basketball fans or basketball players can be further influenced.
The existing basketball rim abnormality detection method generally manages and detects basketball rims by managers of a basketball stadium, but the managers can generally find that the basketball rims are abnormal only when the basketball rims are obviously abnormal, so that some basketball rims with the abnormality are easily judged as basketball rims without the abnormality, and the detection result of whether the basketball rims have the abnormality is not accurate enough.
Disclosure of Invention
The invention provides a basketball rim abnormity detection method and system based on computer vision, which are used for solving the problem that the existing basketball rim abnormity cannot be accurately detected, and the adopted technical scheme is as follows:
in a first aspect, an embodiment of the present invention provides a method and a system for detecting abnormal basketball rim based on computer vision, including the following steps:
acquiring a multi-frame basketball court panoramic image within a set time period;
obtaining each target basketball frame in the basketball court and a standard circle corresponding to each target basketball frame according to the last frame of basketball court panoramic image; obtaining morphological indexes corresponding to the target basketball rims according to pixel point differences between the target basketball rims and the corresponding standard circles;
calculating the minimum circumscribed rectangle of each target basketball frame corresponding to the last frame of basketball court panoramic image; obtaining the bending degree index corresponding to each target basketball hoop according to the difference between the minimum external matrix shape of each target basketball hoop and the corresponding standard minimum external moment;
inputting each frame of basketball court panoramic image into a key point detection network to obtain a key point thermodynamic diagram corresponding to each frame of basketball court panoramic image; obtaining a heat index corresponding to each target basketball hoop according to the key point thermodynamic diagram;
and obtaining the abnormal degree of each target basketball hoop according to the form index, the bending degree index and the heat index corresponding to each target basketball hoop.
The invention also provides a basketball rim abnormality detection system based on computer vision, which comprises a memory and a processor, wherein the processor executes a computer program stored in the memory to realize the basketball rim abnormality detection method based on computer vision.
According to the difference of pixel points between the target basketball hoop and the corresponding standard circle, the form index corresponding to each target basketball hoop is obtained; obtaining a bending degree index corresponding to each target basketball hoop according to the difference between the minimum external matrix shape of each target basketball hoop and the corresponding standard minimum external moment; inputting each frame of basketball court panoramic image into a key point detection network to obtain a key point thermodynamic diagram corresponding to each frame of basketball court panoramic image; obtaining a heat index corresponding to each target basketball hoop according to the key point thermodynamic diagram; and obtaining the abnormal degree of each target basketball hoop according to the form index, the bending degree index and the heat index corresponding to each target basketball hoop. The form index, the bending degree index and the heat index are used as the basis for obtaining the abnormal degree of each target basketball rim, so that the accuracy of abnormal detection of the basketball rim can be improved; the pixel point difference between the target basketball rim and the corresponding standard circle is used as a basis for obtaining the form index corresponding to each target basketball rim, so that the calculation amount can be reduced, and the efficiency of abnormal basketball rim detection can be improved.
Preferably, obtaining each target basketball rim in the basketball court and a standard circle corresponding to each target basketball rim according to the last frame of panoramic image of the basketball court comprises:
inputting the last frame of basketball court panoramic image into a semantic perception network to obtain each target basketball frame in the basketball court;
detecting the last frame of basketball court panoramic image according to a Hough transform algorithm to obtain each detection circle corresponding to each target basketball hoop in the last frame of basketball court panoramic image;
obtaining the number of pixel points which are not on each detection circle in each target basketball frame and the number of pixel points on each detection circle in each target basketball frame; calculating the sum of the shortest distances from all pixel points which are not on all detection circles in all target basketball frames to all detection circles;
and obtaining a standard circle corresponding to each target basketball hoop according to the sum of the number of the pixel points on each detection circle in each target basketball hoop and the shortest distance from each pixel point on each detection circle in each target basketball hoop to each detection circle.
Preferably, the form index corresponding to each target basketball rim is calculated according to the following formula:
wherein the content of the first and second substances,
is as follows
The shape index corresponding to each target ball frame,
is a first
The object is not in the basketball frame
The number of pixel points on the standard circle corresponding to the target basketball rim,
is as follows
Out of the basketball rim
Corresponds to a basketball hoopThe sum of the shortest distances from each pixel point on the standard circle to the corresponding standard circle.
Preferably, the construction process of the key point detection network includes:
acquiring a training sample set, wherein the training sample comprises a multi-frame basketball court sample panoramic image;
marking the positions of key points in panoramic images of basketball court samples of each frame in the training sample set; the marked key points are the head of the human body, the hand of the human body and a target basketball frame;
and inputting the training sample set and the labeled data into a network, and training by adopting a loss function to obtain a trained key point detection network.
Preferably, the method for obtaining the heat index corresponding to each target basketball rim includes:
obtaining a movement area corresponding to each target basketball hoop according to the panoramic image of each basketball court;
obtaining the number of key points in a motion area corresponding to each target basketball hoop in each basketball court panoramic image according to the key point thermodynamic diagram, and taking the number of the key points as the corresponding heat value of each target basketball hoop in each basketball court panoramic image;
constructing a heat value sequence corresponding to each target basketball hoop according to the heat values; obtaining the variation trend of the heat value of each target basketball frame according to the heat value sequence; and obtaining the heat index corresponding to each target basketball hoop according to the heat value change trend of each target basketball hoop.
Preferably, the degree of abnormality of each target basketball rim is calculated according to the following formula:
wherein the content of the first and second substances,
is as follows
The degree of abnormality of the individual target basketball rims,
is a first
The shape index corresponding to the target basketball hoop,
is as follows
The hanging frame index corresponding to the target basketball frame,
is as follows
The bending degree index corresponding to the target basketball rim,
is as follows
The heat index corresponding to the target basketball rim,
is composed of
The corresponding weight of the weight is set to be,
is composed of
The corresponding weight of the weight is set to be,
is composed of
The corresponding weight of the weight is set to be,
is composed of
Corresponding weight, and
。
preferably, the method for obtaining the hanging frame index corresponding to each target basketball frame includes:
judging whether each target basketball hoop in each frame of basketball court panoramic image is in a hung hoop state or not according to the hand key points and the target basketball hoop key points, and if so, calculating the duration of each target basketball hoop in the hung hoop state;
and obtaining a frame hanging index corresponding to each target basketball frame according to the time length of each target basketball frame in the hung state.
Drawings
To more clearly illustrate the embodiments of the present invention or the technical solutions and advantages of the prior art, the following description will be made
While the drawings necessary for describing the embodiments or prior art are briefly described, it should be apparent that the drawings in the following description are merely examples of the invention and that other drawings may be derived from those drawings by those of ordinary skill in the art without inventive step.
Fig. 1 is a flowchart of a basketball rim abnormality detection method based on computer vision according to the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be described clearly and completely with reference to the accompanying drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, rather than all embodiments, and all other embodiments obtained by those skilled in the art based on the embodiments of the present invention belong to the protection scope of the embodiments of the present invention.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.
The embodiment provides a basketball rim abnormality detection method based on computer vision, which is described in detail as follows:
as shown in fig. 1, the basketball rim abnormality detection method based on computer vision includes the following steps:
and S001, acquiring a multi-frame basketball court panoramic image in a set time period.
In the embodiment, a camera is used for collecting multi-frame basketball court local images in a set time period, and the multi-frame basketball court panoramic image in the set time period is obtained by performing projection transformation and image splicing on each frame of corresponding basketball court local images; in the embodiment, the angles of the cameras, the number of the cameras and the positions of the cameras are set according to the actual situation of the basketball court, but the condition that the areas collected by the cameras comprise all the areas of the basketball court is met; the projective transformation and the image stitching are well-known techniques in this embodiment, and therefore this embodiment will not be described in detail.
In this embodiment, the time period is set to within the first 24 hours; as another embodiment, the set time period needs to be set according to actual conditions, and the corresponding acquisition frequency is also set according to actual requirements.
S002, obtaining each target basketball hoop in the basketball court and a standard circle corresponding to each target basketball hoop according to the last frame of basketball court panoramic image; and obtaining the form indexes corresponding to the target basketball rims according to the pixel point difference between the target basketball rims and the corresponding standard circles.
In the embodiment, a semantic perception effect graph of a target basketball hoop in a basketball court is obtained through a semantic perception network; the specific training process of the semantic perception network comprises the following steps: inputting a multi-frame basketball court sample panoramic image into a semantic perception network, marking the pixel value of a region corresponding to a target basketball hoop as 1 by an artificial set label, marking the pixel values corresponding to other regions as 0, and performing iterative training by adopting a cross entropy loss function; inputting the obtained panoramic image of the last frame of basketball court into a trained semantic perception network to obtain a semantic perception effect image of a target basketball hoop in the basketball court; in this embodiment, the obtained semantic perception effect graph of the target basketball hoop is multiplied and cut by the panoramic image of the last frame of basketball court to obtain each target basketball hoop in the basketball court, and the target basketball hoop corresponds to RGB image data; in this embodiment, the adopted semantic perception network is deep labv3, and since the semantic perception network is a known technology, the structural principle and the training method of the semantic perception network are not described in detail in this embodiment.
In the embodiment, each target basketball hoop under the overlooking visual angle is obtained by performing projection transformation on each target basketball hoop; and analyzing the target basketball rims at the overlooking visual angle to obtain the form indexes of the target basketball rims. The target basketball rim under the overlooking visual angle can accurately reflect the form change of each target basketball rim, the form index of each target basketball rim reflects whether each target basketball rim is in a standard smooth circle, and the larger the value of the form index is, the more the shape of the target basketball rim is represented, the more the shape of the target basketball rim is not close to the standard smooth circle, the larger the abnormity of the target basketball rim is.
In the embodiment, a Canny edge detection algorithm is used for extracting the edge of each target basketball under the overlooking visual angle, so as to obtain the edge image of each target basketball frame under the overlooking visual angle; the Canny edge detection algorithm is a well-known technique, so the embodiment is not described in detail; as another embodiment, other algorithms may be used to extract the edge of each target basketball in the top view, such as a Sobel edge detection algorithm or a Roberts edge detection algorithm.
In the embodiment, in the process of analyzing each target basketball hoop under the overlooking visual angle to obtain the form index of each target basketball hoop, the edge image of each target basketball hoop under the overlooking visual angle is detected and analyzed through a Hough transform algorithm to obtain a standard circle corresponding to each target basketball hoop, and then the form index of each target basketball hoop is obtained by analyzing the difference between each target basketball hoop and the corresponding standard circle of each target basketball hoop; the specific analysis process is as follows: detecting the edge images of the target basketball rims at the overlooking visual angle through a Hough transform algorithm to obtain detection circles corresponding to the target basketball rims at the overlooking visual angle, wherein the position difference distance of the circle center coordinates between the detection circles corresponding to the target basketball rims at the overlooking visual angle is smaller; in the embodiment, after the edge detection is performed on each target basketball hoop under the overlooking visual angle, hough transformation is performed, so that the interference of the irrelevant pixel points on the subsequent calculation of the standard circles corresponding to each target basketball hoop can be reduced.
In this embodiment, the number of pixels on each corresponding detection circle in each target basketball frame at the overlooking viewing angle is obtained, and the number of pixels on each corresponding detection circle in each target basketball frame at the overlooking viewing angle is constructed into a sequence, that is:
wherein the content of the first and second substances,
is the first from a top view
The pixel point number sequence on each corresponding detection circle in each target basketball frame,
is the first from a top view
In the corresponding first target basketball frame
The number of pixels on each detection circle,
is the first from a top view
In the corresponding first target basketball frame
The number of pixels on each detection circle,
is the first from a top view
The number of detection circles corresponding to each target basketball hoop is different, and the number of detection circles corresponding to different target basketball hoops is different.
In this embodiment, the number of pixels that are not on the corresponding detection circles in each target basketball frame at the overlooking viewing angle is obtained, the sum of the shortest distances from the pixels that are not on the corresponding detection circles in each target basketball frame at the overlooking viewing angle to the corresponding detection circles is calculated, and a corresponding sequence is constructed:
wherein the content of the first and second substances,
is the first from a top view
The sum sequence of the shortest distances from each pixel point not on the corresponding detection circle to the corresponding detection circle in the target basketball frame,
is the first from a top view
The object is not in the corresponding second position in the basketball frame
Each pixel point on each detection circle reaches the corresponding second
The sum of the shortest distances on the individual detection circles,
is the first from a top view
The object is not in the corresponding second position in the basketball frame
Each pixel point on each detection circle reaches the corresponding second
The sum of the shortest distances on the individual detection circles,
is the first from a top view
The number of detection circles corresponding to each target basketball rim.
In this embodiment, the scoring index of each detection circle corresponding to each target basketball hoop at the overlooking viewing angle is obtained according to the sum of the number of pixels on each detection circle in each target basketball hoop at the overlooking viewing angle and the shortest distance from each pixel not on each detection circle in each target basketball hoop at the overlooking viewing angle to each detection circle; the number of pixel points on each detection circle in each target basketball frame at the overlooking visual angle is in positive correlation with the scoring index of each detection circle corresponding to each target basketball frame at the overlooking visual angle, and the sum of the shortest distances from each pixel point not on each detection circle in each target basketball frame at the overlooking visual angle to each detection circle is in negative correlation with the scoring index of each detection circle corresponding to each target basketball frame at the overlooking visual angle; calculating the scoring index of each detection circle corresponding to each target basketball hoop under the overlooking visual angle according to the following formula:
wherein the content of the first and second substances,
is the first from a top view
The first goal basketball hoop corresponds to
The score index of each detection circle is set,
is the first from a top view
In the corresponding first target basketball frame
The number of pixels on each detection circle,
from a top view
The object is not in the basketball frame
Each pixel point on the detection circle goes to the first
Sum of shortest distances on the individual detection circles.
In this embodiment, the method for calculating the score index of each detection circle corresponding to each target basketball rim in the top view angle is only one preferred method of this embodiment, and as another method, other methods may be used to calculate the score index of each detection circle corresponding to each target basketball rim in the top view angle, but the sum of the shortest distances from each pixel point not on each detection circle in each target basketball rim in the top view angle to each detection circle in the top view angle is in a positive correlation with the score index of each detection circle corresponding to each target basketball rim in the top view angle, and the sum of the shortest distances from each pixel point not on each detection circle in each target basketball rim in the top view angle to each detection circle in the top view angle is in a negative correlation with the score index of each detection circle corresponding to each target basketball rim in the top view angle.
In this embodiment, the scoring indexes of the detection circles corresponding to the target basketball rims at the top view angle can be obtained through the above calculation method, the detection circle corresponding to the maximum scoring index is respectively selected as the standard circle corresponding to the target basketball rim, and the selected standard circle is the closest circle to the target basketball rim in the normal state; in this embodiment, when the number of the detection circles corresponding to the target basketball rim is 1, the detection circle is directly used as the standard circle of the corresponding target basketball rim without calculating the score index of the detection circle corresponding to the target basketball rim.
In this embodiment, the morphological index corresponding to each target basketball hoop is obtained according to the sum of the number of the pixel points of each target basketball hoop that are not on the standard circle corresponding to each target basketball hoop and the shortest distance from each pixel point of each target basketball hoop that is not on the standard circle corresponding to each target basketball hoop to the standard circle corresponding to each target basketball hoop; the sum of the number of the pixel points of each target basketball frame, which are not located on the standard circle corresponding to each target basketball frame, and the shortest distance from each pixel point of each target basketball frame, which is not located on the standard circle corresponding to each target basketball frame, to the standard circle corresponding to each target basketball frame is in negative correlation with the morphological index corresponding to each target basketball frame; and calculating the form indexes corresponding to the target basketball rims according to the following formula:
wherein the content of the first and second substances,
is as follows
The shape index corresponding to the target basketball hoop,
is as follows
The object is not in the basketball frame
The number of pixel points on the standard circle corresponding to the target basketball rim,
is as follows
Out of the basketball rim
The sum of the shortest distances from each pixel point on the standard circle corresponding to each basketball rim to the corresponding standard circle; and is
And
the larger the value of (a) is, the smaller the value of the corresponding shape index is, the closer the target basketball rim is to the standard circle, that is, the smaller the degree of deformation of the target basketball rim is.
In this embodiment, the method for calculating the form index corresponding to each target basketball hoop is only one preferred method of the present embodiment, and as another method, another method may be used to calculate the form index corresponding to each target basketball hoop, but the sum of the number of pixels in the standard circle not corresponding to each target basketball hoop in each target basketball hoop and the shortest distance from each pixel in the standard circle not corresponding to each target basketball hoop to the standard circle corresponding to each target basketball hoop in each target basketball hoop is in positive correlation with the form index corresponding to each target basketball hoop.
Step S003, calculating the minimum circumscribed rectangle of each target basketball frame corresponding to the last frame of basketball court panoramic image; and obtaining the bending degree index corresponding to each target basketball hoop according to the difference between the minimum external matrix shape of each target basketball hoop and the corresponding standard minimum external moment.
In the embodiment, each target basketball hoop under the front view angle is obtained by performing projection transformation on each target basketball hoop; analyzing each target basketball hoop under the front view angle to obtain a bending degree index corresponding to each target basketball hoop; each target basketball hoop under the front view angle can reflect the bending degree index of each target basketball hoop, and the bending degree index of each target basketball hoop can reflect the abnormal degree of each target basketball hoop.
In this embodiment, the minimum circumscribed rectangle of each target basketball frame at the front view angle is obtained, and the aspect ratio of the minimum circumscribed rectangle of each target basketball frame at the front view angle is calculated; obtaining the minimum circumscribed rectangle when each target basketball hoop is in a normal state under the normal visual angle, and calculating the length-width ratio of the minimum circumscribed rectangle when each target basketball hoop is in the normal state under the normal visual angle; the process of obtaining the minimum circumscribed rectangle is a known technology, so the embodiment is not described in detail; obtaining a bending degree index corresponding to each target basketball hoop according to the length-width ratio of the minimum external rectangle of each target basketball hoop under the normal visual angle and the length-width ratio of the minimum external rectangle when each target basketball hoop under the normal visual angle is normal, and calculating the bending degree index corresponding to each target basketball hoop according to the following formula:
wherein the content of the first and second substances,
is as follows
The bending degree index corresponding to the target basketball rim,
is as follows
The aspect ratio of the minimum circumscribed rectangle when the individual target basketball rim is normal,
is the first under the front view
The aspect ratio of the minimum circumscribed rectangle of the individual target basketball rim; in the present embodiment, the first and second electrodes are,
，
wherein, in the step (A),
is as follows
The length of the minimum circumscribed rectangle when the target basketball rim is normal,
is as follows
The width of the minimum circumscribed rectangle when the individual target basketball rim is normal,
is the first under the front view
The length of the minimum circumscribed rectangle of the individual target basketball rim,
is the first under the front view
The width of the smallest circumscribed rectangle of the individual target basketball rim.
Step S004, inputting each frame of basketball court panoramic image into a key point detection network to obtain a key point thermodynamic diagram corresponding to each frame of basketball court panoramic image; and obtaining the heat index corresponding to each target basketball hoop according to the key point thermodynamic diagram.
In the embodiment, the key point thermodynamic diagrams corresponding to the basketball court panoramic images of each frame are obtained by analyzing the basketball court panoramic images of each frame; obtaining a heat index corresponding to each target basketball hoop according to the key point thermodynamic diagrams corresponding to each frame of basketball court panoramic image; and the heat index corresponding to each target basketball hoop can reflect the number of people playing the basketball at the corresponding target basketball hoop, and the number of people playing the basketball can indirectly reflect the abnormal degree of the target basketball hoop.
In this embodiment, the key point thermodynamic diagram is obtained through a key point detection network, and a specific training process of the key point detection network is as follows: firstly, acquiring a training sample set, wherein the training sample comprises a multi-frame basketball court sample panoramic image; marking the positions of key points in panoramic images of basketball court samples of each frame in the training sample set; the marked key points are the head of the human body, the hand of the human body and a target basketball frame; and inputting the training sample set and the labeled data into a network, and training by adopting a mean square error loss function to obtain a trained key point detection network. In this embodiment, each frame of basketball court panoramic image is input to a trained key point detection network, so as to obtain a key point thermodynamic diagram corresponding to each frame of basketball court panoramic image.
In this embodiment, the key point detection network is an Encoder-Decoder structure, where the Encoder performs convolution and downsampling operations on an input multi-frame basketball court sample panoramic image to extract features in the image to obtain a feature map, the Decoder performs upsampling operations on the obtained feature map to obtain a key point thermodynamic diagram equal to the input image, and positions of key points of a human body are marked by gaussian hot spots in the key point thermodynamic diagram; the final output result is recorded as the human body key point thermodynamic diagram in the embodiment.
In the embodiment, a movement area corresponding to each target basketball hoop is obtained according to the panoramic image of the basketball court; in the embodiment, the area in the three-line corresponding to each target basketball hoop is recorded as the movement area corresponding to each target basketball hoop; as another embodiment, another area may be selected as the movement area corresponding to the target basketball hoop according to actual conditions.
In this embodiment, the number of head key points in a motion area corresponding to each target basketball hoop in each basketball court panoramic image is obtained from a key point thermodynamic diagram corresponding to each frame of basketball court panoramic image, and the number of head key points is used as a heat value corresponding to each target basketball hoop in each basketball court panoramic image; according to the heat value corresponding to each target basketball hoop in each basketball court panoramic image, constructing a heat value sequence corresponding to each target basketball hoop:
wherein the content of the first and second substances,
is as follows
The corresponding heat value sequence of the target basketball frame,
is at the first
Frame basketball court panoramic image
The corresponding heat value of the target basketball rim,
is at the first
Frame basketball court panoramic image
The corresponding heat value of the target basketball rim,
the number of the panoramic images of the basketball court.
In this embodiment, the variation trend of the heat value sequence corresponding to each target basketball hoop is analyzed, and when the variation trend of the heat value sequence corresponding to the target basketball hoop is increased, the heat index corresponding to the target basketball hoop is marked as 0, and the probability that the target basketball hoop is abnormal is low; when the variation trend of the heat value sequence corresponding to the target basketball rim is descending, the probability that the target basketball rim is abnormal is high, and at this time, the difference value between the heat value mean value corresponding to the target basketball rim and the heat value mean value corresponding to the target basketball rim in the maximum ascending trend is used as the heat index corresponding to the target basketball rim. In this embodiment, the analysis process of the variation trend of the heat value sequence corresponding to the target basketball rim is a known technology, and therefore, this embodiment is not described in detail.
In the embodiment, the duration of each target basketball hoop in the hung state is calculated according to the hand key point and the target basketball hoop key point on the key point thermodynamic diagram corresponding to each frame of basketball court panoramic image, and the duration of each target basketball hoop in the hung state can reflect the abnormal degree of each target basketball hoop; in this embodiment, the hand key points and the target basketball hoop key points in the motion area corresponding to each target basketball hoop are obtained from the key point thermodynamic diagrams corresponding to each frame of basketball court panoramic images, the distances from the hand key points in the motion area corresponding to each target basketball hoop in each basketball court panoramic image to the corresponding target basketball hoop key points are calculated, whether the distances from the hand key points in the motion area corresponding to each target basketball hoop in each basketball court panoramic image to the corresponding target basketball hoop key points are smaller than a preset distance or not is judged, if yes, the target basketball hoop is judged to be in the hung hoop state, and if not, the target basketball hoop is judged not to be in the hung hoop state.
When the target basketball hoop is in the hung state, the time length of each target basketball hoop in the hung state in a single time is obtained through time sequence analysis, and the target basketball hoop in the basketball court panoramic image in the hung state is analyzed through a target detection network; the training process of the target detection network in this embodiment is as follows: acquiring a training sample set, wherein the training sample set is a multi-frame basketball hoop sample panoramic image, marking a human body enclosure frame in each frame basketball court sample panoramic image in the training sample set, and marking the coordinates of the central point of the enclosure frame as
The length and width of the bounding box are respectively
And
then the target detection network is labeled as
The coordinates, the length and the width of the central point of the surrounding frame corresponding to different people are also different; and inputting the sample set and the label of the target detection network into the target detection network for training, wherein the target detection network adopts a mean square error loss function for iterative training.
In this embodiment, the target detection network is a known technology, and therefore the structural principle and the training method of the target detection network are not described in detail in this embodiment.
In the embodiment, the basketball hoop panoramic images are input into a trained target detection network, and the corresponding human body enclosure frames on the basketball hoop panoramic images are obtained; obtaining the human body surrounding frame of each target basketball frame in the hung state at a single time according to the corresponding human body surrounding frame on each basketball frame panoramic image, and obtaining the area of the human body surrounding frame according to the length and the width of the human body surrounding frame, wherein the area of the human body surrounding frame can indirectly reflect the weight of a person hung on the target basketball frame; in the embodiment, the basketball hoop hanging indexes corresponding to the target basketball hoops are obtained according to the time length of each target basketball hoop in the hung hoop state and the area of the human body surrounding hoop of each target basketball hoop in the hung hoop state; the time length of each target basketball hoop in the hung hoop state at a time and the area of the human body surrounding hoop of each target basketball hoop in the hung hoop state at a time are in positive correlation with the corresponding hanging hoop indexes of each target basketball hoop; calculating the corresponding frame hanging index of each target basketball frame according to the following formula:
wherein, the first and the second end of the pipe are connected with each other,
is as follows
The hanging frame index corresponding to the target basketball frame,
for the first time period
The number of times that the target basketball hoop is hung,
is a first
Individual target basketball rim
The area of the corresponding surrounding frame of the person when the person is in the hung frame state,
is as follows
Individual target basketball rim
The time length when the frame is hung next time, and the meaning of the set time period is the same as that of the set time period.
In this embodiment, the method for calculating the rim hanging index corresponding to each target basketball rim is only one preferred method of the present embodiment, and other calculation methods may be used to calculate the rim hanging index corresponding to each target basketball rim, but the time length of each target basketball rim in the rim hanging state once and the area of the human body surrounding rim of each target basketball rim in the rim hanging state once are in positive correlation with the rim hanging index corresponding to each target basketball rim.
In this embodiment, in the process of pitching, the sound of the basketball fan or the basketball player touching the target basketball hoop can reflect the abnormality of the target basketball hoop, because when the target basketball hoop has abnormality such as loose screws, the sound of the basketball touching the target basketball hoop is greatly different from the sound of the basketball touching the target basketball hoop when the target basketball hoop is normal; therefore, in the embodiment, the miniature audio collector is arranged at the backboard position corresponding to each target basketball hoop to collect the audio data of the basketball contacting the target basketball hoop in a set time period in real time, the audio data of each target basketball hoop collected by the audio collector is detected, and the mel-frequency cepstrum coefficient characteristic, namely the MFCC characteristic is extracted; the process of extracting the mel-frequency cepstrum coefficient features in this embodiment is a known technology, and therefore this embodiment is not described in detail.
And then analyzing the MFCC characteristics corresponding to the audio data of each target basketball frame through an anomaly detection network, wherein the anomaly detection network is a fully-connected network, and the training process of the anomaly detection network is as follows: the method comprises the steps of obtaining a training sample set, wherein the training sample set is sample audio data of a basketball touching a basketball hoop, artificially marking sound of the basketball touching the basketball hoop in the sample audio data, correspondingly marking the sound as 1 and 0, wherein the 1 is the sound of the basketball touching the basketball hoop under the abnormal condition of the basketball hoop, the 0 is the sound of the basketball touching the basketball hoop under the normal condition of the basketball hoop, inputting the data of the training sample set and the marked data into an abnormal detection network, and performing iterative training by adopting a cross entropy loss function.
In the embodiment, the MFCC characteristics corresponding to the audio data of each target basketball hoop within the set time period are input into the trained audio detection network, and the number of times that the basketball touches the target basketball hoop when the target basketball hoop corresponding to each target basketball hoop within the set time period is abnormal is counted, and the total number of times that the basketball corresponding to each target basketball hoop touches the target basketball hoop within the set time period is counted; calculating the proportion of the frequency of the sound of the basketball touching the target basketball hoop when the target basketball hoop corresponding to each target basketball hoop in the set time period is abnormal to the total frequency of the sound of the basketball touching the target basketball hoop corresponding to each target basketball hoop in the set time period to obtain the sound abnormal index corresponding to each target basketball hoop; the sound frequency of the basketball touching the target basketball hoop when the target basketball hoop corresponding to each target basketball hoop is abnormal in the set time period is in positive correlation with the sound abnormal index corresponding to each target basketball hoop, and the total frequency of the basketball touching the target basketball hoop corresponding to each target basketball hoop in the set time period is in negative correlation with the sound abnormal index corresponding to each target basketball hoop; calculating the sound abnormal index corresponding to each target basketball hoop according to the following formula:
wherein the content of the first and second substances,
is as follows
The sound abnormal index corresponding to the target basketball rim,
for the first time period
The sound frequency of the basketball touching the target basketball hoop when the target basketball hoop corresponding to the target basketball hoop is abnormal,
for the first time period
The total times of the sound that the basketball corresponding to the target basketball hoop touches the target basketball hoop.
In the present embodiment, the method for calculating the sound abnormality index corresponding to each target basketball rim is only one of the preferable methods of the present embodiment, and as another method, another method may be used to calculate the sound abnormality index corresponding to each target basketball rim, but the total number of times the basketball corresponding to each target basketball rim touches the target basketball rim in the set time period and the sound abnormality index corresponding to each target basketball rim are in a negative correlation in order to satisfy the condition that the sound abnormality index corresponding to each target basketball rim in the set time period is in a positive correlation with the sound abnormality index corresponding to each target basketball rim, and the total number of times the basketball corresponding to each target basketball rim touches the target basketball rim in the set time period and the sound abnormality index corresponding to each target basketball rim.
And step S005, obtaining the abnormal degree of each target basketball hoop according to the form index, the bending degree index and the heat index corresponding to each target basketball hoop.
In the embodiment, the abnormal degree of each target basketball hoop is obtained according to the form index, the bending degree index, the heat index, the hoop hanging index and the sound abnormal index corresponding to each target basketball hoop; and the shape index, the bending degree index, the heat index, the frame hanging index and the sound abnormal index corresponding to each target basketball hoop form a positive correlation with the abnormal degree of each target basketball hoop; calculating the abnormal degree of each target basketball frame according to the following formula:
wherein the content of the first and second substances,
is as follows
The degree of abnormality of the individual target basketball rims,
is as follows
The shape index corresponding to the target basketball hoop,
is as follows
The hanging frame index corresponding to the target basketball frame,
is as follows
The bending degree index corresponding to the target basketball rim,
is as follows
The heat index corresponding to the target basketball rim,
is a first
The sound abnormal index corresponding to the target basketball rim,
is composed of
The corresponding weight of the weight is set to be,
is composed of
The corresponding weight of the weight is set to be,
is composed of
The corresponding weight of the weight is set to be,
is composed of
The corresponding weight of the weight is set to be,
is composed of
Corresponding weight, and
(ii) a In the present embodiment, the first and second electrodes are,
the value of (A) needs to be set according to actual conditions, but satisfies
。
In the present embodiment, the method of calculating the abnormality degree of each target basketball rim is only one of the preferred embodiments, and other methods may be used to calculate the abnormality degree of each target basketball rim, but the shape index, the bending degree index, the heating degree index, the hanging rim index and the sound abnormality index corresponding to each target basketball rim are positively correlated with the abnormality degree of each target basketball rim.
According to the abnormal degree of the target basketball rim, the abnormal degree of the target basketball rim is obtained according to the form index, the bending degree index, the heat index, the rim hanging index and the sound abnormal index corresponding to the target basketball rim; as another embodiment, the degree of abnormality of each target basketball rim may be obtained based only on the shape index, the bending degree index, and the heat index corresponding to each target basketball rim.
In this embodiment, the obtained value of the degree of abnormality of each target basketball frame is normalized, and the value of the degree of abnormality of each target basketball frame is set between (0, 1); the normalization process is a well-known technique, and therefore, this embodiment is not described in detail.
In this embodiment, the value of the abnormal degree of each target basketball rim after normalization is set to
Judging that the target basketball frame is not abnormal; the abnormal degree value of each target basketball frame after the normalization is at
Judging that the target basketball frame is not abnormal; the abnormal degree value of each target basketball frame after the normalization is at
Then, determine the target blueThe ball frame state is slightly abnormal; the abnormal degree value of each target basketball frame after the normalization is at
Judging that the state of the target basketball frame is seriously abnormal; in the embodiment, when the value of the abnormal degree of the normalized target basketball hoop is higher than the abnormal degree threshold value, the system sends out an abnormal early warning to prompt relevant maintenance personnel to overhaul the target basketball hoop as soon as possible, so that the influence of the abnormal target basketball hoop on basketball enthusiasts or basketball players is avoided; the threshold value of the degree of abnormality in this embodiment is
(ii) a As another embodiment, different abnormality degree thresholds may be set according to different needs.
As another embodiment, the abnormal degree section may be set differently according to the requirement, for example, the abnormal degree of the target basketball frame after normalization may be set to a value
In the meantime, it is determined that the target basketball frame is not abnormal, and the value of the degree of abnormality of the normalized target basketball frame is
In the meantime, it is determined that there is an abnormality in the target basketball rim, and the value of the degree of abnormality of the normalized target basketball rim is larger than
And when the system is in use, the system sends out an abnormity early warning.
According to the embodiment, the morphological indexes corresponding to the target basketball rims are obtained according to the pixel point difference between the target basketball rims and the corresponding standard circles; obtaining a bending degree index corresponding to each target basketball hoop according to the difference between the minimum external matrix shape of each target basketball hoop and the corresponding standard minimum external moment; inputting each frame of basketball court panoramic image into a key point detection network to obtain a key point thermodynamic diagram corresponding to each frame of basketball court panoramic image; obtaining a heat index corresponding to each target basketball hoop according to the key point thermodynamic diagram; and obtaining the abnormal degree of each target basketball hoop according to the form index, the bending degree index and the heat index corresponding to each target basketball hoop. In the embodiment, the form index, the bending degree index and the heat index are used as the basis for obtaining the abnormal degree of each target basketball rim, so that the accuracy of abnormal detection of the basketball rim can be improved; the pixel point difference between the target basketball rim and the corresponding standard circle is used as a basis for obtaining the form index corresponding to each target basketball rim, so that the calculation amount can be reduced, and the efficiency of basketball rim abnormity detection is improved.
The basketball rim abnormality detection system based on the computer vision of the embodiment comprises a memory and a processor, wherein the processor executes a computer program stored in the memory to realize the basketball rim abnormality detection method based on the computer vision.
It should be noted that the order of the above-mentioned embodiments of the present invention is merely for description and does not represent the merits of the embodiments, and in some cases, actions or steps recited in the claims may be executed in an order different from the order of the embodiments and still achieve desirable results.
Claims (8)
1. A basketball rim abnormity detection method based on computer vision is characterized by comprising the following steps:
acquiring a multi-frame basketball court panoramic image within a set time period;
obtaining each target basketball frame in the basketball court and a standard circle corresponding to each target basketball frame according to the last frame of basketball court panoramic image; obtaining morphological indexes corresponding to the target basketball rims according to pixel point differences between the target basketball rims and the corresponding standard circles;
calculating the minimum circumscribed rectangle of each target basketball frame corresponding to the last frame of basketball court panoramic image; obtaining a bending degree index corresponding to each target basketball hoop according to the difference between the minimum external matrix shape of each target basketball hoop and the corresponding standard minimum external moment;
inputting each frame of basketball court panoramic image into a key point detection network to obtain a key point thermodynamic diagram corresponding to each frame of basketball court panoramic image; obtaining a heat index corresponding to each target basketball hoop according to the key point thermodynamic diagram;
and obtaining the abnormal degree of each target basketball hoop according to the form index, the bending degree index and the heat index corresponding to each target basketball hoop.
2. The method for detecting abnormal basketball rim based on computer vision as claimed in claim 1, wherein the step of obtaining the target basketball rim in the basketball court and the corresponding standard circle of each target basketball rim according to the last frame of panoramic image of the basketball court comprises:
inputting the last frame of basketball court panoramic image into a semantic perception network to obtain each target basketball frame in the basketball court;
detecting the last frame of basketball court panoramic image according to a Hough transform algorithm to obtain each detection circle corresponding to each target basketball hoop in the last frame of basketball court panoramic image;
obtaining the number of pixel points which are not on each detection circle in each target basketball frame and the number of pixel points on each detection circle in each target basketball frame; calculating the sum of the shortest distances from all pixel points which are not on all detection circles in all target basketball frames to all detection circles;
and obtaining a standard circle corresponding to each target basketball hoop according to the sum of the number of the pixel points on each detection circle in each target basketball hoop and the shortest distance from each pixel point on each detection circle in each target basketball hoop to each detection circle.
3. The method as claimed in claim 1, wherein the morphological index of each target basketball rim is calculated according to the following formula:
wherein the content of the first and second substances,
is the form index corresponding to the first target ball frame,
for the first target not in the basketball frame
The number of pixel points on the standard circle corresponding to the target basketball rim,
is as follows
Out of the basketball rim
And the sum of the shortest distances from each pixel point on the standard circle corresponding to each basketball rim to the corresponding standard circle.
4. The computer vision-based basketball rim abnormality detection method as claimed in claim 1, wherein the construction process of the key point detection network comprises:
acquiring a training sample set, wherein the training sample comprises a multi-frame basketball court sample panoramic image;
marking the positions of key points in panoramic images of basketball court samples of each frame in the training sample set; the marked key points are the head of the human body, the hand of the human body and a target basketball frame;
and inputting the training sample set and the labeled data into a network, and training by adopting a loss function to obtain a trained key point detection network.
5. The method as claimed in claim 1, wherein the method for detecting abnormality of basketball rim based on computer vision comprises the steps of:
obtaining a movement area corresponding to each target basketball hoop according to the panoramic image of each basketball court;
obtaining the number of key points in a motion area corresponding to each target basketball hoop in each basketball court panoramic image according to the key point thermodynamic diagram, and taking the number of the key points as the corresponding heat value of each target basketball hoop in each basketball court panoramic image;
constructing a heat value sequence corresponding to each target basketball hoop according to the heat values; obtaining the variation trend of the heat value of each target basketball frame according to the heat value sequence; and obtaining the heat index corresponding to each target basketball hoop according to the heat value change trend of each target basketball hoop.
6. The computer vision-based basketball rim abnormality detection method as claimed in claim 4, wherein the abnormality degree of each target basketball rim is calculated according to the following formula:
wherein the content of the first and second substances,
is as follows
The degree of abnormality of the individual target basketball rims,
is as follows
The shape index corresponding to the target basketball hoop,
is as follows
The hanging frame index corresponding to the target basketball frame,
is as follows
The bending degree index corresponding to the target basketball rim,
is as follows
The heat index corresponding to the target basketball rim,
is composed of
The corresponding weight of the weight is set to be,
is composed of
The corresponding weight of the weight is set to be,
is composed of
Corresponding weight is as
Corresponding weight, and
。
7. the method for detecting abnormality of a basketball rim based on computer vision as claimed in claim 6, wherein the method for obtaining the hanging rim index corresponding to each target basketball rim comprises:
judging whether each target basketball hoop in each frame of basketball court panoramic image is in a hung hoop state or not according to the hand key points and the target basketball hoop key points, and if so, calculating the duration of each target basketball hoop in the hung hoop state;
and obtaining a frame hanging index corresponding to each target basketball frame according to the time length of each target basketball frame in the hung state.
8. A computer vision based basketball rim abnormality detection system, comprising a memory and a processor, wherein the processor executes a computer program stored in the memory to implement a computer vision based basketball rim abnormality detection method according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210818930.2A CN114898281B (en) | 2022-07-13 | 2022-07-13 | Basketball hoop abnormity detection method and system based on computer vision |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210818930.2A CN114898281B (en) | 2022-07-13 | 2022-07-13 | Basketball hoop abnormity detection method and system based on computer vision |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114898281A true CN114898281A (en) | 2022-08-12 |
| CN114898281B CN114898281B (en) | 2022-09-30 |
Family
ID=82729326
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210818930.2A Active CN114898281B (en) | 2022-07-13 | 2022-07-13 | Basketball hoop abnormity detection method and system based on computer vision |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114898281B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111597992A (en) * | 2020-05-15 | 2020-08-28 | 哈尔滨工业大学 | Scene object abnormity identification method based on video monitoring |
| CN113446978A (en) * | 2021-06-28 | 2021-09-28 | 南京林业大学 | Basketball roundness detection device and detection method |
| US20210347366A1 (en) * | 2020-11-04 | 2021-11-11 | Beijing Baidu Netcom Science And Technology Co., Ltd. | Abnormal driving behavior determination method, device, vehicle and medium |
| CN113701825A (en) * | 2021-10-27 | 2021-11-26 | 南通高桥体育用品有限公司 | Body-building facility abnormity detection method and system based on artificial intelligence |
| CN113870264A (en) * | 2021-12-02 | 2021-12-31 | 湖北全兴通管业有限公司 | Tubular part port abnormity detection method and system based on image processing |
-
2022
- 2022-07-13 CN CN202210818930.2A patent/CN114898281B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111597992A (en) * | 2020-05-15 | 2020-08-28 | 哈尔滨工业大学 | Scene object abnormity identification method based on video monitoring |
| US20210347366A1 (en) * | 2020-11-04 | 2021-11-11 | Beijing Baidu Netcom Science And Technology Co., Ltd. | Abnormal driving behavior determination method, device, vehicle and medium |
| CN113446978A (en) * | 2021-06-28 | 2021-09-28 | 南京林业大学 | Basketball roundness detection device and detection method |
| CN113701825A (en) * | 2021-10-27 | 2021-11-26 | 南通高桥体育用品有限公司 | Body-building facility abnormity detection method and system based on artificial intelligence |
| CN113870264A (en) * | 2021-12-02 | 2021-12-31 | 湖北全兴通管业有限公司 | Tubular part port abnormity detection method and system based on image processing |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114898281B (en) | 2022-09-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10737158B2 (en) | Method and device for recognizing movement of tennis racket | |
| JP4860749B2 (en) | Apparatus, system, and method for determining compatibility with positioning instruction in person in image | |
| CN112926410B (en) | Target tracking method, device, storage medium and intelligent video system | |
| CN109145708B (en) | Pedestrian flow statistical method based on RGB and D information fusion | |
| CN103688275A (en) | Method of analysing video of sports motion | |
| JP4764332B2 (en) | Parameter information creation device, parameter information creation program, event detection device, and event detection program | |
| CN112801000B (en) | Household old man falling detection method and system based on multi-feature fusion | |
| JP4886707B2 (en) | Object trajectory identification device, object trajectory identification method, and object trajectory identification program | |
| CN109902632A (en) | A kind of video analysis device and video analysis method towards old man's exception | |
| JP4465150B2 (en) | System and method for measuring relative position of an object with respect to a reference point | |
| CN111783751A (en) | Rifle ball linkage and BIM-based breeding house piglet abnormity early warning method | |
| CN114898281B (en) | Basketball hoop abnormity detection method and system based on computer vision | |
| CN114299606A (en) | Sleep detection method and device based on front-end camera | |
| KR101147873B1 (en) | Screen-golf system using face recognition and control method of the same | |
| CN117115624A (en) | Palm vein image quality assessment method | |
| CN114788951A (en) | Hand-held motion analysis system and method | |
| CN115376210B (en) | Drowning behavior identification method, device, equipment and medium for preventing drowning in swimming pool | |
| CN111325198B (en) | Video object feature extraction method and device, and video object matching method and device | |
| CN116863388A (en) | Sperm motility determining method and system based on neural network | |
| JP2010176206A (en) | Mobile object tracking device | |
| CN107273873B (en) | Pedestrian based on irregular video sequence recognition methods and system again | |
| Maglo et al. | KaliCalib: A Framework for Basketball Court Registration | |
| CN111932620B (en) | Method for judging whether volleyball is out of net or not and method for acquiring service speed | |
| CN115272967A (en) | Cross-camera pedestrian real-time tracking and identifying method, device and medium | |
| CN113673327A (en) | Penalty ball hit prediction method based on human body posture estimation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |