CN108491807A - A kind of cow oestrus behavior method of real-time and system - Google Patents
A kind of cow oestrus behavior method of real-time and system Download PDFInfo
- Publication number
- CN108491807A CN108491807A CN201810263803.4A CN201810263803A CN108491807A CN 108491807 A CN108491807 A CN 108491807A CN 201810263803 A CN201810263803 A CN 201810263803A CN 108491807 A CN108491807 A CN 108491807A
- Authority
- CN
- China
- Prior art keywords
- monitored
- milk cow
- behavior
- light stream
- image sequence
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Systems or methods specially adapted for specific business sectors, e.g. utilities or tourism
- G06Q50/02—Agriculture; Fishing; Mining
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
Abstract
A kind of cow oestrus behavior method of real-time of present invention offer and system, the light stream image sequence of milk cow to be monitored is obtained according to the original sequence of milk cow to be monitored, feature extraction is carried out to light stream image sequence, obtains Optical-flow Feature corresponding with light stream image sequence;Milk cow to be monitored is detected from original sequence, and extracts the edge contour feature of milk cow to be monitored;Optical-flow Feature and the input of edge contour feature are preset into neural network, and the estrus behavior of milk cow to be monitored is monitored according to the output result of default neural network.This method and system can accurately identify the mounting behavior of cow oestrus to be monitored from the monitor video image of milk cow to be monitored, so as to timely and accurately find the estrus behavior of milk cow, be conducive to becoming pregnant in time, calving and extending lactation period for healthy cow, improve the economic benefit of milk cattle cultivating;It avoids simultaneously due to missing inspection caused by manual identified, saves the human cost of milk cattle cultivating to a certain extent.
Description
Technical field
The present invention relates to technical field of computer vision, more particularly, to a kind of cow oestrus behavior side of monitoring in real time
Method and system.
Background technology
Milk cow has the life habit and behavioral characteristic of its own, for the milk cow using artificial insemination of modern scale
, it finds that cow oestrus is conducive to becoming pregnant in time, calving and extending lactation period for healthy cow in time, improves milk cow and support cultivation
Economic benefit.
Large-scale milch cow farms detection of oestrus method includes traditional external observation method, examination per rectum method and in recent years at present
Pedometer method based on Internet of Things and ruminate method.External observation method depends primarily on the mounting behavior of cow in estrus, i.e., into
The mounting for entering to receive after heat peak period other oxen, to judge cow oestrus.Cow oestrus activity has certain regularity,
Most of heats concentrate at dusk, night or morning, if wanting to observe 90% heat cow, it is necessary to focus at dusk and the sight in morning
It examines;Therefore, for scale cattle farm, external observation method heat recall rate is often relatively low, especially cold winter.Rectum
Inspection technique not only heavy workload, but also require breeding person experienced.The method of pedometer is based primarily upon milk cow in the heat stage
Activity sharply increases, and principle is mainly converted into peak value to determine the heat condition of milk cow by the activity of milk cow, but
Turn that in the case of group or vaccine injection the interference for judging heat result can be caused.The method of ruminating, which needs to monitor milk cow, ruminates the time
And set ruminate time threshold, and ruminate threshold value setting have it is empirical, influence the subjective judgement of cow oestrus behavior.
To sum up, existing cow oestrus identification method is difficult to the estrus behavior of accurate judgement milk cow, in view of this, there is an urgent need for carry
For a kind of cow oestrus behavior method of real-time and system, to realize the accurate measurements to cow oestrus behavior.
Invention content
The present invention in order to overcome cow oestrus identification method to be in the prior art difficult to the estrus behavior of accurate judgement milk cow,
To delay milk cow breeding, so that the problem of influencing the interests of milk cattle cultivating, a kind of cow oestrus behavior is provided and is monitored in real time
Method and system.
On the one hand, the present invention provides a kind of cow oestrus behavior method of real-time, including:
S1 obtains the light stream image sequence of the milk cow to be monitored according to the original sequence of milk cow to be monitored, to institute
It states light stream image sequence and carries out feature extraction, obtain Optical-flow Feature corresponding with the light stream image sequence;
S2 detects the milk cow to be monitored from the original sequence, and extracts the side of the milk cow to be monitored
Edge contour feature;
The Optical-flow Feature and edge contour feature input are preset neural network by S3, and according to the default god
Output result through network is monitored the estrus behavior of the milk cow to be monitored.
Preferably, the light stream of the milk cow to be monitored is obtained described in step S1 according to the original sequence of milk cow to be monitored
Image sequence further comprises:
The light stream figure of the milk cow to be monitored is obtained according to the original sequence of the milk cow to be monitored using optical flow method
As sequence.
Preferably, feature extraction is carried out to the light stream image sequence described in step S1, obtained and the light stream image sequence
Corresponding Optical-flow Feature is arranged to further comprise:
The convolutional layer and down-sampling layer that the light stream image sequence is inputted to the default neural network, pass through the convolution
Layer and the down-sampling layer carry out feature extraction to the light stream image sequence, obtain light corresponding with the light stream image sequence
Flow feature.
Preferably, the step S2 further comprises:
Notable figure corresponding with the original sequence is obtained, the image entropy of the notable figure is calculated, according to the figure
As entropy detects the milk cow to be monitored;
The edge contour feature of the milk cow to be monitored is extracted using edge detection algorithm.
Preferably, the step S3 further comprises:
The Optical-flow Feature and the edge contour feature input to the full articulamentum of the default neural network, and according to
The output result of the default neural network output layer is monitored the estrus behavior of the milk cow to be monitored.
Preferably, further include being trained to the default neural network before the step S3, specifically include:
The original sequence sample with behavior label is obtained, according to the original sequence with behavior label
Sample obtains the light stream image sequence sample with behavior label;
Feature extraction is carried out to the light stream image sequence sample with behavior label, obtains and carries behavior mark with described
The corresponding Optical-flow Feature of light stream image sequence sample of note;
Milk cow sample is detected from the light stream image sequence sample marked with behavior, and extracts the milk cow sample
This edge contour feature;
By the Optical-flow Feature and the edge contour feature input preset neural network, to the default neural network into
Row training.
On the one hand, the present invention provides a kind of cow oestrus behavior real-time monitoring system, including:
Fisrt feature extraction module, for obtaining the milk cow to be monitored according to the original sequence of milk cow to be monitored
Light stream image sequence carries out feature extraction to the light stream image sequence, obtains light stream corresponding with the light stream image sequence
Feature;
Second feature extraction module for detecting the milk cow to be monitored from the original sequence, and extracts
The edge contour feature of the milk cow to be monitored;
Behavior monitoring module, for the Optical-flow Feature and edge contour feature input to be preset neural network, and
The estrus behavior of the milk cow to be monitored is monitored according to the output result of the default neural network.
Preferably, further include training module, be used for:
The original sequence sample with behavior label is obtained, according to the original sequence with behavior label
Sample obtains the light stream image sequence sample with behavior label;
Feature extraction is carried out to the light stream image sequence sample with behavior label, obtains and carries behavior mark with described
The corresponding Optical-flow Feature of light stream image sequence sample of note;
Milk cow sample is detected from the light stream image sequence sample marked with behavior, and extracts the milk cow sample
This edge contour feature;
By the Optical-flow Feature and the edge contour feature input preset neural network, to the default neural network into
Row training.
On the one hand, the present invention provides a kind of equipment of cow oestrus behavior method of real-time, including:
At least one processor;And
At least one processor being connect with the processor communication, wherein:
The memory is stored with the program instruction that can be executed by the processor, and the processor calls described program to refer to
It enables and is able to carry out any of the above-described method.
On the one hand, the present invention provides a kind of non-transient computer readable storage medium, and the non-transient computer is readable to deposit
Storage media stores computer instruction, and the computer instruction makes the computer execute any of the above-described method.
A kind of cow oestrus behavior method of real-time provided by the invention and system, according to the original graph of milk cow to be monitored
As sequence obtains the light stream image sequence of milk cow to be monitored, feature extraction is carried out to light stream image sequence, is obtained and light stream image
The corresponding Optical-flow Feature of sequence;Milk cow to be monitored is detected from original sequence, and extracts the edge wheel of milk cow to be monitored
Wide feature;Optical-flow Feature and edge contour feature input are preset into neural network, and according to the output of default neural network
As a result the estrus behavior of milk cow to be monitored is monitored.This method and system can be from the monitor video images of milk cow to be monitored
In accurately identify the mounting behavior of cow oestrus to be monitored, so as to timely and accurately find the estrus behavior of milk cow, favorably
In the becoming pregnant in time of healthy cow, calving and extend lactation period, improves the economic benefit of milk cattle cultivating;It avoids simultaneously due to people
Missing inspection caused by work identifies, saves the human cost of milk cattle cultivating to a certain extent.
Description of the drawings
Fig. 1 is a kind of overall flow schematic diagram of cow oestrus behavior method of real-time of the embodiment of the present invention;
Fig. 2 is a kind of overall structure diagram of cow oestrus behavior real-time monitoring system of the embodiment of the present invention;
Fig. 3 is a kind of structural framing signal of the equipment of cow oestrus behavior method of real-time of the embodiment of the present invention
Figure.
Specific implementation mode
With reference to the accompanying drawings and examples, the specific implementation mode of the present invention is described in further detail.Implement below
Example is not limited to the scope of the present invention for illustrating the present invention.
Fig. 1 is a kind of overall flow schematic diagram of cow oestrus behavior method of real-time of the embodiment of the present invention, such as Fig. 1
Shown, the present invention provides a kind of cow oestrus behavior method of real-time, including:
S1 obtains the light stream image sequence of the milk cow to be monitored according to the original sequence of milk cow to be monitored, to institute
It states light stream image sequence and carries out feature extraction, obtain Optical-flow Feature corresponding with the light stream image sequence;
Specifically, popularizing with current internet of things equipment, more and more cattle farms are assembled with monitoring camera-shooting equipment, this
In embodiment, the monitor video of milk cow to be monitored is shot by the monitoring camera-shooting equipment of cattle farm, is regarded from the monitoring of milk cow to be monitored
The original sequence of milk cow to be monitored is obtained in frequency.
Further, the light stream image sequence of milk cow to be monitored is obtained according to the original sequence of milk cow to be monitored,
Each pixel in middle light stream image has been assigned a velocity vector, the corresponding speed of same pixel in light stream image sequence
The variation of degree vector can embody the movable information of the pixel.In light stream image sequence, consistent region, light are moved
Flow table reveals consistency;Inconsistent region is moved, light stream also then shows inconsistent.It will appear mounting row when cow oestrus
For the mounting behavior action slight compared to general walking or each other smelling etc. is with the mixed and disorderly spy of quick action and direction
Point is in particular in that amplitude intensity in light stream image sequence for certain region Optical-flow Feature is big and the direction of motion is inconsistent.There is mirror
In this, the mounting behavior that can occur to milk cow by detecting the Optical-flow Feature of the light stream image sequence of milk cow to be monitored and generally
Walking or each other the slight action such as smelling distinguish, to realize the monitoring of cow oestrus behavior.
Further, feature extraction, acquisition and light stream are carried out to the light stream image sequence of the milk cow to be monitored of above-mentioned acquisition
The corresponding Optical-flow Feature of image sequence.In the present embodiment, described in certain region using the direction histogram based on amplitude weighting
Optical-flow Feature, direction histogram is a kind of method of non-parametric estmation, direction scope is divided into several angular intervals, then
Vector is referred to different sections respectively, there is rotation translation invariance, can reflect that the movement of Optical-flow Feature in certain region becomes
Law, amplitude intensity can reflect the severity of action, the amplitude intensity of the larger point of movement range to a certain extent
Also bigger, therefore it is accordingly also larger to set the big direction weight of amplitude.In other embodiments, other modes can also be used
Optical-flow Feature is described, can be set according to actual demand, be not specifically limited herein.
S2 detects the milk cow to be monitored from the original sequence, and extracts the side of the milk cow to be monitored
Edge contour feature;
Specifically, on the basis of obtaining milk cow original sequence to be monitored, since original sequence includes waiting for
Milk cow and environmental background are monitored, in order to eliminate the influence of environmental background, in the present embodiment, is detected from original sequence first
Go out milk cow to be monitored, then extracts the edge contour feature of milk cow to be monitored.
The Optical-flow Feature and edge contour feature input are preset neural network by S3, and according to the default god
Output result through network is monitored the estrus behavior of the milk cow to be monitored.
Specifically, the Optical-flow Feature of above-mentioned acquisition and the input of edge contour feature are preset into neural network, presets nerve net
Network is trained in advance, and Optical-flow Feature and edge contour feature are identified and are classified by default neural network, finally
According to the output result of default neural network judge milk cow to be monitored whether occur mounting behavior and it is general walking or smell each other
The slight actions such as news, and then realize the real-time monitoring to the estrus behavior of milk cow to be monitored.
A kind of cow oestrus behavior method of real-time provided by the invention, according to the original sequence of milk cow to be monitored
The light stream image sequence for obtaining milk cow to be monitored carries out feature extraction to light stream image sequence, obtains and light stream image sequence pair
The Optical-flow Feature answered;Milk cow to be monitored is detected from original sequence, and extracts the edge contour feature of milk cow to be monitored;
Optical-flow Feature and edge contour feature input are preset into neural network, and treated according to the output result of default neural network
The estrus behavior of monitoring milk cow is monitored.This method can accurately identify from the monitor video image of milk cow to be monitored and wait supervising
The mounting behavior for surveying cow oestrus, so as to timely and accurately find the estrus behavior of milk cow, be conducive to healthy cow and
When become pregnant, calving and extend lactation period, improve the economic benefit of milk cattle cultivating;It is avoided simultaneously due to leakage caused by manual identified
Inspection, saves the human cost of milk cattle cultivating to a certain extent.
Based on any of the above-described embodiment, a kind of cow oestrus behavior method of real-time is provided, basis waits for described in step S1
The light stream image sequence that the original sequence of monitoring milk cow obtains the milk cow to be monitored further comprises:
The light stream figure of the milk cow to be monitored is obtained according to the original sequence of the milk cow to be monitored using optical flow method
As sequence.
Specifically, in the present embodiment, on the basis of obtaining the original sequence of milk cow to be monitored, optical flow method root is utilized
The light stream image sequence of milk cow to be monitored is obtained according to the original sequence of milk cow to be monitored.Wherein optical flow method is a kind of simple reality
The expression way of image motion is normally defined the apparent motion of the brightness of image pattern in an image sequence, i.e., empty
Between point on body surface expression of the movement velocity on the imaging plane of visual sensor.
Optical flow method is broadly divided into three classes:Method based on matched method, the method for frequency domain and gradient.Wherein be based on
The method matched includes feature based and is based on two kinds of region, and the method for feature based constantly positions target main feature
And tracking, movement and brightness change to big target have robustness;Method based on region first determines similar region
Then position calculates light stream by the displacement of similar area.Method based on frequency domain, the also referred to as method based on energy, utilize speed
Spend adjustable filtering group output frequency or phase information.Method based on gradient utilizes the space-time differential calculation of image sequence brightness
2D velocity fields (light stream).
Based on the above technical solution, original graph of the optical flow method to milk cow to be monitored can be chosen according to actual demand
As sequence is handled, to obtain the light stream image sequence of milk cow to be monitored.
A kind of cow oestrus behavior method of real-time provided by the invention, using optical flow method according to the original of milk cow to be monitored
Beginning image sequence obtains the light stream image sequence of milk cow to be monitored, and then analyzes the light stream image sequence of milk cow to be monitored
Optical-flow Feature is obtained, to be monitored to the estrus behavior of milk cow to be monitored according to Optical-flow Feature, is conducive to timely and accurately
It was found that the estrus behavior of milk cow.
Based on any of the above-described embodiment, a kind of cow oestrus behavior method of real-time is provided, to described described in step S1
Light stream image sequence carries out feature extraction, obtains Optical-flow Feature corresponding with the light stream image sequence and further comprises:
The convolutional layer and down-sampling layer that the light stream image sequence is inputted to the default neural network, pass through the convolution
Layer and the down-sampling layer carry out feature extraction to the light stream image sequence, obtain light corresponding with the light stream image sequence
Flow feature.
Specifically, in the present embodiment, feature is carried out to the light stream image sequence of milk cow to be monitored using default neural network
Light stream image sequence is inputted the convolutional layer and down-sampling layer for presetting neural network, is combined by convolutional layer and down-sampling layer by extraction
The operation for alternately carrying out convolution sum down-sampling to the light stream image sequence of input together, obtains corresponding with light stream image sequence
Optical-flow Feature.
In the present embodiment, it includes three-layer coil lamination and two layers of down-sampling layer to preset neural network in total, and each layer is by multiple
Two dimensional surface forms, and each plane is made of multiple independent neurons.Wherein, in convolutional layer each neuron only with it is upper
One layer of one piece of part receives domain and is connected;Each neuron still only receives domain with one piece of part of last layer in down-sampling layer
It is connected, down-sampling is carried out to the characteristic pattern of last layer, spatial resolution is reduced, the local correlation of image can be made full use of in this way
Property, retain useful information while reducing data processing amount.In other embodiments, the structure for presetting neural network can root
It is built according to actual demand, is not specifically limited herein.
A kind of cow oestrus behavior method of real-time provided by the invention, the light stream image sequence of milk cow to be monitored is defeated
The convolutional layer and down-sampling layer for entering default neural network carry out light stream image sequence by convolutional layer and the down-sampling layer special
Sign extraction, obtains Optical-flow Feature corresponding with light stream image sequence, thus according to Optical-flow Feature to the heat row of milk cow to be monitored
To be monitored, be conducive to the estrus behavior for timely and accurately finding milk cow, to be conducive to the becoming pregnant in time of healthy cow, produce
Calf simultaneously extends lactation period, and then is conducive to improve the economic benefit of milk cattle cultivating.
Based on any of the above-described embodiment, a kind of cow oestrus behavior method of real-time is provided, the step S2 is further
Including:
Notable figure corresponding with the original sequence is obtained, the image entropy of the notable figure is calculated, according to the figure
As entropy detects the milk cow to be monitored;
The edge contour feature of the milk cow to be monitored is extracted using edge detection algorithm.
Specifically, the original sequence in view of milk cow to be monitored includes milk cow to be monitored and environmental background, in order to disappear
Except the influence of environmental background, milk cow to be monitored need to be detected from original sequence.
In the present embodiment, for every frame original image in original sequence, every frame original image is carried out first high
This low-pass filtering treatment obtains the fuzzy content per frame original image, then will be transformed into YUV color spaces per frame original image,
And the average value of tri- components of Y, U, V is calculated, calculate each pixel and yuv space in the fuzzy content per frame original image
Euclidean distance between corresponding average value obtains corresponding notable figure.Image entropy can indicate image to a certain extent
Textural characteristics and variation degree calculated notable in view of this, on the basis of obtaining original sequence corresponding notable figure
The image entropy of figure, there are notable difference, Jin Erke for the image entropy of milk cow to be monitored and environmental background region in notable figure
To detect milk cow to be monitored according to image entropy.
Further, the edge contour feature of milk cow to be monitored is extracted using edge detection algorithm, wherein edge detection is calculated
Method includes Laplacian operators, Roberts operators, Sobel operators, log (Laplacian-Gauss) operator, Kirsch operators
With the detection methods such as Prewitt operators, it can be configured according to actual demand, be not specifically limited herein.
A kind of cow oestrus behavior method of real-time provided by the invention obtains corresponding with original sequence notable
Figure, calculates the image entropy of notable figure, milk cow to be monitored is detected according to image entropy;Milk to be monitored is extracted using edge detection algorithm
The edge contour feature of ox, the edge contour that milk cow to be monitored can be accurately extracted from the original image of milk cow to be monitored are special
Sign, be conducive to that the estrus behavior of milk cow to be monitored is identified according to the edge contour feature of milk cow to be monitored, be conducive to and
When accurately find the estrus behavior of milk cow.
Based on any of the above-described embodiment, a kind of cow oestrus behavior method of real-time is provided, the step S3 is further
Including:
The Optical-flow Feature and the edge contour feature input to the full articulamentum of the default neural network, and according to
The output result of the default neural network output layer is monitored the estrus behavior of the milk cow to be monitored.
Specifically, the default neural network in the present embodiment further includes full articulamentum and output layer, is obtaining milk to be monitored
It is after the Optical-flow Feature of ox light stream image and the edge contour feature of milk cow to be monitored, Optical-flow Feature and the edge contour is special
Sign inputs the full articulamentum of the default neural network, and the full articulamentum of default neural network plays in entire neural network
The effect of " grader ", can be by Optical-flow Feature and edge contour Feature Mapping to sample labeling space, i.e., using full articulamentum
Can be mounting estrus behavior and general walking behavior by Optical-flow Feature and edge contour tagsort.The complete of default neural network connects
It connects layer and classification results is transmitted to output layer, the final output result according to the output layer for presetting neural network can be to be monitored
The estrus behavior of milk cow is monitored.
A kind of cow oestrus behavior method of real-time provided by the invention inputs Optical-flow Feature and edge contour feature
The full articulamentum of default neural network, and according to the output result of default neural network output layer to the heat row of milk cow to be monitored
To be monitored.This method combines the Optical-flow Feature and edge contour feature of milk cow to be monitored, can be from the prison of milk cow to be monitored
The mounting behavior that cow oestrus to be monitored is accurately identified in control video image, so as to timely and accurately find the heat of milk cow
Behavior.
Based on any of the above-described embodiment, a kind of cow oestrus behavior method of real-time is provided, is gone back before the step S3
Including being trained to the default neural network, specifically include:
The original sequence sample with behavior label is obtained, according to the original sequence with behavior label
Sample obtains the light stream image sequence sample with behavior label;
Feature extraction is carried out to the light stream image sequence sample with behavior label, obtains and carries behavior mark with described
The corresponding Optical-flow Feature of light stream image sequence sample of note;
Milk cow sample is detected from the light stream image sequence sample marked with behavior, and extracts the milk cow sample
This edge contour feature;
By the Optical-flow Feature and the edge contour feature input preset neural network, to the default neural network into
Row training.
Specifically, in the present embodiment, it is being monitored to the estrus behavior of milk cow to be monitored using default neural network
Before, it also needs to be trained default neural network, specific training process is as follows:
Suitable milk cow sample is chosen, by the estrus behavior and general behavior of manual identified milk cow sample, and it is right respectively
The milk cow sample of milk cow sample and general behavior with estrus behavior carries out video capture, obtains the original labeled as estrus behavior
Beginning image sequence sample and original sequence sample labeled as general behavior, are combined into the original image marked with behavior
Sequence samples.
Further, the light stream image with behavior label is obtained according to the original sequence sample marked with behavior
Sequence samples, wherein each pixel in light stream image have been assigned a velocity vector, same picture in light stream image sequence
The variation of the corresponding velocity vector of vegetarian refreshments can embody the movable information of the pixel.
Further, feature extraction is carried out to the light stream image sequence sample with behavior label of above-mentioned acquisition, obtained
Optical-flow Feature corresponding with the light stream image sequence sample marked with behavior.
Further, on the basis of obtaining the original sequence sample with behavior label, due in original image
Including waiting for milk cow sample and environmental background, in order to eliminate the influence of environmental background, in the present embodiment, marked first from behavior
Original sequence sample in detect milk cow sample, then extract the edge contour feature of milk cow sample.
Further, the Optical-flow Feature of above-mentioned acquisition and the input of edge contour feature are preset into neural network, to default god
It is trained through network.Wherein default neural network is built in advance, including input layer, convolutional layer, down-sampling layer, Quan Lian
Layer and output layer are connect, each layer is made of multiple two dimensional surfaces, and each plane is made of multiple independent neurons.Convolutional layer
In, each neuron only receives domain with one piece of part of last layer and is connected;In down-sampling layer, each neuron still only with
One piece of part of last layer receives domain and is connected.Wherein the number of plies of convolutional layer and down-sampling layer can be set according to actual demand
It sets, is not specifically limited herein.
A kind of cow oestrus behavior method of real-time provided by the invention obtains the original image sequence with behavior label
Row sample obtains the light stream image sequence sample with behavior label according to the original sequence sample marked with behavior;
Feature extraction is carried out to the light stream image sequence sample marked with behavior, is obtained and the light stream image sequence with behavior label
The corresponding Optical-flow Feature of sample;Milk cow sample is detected from the light stream image sequence sample marked with behavior, and extracts milk
The edge contour feature of ox sample;Optical-flow Feature and the input of edge contour feature are preset into neural network, to presetting neural network
It is trained.This method is trained default neural network according to Optical-flow Feature and edge contour feature, to be conducive to profit
The estrus behavior of milk cow to be monitored is monitored with trained default neural network, can timely and accurately be found to be monitored
The estrus behavior of milk cow.
Fig. 2 is a kind of overall structure diagram of cow oestrus behavior real-time monitoring system of the embodiment of the present invention, such as Fig. 2
Shown, the present invention provides a kind of cow oestrus behavior real-time monitoring system, including fisrt feature extraction module 1, second feature carry
Modulus block 2 and behavior monitoring module 3 realize that the cow oestrus behavior in any of the above-described embodiment is real-time by the cooperation of each module
Monitoring method is implemented as follows:
Fisrt feature extraction module 1, for obtaining the milk cow to be monitored according to the original sequence of milk cow to be monitored
Light stream image sequence, feature extraction is carried out to the light stream image sequence, obtains corresponding with light stream image sequence light
Flow feature;
Specifically, the monitor video that milk cow to be monitored is shot by the monitoring camera-shooting equipment of cattle farm, from milk cow to be monitored
The original sequence of milk cow to be monitored is obtained in monitor video.Using fisrt feature extraction module 1 according to milk cow to be monitored
Original sequence obtains the light stream image sequence of milk cow to be monitored, and wherein each pixel in light stream image has been assigned one
A velocity vector, in light stream image sequence the variation of the corresponding velocity vector of same pixel can embody the pixel
Movable information.
Further, using fisrt feature extraction module 1 to the light stream image sequence of the milk cow to be monitored of above-mentioned acquisition into
Row feature extraction obtains Optical-flow Feature corresponding with light stream image sequence.
Second feature extraction module 2 for detecting the milk cow to be monitored from the original sequence, and extracts
The edge contour feature of the milk cow to be monitored;
Specifically, on the basis of obtaining milk cow original sequence to be monitored, since original sequence includes waiting for
Milk cow and environmental background are monitored, in order to eliminate the influence of environmental background, in the present embodiment, utilizes second feature extraction module 2 first
Milk cow to be monitored is first detected from original sequence, then extracts the edge contour feature of milk cow to be monitored.
Behavior monitoring module 3, for the Optical-flow Feature and edge contour feature input to be preset neural network, and
The estrus behavior of the milk cow to be monitored is monitored according to the output result of the default neural network.
Specifically, the Optical-flow Feature of above-mentioned acquisition and edge contour feature are inputted into default god using behavior monitoring module 3
Through network, default neural network be it is trained in advance, by default neural network to Optical-flow Feature and edge contour feature into
Row identification and classification, it is final to judge whether milk cow to be monitored mounting behavior and one occurs according to the output result for presetting neural network
As walking or the slight action such as smelling each other, and then realize the real-time monitoring to the estrus behavior of milk cow to be monitored.
A kind of cow oestrus behavior real-time monitoring system provided by the invention, according to the original sequence of milk cow to be monitored
The light stream image sequence for obtaining milk cow to be monitored carries out feature extraction to light stream image sequence, obtains and light stream image sequence pair
The Optical-flow Feature answered;Milk cow to be monitored is detected from original sequence, and extracts the edge contour feature of milk cow to be monitored;
Optical-flow Feature and edge contour feature input are preset into neural network, and treated according to the output result of default neural network
The estrus behavior of monitoring milk cow is monitored.The system can accurately identify from the monitor video image of milk cow to be monitored and wait supervising
The mounting behavior for surveying cow oestrus, so as to timely and accurately find the estrus behavior of milk cow, be conducive to healthy cow and
When become pregnant, calving and extend lactation period, improve the economic benefit of milk cattle cultivating;It is avoided simultaneously due to leakage caused by manual identified
Inspection, saves the human cost of milk cattle cultivating to a certain extent.
Based on any of the above-described embodiment, a kind of cow oestrus behavior real-time monitoring system is provided, further includes training module, used
In:
The original sequence sample with behavior label is obtained, according to the original sequence with behavior label
Sample obtains the light stream image sequence sample with behavior label;
Feature extraction is carried out to the light stream image sequence sample with behavior label, obtains and carries behavior mark with described
The corresponding Optical-flow Feature of light stream image sequence sample of note;
Milk cow sample is detected from the light stream image sequence sample marked with behavior, and extracts the milk cow sample
This edge contour feature;
By the Optical-flow Feature and the edge contour feature input preset neural network, to the default neural network into
Row training.
Specifically, cow oestrus behavior real-time monitoring system provided in this embodiment further includes training module, passes through training
Module is trained default neural network, and specific training process is as follows:
Suitable milk cow sample is chosen, by the estrus behavior and general behavior of manual identified milk cow sample, and it is right respectively
The milk cow sample of milk cow sample and general behavior with estrus behavior carries out video capture, obtains the original labeled as estrus behavior
Beginning image sequence sample and original sequence sample labeled as general behavior, are combined into the original image marked with behavior
Sequence samples.
Further, the light stream image with behavior label is obtained according to the original sequence sample marked with behavior
Sequence samples, wherein each pixel in light stream image have been assigned a velocity vector, same picture in light stream image sequence
The variation of the corresponding velocity vector of vegetarian refreshments can embody the movable information of the pixel.
Further, feature extraction is carried out to the light stream image sequence sample with behavior label of above-mentioned acquisition, obtained
Optical-flow Feature corresponding with the light stream image sequence sample marked with behavior.
Further, on the basis of obtaining the original sequence sample with behavior label, due in original image
Including waiting for milk cow sample and environmental background, in order to eliminate the influence of environmental background, in the present embodiment, marked first from behavior
Original sequence sample in detect milk cow sample, then extract the edge contour feature of milk cow sample.
Further, the Optical-flow Feature of above-mentioned acquisition and the input of edge contour feature are preset into neural network, to default god
It is trained through network.Wherein default neural network is built in advance, including input layer, convolutional layer, down-sampling layer, Quan Lian
Layer and output layer are connect, each layer is made of multiple two dimensional surfaces, and each plane is made of multiple independent neurons.Convolutional layer
In, each neuron only receives domain with one piece of part of last layer and is connected;In down-sampling layer, each neuron still only with
One piece of part of last layer receives domain and is connected.Wherein the number of plies of convolutional layer and down-sampling layer can be set according to actual demand
It sets, is not specifically limited herein.
A kind of cow oestrus behavior real-time monitoring system provided by the invention obtains the original image sequence with behavior label
Row sample obtains the light stream image sequence sample with behavior label according to the original sequence sample marked with behavior;
Feature extraction is carried out to the light stream image sequence sample marked with behavior, is obtained and the light stream image sequence with behavior label
The corresponding Optical-flow Feature of sample;Milk cow sample is detected from the light stream image sequence sample marked with behavior, and extracts milk
The edge contour feature of ox sample;Optical-flow Feature and the input of edge contour feature are preset into neural network, to presetting neural network
It is trained.The system is trained default neural network according to Optical-flow Feature and edge contour feature, to be conducive to profit
The estrus behavior of milk cow to be monitored is monitored with trained default neural network, can timely and accurately be found to be monitored
The estrus behavior of milk cow.
Fig. 3 shows a kind of structure diagram of the equipment of cow oestrus behavior method of real-time of the embodiment of the present invention.Ginseng
According to Fig. 3, the equipment of the cow oestrus behavior method of real-time, including:Processor (processor) 31, memory
(memory) 32 and bus 33;Wherein, the processor 31 and memory 32 complete mutual communication by the bus 33;
The processor 31 is used to call the program instruction in the memory 32, to execute the side that above-mentioned each method embodiment is provided
Method, such as including:The light stream image sequence that milk cow to be monitored is obtained according to the original sequence of milk cow to be monitored, to light stream figure
As sequence carries out feature extraction, acquisition Optical-flow Feature corresponding with light stream image sequence;It detects to wait for from original sequence
Milk cow is monitored, and extracts the edge contour feature of milk cow to be monitored;By Optical-flow Feature and the default nerve of edge contour feature input
Network, and the estrus behavior of milk cow to be monitored is monitored according to the output result of default neural network.
The present embodiment discloses a kind of computer program product, and the computer program product includes being stored in non-transient calculating
Computer program on machine readable storage medium storing program for executing, the computer program include program instruction, when described program instruction is calculated
When machine executes, computer is able to carry out the method that above-mentioned each method embodiment is provided, such as including:According to milk cow to be monitored
Original sequence obtains the light stream image sequence of milk cow to be monitored, and feature extraction, acquisition and light are carried out to light stream image sequence
The corresponding Optical-flow Feature of flow image sequences;Milk cow to be monitored is detected from original sequence, and extracts milk cow to be monitored
Edge contour feature;Optical-flow Feature and the input of edge contour feature are preset into neural network, and according to the defeated of default neural network
Go out result to be monitored the estrus behavior of milk cow to be monitored.
The present embodiment provides a kind of non-transient computer readable storage medium, the non-transient computer readable storage medium
Computer instruction is stored, the computer instruction makes the computer execute the method that above-mentioned each method embodiment is provided, example
Such as include:The light stream image sequence that milk cow to be monitored is obtained according to the original sequence of milk cow to be monitored, to light stream image sequence
Row carry out feature extraction, obtain Optical-flow Feature corresponding with light stream image sequence;It is detected from original sequence to be monitored
Milk cow, and extract the edge contour feature of milk cow to be monitored;Optical-flow Feature and the input of edge contour feature are preset into neural network,
And the estrus behavior of milk cow to be monitored is monitored according to the output result of default neural network.
One of ordinary skill in the art will appreciate that:Realize that all or part of step of above method embodiment can pass through
The relevant hardware of program instruction is completed, and program above-mentioned can be stored in a computer read/write memory medium, the program
When being executed, step including the steps of the foregoing method embodiments is executed;And storage medium above-mentioned includes:ROM, RAM, magnetic disc or light
The various media that can store program code such as disk.
The embodiments such as the equipment of cow oestrus behavior method of real-time described above are only schematical, wherein
The unit illustrated as separating component may or may not be physically separated, the component shown as unit
It may or may not be physical unit, you can be located at a place, or may be distributed over multiple network element
On.Some or all of module therein can be selected according to the actual needs to achieve the purpose of the solution of this embodiment.Ability
Domain those of ordinary skill is not in the case where paying performing creative labour, you can to understand and implement.
Through the above description of the embodiments, those skilled in the art can be understood that each embodiment can
It is realized by the mode of software plus required general hardware platform, naturally it is also possible to pass through hardware.Based on this understanding, on
Stating technical solution, substantially the part that contributes to existing technology can be expressed in the form of software products in other words, should
Computer software product can store in a computer-readable storage medium, such as ROM/RAM, magnetic disc, CD, including several fingers
It enables and using so that a computer equipment (can be personal computer, server or the network equipment etc.) executes each implementation
Method described in certain parts of example or embodiment.
Finally, the present processes are only preferable embodiment, are not intended to limit the scope of the present invention.It is all
Within the spirit and principles in the present invention, any modification, equivalent replacement, improvement and so on should be included in the protection of the present invention
Within the scope of.
Claims (10)
1. a kind of cow oestrus behavior method of real-time, which is characterized in that including:
S1 obtains the light stream image sequence of the milk cow to be monitored according to the original sequence of milk cow to be monitored, to the light
Flow image sequences carry out feature extraction, obtain Optical-flow Feature corresponding with the light stream image sequence;
S2 detects the milk cow to be monitored from the original sequence, and extracts the edge wheel of the milk cow to be monitored
Wide feature;
The Optical-flow Feature and edge contour feature input are preset neural network by S3, and according to the default nerve net
The output result of network is monitored the estrus behavior of the milk cow to be monitored.
2. according to the method described in claim 1, it is characterized in that, according to the original image sequence of milk cow to be monitored described in step S1
The light stream image sequence that row obtain the milk cow to be monitored further comprises:
The light stream image sequence of the milk cow to be monitored is obtained according to the original sequence of the milk cow to be monitored using optical flow method
Row.
3. according to the method described in claim 1, it is characterized in that, carrying out feature to the light stream image sequence described in step S1
Extraction obtains Optical-flow Feature corresponding with the light stream image sequence and further comprises:
The convolutional layer and down-sampling layer that the light stream image sequence is inputted to the default neural network, by the convolutional layer and
The down-sampling layer carries out feature extraction to the light stream image sequence, and it is special to obtain light stream corresponding with the light stream image sequence
Sign.
4. according to the method described in claim 1, it is characterized in that, the step S2 further comprises:
Notable figure corresponding with the original sequence is obtained, the image entropy of the notable figure is calculated, according to described image entropy
Detect the milk cow to be monitored;
The edge contour feature of the milk cow to be monitored is extracted using edge detection algorithm.
5. according to the method described in claim 1, it is characterized in that, the step S3 further comprises:
The Optical-flow Feature and the edge contour feature are inputted to the full articulamentum of the default neural network, and according to described
The output result of default neural network output layer is monitored the estrus behavior of the milk cow to be monitored.
6. according to the method described in claim 1, it is characterized in that, further including to the default nerve net before the step S3
Network is trained, and is specifically included:
The original sequence sample with behavior label is obtained, according to the original sequence sample with behavior label
Obtain the light stream image sequence sample with behavior label;
To the light stream image sequence sample progress feature extraction for carrying behavior and marking, obtains and carry what behavior marked with described
The corresponding Optical-flow Feature of light stream image sequence sample;
Milk cow sample is detected from the light stream image sequence sample marked with behavior, and extracts the milk cow sample
Edge contour feature;
The Optical-flow Feature and edge contour feature input are preset into neural network, the default neural network is instructed
Practice.
7. a kind of cow oestrus behavior real-time monitoring system, which is characterized in that including:
Fisrt feature extraction module, the light stream for obtaining the milk cow to be monitored according to the original sequence of milk cow to be monitored
Image sequence carries out feature extraction to the light stream image sequence, obtains Optical-flow Feature corresponding with the light stream image sequence;
Second feature extraction module, for detecting the milk cow to be monitored from the original sequence, and described in extraction
The edge contour feature of milk cow to be monitored;
Behavior monitoring module presets neural network for inputting the Optical-flow Feature and the edge contour feature, and according to
The output result of the default neural network is monitored the estrus behavior of the milk cow to be monitored.
8. system according to claim 7, which is characterized in that further include training module, be used for:
The original sequence sample with behavior label is obtained, according to the original sequence sample with behavior label
Obtain the light stream image sequence sample with behavior label;
To the light stream image sequence sample progress feature extraction for carrying behavior and marking, obtains and carry what behavior marked with described
The corresponding Optical-flow Feature of light stream image sequence sample;
Milk cow sample is detected from the light stream image sequence sample marked with behavior, and extracts the milk cow sample
Edge contour feature;
The Optical-flow Feature and edge contour feature input are preset into neural network, the default neural network is instructed
Practice.
9. a kind of equipment of cow oestrus behavior method of real-time, which is characterized in that including:
At least one processor;And
At least one processor being connect with the processor communication, wherein:
The memory is stored with the program instruction that can be executed by the processor, and the processor calls described program to instruct energy
Enough methods executed as described in claim 1 to 6 is any.
10. a kind of non-transient computer readable storage medium, which is characterized in that the non-transient computer readable storage medium is deposited
Computer instruction is stored up, the computer instruction makes the computer execute the method as described in claim 1 to 6 is any.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201810263803.4A CN108491807B (en) | 2018-03-28 | 2018-03-28 | Real-time monitoring method and system for oestrus of dairy cows |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201810263803.4A CN108491807B (en) | 2018-03-28 | 2018-03-28 | Real-time monitoring method and system for oestrus of dairy cows |
Publications (2)
Publication Number | Publication Date |
---|---|
CN108491807A true CN108491807A (en) | 2018-09-04 |
CN108491807B CN108491807B (en) | 2020-08-28 |
Family
ID=63316493
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201810263803.4A Active CN108491807B (en) | 2018-03-28 | 2018-03-28 | Real-time monitoring method and system for oestrus of dairy cows |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN108491807B (en) |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN109919445A (en) * | 2019-02-01 | 2019-06-21 | 广州影子科技有限公司 | Cultivate task management method and device, cultivation task management equipment and system |
CN110070051A (en) * | 2019-04-24 | 2019-07-30 | 卿胜奎 | Ox heat monitoring method and system based on computer vision technique |
CN110866481A (en) * | 2019-11-07 | 2020-03-06 | 北京小龙潜行科技有限公司 | Sow oestrus detection method and device |
JP6833231B1 (en) * | 2020-04-13 | 2021-02-24 | Assest株式会社 | Livestock estrous cycle discrimination program |
CN113743368A (en) * | 2021-09-18 | 2021-12-03 | 内蒙古工业大学 | Behavior monitoring method and device, storage medium and electronic equipment |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN104881647A (en) * | 2015-05-27 | 2015-09-02 | 北京旷视科技有限公司 | Information processing method, information processing system and information processing apparatus |
CN106778902A (en) * | 2017-01-03 | 2017-05-31 | 河北工业大学 | Milk cow individual discrimination method based on depth convolutional neural networks |
CN107038713A (en) * | 2017-04-12 | 2017-08-11 | 南京航空航天大学 | A kind of moving target method for catching for merging optical flow method and neutral net |
US20170278289A1 (en) * | 2016-03-22 | 2017-09-28 | Uru, Inc. | Apparatus, systems, and methods for integrating digital media content into other digital media content |
CN107609635A (en) * | 2017-08-28 | 2018-01-19 | 哈尔滨工业大学深圳研究生院 | A kind of physical object speed estimation method based on object detection and optical flow computation |
-
2018
- 2018-03-28 CN CN201810263803.4A patent/CN108491807B/en active Active
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN104881647A (en) * | 2015-05-27 | 2015-09-02 | 北京旷视科技有限公司 | Information processing method, information processing system and information processing apparatus |
US20170278289A1 (en) * | 2016-03-22 | 2017-09-28 | Uru, Inc. | Apparatus, systems, and methods for integrating digital media content into other digital media content |
CN106778902A (en) * | 2017-01-03 | 2017-05-31 | 河北工业大学 | Milk cow individual discrimination method based on depth convolutional neural networks |
CN107038713A (en) * | 2017-04-12 | 2017-08-11 | 南京航空航天大学 | A kind of moving target method for catching for merging optical flow method and neutral net |
CN107609635A (en) * | 2017-08-28 | 2018-01-19 | 哈尔滨工业大学深圳研究生院 | A kind of physical object speed estimation method based on object detection and optical flow computation |
Non-Patent Citations (2)
Title |
---|
YANG QIN ETAL.: ""Multi-channel Features Fitted 3D CNNs and LSTMs for Human Activity Recognition"", 《2016 TENTH INTERNATIONAL CONFERENCE ON SENSING TECHNOLOGY》 * |
赵宏伟 等: ""视觉显著目标的自适应分割"", 《光学精密工程》 * |
Cited By (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN109919445A (en) * | 2019-02-01 | 2019-06-21 | 广州影子科技有限公司 | Cultivate task management method and device, cultivation task management equipment and system |
CN110070051A (en) * | 2019-04-24 | 2019-07-30 | 卿胜奎 | Ox heat monitoring method and system based on computer vision technique |
CN110866481A (en) * | 2019-11-07 | 2020-03-06 | 北京小龙潜行科技有限公司 | Sow oestrus detection method and device |
CN110866481B (en) * | 2019-11-07 | 2023-06-30 | 北京小龙潜行科技有限公司 | Sow oestrus detection method and device |
JP6833231B1 (en) * | 2020-04-13 | 2021-02-24 | Assest株式会社 | Livestock estrous cycle discrimination program |
WO2021210459A1 (en) * | 2020-04-13 | 2021-10-21 | Assest株式会社 | Livestock estrous period determination program |
JP2021168060A (en) * | 2020-04-13 | 2021-10-21 | Assest株式会社 | Livestock estrus determination program |
CN113743368A (en) * | 2021-09-18 | 2021-12-03 | 内蒙古工业大学 | Behavior monitoring method and device, storage medium and electronic equipment |
Also Published As
Publication number | Publication date |
---|---|
CN108491807B (en) | 2020-08-28 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN108491807A (en) | A kind of cow oestrus behavior method of real-time and system | |
Ranjan et al. | Detection and classification of leaf disease using artificial neural network | |
Wu et al. | Detection and counting of banana bunches by integrating deep learning and classic image-processing algorithms | |
CN110321767A (en) | Image acquiring apparatus and method, behavior analysis system and storage medium | |
Yang et al. | Pig mounting behaviour recognition based on video spatial–temporal features | |
WO2010117576A2 (en) | Image processing and machine learning for diagnostic analysis of microcirculation | |
Jadhav et al. | Soybean leaf disease detection and severity measurement using multiclass SVM and KNN classifier | |
CN110263920A (en) | Convolutional neural networks model and its training method and device, method for inspecting and device | |
CN112257702A (en) | Crop disease identification method based on incremental learning | |
Li et al. | Cow individual identification based on convolutional neural network | |
CN108460370B (en) | Fixed poultry life information alarm device | |
Ratnayake et al. | Towards computer vision and deep learning facilitated pollination monitoring for agriculture | |
Niu et al. | Automatic localization of optic disc based on deep learning in fundus images | |
CN112528823A (en) | Striped shark movement behavior analysis method and system based on key frame detection and semantic component segmentation | |
CN108664942A (en) | The extracting method and video classification methods of mouse video multidimensional characteristic value | |
Tenorio et al. | Automatic visual estimation of tomato cluster maturity in plant rows | |
de Melo et al. | A robust segmentation method for counting bovine milk somatic cells in microscope slide images | |
Ravivarma et al. | An efficient system for the detection of exudates in colour fundus images using image processing technique | |
Nooraiyeen | Robotic vehicle for automated detection of leaf diseases | |
Tangtisanon et al. | Holy basil curl leaf disease classification using edge detection and machine learning | |
Jiang et al. | Navigation path curve extraction method based on depth image for combine harvester | |
CN103699907B (en) | Pesticide spraying detection method based on machine learning | |
Maheswaran et al. | Efficient plant leaf disease identification Material Fabrication using lightweight device | |
Zelman et al. | Nearly automatic motion capture system for tracking octopus arm movements in 3D space | |
Charoenpong et al. | Pupil extraction system for Nystagmus diagnosis by using K-mean clustering and Mahalanobis distance technique |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |