CN109784181A - Picture watermark recognition methods, device, equipment and computer readable storage medium - Google Patents
Picture watermark recognition methods, device, equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN109784181A CN109784181A CN201811540605.4A CN201811540605A CN109784181A CN 109784181 A CN109784181 A CN 109784181A CN 201811540605 A CN201811540605 A CN 201811540605A CN 109784181 A CN109784181 A CN 109784181A
- Authority
- CN
- China
- Prior art keywords
- picture
- watermark
- identified
- convolutional neural
- neural networks
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 48
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 62
- 238000003062 neural network model Methods 0.000 claims abstract description 37
- 230000002708 enhancing effect Effects 0.000 claims abstract description 13
- 230000003252 repetitive effect Effects 0.000 claims abstract description 12
- 238000012549 training Methods 0.000 claims description 27
- 239000000284 extract Substances 0.000 claims description 12
- 238000013475 authorization Methods 0.000 claims description 5
- 238000012935 Averaging Methods 0.000 claims description 4
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 claims description 2
- 235000013399 edible fruits Nutrition 0.000 claims 1
- 238000001514 detection method Methods 0.000 abstract description 9
- 210000005036 nerve Anatomy 0.000 abstract description 3
- 238000005457 optimization Methods 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 11
- 238000000605 extraction Methods 0.000 description 8
- 238000004891 communication Methods 0.000 description 5
- 239000000047 product Substances 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 238000012015 optical character recognition Methods 0.000 description 3
- 238000012795 verification Methods 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 230000001629 suppression Effects 0.000 description 2
- 241000208340 Araliaceae Species 0.000 description 1
- 102000008169 Co-Repressor Proteins Human genes 0.000 description 1
- 108010060434 Co-Repressor Proteins Proteins 0.000 description 1
- 235000005035 Panax pseudoginseng ssp. pseudoginseng Nutrition 0.000 description 1
- 235000003140 Panax quinquefolius Nutrition 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 238000012550 audit Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 239000006227 byproduct Substances 0.000 description 1
- 238000012512 characterization method Methods 0.000 description 1
- 238000012790 confirmation Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 235000008434 ginseng Nutrition 0.000 description 1
- 210000004218 nerve net Anatomy 0.000 description 1
- 230000001537 neural effect Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
Abstract
The present invention relates to technical field of image detection, disclosing a kind of picture watermark recognition methods, device, equipment and computer readable storage medium, picture watermark recognition methods includes: to carry out data enhancing to existing picture library, obtains sample graph valut；Pass through the picture in sample graph valut, preset convolutional neural networks model is trained based on the mode of repetitive exercise, obtains target convolution neural network model, wherein, target convolution neural network model includes inception network and prediction interval, and prediction interval is FPN structure；Picture to be identified is inputted into target convolution neural network model, the characteristic information of picture to be identified is extracted by inception network；According to prediction interval and characteristic information, the watermark in picture to be identified is identified, obtain the watermark recognition result of picture to be identified.Through the invention, former SSD network structure is optimized, according to the target nerve network model that optimization obtains, carries out picture watermark identification, recognition accuracy is greatly improved.
Description
Technical field
The present invention relates to technical field of image detection more particularly to picture watermark recognition methods, device, equipment and computers
Readable storage medium storing program for executing.
Background technique
Image contains abundant and intuitive information, currently in fields such as social activity, the shopping of internet, requires a large amount of
Image transmits information to user.Since the propagation of internet information is extremely quick and portable, more and more individuals and organizations
It selects to be embedded in watermark information to owned image, the watermark of trade mark or network address is such as stamped in image section region, is protected with this
Protect the ownership of image information.Therefore, the provider of image information needs to audit image before using image, identifies
Whether contain watermark information in image, avoids the occurrence of the behavior of misuse and infringement.
On the one hand, with the rapid development of Internet, image provider can be uploaded daily using user, crawler is downloaded etc.
All multipaths obtain great amount of images information, and quantity is considerably beyond the limit of manual examination and verification.On the other hand, watermark information is in image
In vision significance it is very low, have area small, of light color, the features such as transparency is high, watermarked image and non-watermarked image
Between difference often very little, discrimination are lower.Therefore, by way of manual examination and verification, it is difficult to realize to watermark picture
It accurately identifies.
Summary of the invention
The main purpose of the present invention is to provide a kind of picture watermark recognition methods, device, equipment and computer-readable deposit
Storage media, it is intended to solve in the prior art by way of manual examination and verification, it is difficult to realize to accurately identifying with watermark picture
Technical problem.
To achieve the above object, the present invention provides a kind of picture watermark recognition methods, the picture watermark recognition methods packet
Include following steps:
Data enhancing is carried out to existing picture library, obtains sample graph valut；
By the picture in sample graph valut, preset convolutional neural networks model is instructed based on the mode of repetitive exercise
Practice, obtain target convolution neural network model, wherein the target convolution neural network model include inception network with
And prediction interval, the prediction interval are FPN structure；
Picture to be identified is inputted into the target convolution neural network model, institute is extracted by the inception network
State the characteristic information of picture to be identified；
According to the prediction interval and the characteristic information, the watermark in the picture to be identified is identified, obtain described wait know
The watermark recognition result of other picture.
Optionally, described the step of carrying out data enhancing to existing picture library, obtaining sample graph valut, includes:
The watermark of original image in existing picture library is adjusted, newly-increased picture is obtained；
Based on original image and newly-increased picture, sample graph valut is obtained.
Optionally, the preset convolutional neural networks model includes inception network and prediction interval, the prediction interval
For FPN structure, the picture by sample graph valut, based on the mode of repetitive exercise to preset convolutional neural networks model
The step of being trained, obtaining target convolution neural network model include:
Picture in sample graph valut is pre-processed, obtains training picture, the quantity of the trained picture is at least
Two；
Every trained picture is inputted into preset convolutional neural networks model, extracts every by the inception network
The characteristic information of training picture；
According to the characteristic information of the prediction interval and every trained picture, identifies the watermark in every trained picture, obtain
The watermark recognition result of every trained picture；
The watermark recognition result of every trained picture and corresponding original markup information are compared, obtain every
The corresponding similarity of training picture, carries out calculating of averaging to obtained similarity, obtains map value；
Detect whether the map value meets accuracy requirement；
If the map value meets accuracy requirement, using the preset convolutional neural networks model as target convolutional Neural
Network model；
If the map value is unsatisfactory for accuracy requirement, parameter adjustment is carried out to the preset convolutional neural networks model, is obtained
To new convolutional neural networks model；
Using the new convolutional neural networks model as preset convolutional neural networks model, and execute described by every instruction
Practice picture and input preset convolutional neural networks model, is believed by the feature that the inception network extracts every trained picture
The step of breath.
Optionally, the picture in sample graph valut pre-processes, obtain train picture the step of include:
The picture format of picture in sample graph valut is adjusted to preset picture format, obtains training picture.
Optionally, described according to the prediction interval and the characteristic information, it identifies the watermark in the picture to be identified, obtains
To the picture to be identified watermark recognition result the step of after, further includes:
If determining that the picture to be identified is without watermark picture, then by described wait know according to the watermark recognition result
Other picture is stored into stand-by picture library.
Optionally, described according to the prediction interval and the characteristic information, it identifies the watermark in the picture to be identified, obtains
To the picture to be identified watermark recognition result the step of after, further includes:
If determining the picture to be identified for band watermark picture according to the watermark recognition result；
It determines the affiliated obligee of the picture to be identified, and send authorization to the affiliated right human hair of the picture to be identified
Application.
Optionally, the affiliated obligee of the determination picture to be identified, and to the ownership of the picture to be identified
The step of sharp human hair send authorized application include:
According to the watermark recognition result, the watermark in the picture to be identified is obtained；
According to the watermark, the affiliated obligee of the picture to be identified is determined；
It detects in preset contact information record sheet, if there are the corresponding targets of affiliated obligee of the picture to be identified
Contact information；
The corresponding target contact information of the affiliated obligee if it exists is then based on the target contact information, Xiang Suoshu
The affiliated right human hair of picture to be identified send authorized application.
In addition, to achieve the above object, the present invention also provides a kind of picture watermark identification device, the picture watermark identification
Device includes:
Data enhance module, for carrying out data enhancing to existing picture library, obtain sample graph valut；
Training module, for passing through the picture in sample graph valut, based on the mode of repetitive exercise to preset convolutional Neural
Network model is trained, and obtains target convolution neural network model, wherein the target convolution neural network model includes
Inception network and prediction interval, the prediction interval are FPN structure；
Characteristic information extracting module passes through institute for picture to be identified to be inputted the target convolution neural network model
State the characteristic information that inception network extracts the picture to be identified；
Identification module, for identifying the watermark in the picture to be identified according to the prediction interval and the characteristic information,
Obtain the watermark recognition result of the picture to be identified.
In addition, to achieve the above object, the present invention also provides a kind of picture watermarks to identify equipment, the picture watermark identification
Equipment includes: that the picture watermark that can run on the memory and on the processor of memory, processor and being stored in is known
Other program, the picture watermark recognizer realize picture watermark recognition methods as described above when being executed by the processor
Step.
In addition, to achieve the above object, it is described computer-readable the present invention also provides a kind of computer readable storage medium
Picture watermark recognizer is stored on storage medium, the picture watermark recognizer realizes institute as above when being executed by processor
The step of picture watermark recognition methods stated.
In the present invention, data enhancing is carried out to existing picture library, obtains sample graph valut；Pass through the figure in sample graph valut
Piece is trained preset convolutional neural networks model based on the mode of repetitive exercise, obtains target convolution neural network model,
Wherein, the target convolution neural network model includes inception network and prediction interval, and the prediction interval is FPN structure；
Picture to be identified is inputted into the target convolution neural network model, is extracted by the inception network described to be identified
The characteristic information of picture；According to the prediction interval and the characteristic information, identifies the watermark in the picture to be identified, obtain institute
State the watermark recognition result of picture to be identified.Through the invention, former SSD network structure is optimized, is obtained according to optimization
Target nerve network model, carry out picture watermark identification, recognition accuracy is greatly improved.
Detailed description of the invention
Fig. 1 is that the picture watermark for the hardware running environment that the embodiment of the present invention is related to identifies device structure schematic diagram；
Fig. 2 is the flow diagram of picture watermark recognition methods first embodiment of the present invention；
Fig. 3 is SSD schematic network structure in the prior art；
Fig. 4 is the structural representation of preset convolutional neural networks model in one embodiment of picture watermark recognition methods of the present invention
Figure；
Fig. 5 is the structural schematic diagram of the prediction interval of original SSD network；
Fig. 6 is the structural schematic diagram of prediction interval in picture watermark recognition methods first embodiment of the present invention；
Fig. 7 is the functional block diagram of picture watermark identification device first embodiment of the present invention.
The embodiments will be further described with reference to the accompanying drawings for the realization, the function and the advantages of the object of the present invention.
Specific embodiment
It should be appreciated that the specific embodiments described herein are merely illustrative of the present invention, it is not intended to limit the present invention.
As shown in FIG. 1, FIG. 1 is the picture watermark identification equipment knots for the hardware running environment that the embodiment of the present invention is related to
Structure schematic diagram.
Picture watermark of embodiment of the present invention identification equipment can be PC, portable computer, server etc. with data processing
Function terminal equipment.
As shown in Figure 1, picture watermark identification equipment may include: processor 1001, such as CPU, network interface 1004,
User interface 1003, memory 1005, communication bus 1002.Wherein, communication bus 1002 is for realizing between these components
Connection communication.User interface 1003 may include display screen (Display), input unit such as keyboard (Keyboard), optional
User interface 1003 can also include standard wireline interface and wireless interface.Network interface 1004 optionally may include standard
Wireline interface, wireless interface (such as WI-FI interface).Memory 1005 can be high speed RAM memory, be also possible to stable
Memory (non-volatile memory), such as magnetic disk storage.Memory 1005 optionally can also be independently of aforementioned
The storage device of processor 1001.
It will be understood by those skilled in the art that the identification device structure of picture watermark shown in Fig. 1 is not constituted to picture
Watermark identifies the restriction of equipment, may include perhaps combining certain components or different than illustrating more or fewer components
Component layout.
As shown in Figure 1, as may include that operating system, network are logical in a kind of memory 1005 of computer storage medium
Believe module, Subscriber Interface Module SIM and picture watermark recognizer.
In picture watermark identification equipment shown in Fig. 1, network interface 1004 is mainly used for connecting background server, and rear
Platform server carries out data communication；User interface 1003 is mainly used for connecting client (user terminal), carries out data with client
Communication；And processor 1001 can be used for calling the picture watermark recognizer stored in memory 1005, and execute with the following figure
Operation in each embodiment of piece watermark recognition methods.
It is the flow diagram of picture watermark recognition methods first embodiment of the present invention referring to Fig. 2, Fig. 2.
In the present embodiment, the picture watermark recognition methods includes:
Step S10 carries out data enhancing to existing picture library, obtains sample graph valut；
In one embodiment, step S10 includes:
The watermark of original image in existing picture library is adjusted, newly-increased picture is obtained；
Based on original image and newly-increased picture, sample graph valut is obtained.
In the present embodiment, the watermark in the original image in existing picture library is adjusted, including watermark is turned over
Turn and/or to position of the watermark in picture is changed, in this way, which several can be obtained after an original image is modified
Newly-increased picture.For example, watermark to be rotated to 90 degree respectively for a picture A1 with watermark a, rotated by 3 times, this
When, the picture in picture library are as follows: picture A1, picture A2, picture A3, picture A4, then the watermark location in every picture is changed
Once, at this point, picture in picture library are as follows: picture A1, picture A2, picture A3, picture A4, picture A11, picture A21, picture
A31, picture A41.As it can be seen that by the above-mentioned means, can derive to obtain 7 additional pictures by an original image, even
Data enhancing is carried out to every original image in existing picture library in the manner described above, then can make the figure in existing picture library
Piece quantity rises to 8n by n, greatly improves the quantity of picture in existing picture library.I.e. by carrying out data to existing picture library
The mode of enhancing obtains sample graph valut.It is subsequent, preset convolutional neural networks model is trained by sample graph valut,
Since the picture number in sample graph valut is enough, it may make that training effect is more preferable, the convolutional neural networks mould that training obtains
Type has better generalization ability.
Step S20, by the picture in sample graph valut, based on the mode of repetitive exercise to preset convolutional neural networks mould
Type is trained, and obtains target convolution neural network model, wherein the target convolution neural network model includes
Inception network and prediction interval, the prediction interval are FPN structure；
In the present embodiment, the nuclear structure of preset convolutional neural networks model is former using improved SSD network structure
The SSD of beginning uses VGG-16 as basic network, and adds feature extraction layer on VGG-16 basic network.Referring to Fig. 3, Fig. 3
For SSD schematic network structure in the prior art.As shown in figure 3, SSD network is divided into 3 parts, first part is using VGG-16
As the feature extraction layer of basic network, for carrying out feature extraction to the picture of input, second part is prediction interval, is used for base
Prediction of result is carried out in the characteristic information that first part extracts, Part III is output layer, for exporting second part prediction
Obtained result.
It is the knot of preset convolutional neural networks model in one embodiment of picture watermark recognition methods of the present invention referring to Fig. 4, Fig. 4
Structure schematic diagram.As shown in figure 4, preset convolutional neural networks model is divided into part A, part B and part C.Wherein, part A is
Inception network, for carrying out feature extraction to the picture of input；Part B is using FPN (feature pyramid
Networks) the prediction interval of structure, the characteristic information for being extracted based on inception network carry out prediction of result；Part C
For output layer, the result predicted for output par, c B.
Since object interested in different pictures (watermark of the present embodiment middle finger) has different size and accounting, and it is big
Convolution kernel be easy to be abstracted big object, small convolution kernel is easy to be abstracted small object, different size of object picture is wanted
Think to obtain preferable feature abstraction simultaneously, it is necessary to which constantly (different layers use different volumes to the number of plies of intensification convolutional neural networks
Product core), but a large amount of calculating can be brought simultaneously.Such as the convolution kernel of a 3*3 in VGG-16, outputting and inputting all is 256
If, the parameter amount of one layer of needs is 3*3*256*256, and the parameter amount that this is needed for multilayer convolutional neural networks is undoubtedly
It is huge, so original SSD algorithm has only got the layer 5 of VGG-16 network, caused consequence is that picture feature
Abstracting power is inadequate.And Inception network is by the way of partially connected and parallel-convolution, effective solution this ask
Topic, so bring calculation amount does not increase while deepening the convolutional neural networks number of plies using Inception network.That is phase
VGG-16 is used compared with former SSD network, uses inception network that can extract more features from picture in the present embodiment
Information.
It is the structural schematic diagram of the prediction interval of original SSD network referring to Fig. 5, Fig. 5.As shown in Figure 5, wherein white blocks are volume
Product, black block is prediction interval.Predicted operation is not carried out in the prediction interval of former SSD network, the characteristic information obtained for the bottom,
Characteristic information only is obtained to high level and carries out predicted operation, high-level characteristic be it is after convolutional neural networks higher level of abstraction as a result, so
Lead to detect wisp that accuracy is poor, and watermark is generally smaller, that is, causes former SSD network to the prediction result of watermark
Accuracy rate is poor.It is prediction interval in picture watermark recognition methods first embodiment of the present invention referring to Fig. 6, Fig. 6 in the present embodiment
Structural schematic diagram.As shown in Figure 6, wherein white blocks are convolution, and black block is prediction interval.Prediction interval uses FPN structure, passes through
After lateral connection, each layer is predicted, while high-rise information can be passed to bottom, finally all predictions are melted
Conjunction obtains final result.Specific implementation are as follows: rebuild to obtain in inception network and rebuild layer and (adopted in the present embodiment
With the prediction interval of FPN structure), the reconstruction layer and feature extraction layer (feature extraction layer is inception network in the present embodiment)
Carry out lateral mapping.Prediction interval using FPN structure the Semantic detection of low feature extraction layer can be added to detection in come, while cross
The accuracy that ensure that object space on different resolution to mapping improves the detection accuracy to watermark.
In one embodiment, the process of repetitive exercise are as follows:
Picture in sample graph valut is pre-processed, obtains training picture, the quantity of training picture is at least two,
For example, the picture format of picture in sample graph valut is adjusted to preset picture format, obtain training picture, in the present embodiment,
It before picture in sample graph valut is inputted preset convolutional neural networks model, needs to pre-process picture, pre-process
Mode can be, picture is adjusted to unified format (such as JPG), ensure that picture input consistency, be conducive to improve instruction
Practice effect；Every trained picture is inputted into preset convolutional neural networks model, every training figure is extracted by inception network
The characteristic information of piece；According to the characteristic information of prediction interval and every trained picture, identifies the watermark in every trained picture, obtain
The watermark recognition result of every trained picture；The watermark recognition result of every trained picture and corresponding original mark are believed
Breath compares, and obtains the corresponding similarity of every trained picture, carries out calculating of averaging to obtained similarity, obtain
Map value；Whether detection map value meets accuracy requirement；If map value meets accuracy requirement, with preset convolutional neural networks model
As target convolution neural network model；If map value is unsatisfactory for accuracy requirement, preset convolutional neural networks model is joined
Number adjustment, obtains new convolutional neural networks model；Using new convolutional neural networks model as preset convolutional neural networks mould
Type, and execute and every trained picture is inputted into preset convolutional neural networks model, every training is extracted by inception network
The step of characteristic information of picture.Wherein, original markup information includes the location information of watermark in trained picture.If watermark is known
The watermark location information that other result includes is more close with the location information that original markup information includes, then similarity is higher, causes
Map value is higher, conversely, map is lower.Map has reacted the superiority and inferiority of convolutional neural networks, and map is higher, illustrates current convolutional Neural
The result of network output is more accurate.In the present embodiment, a threshold value can be preset, if map value is higher than threshold value, i.e. map value meets essence
Degree demand, then current convolutional neural networks model can be used for watermark detection, then using current convolutional neural networks model as
Target convolution neural network model.If map value is lower than threshold value, parameter adjustment (ginseng is carried out to preset convolutional neural networks model
The embodiment of number adjustment can refer to the prior art, and neural network is substantially a calculation process, inputs and believes in front end receiver
After number, process complicated operation from level to level exports result in least significant end.Then calculated result is compared with correct result, is obtained
To error, further according to error by the relevant parameter of corresponding advance in caculating means network internal, so that network receives next time again
When same data, the final error calculated between the obtained result of output and correct result can be smaller and smaller), obtain new volume
Then training picture is inputted currently available new convolutional neural networks model by product neural network model, and according to new volume
Product neural network model is directed to the watermark recognition result of training picture output and the original markup information of training picture compares,
New map value is obtained, if map value is greater than preset threshold, using new convolutional neural networks model as target convolution nerve net
Network model, otherwise repeats the above steps, until map value be greater than preset threshold when, using corresponding convolutional neural networks model as
Target convolution neural network model.
Picture to be identified is inputted the target convolution neural network model, passes through the inception net by step S30
Network extracts the characteristic information of the picture to be identified；
In the present embodiment, inception network, can be used inceptionV1, inceptionV2, inceptionV3,
Any one in inceptionV4.Wherein, the characteristic information of picture refers to the main information that can indicate this picture.By one
It opens picture to be identified to be input in target convolution neural network model, target convolution neural network model should by convolution Nuclear receptor co repressor
The pixel value of picture, comprising many convolution kernels, (particular number carries out according to actual needs in target convolution neural network model
Setting), it after the completion of preceding layer convolution, then gives next layer of convolution and carries out operation, each convolution kernel can extract different in picture
Information (extracting mode is that the local picture pixel value of each convolution kernel and scanning does dot-product operation).And use inception
Network be because it using parallel-convolution core by the way of, comparison vgg16 for, abstract energy of the identical convolution number of plies for image
Power is stronger, so that the picture feature indicated is more.
Step S40 identifies the watermark in the picture to be identified, obtains according to the prediction interval and the characteristic information
The watermark recognition result of the picture to be identified.
In the present embodiment, the characteristic information of picture to be identified is extracted by inception network, is tied according to FPN is used
The prediction interval of structure is based on characteristic information, obtains the watermark recognition result for picture to be identified.Watermark recognition result includes three kinds
Type, type one: unidentified to arrive watermark；Type two: watermark is recognized；Type three: watermark is recognized, and watermark location is
xxxx。
By way of step S30, characteristic information is obtained, this feature information, i.e. the pixel position of characteristic point are to get arriving
Characterize the characteristic pattern of characteristics of image.It is raw first centered on each unit on characteristic pattern using the prediction interval of FPN structure
At the first verifying frame of length and width difference ratio, then in each frame picture carry out object prediction, prediction classification confidence level and
Bezel locations are compared with the classification of the middle object of truthful data and bezel locations, because might have multiple priori frame predictions
At the same classification, so selecting most suitable frame using the method for NMS (non-maxima suppression), that is, last object is being schemed
In position.In the present embodiment, i.e. progress watermark prediction, and be watermark by classification, and the position of the highest frame of confidence level, as
The output of watermark recognition result, the watermark recognition result, i.e. position of the watermark in picture to be identified.
In the present embodiment, data enhancing is carried out to existing picture library, obtains sample graph valut；By in sample graph valut
Picture is trained preset convolutional neural networks model based on the mode of repetitive exercise, obtains target convolutional neural networks mould
Type, wherein the target convolution neural network model includes inception network and prediction interval, and the prediction interval is FPN knot
Structure；Picture to be identified is inputted into the target convolution neural network model, is extracted by the inception network described wait know
The characteristic information of other picture；According to the prediction interval and the characteristic information, identifies the watermark in the picture to be identified, obtain
The watermark recognition result of the picture to be identified.Through this embodiment, former SSD network structure is optimized, according to optimization
Obtained target nerve network model, carries out picture watermark identification, and recognition accuracy is greatly improved.
Optionally, in an embodiment, after step S40, further includes:
If determining that the picture to be identified is without watermark picture, then by described wait know according to the watermark recognition result
Other picture is stored into stand-by picture library.
In the present embodiment, if watermark recognition result is " unidentified to arrive watermark ", which is particularly likely that energy
Enough use without compensations, just directly picture to be identified is stored into stand-by picture library, when for subsequent needing, from stand-by picture library
Middle calling picture to be identified.
Optionally, in an embodiment, after step S40, further includes:
If determining the picture to be identified for band watermark picture according to the watermark recognition result；
It determines the affiliated obligee of the picture to be identified, and send authorization to the affiliated right human hair of the picture to be identified
Application.
In the present embodiment, if recognizing watermark, it can be determined according to the network originating or watermark style of the picture to be identified
The obligee of the picture to be identified, and authorized application is sent to the affiliated right human hair of the picture to be identified.
In one alternative embodiment, the affiliated obligee of the picture to be identified is determined, and to the institute of the picture to be identified
Belonging to the step of obligee sends authorized application includes:
According to the watermark recognition result, the watermark in the picture to be identified is obtained；
In the present embodiment, watermark recognition result can be pixel coordinate of the watermark in picture to be identified, which sits
Mark, i.e. position of the watermark in picture to be identified.In the present embodiment, output watermark location is set by watermark recognition result, i.e.,
Watermark recognition result is " to recognize watermark, and when watermark location is xxxx ", " xxxx " indicates watermark in picture to be identified
Coordinate information.In the present embodiment, by way of step S30, characteristic information, this feature information, the i.e. pixel of characteristic point are obtained
Point position, can be obtained the characteristic pattern of characterization characteristics of image.Using the prediction interval of FPN structure, first with each on characteristic pattern
Centered on a unit, the not year-on-year first verifying frame of length and width is generated, object prediction then is carried out for the picture in each frame, in advance
It surveys classification confidence level and bezel locations is compared with the classification of the middle object of truthful data and bezel locations, because might have
Multiple priori frame predictions are at the same classification, so selecting most suitable frame using the method for NMS (non-maxima suppression), also
It is position of the last object in figure.In the present embodiment, i.e. progress watermark prediction, and be watermark, and confidence level highest by classification
Frame position, as watermark recognition result export, the watermark recognition result, i.e. position of the watermark in picture to be identified.
According to the watermark, the affiliated obligee of the picture to be identified is determined；
According to the watermark location information of above-mentioned output, the picture region of corresponding position in picture to be identified is extracted, and to this
Picture region, which carries out Text region, (can pass through OCR, Optical Character Recognition, optical character recognition technology
Realize), such as Text region result is " company A ", it is determined that the obligee of the picture to be identified is company A.
It detects in preset contact information record sheet, if there are the corresponding targets of affiliated obligee of the picture to be identified
Contact information；
The contact information that whether there is company A in preset contact information record sheet is searched, in preset contact information record sheet
The contact information of multiple contact persons can be stored in advance.
The corresponding target contact information of the affiliated obligee if it exists is then based on the target contact information, Xiang Suoshu
The affiliated right human hair of picture to be identified send authorized application.
The contact information of company A if it exists is then based on the contact information, sends authorized application to company A.For example, preset
There are the mailboxes of company A in contact information record sheet, then the authorized application editted in advance are sent to the mailbox.
It is subsequent, if the license confirmation that the affiliated right human hair for receiving picture to be identified is sent, which is stored
Into stand-by picture library, when for subsequent needing, the picture to be identified is called from stand-by picture library.
In the present embodiment, when, there are when watermark, authorized application being sent to the institute of the picture to be identified in picture to be identified
Belong to obligee, can avoid the directly bring risk of infringement using the picture to be identified.
In addition, the embodiment of the present invention also proposes a kind of picture watermark identification device.
It is the functional block diagram of picture watermark identification device first embodiment of the present invention referring to Fig. 7, Fig. 7.
In the present embodiment, picture watermark identification device includes:
Data enhance module 10, for carrying out data enhancing to existing picture library, obtain sample graph valut；
Training module 20, for passing through the picture in sample graph valut, based on the mode of repetitive exercise to preset convolution mind
It is trained through network model, obtains target convolution neural network model, wherein the target convolution neural network model includes
Inception network and prediction interval, the prediction interval are FPN structure；
Characteristic information extracting module 30 passes through for picture to be identified to be inputted the target convolution neural network model
The inception network extracts the characteristic information of the picture to be identified；
Identification module 40, for identifying the water in the picture to be identified according to the prediction interval and the characteristic information
Print, obtains the watermark recognition result of the picture to be identified.
Further, data enhancing module 10 includes:
Adjustment unit is adjusted for the watermark to original image in existing picture library, obtains newly-increased picture；
Integral unit obtains sample graph valut for being based on original image and newly-increased picture.
Further, the preset convolutional neural networks model includes inception network and prediction interval, the prediction
Layer is FPN structure, and training module 20 includes:
Pretreatment unit obtains training picture, the training figure for pre-processing the picture in sample graph valut
The quantity of piece is at least two；
Feature extraction unit, for every trained picture to be inputted preset convolutional neural networks model, by described
Inception network extracts the characteristic information of every trained picture；
Recognition unit identifies every trained picture for the characteristic information according to the prediction interval and every trained picture
In watermark, obtain the watermark recognition result of every trained picture；
Comparison unit, for carrying out pair the watermark recognition result of every trained picture and corresponding original markup information
Than obtaining the corresponding similarity of every trained picture, carrying out calculating of averaging to obtained similarity, obtain map value；
Detection unit, for detecting whether the map value meets accuracy requirement；
Determination unit, if meeting accuracy requirement for the map value, using the preset convolutional neural networks model as
Target convolution neural network model；
Adjustment unit, if being unsatisfactory for accuracy requirement for the map value, to the preset convolutional neural networks model into
The adjustment of row parameter, obtains new convolutional neural networks model；
Execution unit is returned, for using the new convolutional neural networks model as preset convolutional neural networks model,
And execute it is described every trained picture is inputted into preset convolutional neural networks model, extracted by the inception network every
The step of characteristic information of Zhang Xunlian picture.
Further, pretreatment unit includes:
Format adjusting subelement is obtained for the picture format of picture in sample graph valut to be adjusted to preset picture format
To training picture.
Further, picture watermark identification device further include:
Memory module 50, if determining that the picture to be identified is without watermark figure for according to the watermark recognition result
Piece then stores the picture to be identified into stand-by picture library.
Further, picture watermark identification device further include:
Judgment module 60, if determining the picture to be identified for band watermark picture for according to the watermark recognition result；
Authorization module 70 is requested, for determining the affiliated obligee of the picture to be identified, and to the picture to be identified
Affiliated right human hair send authorized application.
Further, request authorization module includes:
Acquiring unit, for obtaining the watermark in the picture to be identified according to the watermark recognition result；
Obligee's determination unit, for determining the affiliated obligee of the picture to be identified according to the watermark；
Detection unit, for detecting in preset contact information record sheet, if there are the ownerships of the picture to be identified
The corresponding target contact information of sharp people；
Authorized application transmission unit is then based on institute for the corresponding target contact information of the affiliated obligee if it exists
Target contact information is stated, send authorized application to the affiliated right human hair of the picture to be identified.
Each embodiment of the specific embodiment of picture watermark identification device of the present invention and above-mentioned picture watermark recognition methods
Essentially identical, this will not be repeated here.
In addition, the embodiment of the present invention also proposes a kind of computer readable storage medium, the computer readable storage medium
On be stored with picture watermark recognizer, the picture watermark recognizer realizes picture as described above when being executed by processor
The step of watermark recognition methods.
Each implementation of the specific embodiment of computer readable storage medium of the present invention and above-mentioned picture watermark recognition methods
Example is essentially identical, and this will not be repeated here.
It should be noted that, in this document, the terms "include", "comprise" or its any other variant are intended to non-row
His property includes, so that the process, method, article or the system that include a series of elements not only include those elements, and
And further include other elements that are not explicitly listed, or further include for this process, method, article or system institute it is intrinsic
Element.In the absence of more restrictions, the element limited by sentence "including a ...", it is not excluded that including being somebody's turn to do
There is also other identical elements in the process, method of element, article or system.
The serial number of the above embodiments of the invention is only for description, does not represent the advantages or disadvantages of the embodiments.
Through the above description of the embodiments, those skilled in the art can be understood that above-described embodiment side
Method can be realized by means of software and necessary general hardware platform, naturally it is also possible to by hardware, but in many cases
The former is more preferably embodiment.Based on this understanding, technical solution of the present invention substantially in other words does the prior art
The part contributed out can be embodied in the form of software products, which is stored in one as described above
In storage medium (such as ROM/RAM, magnetic disk, CD), including some instructions are used so that terminal device (it can be mobile phone,
Computer, server, air conditioner or network equipment etc.) execute method described in each embodiment of the present invention.
The above is only a preferred embodiment of the present invention, is not intended to limit the scope of the invention, all to utilize this hair
Equivalent structure or equivalent flow shift made by bright specification and accompanying drawing content is applied directly or indirectly in other relevant skills
Art field, is included within the scope of the present invention.
Claims (10)
1. a kind of picture watermark recognition methods, which is characterized in that the picture watermark recognition methods the following steps are included:
Data enhancing is carried out to existing picture library, obtains sample graph valut；
By the picture in sample graph valut, preset convolutional neural networks model is trained based on the mode of repetitive exercise,
Obtain target convolution neural network model, wherein the target convolution neural network model is comprising inception network and in advance
Layer is surveyed, the prediction interval is FPN structure；
Picture to be identified is inputted into the target convolution neural network model, by the inception network extract it is described to
Identify the characteristic information of picture；
According to the prediction interval and the characteristic information, identifies the watermark in the picture to be identified, obtain the figure to be identified
The watermark recognition result of piece.
2. picture watermark recognition methods as described in claim 1, which is characterized in that described to carry out data increasing to existing picture library
By force, the step of obtaining sample graph valut include:
The watermark of original image in existing picture library is adjusted, newly-increased picture is obtained；
Based on original image and newly-increased picture, sample graph valut is obtained.
3. picture watermark recognition methods as described in claim 1, which is characterized in that the preset convolutional neural networks model packet
Network containing inception and prediction interval, the prediction interval are FPN structure, and the picture by sample graph valut is based on
The step of mode of repetitive exercise is trained preset convolutional neural networks model, obtains target convolution neural network model packet
It includes:
Picture in sample graph valut is pre-processed, obtains training picture, the quantity of the trained picture is at least two；
Every trained picture is inputted into preset convolutional neural networks model, every training is extracted by the inception network
The characteristic information of picture；
According to the characteristic information of the prediction interval and every trained picture, identifies the watermark in every trained picture, obtain every
The watermark recognition result of training picture；
The watermark recognition result of every trained picture and corresponding original markup information are compared, every training is obtained
The corresponding similarity of picture carries out calculating of averaging to obtained similarity, obtains map value；
Detect whether the map value meets accuracy requirement；
If the map value meets accuracy requirement, using the preset convolutional neural networks model as target convolutional neural networks
Model；
If the map value is unsatisfactory for accuracy requirement, parameter adjustment is carried out to the preset convolutional neural networks model, is obtained new
Convolutional neural networks model；
Using the new convolutional neural networks model as preset convolutional neural networks model, and execute described by every training figure
Piece inputs preset convolutional neural networks model, extracts the characteristic information of every trained picture by the inception network
Step.
4. picture watermark recognition methods as claimed in claim 3, which is characterized in that the picture in sample graph valut into
Row pretreatment, obtain train picture the step of include:
The picture format of picture in sample graph valut is adjusted to preset picture format, obtains training picture.
5. picture watermark recognition methods as described in claim 1, which is characterized in that described according to the prediction interval and the spy
After the step of reference ceases, and identifies the watermark in the picture to be identified, obtains the watermark recognition result of the picture to be identified,
Further include:
If determining that the picture to be identified is without watermark picture, then by the figure to be identified according to the watermark recognition result
Piece is stored into stand-by picture library.
6. the picture watermark recognition methods as described in any one of claims 1 to 5, which is characterized in that described according to described pre-
Layer and the characteristic information are surveyed, identifies the watermark in the picture to be identified, obtains the watermark identification knot of the picture to be identified
After the step of fruit, further includes:
If determining the picture to be identified for band watermark picture according to the watermark recognition result；
It determines the affiliated obligee of the picture to be identified, and send authorization Shen to the affiliated right human hair of the picture to be identified
Please.
7. picture watermark recognition methods as claimed in claim 6, which is characterized in that the institute of the determination picture to be identified
Belong to obligee, and the step of sending authorized application to the affiliated right human hair of the picture to be identified includes:
According to the watermark recognition result, the watermark in the picture to be identified is obtained；
According to the watermark, the affiliated obligee of the picture to be identified is determined；
It detects in preset contact information record sheet, if there are the corresponding target contacts of the affiliated obligee of the picture to be identified
Information；
The corresponding target contact information of the affiliated obligee if it exists is then based on the target contact information, to described wait know
The affiliated right human hair of other picture send authorized application.
8. a kind of picture watermark identification device, which is characterized in that the picture watermark identification device includes:
Data enhance module, for carrying out data enhancing to existing picture library, obtain sample graph valut；
Training module, for passing through the picture in sample graph valut, based on the mode of repetitive exercise to preset convolutional neural networks
Model is trained, and obtains target convolution neural network model, wherein the target convolution neural network model includes
Inception network and prediction interval, the prediction interval are FPN structure；
Characteristic information extracting module, for picture to be identified to be inputted the target convolution neural network model, by described
Inception network extracts the characteristic information of the picture to be identified；
Identification module, for identifying the watermark in the picture to be identified, obtaining according to the prediction interval and the characteristic information
The watermark recognition result of the picture to be identified.
9. a kind of picture watermark identifies equipment, which is characterized in that the picture watermark identification equipment includes: memory, processor
And the picture watermark recognizer that is stored on the memory and can run on the processor, the picture watermark identification
The step of picture watermark recognition methods as described in any one of claims 1 to 7 is realized when program is executed by the processor.
10. a kind of computer readable storage medium, which is characterized in that be stored with picture water on the computer readable storage medium
Recognizer is printed, is realized as described in any one of claims 1 to 7 when the picture watermark recognizer is executed by processor
The step of picture watermark recognition methods.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811540605.4A CN109784181B (en) | 2018-12-14 | 2018-12-14 | Picture watermark identification method, device, equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811540605.4A CN109784181B (en) | 2018-12-14 | 2018-12-14 | Picture watermark identification method, device, equipment and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109784181A true CN109784181A (en) | 2019-05-21 |
| CN109784181B CN109784181B (en) | 2024-03-22 |
Family
ID=66497081
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811540605.4A Active CN109784181B (en) | 2018-12-14 | 2018-12-14 | Picture watermark identification method, device, equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109784181B (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110428357A (en) * | 2019-08-09 | 2019-11-08 | 厦门美图之家科技有限公司 | The detection method of watermark, device, electronic equipment and storage medium in image |
| CN110675399A (en) * | 2019-10-28 | 2020-01-10 | 上海悦易网络信息技术有限公司 | Screen appearance flaw detection method and equipment |
| CN110766598A (en) * | 2019-10-29 | 2020-02-07 | 厦门大学嘉庚学院 | Intelligent model watermark embedding and extracting method and system based on convolutional neural network |
| CN110852242A (en) * | 2019-11-06 | 2020-02-28 | 北京字节跳动网络技术有限公司 | Watermark identification method, device, equipment and storage medium based on multi-scale network |
| CN110991488A (en) * | 2019-11-08 | 2020-04-10 | 广州坚和网络科技有限公司 | Image watermark identification method using deep learning model |
| CN111062854A (en) * | 2019-12-26 | 2020-04-24 | Oppo广东移动通信有限公司 | Method, device, terminal and storage medium for detecting watermark |
| CN111104542A (en) * | 2020-01-07 | 2020-05-05 | 成都睿琪科技有限责任公司 | Part identification management method and device |
| CN111324871A (en) * | 2020-03-09 | 2020-06-23 | 河南大学 | Big data watermarking method and device based on artificial intelligence |
| CN112200711A (en) * | 2020-10-27 | 2021-01-08 | 支付宝(杭州)信息技术有限公司 | Training method and system of watermark classification model |
| CN112419135A (en) * | 2020-11-19 | 2021-02-26 | 广州华多网络科技有限公司 | Watermark recognition online training, sampling and removing method, device, equipment and medium |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070252073A1 (en) * | 2005-08-04 | 2007-11-01 | Korea Advanced Institute Of Science And Technology | High sensitivity and high dynamic-range CMOS image sensor pixel structure with dynamic C-V characteristics |
| CN105740910A (en) * | 2016-02-02 | 2016-07-06 | 北京格灵深瞳信息技术有限公司 | Vehicle object detection method and device |
| CN106096668A (en) * | 2016-08-18 | 2016-11-09 | 携程计算机技术（上海）有限公司 | The recognition methods of watermarked image and the system of identification |
| CN106504064A (en) * | 2016-10-25 | 2017-03-15 | 清华大学 | Clothes classification based on depth convolutional neural networks recommends method and system with collocation |
| CN107808358A (en) * | 2017-11-13 | 2018-03-16 | 携程计算机技术（上海）有限公司 | Image watermark automatic testing method |
| CN108764164A (en) * | 2018-05-30 | 2018-11-06 | 华中科技大学 | A kind of method for detecting human face and system based on deformable convolutional network |
| CN108846826A (en) * | 2018-04-24 | 2018-11-20 | 深圳大学 | Object detecting method, device, image processing equipment and storage medium |
| CN108875595A (en) * | 2018-05-29 | 2018-11-23 | 重庆大学 | A kind of Driving Scene object detection method merged based on deep learning and multilayer feature |
-
2018
- 2018-12-14 CN CN201811540605.4A patent/CN109784181B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070252073A1 (en) * | 2005-08-04 | 2007-11-01 | Korea Advanced Institute Of Science And Technology | High sensitivity and high dynamic-range CMOS image sensor pixel structure with dynamic C-V characteristics |
| CN105740910A (en) * | 2016-02-02 | 2016-07-06 | 北京格灵深瞳信息技术有限公司 | Vehicle object detection method and device |
| CN106096668A (en) * | 2016-08-18 | 2016-11-09 | 携程计算机技术（上海）有限公司 | The recognition methods of watermarked image and the system of identification |
| CN106504064A (en) * | 2016-10-25 | 2017-03-15 | 清华大学 | Clothes classification based on depth convolutional neural networks recommends method and system with collocation |
| CN107808358A (en) * | 2017-11-13 | 2018-03-16 | 携程计算机技术（上海）有限公司 | Image watermark automatic testing method |
| CN108846826A (en) * | 2018-04-24 | 2018-11-20 | 深圳大学 | Object detecting method, device, image processing equipment and storage medium |
| CN108875595A (en) * | 2018-05-29 | 2018-11-23 | 重庆大学 | A kind of Driving Scene object detection method merged based on deep learning and multilayer feature |
| CN108764164A (en) * | 2018-05-30 | 2018-11-06 | 华中科技大学 | A kind of method for detecting human face and system based on deformable convolutional network |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110428357A (en) * | 2019-08-09 | 2019-11-08 | 厦门美图之家科技有限公司 | The detection method of watermark, device, electronic equipment and storage medium in image |
| CN110675399A (en) * | 2019-10-28 | 2020-01-10 | 上海悦易网络信息技术有限公司 | Screen appearance flaw detection method and equipment |
| CN110766598B (en) * | 2019-10-29 | 2023-04-07 | 厦门大学嘉庚学院 | Intelligent model watermark embedding and extracting method and system based on convolutional neural network |
| CN110766598A (en) * | 2019-10-29 | 2020-02-07 | 厦门大学嘉庚学院 | Intelligent model watermark embedding and extracting method and system based on convolutional neural network |
| CN110852242A (en) * | 2019-11-06 | 2020-02-28 | 北京字节跳动网络技术有限公司 | Watermark identification method, device, equipment and storage medium based on multi-scale network |
| CN110991488A (en) * | 2019-11-08 | 2020-04-10 | 广州坚和网络科技有限公司 | Image watermark identification method using deep learning model |
| CN110991488B (en) * | 2019-11-08 | 2023-10-20 | 广州坚和网络科技有限公司 | Picture watermark identification method using deep learning model |
| CN111062854A (en) * | 2019-12-26 | 2020-04-24 | Oppo广东移动通信有限公司 | Method, device, terminal and storage medium for detecting watermark |
| CN111062854B (en) * | 2019-12-26 | 2023-08-25 | Oppo广东移动通信有限公司 | Method, device, terminal and storage medium for detecting watermark |
| CN111104542A (en) * | 2020-01-07 | 2020-05-05 | 成都睿琪科技有限责任公司 | Part identification management method and device |
| CN111104542B (en) * | 2020-01-07 | 2023-04-18 | 成都睿琪科技有限责任公司 | Part identification management method and device |
| CN111324871A (en) * | 2020-03-09 | 2020-06-23 | 河南大学 | Big data watermarking method and device based on artificial intelligence |
| CN112200711A (en) * | 2020-10-27 | 2021-01-08 | 支付宝(杭州)信息技术有限公司 | Training method and system of watermark classification model |
| CN112419135A (en) * | 2020-11-19 | 2021-02-26 | 广州华多网络科技有限公司 | Watermark recognition online training, sampling and removing method, device, equipment and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109784181B (en) | 2024-03-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109784181A (en) | Picture watermark recognition methods, device, equipment and computer readable storage medium | |
| Liu et al. | Cross‐ethnicity face anti‐spoofing recognition challenge: A review | |
| CN110276366A (en) | Carry out test object using Weakly supervised model | |
| CN107844748A (en) | Auth method, device, storage medium and computer equipment | |
| WO2017087568A1 (en) | A digital image capturing device system and method | |
| CN110135246A (en) | A kind of recognition methods and equipment of human action | |
| CN107657259A (en) | Distorted image detection method, electronic installation and readable storage medium storing program for executing | |
| EP4109332A1 (en) | Certificate authenticity identification method and apparatus, computer-readable medium, and electronic device | |
| JP2014505896A5 (en) | ||
| CN112200187A (en) | Target detection method, device, machine readable medium and equipment | |
| CN111539412B (en) | Image analysis method, system, device and medium based on OCR | |
| CN111401374A (en) | Model training method based on multiple tasks, character recognition method and device | |
| CN111652087A (en) | Car checking method and device, electronic equipment and storage medium | |
| CN110795714A (en) | Identity authentication method and device, computer equipment and storage medium | |
| CN111753721A (en) | Human body posture recognition method and device | |
| CN113988180A (en) | Model fingerprint-based generated image tracing method | |
| CN117058723B (en) | Palmprint recognition method, palmprint recognition device and storage medium | |
| CN109391836B (en) | Supplementing a media stream with additional information | |
| JP6498674B2 (en) | A method for annotating objects in multimedia assets | |
| WO2020224127A1 (en) | Video stream capturing method and apparatus, and storage medium | |
| CN108369647A (en) | Quality control based on image | |
| CN110442807A (en) | A kind of webpage type identification method, device, server and storage medium | |
| CN114448664A (en) | Phishing webpage identification method and device, computer equipment and storage medium | |
| CN113177409A (en) | Intelligent sensitive word recognition system | |
| CN111753852A (en) | Tea leaf identification method, recommendation method, tea leaf identification device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |