CN114842476A - Watermark detection method and device and model training method and device - Google Patents

Watermark detection method and device and model training method and device Download PDF

Info

Publication number
CN114842476A
CN114842476A CN202210745781.1A CN202210745781A CN114842476A CN 114842476 A CN114842476 A CN 114842476A CN 202210745781 A CN202210745781 A CN 202210745781A CN 114842476 A CN114842476 A CN 114842476A
Authority
CN
China
Prior art keywords
image
watermark
detected
sample
prediction result
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210745781.1A
Other languages
Chinese (zh)
Inventor
崔东林
邓天生
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Baidu Netcom Science and Technology Co Ltd
Original Assignee
Beijing Baidu Netcom Science and Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Baidu Netcom Science and Technology Co Ltd filed Critical Beijing Baidu Netcom Science and Technology Co Ltd
Priority to CN202210745781.1A priority Critical patent/CN114842476A/en
Publication of CN114842476A publication Critical patent/CN114842476A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/10Character recognition
    • G06V30/14Image acquisition
    • G06V30/146Aligning or centring of the image pick-up or image-field
    • G06V30/147Determination of region of interest
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/70Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/10Character recognition
    • G06V30/19Recognition using electronic means
    • G06V30/191Design or setup of recognition systems or techniques; Extraction of features in feature space; Clustering techniques; Blind source separation
    • G06V30/19147Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00Indexing scheme relating to image or video recognition or understanding
    • G06V2201/09Recognition of logos

Abstract

The disclosure provides a watermark detection method and device and a model training method and device, and relates to the field of image processing, in particular to the fields of artificial intelligence and deep learning. The implementation scheme is as follows: carrying out predictive analysis on an image to be detected to obtain a predictive result of the image to be detected; responding to the image to be detected including the image area, and performing character recognition on the image to be detected in the image area; and in response to identifying the text content in the image region, determining whether the target watermark is contained in the image to be detected based on the text content.

Description

Watermark detection method and device and model training method and device
Technical Field
The present disclosure relates to the field of image processing, particularly to the field of artificial intelligence and deep learning, and more particularly to a watermark detection method, a model training method, a watermark detection apparatus, a model training apparatus, an electronic device, and a computer-readable storage medium.
Background
Artificial intelligence is the subject of research that makes computers simulate some human mental processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.), both at the hardware level and at the software level. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing, and the like; the artificial intelligence software technology mainly comprises a computer vision technology, a voice recognition technology, a natural language processing technology, machine learning/deep learning, a big data processing technology, a knowledge map technology and the like.
With the rapid development of internet technology, a plurality of visual products of different types emerge, and different products have unique watermark identifications, so that mutual carrying of resources among each other is avoided, infringement is prevented, and when resources are distributed or searched for are captured, watermarks in the resources are generally required to be effectively detected.
The approaches described in this section are not necessarily approaches that have been previously conceived or pursued. Unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Similarly, unless otherwise indicated, the problems mentioned in this section should not be considered as having been acknowledged in any prior art.
Disclosure of Invention
The present disclosure provides a watermark detection method, a model training method, a watermark detection apparatus, a model training apparatus, an electronic device, and a computer-readable storage medium.
According to an aspect of the present disclosure, there is provided a watermark detection method, including: performing predictive analysis on an image to be detected to obtain a prediction result of the image to be detected, wherein the prediction result indicates whether the image to be detected comprises an image area possibly containing a watermark or not; responding to the image to be detected including the image area, and performing character recognition on the image to be detected in the image area; and in response to identifying the text content in the image region, determining whether the target watermark is contained in the image to be detected based on the text content.
According to another aspect of the present disclosure, there is provided a model training method, including: the method comprises the steps of obtaining a sample data set, wherein the sample data set comprises at least one positive sample image and at least one negative sample image, watermark marking information of each positive sample image in the at least one positive sample image comprises at least one watermark marking area and at least one first watermark label corresponding to the at least one watermark marking area, and watermark marking information of each negative sample image in the at least one negative sample image comprises a second watermark label; based on the sample data set, the following steps are executed until the model converges: acquiring a first sample image based on the sample data set, wherein the first sample image comprises at least one piece of watermark marking information; inputting the first sample image into a model to obtain at least one watermark prediction result; and adjusting parameters of the model based on each watermark prediction result in the at least one watermark prediction result and the watermark marking information corresponding to the watermark prediction result.
According to another aspect of the present disclosure, there is provided a watermark detection apparatus including: the detection device comprises a first acquisition unit, a second acquisition unit and a third acquisition unit, wherein the first acquisition unit is configured to perform prediction analysis on an image to be detected so as to acquire a prediction result of the image to be detected, and the prediction result indicates whether an image area in which a watermark possibly exists is included in the image to be detected or not; the recognition unit is configured to respond to the image to be detected including the image area, and perform character recognition on the image to be detected in the image area; and a first determination unit configured to determine whether the target watermark is contained in the image to be detected based on the text content in response to the text content being identified in the image area.
According to another aspect of the present disclosure, there is provided a model training apparatus including: the fourth obtaining unit is configured to obtain a sample data set, wherein the sample data set comprises at least one positive sample image and at least one negative sample image, the watermark labeling information of each positive sample image in the at least one positive sample image comprises at least one watermark labeling area and at least one first watermark label corresponding to the at least one watermark labeling area, and the watermark labeling information of each negative sample image in the at least one negative sample image comprises a second watermark label; an execution unit configured to execute the following sub-unit operations based on the sample data set until the model converges: an obtaining subunit, configured to obtain a first sample image based on the sample data set, where the first sample image includes at least one piece of watermark annotation information; a second input subunit configured to input the first sample image into the model to obtain at least one watermark prediction result; and an adjusting subunit, configured to adjust parameters of the model based on each watermark prediction result of the at least one watermark prediction result and corresponding watermark marking information of the watermark prediction result.
According to another aspect of the present disclosure, there is provided an electronic device including: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the watermark detection method or the model training method described above.
According to another aspect of the present disclosure, there is provided a non-transitory computer readable storage medium storing computer instructions for causing a computer to execute the above watermark detection method or model training method.
According to one or more embodiments of the disclosure, after a watermark prediction region in an image is acquired, the region is further subjected to character recognition, and whether the region contains a target watermark is judged based on a character recognition result, so that verification of a model recognition result can be realized, and the accuracy of watermark recognition is improved.
It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present disclosure, nor do they limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the embodiments and, together with the description, serve to explain the exemplary implementations of the embodiments. The illustrated embodiments are for purposes of illustration only and do not limit the scope of the claims. Throughout the drawings, identical reference numbers designate similar, but not necessarily identical, elements.
FIG. 1 illustrates a schematic diagram of an exemplary system in which various methods described herein may be implemented, according to an embodiment of the present disclosure;
fig. 2 shows a flow diagram of a watermark detection method according to an embodiment of the present disclosure;
fig. 3 shows a flow chart of a watermark detection method according to an embodiment of the present disclosure;
fig. 4 shows a block flow diagram of a watermark detection method according to an exemplary embodiment of the present disclosure;
FIG. 5 shows a flow diagram of a model training method according to an embodiment of the present disclosure;
fig. 6 shows a block diagram of a watermark detection apparatus according to an embodiment of the present disclosure;
FIG. 7 shows a block diagram of a model training apparatus according to an embodiment of the present disclosure;
FIG. 8 illustrates a block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure.
Detailed Description
Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which various details of the embodiments of the disclosure are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
In the present disclosure, unless otherwise specified, the use of the terms "first", "second", and the like to describe various elements is not intended to limit the positional relationship, the temporal relationship, or the importance relationship of the elements, and such terms are used only to distinguish one element from another. In some examples, a first element and a second element may refer to the same instance of the element, and in some cases, based on the context, they may also refer to different instances.
The terminology used in the description of the various described examples in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the elements may be one or more. Furthermore, the term "and/or" as used in this disclosure is intended to encompass any and all possible combinations of the listed items.
In the related art, some watermark detection methods based on deep learning often use different types of watermarks as different detection targets, and detect the watermarks through a model, and if a new product watermark appears, the model cannot accurately identify the new product watermark, so that a large number of false detection conditions often exist, and the identification accuracy is difficult to guarantee.
The embodiment of the disclosure provides a watermark detection method, which further performs character recognition on an area after acquiring a watermark prediction area in an image, and judges whether the area contains a target watermark or not based on a character recognition result, so that verification of a model recognition result is realized, and the accuracy of watermark recognition is improved.
Embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.
Fig. 1 illustrates a schematic diagram of an exemplary system 100 in which various methods and apparatus described herein may be implemented in accordance with embodiments of the present disclosure. Referring to fig. 1, the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. Client devices 101, 102, 103, 104, 105, and 106 may be configured to execute one or more applications.
In embodiments of the present disclosure, the server 120 may run one or more services or software applications that enable the watermark detection method to be performed.
In some embodiments, the server 120 may also provide other services or software applications, which may include non-virtual environments and virtual environments. In certain embodiments, these services may be provided as web-based services or cloud services, for example, provided to users of client devices 101, 102, 103, 104, 105, and/or 106 under a software as a service (SaaS) model.
In the configuration shown in fig. 1, server 120 may include one or more components that implement the functions performed by server 120. These components may include software components, hardware components, or a combination thereof, which may be executed by one or more processors. A user operating a client device 101, 102, 103, 104, 105, and/or 106 may, in turn, utilize one or more client applications to interact with the server 120 to take advantage of the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from system 100. Accordingly, fig. 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.
A user may use client devices 101, 102, 103, 104, 105, and/or 106 to acquire images of a sample to be detected. The client device may provide an interface that enables a user of the client device to interact with the client device. The client device may also output information to the user via the interface. Although fig. 1 depicts only six client devices, those skilled in the art will appreciate that any number of client devices may be supported by the present disclosure.
Client devices 101, 102, 103, 104, 105, and/or 106 may include various types of computer devices, such as portable handheld devices, general purpose computers (such as personal computers and laptops), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, and so forth. These computer devices may run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux, or Linux-like operating systems (e.g., GOOGLE Chrome OS); or include various Mobile operating systems such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android. Portable handheld devices may include cellular telephones, smart phones, tablet computers, Personal Digital Assistants (PDAs), and the like. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. The gaming system may include a variety of handheld gaming devices, internet-enabled gaming devices, and the like. The client device is capable of executing a variety of different applications, such as various Internet-related applications, communication applications (e.g., email applications), Short Message Service (SMS) applications, and may use a variety of communication protocols.
Network 110 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including but not limited to TCP/IP, SNA, IPX, etc. By way of example only, one or more networks 110 may be a Local Area Network (LAN), an ethernet-based network, a token ring, a Wide Area Network (WAN), the internet, a virtual network, a Virtual Private Network (VPN), an intranet, an extranet, a blockchain network, a Public Switched Telephone Network (PSTN), an infrared network, a wireless network (e.g., bluetooth, WIFI), and/or any combination of these and/or other networks.
The server 120 may include one or more general purpose computers, special purpose server computers (e.g., PC (personal computer) servers, UNIX servers, mid-end servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and/or combination. The server 120 may include one or more virtual machines running a virtual operating system, or other computing architecture involving virtualization (e.g., one or more flexible pools of logical storage that may be virtualized to maintain virtual storage for the server). In various embodiments, the server 120 may run one or more services or software applications that provide the functionality described below.
The computing units in server 120 may run one or more operating systems including any of the operating systems described above, as well as any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and/or middle tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, and the like.
In some implementations, the server 120 may include one or more applications to analyze and consolidate data feeds and/or event updates received from users of the client devices 101, 102, 103, 104, 105, and/or 106. Server 120 may also include one or more applications to display data feeds and/or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and/or 106.
In some embodiments, the server 120 may be a server of a distributed system, or a server incorporating a blockchain. The server 120 may also be a cloud server, or a smart cloud computing server or a smart cloud host with artificial intelligence technology. The cloud Server is a host product in a cloud computing service system, and is used for solving the defects of high management difficulty and weak service expansibility in the traditional physical host and Virtual Private Server (VPS) service.
The system 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. The database 130 may reside in various locations. For example, the database used by the server 120 may be local to the server 120, or may be remote from the server 120 and may communicate with the server 120 via a network-based or dedicated connection. The database 130 may be of different types. In certain embodiments, the database used by the server 120 may be, for example, a relational database. One or more of these databases may store, update, and retrieve data to and from the database in response to the command.
In some embodiments, one or more of the databases 130 may also be used by applications to store application data. The databases used by the application may be different types of databases, such as key-value stores, object stores, or regular stores supported by a file system.
The system 100 of fig. 1 may be configured and operated in various ways to enable application of the various methods and apparatus described in accordance with the present disclosure.
According to an embodiment of the present disclosure, as shown in fig. 2, there is provided a watermark detection method including: step S201, carrying out prediction analysis on an image to be detected to obtain a prediction result of the image to be detected, wherein the prediction result indicates whether the image to be detected comprises an image area possibly containing a watermark or not; s202, in response to the image to be detected including an image area, performing character recognition on the image to be detected in the image area; and step S203, responding to the character content identified in the image area, and determining whether the target watermark is contained in the image to be detected or not based on the character content.
Therefore, after the watermark prediction area in the image is obtained, character recognition is further carried out on the area, and whether the area contains the target watermark or not is judged based on the character recognition result, so that the model recognition result can be verified, and the accuracy of watermark recognition is improved.
In some embodiments, the image to be detected may be a picture or may be derived from a video. For the video watermark detection by the method, the watermark detection can be carried out on each frame of image or some frames of images in the video, and whether the video contains the watermark or not is judged according to the watermark detection result of each frame.
In some embodiments, the image to be detected may be an image obtained after preprocessing based on an original image derived from picture data or video data. The pre-processing may include converting the image into a predetermined format (e.g., RGB format), resizing the image to a predetermined size (e.g., 640 × 640), and normalizing each pixel value of the image.
In some embodiments, after the image to be detected is acquired, the image to be detected may be subjected to predictive analysis first.
According to some embodiments, performing predictive analysis on an image to be detected to obtain a prediction result of the image to be detected may include: inputting an image to be detected into a watermark detection model to obtain an image area output by the watermark detection model and a corresponding prediction probability of the image area, wherein the prediction probability indicates the possibility that the image area contains the watermark; and determining that the image to be detected comprises the image area in response to the prediction probability being greater than the preset probability threshold.
In some embodiments, the preprocessed image to be detected may be input into a watermark detection model, and the image to be detected is subjected to predictive analysis by the watermark detection model, so as to obtain an image area in which a watermark may exist in the image to be detected and a predictive probability of the watermark in the image area.
The watermark detection model can be constructed based on a convolutional neural network model, for example. The image area output by the watermark detection model may be, for example, a watermark detection box, and the area may be represented by outputting coordinates of the watermark detection box.
In some embodiments, the probability of the watermark being present in the image to be detected can be determined by comparing the prediction probability with a preset probability threshold. Exemplarily, when the prediction probability is greater than a preset probability threshold, determining that the image to be detected includes an image region where the watermark may exist; and when the prediction probability is smaller than or equal to a preset probability threshold, judging that the image to be detected does not include an image area in which the watermark possibly exists, namely judging that the image does not have the watermark.
In some embodiments, the preset probability threshold may be adaptively set according to actual needs of different service scenarios, so that the prediction accuracy of the watermark detection model can be further improved.
According to some embodiments, as shown in fig. 3, the watermark detection method may further include: step S301, obtaining a plurality of first images, wherein each first image in the plurality of first images comprises a watermark label, and the watermark label indicates whether the first image corresponding to the watermark label contains a watermark; step S302, acquiring a watermark prediction result of each first image in a plurality of first images based on a watermark detection model and a preset probability threshold; step S303, determining a recall rate and a false detection rate of a watermark detection model based on a watermark prediction result of each first image in a plurality of first images and a watermark label of the first image; and step S304, adjusting a preset probability threshold value based on the recall rate and the false detection rate, and judging whether the image to be detected comprises an image area based on the adjusted preset probability threshold value.
In some embodiments, the collection of relevant image data may also be performed for the business scenario to be applied. In some embodiments, the collected image data may be first preprocessed and each image data may be labeled, so as to obtain a plurality of first images for evaluating the model in the business scenario.
In some embodiments, the plurality of first images may be respectively input into a watermark detection model, and a prediction result of each first image is obtained based on a preset probability threshold and a model output result; and then, counting the false detection rate and the recall rate of the model according to the prediction result of each first image and the watermark label thereof, and adjusting the preset probability threshold according to the statistical result so that the model has higher accuracy based on the adjusted preset probability threshold in the service scene.
In some embodiments, a plurality of different preset probability thresholds (e.g., 9 different preset probability thresholds such as 0.2,0.3, …, 0.9, etc.) may also be preset respectively, and prediction analysis is performed through the model based on the plurality of first images and the different preset probability thresholds respectively, and false detection rate and recall rate of the model under the different preset probability thresholds are counted respectively, and an optimal preset probability threshold is determined by comparing different statistical results, so that prediction accuracy of the model in a specific service scene can be further improved.
In some embodiments, after an image region that may include a watermark in an image to be detected is obtained by the watermark detection model, text recognition may be further performed on the image in the image region (for example, an OCR text recognition technology may be applied), and when text content is recognized in the region, it may be determined that the image to be detected includes the watermark according to the text content. In an actual application scene, most watermarks contain characters, so that after the watermarks are detected through the model, whether character contents exist in an image area output by the model is identified, secondary judgment is carried out based on the character contents, the accuracy of watermark detection can be further improved, and the condition of false detection is reduced.
Characters and icons in watermarks of different products are greatly different, so that when watermarks of new products appear, a trained watermark detection model cannot accurately identify the newly appearing watermarks, and a larger detection error may be generated. In the related art, when such a situation occurs, the training data set needs to be reconstructed to retrain the model, but in this way, the iterative update efficiency of the model is low, and a large amount of labor and time cost is often consumed.
According to some embodiments, determining whether the target watermark is included in the image to be detected based on the text content comprises: matching with at least one preset vocabulary based on the text content; and in response to the preset vocabulary matched with the text content contained in at least one preset vocabulary, determining that the image to be detected contains the target watermark.
Therefore, through the matching of the text content and at least one preset vocabulary, the watermark of a newly added product does not need to be retrained by reconstructing a training set and can be expanded and identified by only adding a corresponding text into the preset vocabulary. Therefore, the expandability of the watermark identification is improved, the cost of model iteration is saved, and the efficiency of expanding the watermark category of the watermark identification system is improved.
In some embodiments, after the text content is recognized in the image region, the image to be detected may further include a target watermark by matching with at least one preset vocabulary, and after a certain preset vocabulary is successfully matched in the preset vocabulary.
In some embodiments, at least one preset vocabulary may be stored, for example, in a preset vocabulary. The matching of the text content can be performed by a method based on text encoding, for example.
In some embodiments, when the recognized text content is completely consistent with a certain preset vocabulary in a preset vocabulary, the matching is judged to be successful, and the image to be detected can be determined to include the target watermark; and when the matching is unsuccessful, the target watermark is not included in the image to be detected.
In some embodiments, the preset vocabulary in the preset vocabulary can be added or deleted according to actual situations. Therefore, when a new product watermark appears, the characters in the watermark can be stored as preset words in the preset word list, so that when the subsequent watermark detection is carried out, the updated preset word list can be matched, and the accurate identification of the new product watermark is realized.
Fig. 4 shows a block flow diagram of a watermark detection method according to an example embodiment of the present disclosure. As shown in fig. 4, the watermark detection method may include: s401, acquiring an image to be detected; step S402, preprocessing the image to be detected in a uniform format, a uniform image size and the like; step S403, inputting the preprocessed image into a watermark detection model, and performing predictive analysis, thereby obtaining a predictive analysis result (including an image region that may include a watermark and a predictive probability); step S404, comparing the prediction probability with a preset probability threshold value; step S405, when the prediction probability is smaller than or equal to a preset probability threshold, judging that no watermark exists in the image; step S406, when the prediction probability is greater than a preset probability threshold, outputting an image area which is possibly provided with a watermark in the image to be detected; step S407, performing character recognition on the image to be detected in the image area; step S408, judging whether the text content is detected; step S409, when the text content is not detected, judging that the image does not contain the target watermark; step S410, when the text content is detected, matching with at least one preset vocabulary based on the text content; step S411, when the matching is not successful, judging that the image does not contain the target watermark; and step S412, when the matching is successful, determining that the target watermark is contained in the image.
According to some embodiments, as shown in fig. 5, there is provided a model training method, comprising: step S501, a sample data set is obtained, wherein the sample data set comprises at least one positive sample image and at least one negative sample image, watermark labeling information of each positive sample image in the at least one positive sample image comprises at least one watermark labeling area and at least one first watermark label corresponding to the at least one watermark labeling area, and watermark labeling information of each negative sample image in the at least one negative sample image comprises a second watermark label; based on the sample data set, the following steps are executed until the model converges: step S502, acquiring a first sample image based on a sample data set, wherein the first sample image comprises at least one piece of watermark marking information; step S503, inputting the first sample image into a model to obtain at least one watermark prediction result; and step S504, based on each watermark prediction result in at least one watermark prediction result and the corresponding watermark marking information of the watermark prediction result, adjusting the parameters of the model.
In some embodiments, for training of the watermark detection model, the sample data set may be obtained by collecting data on the line and labeling the watermark in the collected image. In some embodiments, labeling the watermark in the image includes labeling an area in which the watermark is located (including an icon and text of the watermark) and a first watermark label indicating that the area includes the watermark.
In some embodiments, some images not containing watermarks may be acquired at the same time, and labeled with a second watermark label indicating that the area does not contain watermarks, so as to acquire a certain number of negative sample images.
In some embodiments, data enhancement and pre-processing operations may be performed on the labeled images to construct a sample data set. The data enhancement may include, for example, randomly cropping, randomly flipping, color space transforming, and the like of the image; the pre-processing may include, for example, converting the image to some preset format, resizing the image to a predetermined size, normalizing the image by pixel value, and the like.
According to some embodiments, based on the sample data set, acquiring the first sample image may comprise: randomly selecting at least one sample image in the sample data set; performing random sample enhancement on each sample image of the at least one sample image; and synthesizing a first sample image based on the at least one sample image subjected to the random sample enhancement.
In some embodiments, before the sample images are input to the watermark detection model, at least one sample image (for example, 4 sample images) may be first selected from the sample data set, and each sample image is subjected to random data enhancement, and the at least one sample image subjected to random sample enhancement is synthesized into a first sample image by means of stitching and input to the watermark detection model, so as to train the model. The random data enhancement may include, for example, operations such as random scaling, random cropping, random arrangement, and the like.
Therefore, the first sample image is obtained in the mode and is input into the model for model training, the detection effect of the model on the small target can be enhanced while sample data is further enriched, and the use of computing resources is further saved.
According to some embodiments, as shown in fig. 6, there is provided a watermark detection apparatus 600 comprising: a first obtaining unit 610, configured to perform prediction analysis on an image to be detected to obtain a prediction result of the image to be detected, where the prediction result indicates whether an image area in which a watermark may exist is included in the image to be detected; a recognition unit 620 configured to perform character recognition on the image to be detected within the image area in response to the image area being included in the image to be detected; and a first determining unit 630 configured to determine whether the target watermark is included in the image to be detected based on the text content in response to the text content being identified within the image area.
The operations of the units 610 to 630 in the watermark detection apparatus 600 are similar to the operations of the steps S201 to S203 in the above watermark detection method, and are not described herein again.
According to some embodiments, the first determination unit comprises: the matching subunit is configured to match with at least one preset vocabulary based on the text content; and the first determining subunit is configured to determine that the image to be detected contains the target watermark in response to the at least one preset vocabulary containing a preset vocabulary matched with the text content.
According to some embodiments, the first obtaining unit comprises: a first input subunit, configured to input the image to be detected into a watermark detection model to obtain the image region output by the watermark detection model and a corresponding prediction probability of the image region, where the prediction probability indicates a possibility that the image region contains a watermark; and a second determining subunit configured to determine that the image to be detected includes the image region in response to the prediction probability being greater than a preset probability threshold.
According to some embodiments, the watermark detection apparatus further comprises: a second obtaining unit configured to obtain a plurality of first images, wherein each of the plurality of first images includes a watermark tag indicating whether a watermark is included in the corresponding first image of the watermark tag; a third obtaining unit configured to obtain a watermark prediction result of each of the plurality of first images based on the watermark detection model and the preset probability threshold; a second determination unit configured to determine a recall rate and a false detection rate of the watermark detection model based on a watermark prediction result of each of the plurality of first images and a watermark label of the first image; and the adjusting unit is configured to adjust the preset probability threshold value based on the recall rate and the false detection rate so as to judge whether the image to be detected comprises the image area based on the adjusted preset probability threshold value.
According to some embodiments, as shown in fig. 7, there is provided a model training apparatus 700 comprising: a fourth obtaining unit 710, configured to obtain a sample data set, where the sample data set includes at least one positive sample image and at least one negative sample image, watermark annotation information of each positive sample image in the at least one positive sample image includes at least one watermark annotation area and at least one first watermark label corresponding to the at least one watermark annotation area, and watermark annotation information of each negative sample image in the at least one negative sample image includes a second watermark label; an execution unit 720 configured to perform the following sub-unit operations based on the sample data set until the model converges: an obtaining subunit 721 configured to obtain, based on the sample data set, a first sample image, where the first sample image includes at least one piece of watermark annotation information; a second input subunit 722 configured to input the first sample image into a model to obtain at least one watermark prediction result; and an adjusting subunit 723, configured to adjust parameters of the model based on each watermark prediction result of the at least one watermark prediction result and corresponding watermark labeling information of the watermark prediction result.
The operations of the units 710 to 720 and the sub-units 721 to 723 in the model training apparatus 700 are similar to the operations of the steps S501 to S504 in the above model training method, and are not described herein again.
According to some embodiments, the obtaining subunit comprises: a selection module configured to randomly select at least one sample image in the sample data set; a sample enhancement module configured to perform random sample enhancement on each sample image of the at least one sample image; and a synthesis module configured to synthesize the first sample image based on the at least one sample image subjected to the random sample enhancement.
According to an embodiment of the present disclosure, there is also provided an electronic device, a readable storage medium, and a computer program product.
Referring to fig. 8, a block diagram of a structure of an electronic device 800, which may be a server or a client of the present disclosure, which is an example of a hardware device that may be applied to aspects of the present disclosure, will now be described. Electronic device is intended to represent various forms of digital electronic computer devices, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the disclosure described and/or claimed herein.
As shown in fig. 8, the electronic device 800 includes a computing unit 801 that can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM) 802 or a computer program loaded from a storage unit 808 into a Random Access Memory (RAM) 803. In the RAM 803, various programs and data necessary for the operation of the electronic apparatus 800 can also be stored. The calculation unit 801, the ROM 802, and the RAM 803 are connected to each other by a bus 804. An input/output (I/O) interface 805 is also connected to bus 804.
A number of components in the electronic device 800 are connected to the I/O interface 805, including: an input unit 806, an output unit 807, a storage unit 808, and a communication unit 809. The input unit 806 may be any type of device capable of inputting information to the electronic device 800, and the input unit 806 may receive input numeric or character information and generate key signal inputs related to user settings and/or function controls of the electronic device, and may include, but is not limited to, a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and/or a remote controller. Output unit 807 can be any type of device capable of presenting information and can include, but is not limited to, a display, speakers, a video/audio output terminal, a vibrator, and/or a printer. The storage unit 808 may include, but is not limited to, a magnetic disk, an optical disk. The communication unit 809 allows the electronic device 800 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers and/or chipsets, such as bluetooth (TM) devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication devices, and/or the like.
Computing unit 801 may be a variety of general and/or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various dedicated Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, and the like. The calculation unit 801 performs the various methods and processes described above, such as the watermark detection method or the model training method described above. For example, in some embodiments, the watermark detection method or the model training method described above may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and/or installed onto the electronic device 800 via the ROM 802 and/or the communication unit 809. When loaded into RAM 803 and executed by the computing unit 801, a computer program may perform one or more steps of the above described watermark detection method or model training method described above. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the above-described watermark detection method or model training method by any other suitable means (e.g., by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), system on a chip (SOCs), Complex Programmable Logic Devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.
The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server of a distributed system, or a server with a combined blockchain.
It should be understood that various forms of the flows shown above, reordering, adding or deleting steps, may be used. For example, the steps described in the present disclosure may be performed in parallel, sequentially or in different orders, and are not limited herein as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved.
Although embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it is to be understood that the above-described methods, systems and apparatus are merely exemplary embodiments or examples and that the scope of the present invention is not limited by these embodiments or examples, but only by the claims as issued and their equivalents. Various elements in the embodiments or examples may be omitted or may be replaced with equivalents thereof. Further, the steps may be performed in an order different from that described in the present disclosure. Further, various elements in the embodiments or examples may be combined in various ways. It is important that as technology evolves, many of the elements described herein may be replaced with equivalent elements that appear after the present disclosure.

Claims (14)

1. A watermark detection method, characterized in that the method comprises:
performing predictive analysis on an image to be detected to obtain a prediction result of the image to be detected, wherein the prediction result indicates whether the image to be detected comprises an image area in which a watermark possibly exists;
responding to the image to be detected comprising the image area, and performing character recognition on the image to be detected in the image area; and
in response to identifying textual content within the image region, determining whether a target watermark is included in the image to be detected based on the textual content.
2. The method of claim 1, wherein the determining whether the target watermark is included in the image to be detected based on the text content comprises:
matching with at least one preset vocabulary based on the text content; and
and determining that the image to be detected contains the target watermark in response to the fact that the preset vocabulary matched with the text content is contained in the at least one preset vocabulary.
3. The method according to claim 1 or 2, wherein the performing of the prediction analysis on the image to be detected to obtain the prediction result of the image to be detected comprises:
inputting the image to be detected into a watermark detection model to obtain the image region output by the watermark detection model and a corresponding prediction probability of the image region, wherein the prediction probability indicates the possibility that the image region contains the watermark; and
and determining that the image to be detected comprises the image area in response to the prediction probability being greater than a preset probability threshold.
4. The method of claim 3, further comprising:
acquiring a plurality of first images, wherein each first image in the plurality of first images comprises a watermark label, and the watermark label indicates whether the corresponding first image of the watermark label contains a watermark;
acquiring a watermark prediction result of each first image in the plurality of first images based on the watermark detection model and the preset probability threshold;
determining a recall rate and a false detection rate of the watermark detection model based on a watermark prediction result of each first image in the plurality of first images and a watermark label of the first image; and
and adjusting the preset probability threshold value based on the recall rate and the false detection rate so as to judge whether the image to be detected comprises the image area based on the adjusted preset probability threshold value.
5. A method of model training, the method comprising:
acquiring a sample data set, wherein the sample data set comprises at least one positive sample image and at least one negative sample image, watermark labeling information of each positive sample image in the at least one positive sample image comprises at least one watermark labeling area and at least one first watermark label corresponding to the at least one watermark labeling area, and watermark labeling information of each negative sample image in the at least one negative sample image comprises a second watermark label;
based on the sample data set, performing the following steps until the model converges:
acquiring a first sample image based on the sample data set, wherein the first sample image comprises at least one piece of watermark marking information;
inputting the first sample image into the model to obtain at least one watermark prediction; and
and adjusting parameters of the model based on each watermark prediction result in the at least one watermark prediction result and the corresponding watermark marking information of the watermark prediction result.
6. The method of claim 5, wherein said obtaining a first sample image based on the sample data set comprises:
randomly selecting at least one sample image in the sample data set;
performing random sample enhancement on each sample image of the at least one sample image; and
synthesizing the first sample image based on the at least one sample image after the random sample enhancement.
7. An apparatus for detecting a watermark, the apparatus comprising:
the detection device comprises a first acquisition unit, a second acquisition unit and a third acquisition unit, wherein the first acquisition unit is configured to perform prediction analysis on an image to be detected so as to acquire a prediction result of the image to be detected, and the prediction result indicates whether an image area in which a watermark possibly exists is included in the image to be detected;
the recognition unit is configured to respond to the image area included in the image to be detected, and perform character recognition on the image to be detected in the image area; and
a first determination unit configured to determine whether a target watermark is included in the image to be detected based on the text content in response to the text content being identified in the image area.
8. The apparatus of claim 7, wherein the first determining unit comprises:
the matching subunit is configured to match with at least one preset vocabulary based on the text content; and
a first determining subunit, configured to determine that the image to be detected contains the target watermark in response to a preset vocabulary matching the text content being included in the at least one preset vocabulary.
9. The apparatus according to claim 7 or 8, wherein the first obtaining unit comprises:
a first input subunit, configured to input the image to be detected into a watermark detection model to obtain the image region output by the watermark detection model and a corresponding prediction probability of the image region, where the prediction probability indicates a possibility that the image region contains a watermark; and
a second determining subunit configured to determine that the image to be detected includes the image region in response to the prediction probability being greater than a preset probability threshold.
10. The apparatus of claim 9, further comprising:
a second obtaining unit configured to obtain a plurality of first images, wherein each of the plurality of first images includes a watermark tag indicating whether a watermark is included in the corresponding first image of the watermark tag;
a third obtaining unit configured to obtain a watermark prediction result of each of the plurality of first images based on the watermark detection model and the preset probability threshold;
a second determination unit configured to determine a recall rate and a false detection rate of the watermark detection model based on a watermark prediction result of each of the plurality of first images and a watermark label of the first image; and
and the adjusting unit is configured to adjust the preset probability threshold value based on the recall rate and the false detection rate so as to judge whether the image to be detected comprises the image area based on the adjusted preset probability threshold value.
11. A model training apparatus, the apparatus comprising:
a fourth obtaining unit, configured to obtain a sample data set, where the sample data set includes at least one positive sample image and at least one negative sample image, watermark annotation information of each positive sample image in the at least one positive sample image includes at least one watermark annotation area and at least one first watermark label corresponding to the at least one watermark annotation area, and watermark annotation information of each negative sample image in the at least one negative sample image includes a second watermark label;
an execution unit configured to execute, based on the sample data set, operations of the following sub-units until the model converges:
an obtaining subunit configured to obtain a first sample image based on the sample data set, wherein the first sample image includes at least one watermark marking information;
a second input subunit configured to input the first sample image into the model to obtain at least one watermark prediction result; and
and the adjusting subunit is configured to adjust the parameters of the model based on each watermark prediction result in the at least one watermark prediction result and the corresponding watermark marking information of the watermark prediction result.
12. The apparatus of claim 11, wherein the acquisition subunit comprises:
a selection module configured to randomly select at least one sample image in the sample data set;
a sample enhancement module configured to perform random sample enhancement on each sample image of the at least one sample image; and
a synthesis module configured to synthesize the first sample image based on the at least one sample image subjected to the random sample enhancement.
13. An electronic device, characterized in that the electronic device comprises:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein
The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.
14. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any one of claims 1-6.
CN202210745781.1A 2022-06-29 2022-06-29 Watermark detection method and device and model training method and device Pending CN114842476A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210745781.1A CN114842476A (en) 2022-06-29 2022-06-29 Watermark detection method and device and model training method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210745781.1A CN114842476A (en) 2022-06-29 2022-06-29 Watermark detection method and device and model training method and device

Publications (1)

Publication Number Publication Date
CN114842476A true CN114842476A (en) 2022-08-02

Family

ID=82574399

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210745781.1A Pending CN114842476A (en) 2022-06-29 2022-06-29 Watermark detection method and device and model training method and device

Country Status (1)

Country Link
CN (1) CN114842476A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116701640A (en) * 2023-08-04 2023-09-05 腾讯科技(深圳)有限公司 Watermark identification model generation method, watermark identification device and electronic equipment

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180146245A1 (en) * 2015-04-30 2018-05-24 Verance Corporation Watermark based content recognition improvements
CN111062903A (en) * 2019-12-06 2020-04-24 携程计算机技术(上海)有限公司 Automatic processing method and system for image watermark, electronic equipment and storage medium
CN111798360A (en) * 2020-06-30 2020-10-20 百度在线网络技术(北京)有限公司 Watermark detection method, watermark detection device, electronic equipment and storage medium
CN112419135A (en) * 2020-11-19 2021-02-26 广州华多网络科技有限公司 Watermark recognition online training, sampling and removing method, device, equipment and medium
CN113822126A (en) * 2021-06-25 2021-12-21 腾讯科技(深圳)有限公司 Icon recognition method, device and computer-readable storage medium
CN113887438A (en) * 2021-09-30 2022-01-04 平安银行股份有限公司 Watermark detection method, device, equipment and medium for face image
CN113989559A (en) * 2021-10-27 2022-01-28 上海瑾盛通信科技有限公司 Method, device and equipment for determining probability threshold of classification model and storage medium
CN114596442A (en) * 2022-02-22 2022-06-07 北京百度网讯科技有限公司 Image identification method, device, equipment and storage medium

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180146245A1 (en) * 2015-04-30 2018-05-24 Verance Corporation Watermark based content recognition improvements
CN111062903A (en) * 2019-12-06 2020-04-24 携程计算机技术(上海)有限公司 Automatic processing method and system for image watermark, electronic equipment and storage medium
CN111798360A (en) * 2020-06-30 2020-10-20 百度在线网络技术(北京)有限公司 Watermark detection method, watermark detection device, electronic equipment and storage medium
CN112419135A (en) * 2020-11-19 2021-02-26 广州华多网络科技有限公司 Watermark recognition online training, sampling and removing method, device, equipment and medium
CN113822126A (en) * 2021-06-25 2021-12-21 腾讯科技(深圳)有限公司 Icon recognition method, device and computer-readable storage medium
CN113887438A (en) * 2021-09-30 2022-01-04 平安银行股份有限公司 Watermark detection method, device, equipment and medium for face image
CN113989559A (en) * 2021-10-27 2022-01-28 上海瑾盛通信科技有限公司 Method, device and equipment for determining probability threshold of classification model and storage medium
CN114596442A (en) * 2022-02-22 2022-06-07 北京百度网讯科技有限公司 Image identification method, device, equipment and storage medium

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116701640A (en) * 2023-08-04 2023-09-05 腾讯科技(深圳)有限公司 Watermark identification model generation method, watermark identification device and electronic equipment
CN116701640B (en) * 2023-08-04 2024-01-26 腾讯科技(深圳)有限公司 Watermark identification model generation method, watermark identification device and electronic equipment

Similar Documents

Publication Publication Date Title
CN112749758B (en) Image processing method, neural network training method, device, equipment and medium
CN112857268B (en) Object area measuring method, device, electronic equipment and storage medium
CN113256583A (en) Image quality detection method and apparatus, computer device, and medium
CN114445667A (en) Image detection method and method for training image detection model
CN114494935B (en) Video information processing method and device, electronic equipment and medium
CN113723305A (en) Image and video detection method, device, electronic equipment and medium
CN114723949A (en) Three-dimensional scene segmentation method and method for training segmentation model
CN114842476A (en) Watermark detection method and device and model training method and device
CN115422389B (en) Method and device for processing text image and training method of neural network
CN113868453B (en) Object recommendation method and device
CN114998963A (en) Image detection method and method for training image detection model
CN114429678A (en) Model training method and device, electronic device and medium
CN114494797A (en) Method and apparatus for training image detection model
CN114547252A (en) Text recognition method and device, electronic equipment and medium
CN115359309A (en) Training method, device, equipment and medium of target detection model
CN113486853A (en) Video detection method and device, electronic equipment and medium
CN112579587A (en) Data cleaning method and device, equipment and storage medium
CN115170536B (en) Image detection method, training method and device of model
CN116070711B (en) Data processing method, device, electronic equipment and storage medium
CN115512131B (en) Image detection method and training method of image detection model
CN114140851B (en) Image detection method and method for training image detection model
CN114842474B (en) Character recognition method, device, electronic equipment and medium
CN114677691B (en) Text recognition method, device, electronic equipment and storage medium
CN114140852B (en) Image detection method and device
CN115620271B (en) Image processing and model training method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication
RJ01 Rejection of invention patent application after publication

Application publication date: 20220802