CN112884787B - Image clipping method and device, readable medium and electronic equipment - Google Patents
Image clipping method and device, readable medium and electronic equipment Download PDFInfo
- Publication number
- CN112884787B CN112884787B CN202110118673.7A CN202110118673A CN112884787B CN 112884787 B CN112884787 B CN 112884787B CN 202110118673 A CN202110118673 A CN 202110118673A CN 112884787 B CN112884787 B CN 112884787B
- Authority
- CN
- China
- Prior art keywords
- image
- candidate
- target
- nodes
- determining
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 51
- 238000001514 detection method Methods 0.000 claims abstract description 100
- 238000004891 communication Methods 0.000 claims abstract description 91
- 238000012545 processing Methods 0.000 claims description 18
- 238000004590 computer program Methods 0.000 claims description 16
- 238000010586 diagram Methods 0.000 claims description 15
- 230000006870 function Effects 0.000 description 9
- 230000003287 optical effect Effects 0.000 description 5
- 238000013136 deep learning model Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 230000000717 retained effect Effects 0.000 description 3
- 238000012549 training Methods 0.000 description 3
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 206010043417 Therapeutic response unexpected Diseases 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/12—Edge-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/04—Context-preserving transformations, e.g. by using an importance map
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/181—Segmentation; Edge detection involving edge growing; involving edge linking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20021—Dividing image into blocks, subimages or windows
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20112—Image segmentation details
- G06T2207/20132—Image cropping
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
Abstract
The disclosure relates to an image cropping method, an image cropping device, a readable medium and an electronic device, wherein the image cropping method comprises the following steps: obtaining an external detection frame of each object contour in the image to be cut through a pre-trained object detection model; taking the external detection frames as nodes in the communication graph, taking the distance between the external detection frames as the weight of the edges between the nodes, constructing the communication graph corresponding to the object in the image to be cut as a first communication graph, and determining a target communication graph according to the first communication graph; determining a connected subgraph of the target connected graph, and determining the connected subgraph comprising the target nodes as a candidate subgraph; and determining a target clipping region according to the candidate subgraph, and clipping the image to be clipped according to the target clipping region. Therefore, the image can be cut by the connected graph formed by the objects in the image, so that the interdependence relationship among the objects in the image can be clearly shown, and the corresponding intelligent cutting can be performed according to different cutting requirements.
Description
Technical Field
The present disclosure relates to the field of image processing, and in particular, to an image cropping method, an image cropping device, a readable medium, and an electronic device.
Background
In intelligent clipping of pictures, not only is time-consuming but also the effect will be unstable according to human factors, so in order to be able to detect and obtain a good-effect picture, we can use a machine to clip, usually can use an object detection model to detect as many objects as possible in the picture, and then clip according to the detected objects. Alternatively, the region to be cropped may be determined from the image directly from the trained model.
Disclosure of Invention
This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.
In a first aspect, the present disclosure provides an image cropping method, the method comprising:
obtaining an external detection frame of each object contour in the image to be cut through a pre-trained object detection model, and determining the object category of the external detection frame;
taking the external detection frames as nodes in a communication graph, taking the distance between the external detection frames as the weight of edges between the nodes, constructing the communication graph corresponding to the object in the image to be cut as a first communication graph, and determining a target communication graph according to the first communication graph;
Determining a connected subgraph of the target connected graph, and determining the connected subgraph comprising target nodes as candidate subgraphs, wherein one or more candidate subgraphs are provided, and the object class of the external detection frame corresponding to the target nodes is the object class to be reserved in the image to be cut;
and determining a target clipping region according to the candidate subgraph, and clipping the image to be clipped according to the target clipping region.
In a second aspect, the present disclosure provides an image cropping device, the device comprising:
the object detection module is used for acquiring an external detection frame of each object contour in the image to be cut through a pre-trained object detection model and determining the object category of the external detection frame;
the processing module is used for taking the external detection frames as nodes in the communication graph, taking the distance between the external detection frames as the weight of the edges between the nodes, constructing the communication graph corresponding to the object in the image to be cut as a first communication graph, and determining a target communication graph according to the first communication graph;
the determining module is used for determining a connected subgraph of the target connected graph, determining the connected subgraph comprising target nodes as candidate subgraphs, wherein one or more candidate subgraphs are provided, and the object class of the external detection frame corresponding to the target nodes is a target object class to be reserved in the image to be cut;
And the cropping module is used for determining a target cropping area according to the candidate subgraph and cropping the image to be cropped according to the target cropping area.
In a third aspect, the present disclosure provides a computer readable medium having stored thereon a computer program which when executed by a processing device implements the steps of the method described in the first aspect.
In a fourth aspect, the present disclosure provides an electronic device comprising:
a storage device having a computer program stored thereon;
processing means for executing said computer program in said storage means to carry out the steps of the method described in the first aspect.
Through the technical scheme, the objects in the image can be used as nodes, and the corresponding communication graph is constructed, so that the image is cut according to the communication graph, the interdependence relationship among the objects in the image can be clearly shown, and the corresponding intelligent cutting can be performed according to different cutting requirements.
Additional features and advantages of the present disclosure will be set forth in the detailed description which follows.
Drawings
The above and other features, advantages, and aspects of embodiments of the present disclosure will become more apparent by reference to the following detailed description when taken in conjunction with the accompanying drawings. The same or similar reference numbers will be used throughout the drawings to refer to the same or like elements. It should be understood that the figures are schematic and that elements and components are not necessarily drawn to scale. In the drawings:
Fig. 1 is a flowchart illustrating an image cropping method according to an exemplary embodiment of the present disclosure.
Fig. 2 is a flowchart illustrating an image cropping method according to still another exemplary embodiment of the present disclosure.
Fig. 3 is a flowchart illustrating an image cropping method according to still another exemplary embodiment of the present disclosure.
Fig. 4 is a flowchart illustrating an image cropping method according to still another exemplary embodiment of the present disclosure.
Fig. 5 is a block diagram illustrating a structure of an image cropping device according to an exemplary embodiment of the present disclosure.
Fig. 6 shows a schematic structural diagram of an electronic device suitable for use in implementing embodiments of the present disclosure.
Detailed Description
Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure have been shown in the accompanying drawings, it is to be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustration purposes only and are not intended to limit the scope of the present disclosure.
It should be understood that the various steps recited in the method embodiments of the present disclosure may be performed in a different order and/or performed in parallel. Furthermore, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the present disclosure is not limited in this respect.
The term "including" and variations thereof as used herein are intended to be open-ended, i.e., including, but not limited to. The term "based on" is based at least in part on. The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments. Related definitions of other terms will be given in the description below.
It should be noted that the terms "first," "second," and the like in this disclosure are merely used to distinguish between different devices, modules, or units and are not used to define an order or interdependence of functions performed by the devices, modules, or units.
It should be noted that references to "one", "a plurality" and "a plurality" in this disclosure are intended to be illustrative rather than limiting, and those of ordinary skill in the art will appreciate that "one or more" is intended to be understood as "one or more" unless the context clearly indicates otherwise.
The names of messages or information interacted between the various devices in the embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.
Fig. 1 is a flowchart illustrating an image cropping method according to an exemplary embodiment of the present disclosure. As shown in fig. 1, the method includes steps 101 to 104.
In step 101, an external detection frame of each object contour in the image to be cut is obtained through a pre-trained object detection model, and an object category to which the external detection frame belongs is determined.
The object detection model may be, for example, a deep learning model using RetinaNet, or a deep learning model based on YOLO (You Only Look Once), pre-trained with a large amount of training data.
The image to be cropped may be a partially sampled video frame in a video, a partially sampled image included in an article, or any one of a group of images including a plurality of images. For example, when a cover image of a short video needs to be determined, a frame used as a cover image may be determined from all video frames in the short video, and the frame is used as the image to be cropped, and then the cover image of the short video may be obtained by cropping the video frame. The method of determining the image to be cropped is not limited in this disclosure.
The external detection frame can completely comprise the outline of the object, the external detection frame can be in the form of a rectangular detection frame, a round detection frame, an oval detection frame, other polygonal detection frames and the like, and the external detection frame can be a minimum external frame or not.
The object type to which the external detection frame belongs, that is, the object type to which the external detection frame corresponds, for example, if the external detection frame is an external detection frame formed according to the outline of a section of text, the object type to which the external detection frame belongs is the text type; if the external detection frame is an external detection frame formed by the outline of the object with significance detected by the object detection model, the object category of the external detection frame is the significance object category.
The plurality of object types may be detected by different object detection models, or may be detected directly from the same object detection model.
In step 102, the external detection frames are used as nodes in the connected graph, distances between the external detection frames are used as weights of edges between the nodes, so as to construct the connected graph corresponding to the object in the image to be cut as a first connected graph, and a target connected graph is determined according to the first connected graph.
In step 103, determining a connected subgraph of the target connected graph, and determining the connected subgraph including target nodes as candidate subgraphs, wherein one or more candidate subgraphs are provided, and the object class to which the external detection frame corresponding to the target nodes belongs is the object class to be reserved in the image to be cut.
The distance between the external detection frames may be the distance between the center points of the external detection frames.
When the target communication map is determined according to the first communication map, the first communication map may be directly determined as the target communication map, or the first communication map may be first simplified to a certain extent or subjected to operations such as transformation according to different requirements, and then the modified map is used as the target communication map. The simplification or transformation operations that may be performed in particular will be described in the following embodiments.
The object class to which the external detection frame corresponding to the target node belongs, namely the object class to which the target node belongs. The target object category to be reserved in the image to be cut can be one or more. For example, in one possible application scenario, only the face class in the image to be cropped may be retained, in another possible application scenario, both the face class and the text class may be retained, and so on.
Because a plurality of connected subgraphs can be included in one connected graph, through selecting the target node, the connected subgraphs required by a user in the plurality of connected subgraphs can be selected as candidate subgraphs according to requirements, and then the cutting area of the image to be cut is divided.
In step 104, a target cropping zone is determined according to the candidate subgraph, and the image to be cropped is cropped according to the target cropping zone.
The number of the obtained clipping regions determined by each candidate sub-graph can be one or more, and after all clipping regions corresponding to all candidate sub-graphs are determined, the most suitable clipping region is selected as the target clipping region, so that the image to be clipped can be clipped only according to the requirement of a user.
Through the technical scheme, the objects in the image can be used as nodes, and the corresponding communication graph is constructed, so that the image is cut according to the communication graph, the interdependence relationship among the objects in the image can be clearly shown, and the corresponding intelligent cutting can be performed according to different cutting requirements.
Fig. 2 is a flowchart illustrating an image cropping method according to still another exemplary embodiment of the present disclosure. As shown in fig. 2, the method further comprises steps 201 to 205.
In step 201, the external detection frames are used as nodes in the connected graph, and distances between the external detection frames are used as weights of edges between the nodes, so as to construct the connected graph corresponding to the object in the image to be clipped as a first connected graph.
In step 202, the smallest spanning tree in the first connectivity graph is found as the second connectivity graph.
In step 203, the target connectivity map is determined from the second connectivity map.
By searching the minimum spanning tree in the first communication graph, the relation among all nodes in the first communication graph can be simplified to a simplest state, the number of combinations among all nodes is reduced under the condition that all nodes can be communicated with other nodes, namely the number of combinations among all objects in an image to be cut is reduced, namely the possible cutting mode is reduced, and operation is reduced under the condition that the number of the objects in the image to be cut is more.
In step 204, according to the distance between the nodes in the second connected graph, the nodes belonging to the same object class in the second connected graph are clustered, and the nodes clustered into the same class of clusters are combined into the same node, wherein the combined nodes corresponding to the various clusters include all the nodes combined in the class of clusters, and the connected graph formed by all the nodes after the clustering is used as a third connected graph.
In step 205, the third communication map is determined as the target communication map.
On the basis of searching the minimum spanning tree of the first communication diagram, the nodes with similar distances can be clustered, so that the number of combinations among the nodes is further reduced. In the clustering process, the object category of the node, namely the object category of the detection frame corresponding to the node, and only the nodes belonging to the same object category are clustered, so that the object category of the combined nodes is unchanged after the plurality of nodes clustered into the same class cluster are combined.
Because each node in the connected graph corresponds to an external detection frame, and the external detection frame corresponds to a fixed area in the image to be cut, when a plurality of nodes in the same cluster are combined, all image areas of the external detection frame corresponding to the combined nodes are needed to be included in the combined nodes, and the form of the external detection frame is not changed. For example, when the circumscribed detection frame is a rectangular detection frame, if there are two nodes currently belonging to the same cluster, and the position of the circumscribed detection frame corresponding to one node in the image to be cut is (X1, Y1, X2, Y2), where (X1, Y1) is the upper left corner point of the rectangular region, (X2, Y2) is the lower right corner point of the rectangular region, the position of the circumscribed detection frame corresponding to the other node in the image to be cut is (X3, Y3, X4, Y4), where (X3, Y3) is the upper left corner point of the rectangular region, (X4, Y4) is the lower right corner point of the rectangular region, and X1< X2< X3< X4, Y1< Y2< Y3< Y4), the position of the rectangular region corresponding to the node after combining in the image to be cut should be (X1, Y1, X4, Y4), where (X1, Y1) is the upper right corner point of the rectangular region corresponding to the node after combining, and (X4, Y4) is the lower right corner point of the rectangular region after combining.
Fig. 3 is a flowchart illustrating an image cropping method according to still another exemplary embodiment of the present disclosure. As shown in fig. 3, the method further comprises step 301 and step 302.
In step 301, a candidate cropping zone is determined from the candidate subgraph, wherein the candidate cropping zone and the candidate subgraph are in a one-to-one or many-to-one relationship.
The method for determining the candidate clipping region according to the candidate subgraph may be various, for example, a maximum image region including all nodes in the candidate subgraph and only including all nodes in the candidate subgraph may be searched in the image to be clipped, and the maximum image region corresponding to the candidate subgraph is determined as the candidate clipping region. In the case that the image to be cropped from the image to be cropped is rectangular, the preset cropping ratio may be, for example, (1:1), and when searching the maximum image area satisfying the above requirement, the center of the circumscribed rectangle formed by all the nodes in the candidate subgraph may be searched for in four directions.
Or if the target object category to be reserved in the image to be cut includes a face category, searching a maximum image area which includes and only includes all nodes in the candidate subgraph, satisfies a preset cutting proportion and is in a preset position in the image to be cut, and determining the maximum image area corresponding to the candidate subgraph as the candidate cutting area. The preset position may be determined according to the target object category, for example, in the case that the target object category includes a face category, since the face of the human body will generally need to be located above in the picture, the preset position may be set to be a proportion of the upper space of the maximum image area to the height of the image to be cropped, which may be, for example, (0,0.1,0.2,0.33,0.5) or the like.
In step 302, the target cropping zone is selected from the candidate cropping zones by the ordering model, and the image to be cropped is cropped according to the target cropping zone.
The ranking model may be a deep learning model obtained by training in advance, and the loss function when training the ranking model may be as follows:
wherein phi is ResNet depth convolution network model, X t For other pictures, X k For the marked target picture, the loss function canTo be rank loss.
Fig. 4 is a flowchart illustrating an image cropping method according to still another exemplary embodiment of the present disclosure. As shown in fig. 4, the method further comprises a step 401 and a step 402.
In step 401, an object class selection instruction input by a user is received.
In step 402, the target object class to be retained in the image to be cropped is determined according to the object class selection instruction. That is, before clipping an image to be clipped, a user can select any object type possibly existing in the image to be clipped to clip according to his own needs.
Fig. 5 is a block diagram illustrating a structure of an image cropping device according to an exemplary embodiment of the present disclosure. As shown in fig. 5, the apparatus includes: the object detection module 10 is configured to obtain an external detection frame of each object contour in the image to be cropped through a pre-trained object detection model, and determine an object class to which the external detection frame belongs; the processing module 20 is configured to take the external detection frames as nodes in the connected graph, take distances between the external detection frames as weights of edges between the nodes, construct a connected graph corresponding to an object in the image to be cropped as a first connected graph, and determine a target connected graph according to the first connected graph; the determining module 30 is configured to determine a connected subgraph of the target connected graph, and determine the connected subgraph including the target node as a candidate subgraph, where the candidate subgraph is one or more, and an object class to which the external detection frame corresponding to the target node belongs is a target object class to be reserved in the image to be cut; and the cropping module 40 is used for determining a target cropping area according to the candidate subgraph and cropping the image to be cropped according to the target cropping area.
In one possible implementation, the processing module 20 is further configured to: searching a minimum spanning tree in the first communication graph as a second communication graph; and determining the target communication diagram according to the second communication diagram.
In one possible implementation, the processing module 20 is further configured to: clustering nodes belonging to the same object class in the second communication graph according to the distance between the nodes in the second communication graph, and merging the nodes clustered into the same class of clusters into the same node, wherein the merged nodes corresponding to the various clusters comprise all the nodes merged in the class of clusters, and the communication graph formed by all the nodes after the clustering is used as a third communication graph; and determining the third communication map as the target communication map.
In one possible implementation, the clipping module 40 is further configured to: determining a candidate clipping region according to the candidate subgraph, wherein the candidate clipping region and the candidate subgraph are in one-to-one or many-to-one relation; and selecting the target clipping region from the candidate clipping regions through the sorting model, and clipping the image to be clipped according to the target clipping region.
In one possible implementation, the clipping module 40 is further configured to: searching a maximum image area which comprises all nodes in the candidate subgraph and only comprises all nodes in the image to be cut and meets a preset cutting proportion; and determining the maximum image area corresponding to the candidate subgraph as the candidate clipping area.
In one possible implementation, the clipping module 40 is further configured to: searching a maximum image area which comprises and only comprises all nodes in the candidate subgraph, satisfies a preset clipping proportion and is in a preset position in the image to be clipped under the condition that the target object category to be reserved in the image to be clipped comprises a face category; and determining the maximum image area corresponding to the candidate subgraph as the candidate clipping area.
In one possible embodiment, the apparatus further comprises: the receiving module is used for receiving an object category selection instruction input by a user; and the selection module is used for determining the target object type to be reserved in the image to be cut according to the object type selection instruction.
Referring now to fig. 6, a schematic diagram of an electronic device 600 suitable for use in implementing embodiments of the present disclosure is shown. The terminal devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and the like, and stationary terminals such as digital TVs, desktop computers, and the like. The electronic device shown in fig. 6 is merely an example and should not be construed to limit the functionality and scope of use of the disclosed embodiments.
As shown in fig. 6, the electronic device 600 may include a processing means (e.g., a central processing unit, a graphics processor, etc.) 601, which may perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 602 or a program loaded from a storage means 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic apparatus 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
In general, the following devices may be connected to the I/O interface 605: input devices 606 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, and the like; an output device 607 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 608 including, for example, magnetic tape, hard disk, etc.; and a communication device 609. The communication means 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. While fig. 6 shows an electronic device 600 having various means, it is to be understood that not all of the illustrated means are required to be implemented or provided. More or fewer devices may be implemented or provided instead.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a non-transitory computer readable medium, the computer program comprising program code for performing the method shown in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via communication means 609, or from storage means 608, or from ROM 602. The above-described functions defined in the methods of the embodiments of the present disclosure are performed when the computer program is executed by the processing device 601.
It should be noted that the computer readable medium described in the present disclosure may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present disclosure, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, fiber optic cables, RF (radio frequency), and the like, or any suitable combination of the foregoing.
In some implementations, the clients, servers may communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol ), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the internet (e.g., the internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed networks.
The computer readable medium may be contained in the electronic device; or may exist alone without being incorporated into the electronic device.
The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: obtaining an external detection frame of each object contour in the image to be cut through a pre-trained object detection model, and determining the object category of the external detection frame; taking the external detection frames as nodes in a communication graph, taking the distance between the external detection frames as the weight of edges between the nodes, constructing the communication graph corresponding to the object in the image to be cut as a first communication graph, and determining a target communication graph according to the first communication graph; determining a connected subgraph of the target connected graph, and determining the connected subgraph comprising target nodes as candidate subgraphs, wherein one or more candidate subgraphs are provided, and the object class of the external detection frame corresponding to the target nodes is the object class to be reserved in the image to be cut; and determining a target clipping region according to the candidate subgraph, and clipping the image to be clipped according to the target clipping region.
Computer program code for carrying out operations of the present disclosure may be written in one or more programming languages, including, but not limited to, an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules described in the embodiments of the present disclosure may be implemented in software or hardware. The name of the module is not limited to the module itself in some cases, for example, the object detection module may also be described as a "module that obtains an external detection frame of each object contour in the image to be cropped through a pre-trained object detection model, and determines an object class to which the external detection frame belongs".
The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), an Application Specific Standard Product (ASSP), a system on a chip (SOC), a Complex Programmable Logic Device (CPLD), and the like.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
According to one or more embodiments of the present disclosure, example 1 provides an image cropping method, the method comprising:
obtaining an external detection frame of each object contour in the image to be cut through a pre-trained object detection model, and determining the object category of the external detection frame;
taking the external detection frames as nodes in a communication graph, taking the distance between the external detection frames as the weight of edges between the nodes, constructing the communication graph corresponding to the object in the image to be cut as a first communication graph, and determining a target communication graph according to the first communication graph;
determining a connected subgraph of the target connected graph, and determining the connected subgraph comprising target nodes as candidate subgraphs, wherein one or more candidate subgraphs are provided, and the object class of the external detection frame corresponding to the target nodes is the object class to be reserved in the image to be cut;
and determining a target clipping region according to the candidate subgraph, and clipping the image to be clipped according to the target clipping region.
According to one or more embodiments of the present disclosure, example 2 provides the method of example 1, the determining a target connectivity graph from the first connectivity graph comprising:
Searching a minimum spanning tree in the first communication graph as a second communication graph;
and determining the target communication diagram according to the second communication diagram.
According to one or more embodiments of the present disclosure, example 3 provides the method of example 2, the determining the target connectivity graph from the second connectivity graph comprising:
clustering nodes belonging to the same object class in the second communication graph according to the distance between the nodes in the second communication graph, and merging the nodes clustered into the same class of clusters into the same node, wherein the merged nodes corresponding to the various clusters comprise all the nodes merged in the class of clusters, and the communication graph formed by all the nodes after the clustering is used as a third communication graph;
and determining the third communication map as the target communication map.
According to one or more embodiments of the present disclosure, example 4 provides the method of example 1, the determining a target cropping zone according to the candidate subgraph, and cropping the image to be cropped according to the target cropping zone comprising:
determining a candidate clipping region according to the candidate subgraph, wherein the candidate clipping region and the candidate subgraph are in one-to-one or many-to-one relation;
And selecting the target clipping region from the candidate clipping regions through the sorting model, and clipping the image to be clipped according to the target clipping region.
In accordance with one or more embodiments of the present disclosure, example 5 provides the method of example 4, the determining a candidate clipping region from the candidate subgraph comprising:
searching a maximum image area which comprises all nodes in the candidate subgraph and only comprises all nodes in the image to be cut and meets a preset cutting proportion;
and determining the maximum image area corresponding to the candidate subgraph as the candidate clipping area.
In accordance with one or more embodiments of the present disclosure, example 6 provides the method of example 4, the determining a candidate clipping region from the candidate subgraph comprising:
searching a maximum image area which comprises and only comprises all nodes in the candidate subgraph, satisfies a preset clipping proportion and is in a preset position in the image to be clipped in the condition that the target object category to be reserved in the image to be clipped comprises a face category;
and determining the maximum image area corresponding to the candidate subgraph as the candidate clipping area.
Example 7 provides the method of example 1, according to one or more embodiments of the present disclosure, the method further comprising:
receiving an object category selection instruction input by a user;
and determining the target object type to be reserved in the image to be cut according to the object type selection instruction.
In accordance with one or more embodiments of the present disclosure, example 8 provides an image cropping device, the device comprising:
the object detection module is used for acquiring an external detection frame of each object contour in the image to be cut through a pre-trained object detection model and determining the object category of the external detection frame;
the processing module is used for taking the external detection frames as nodes in the communication graph, taking the distance between the external detection frames as the weight of the edges between the nodes, constructing the communication graph corresponding to the object in the image to be cut as a first communication graph, and determining a target communication graph according to the first communication graph;
the determining module is used for determining a connected subgraph of the target connected graph, determining the connected subgraph comprising target nodes as candidate subgraphs, wherein one or more candidate subgraphs are provided, and the object class of the external detection frame corresponding to the target nodes is a target object class to be reserved in the image to be cut;
And the cropping module is used for determining a target cropping area according to the candidate subgraph and cropping the image to be cropped according to the target cropping area.
According to one or more embodiments of the present disclosure, example 9 provides a computer-readable medium having stored thereon a computer program which, when executed by a processing device, implements the steps of the method of any of examples 1-7.
In accordance with one or more embodiments of the present disclosure, example 10 provides an electronic device, comprising:
a storage device having a computer program stored thereon;
processing means for executing the computer program in the storage means to implement the steps of the method of any one of examples 1-7.
The foregoing description is only of the preferred embodiments of the present disclosure and description of the principles of the technology being employed. It will be appreciated by persons skilled in the art that the scope of the disclosure referred to in this disclosure is not limited to the specific combinations of features described above, but also covers other embodiments which may be formed by any combination of features described above or equivalents thereof without departing from the spirit of the disclosure. Such as those described above, are mutually substituted with the technical features having similar functions disclosed in the present disclosure (but not limited thereto).
Moreover, although operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order. In certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the present disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.
Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are example forms of implementing the claims. The specific manner in which the various modules perform the operations in the apparatus of the above embodiments have been described in detail in connection with the embodiments of the method, and will not be described in detail herein.
Claims (7)
1. A method of cropping an image, the method comprising:
obtaining an external detection frame of each object contour in an image to be cut through a pre-trained object detection model, and determining the object category of the external detection frame;
taking the external detection frames as nodes in a communication graph, taking the distance between the external detection frames as the weight of edges between the nodes, constructing the communication graph corresponding to the object in the image to be cut as a first communication graph, and determining a target communication graph according to the first communication graph;
determining a connected subgraph of the target connected graph, and determining the connected subgraph comprising target nodes as candidate subgraphs, wherein one or more candidate subgraphs are provided, and the object class of the external detection frame corresponding to the target nodes is the object class to be reserved in the image to be cut; wherein the target object class is at least one;
determining a candidate clipping region according to the candidate subgraph, wherein the candidate clipping region and the candidate subgraph are in one-to-one or many-to-one relation;
selecting the target clipping region from the candidate clipping regions through an ordering model, and clipping the image to be clipped according to the target clipping region;
Wherein said determining a candidate clipping region from said candidate subgraph comprises: searching a maximum image area which comprises all nodes in the candidate subgraph and only comprises all nodes in the image to be cut and meets a preset cutting proportion; determining the maximum image area corresponding to the candidate subgraph as the candidate clipping area;
alternatively, the determining a candidate clipping region according to the candidate subgraph includes: searching a maximum image area which comprises and only comprises all nodes in the candidate subgraph, satisfies a preset clipping proportion and is in a preset position in the image to be clipped in the condition that the target object category to be reserved in the image to be clipped comprises a face category; and determining the maximum image area corresponding to the candidate subgraph as the candidate clipping area.
2. The method of claim 1, wherein the determining a target connectivity map from the first connectivity map comprises:
searching a minimum spanning tree in the first communication graph as a second communication graph;
and determining the target communication diagram according to the second communication diagram.
3. The method of claim 2, wherein the determining the target connectivity map from the second connectivity map comprises:
Clustering nodes belonging to the same object class in the second communication graph according to the distance between the nodes in the second communication graph, and merging the nodes clustered into the same class of clusters into the same node, wherein the merged nodes corresponding to the various clusters comprise all the nodes merged in the class of clusters, and the communication graph formed by all the nodes after the clustering is used as a third communication graph;
and determining the third communication map as the target communication map.
4. The method according to claim 1, wherein the method further comprises:
receiving an object category selection instruction input by a user;
and determining the target object type to be reserved in the image to be cut according to the object type selection instruction.
5. An image cropping device, the device comprising:
the object detection module is used for acquiring an external detection frame of each object contour in the image to be cut through a pre-trained object detection model, and determining the object category of the external detection frame;
the processing module is used for taking the external detection frames as nodes in the communication graph, taking the distance between the external detection frames as the weight of the edges between the nodes, constructing the communication graph corresponding to the object in the image to be cut as a first communication graph, and determining a target communication graph according to the first communication graph;
The determining module is used for determining a connected subgraph of the target connected graph, determining the connected subgraph comprising target nodes as candidate subgraphs, wherein one or more candidate subgraphs are provided, and the object class of the external detection frame corresponding to the target nodes is a target object class to be reserved in the image to be cut; wherein the target object class is at least one;
the cropping module is used for determining a target cropping area according to the candidate subgraph and cropping the image to be cropped according to the target cropping area;
the clipping module is further configured to determine a candidate clipping region according to the candidate subgraph, where the candidate clipping region and the candidate subgraph are in a one-to-one or many-to-one relationship; selecting the target clipping region from the candidate clipping regions through an ordering model, and clipping the image to be clipped according to the target clipping region;
the clipping module is further configured to search a maximum image area that includes all nodes in the candidate subgraph and only includes all nodes in the candidate subgraph, where the maximum image area meets a preset clipping proportion; determining the maximum image area corresponding to the candidate subgraph as the candidate clipping area;
Or, the cropping module is further configured to, when the target object class to be kept in the image to be cropped includes a face class, search for a maximum image area that includes and includes only all nodes in the candidate subgraph, satisfies a preset cropping proportion, and is in a preset position in the image to be cropped; and determining the maximum image area corresponding to the candidate subgraph as the candidate clipping area.
6. A computer readable medium on which a computer program is stored, characterized in that the program, when being executed by a processing device, carries out the steps of the method according to any one of claims 1-4.
7. An electronic device, comprising:
a storage device having a computer program stored thereon;
processing means for executing said computer program in said storage means to carry out the steps of the method according to any one of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110118673.7A CN112884787B (en) | 2021-01-28 | 2021-01-28 | Image clipping method and device, readable medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110118673.7A CN112884787B (en) | 2021-01-28 | 2021-01-28 | Image clipping method and device, readable medium and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112884787A CN112884787A (en) | 2021-06-01 |
| CN112884787B true CN112884787B (en) | 2023-09-15 |
Family
ID=76053010
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110118673.7A Active CN112884787B (en) | 2021-01-28 | 2021-01-28 | Image clipping method and device, readable medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112884787B (en) |
Citations (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103544712A (en) * | 2013-11-12 | 2014-01-29 | 中国科学院自动化研究所 | Method for automatically segmenting human lateral geniculate nucleus through prior knowledge |
| CN104809721A (en) * | 2015-04-09 | 2015-07-29 | 香港中文大学深圳研究院 | Segmentation method and device of cartoon |
| US9327406B1 (en) * | 2014-08-19 | 2016-05-03 | Google Inc. | Object segmentation based on detected object-specific visual cues |
| CN105787872A (en) * | 2011-11-09 | 2016-07-20 | 乐天株式会社 | Image Processing Device, Method For Controlling Image Processing Device |
| CN106127782A (en) * | 2016-06-30 | 2016-11-16 | 北京奇艺世纪科技有限公司 | A kind of image partition method and system |
| CN106778864A (en) * | 2016-12-13 | 2017-05-31 | 东软集团股份有限公司 | Initial sample selection method and device |
| CN108597003A (en) * | 2018-04-20 | 2018-09-28 | 腾讯科技（深圳）有限公司 | A kind of article cover generation method, device, processing server and storage medium |
| CN109712164A (en) * | 2019-01-17 | 2019-05-03 | 上海携程国际旅行社有限公司 | Image intelligent cut-out method, system, equipment and storage medium |
| CN110009654A (en) * | 2019-04-10 | 2019-07-12 | 大连理工大学 | Three-dimensional data dividing method based on maximum Flow Policy |
| CN110136142A (en) * | 2019-04-26 | 2019-08-16 | 微梦创科网络科技（中国）有限公司 | A kind of image cropping method, apparatus, electronic equipment |
| CN110147833A (en) * | 2019-05-09 | 2019-08-20 | 北京迈格威科技有限公司 | Facial image processing method, apparatus, system and readable storage medium storing program for executing |
| CN110189340A (en) * | 2019-06-03 | 2019-08-30 | 北京达佳互联信息技术有限公司 | Image partition method, device, electronic equipment and storage medium |
| CN110795925A (en) * | 2019-10-12 | 2020-02-14 | 腾讯科技（深圳）有限公司 | Image-text typesetting method based on artificial intelligence, image-text typesetting device and electronic equipment |
| CN110796663A (en) * | 2019-09-17 | 2020-02-14 | 北京迈格威科技有限公司 | Picture clipping method, device, equipment and storage medium |
| CN111274981A (en) * | 2020-02-03 | 2020-06-12 | 中国人民解放军国防科技大学 | Target detection network construction method and device and target detection method |
| CN112001406A (en) * | 2019-05-27 | 2020-11-27 | 杭州海康威视数字技术股份有限公司 | Text region detection method and device |
| CN112017189A (en) * | 2020-10-26 | 2020-12-01 | 腾讯科技（深圳）有限公司 | Image segmentation method and device, computer equipment and storage medium |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8447139B2 (en) * | 2010-04-13 | 2013-05-21 | International Business Machines Corporation | Object recognition using Haar features and histograms of oriented gradients |
| US8831339B2 (en) * | 2012-06-19 | 2014-09-09 | Palo Alto Research Center Incorporated | Weighted feature voting for classification using a graph lattice |
| CN109145931B (en) * | 2018-09-03 | 2019-11-05 | 百度在线网络技术（北京）有限公司 | Object detecting method, device and storage medium |
-
2021
- 2021-01-28 CN CN202110118673.7A patent/CN112884787B/en active Active
Patent Citations (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105787872A (en) * | 2011-11-09 | 2016-07-20 | 乐天株式会社 | Image Processing Device, Method For Controlling Image Processing Device |
| CN103544712A (en) * | 2013-11-12 | 2014-01-29 | 中国科学院自动化研究所 | Method for automatically segmenting human lateral geniculate nucleus through prior knowledge |
| US9327406B1 (en) * | 2014-08-19 | 2016-05-03 | Google Inc. | Object segmentation based on detected object-specific visual cues |
| CN104809721A (en) * | 2015-04-09 | 2015-07-29 | 香港中文大学深圳研究院 | Segmentation method and device of cartoon |
| CN106127782A (en) * | 2016-06-30 | 2016-11-16 | 北京奇艺世纪科技有限公司 | A kind of image partition method and system |
| CN106778864A (en) * | 2016-12-13 | 2017-05-31 | 东软集团股份有限公司 | Initial sample selection method and device |
| CN108597003A (en) * | 2018-04-20 | 2018-09-28 | 腾讯科技（深圳）有限公司 | A kind of article cover generation method, device, processing server and storage medium |
| CN109712164A (en) * | 2019-01-17 | 2019-05-03 | 上海携程国际旅行社有限公司 | Image intelligent cut-out method, system, equipment and storage medium |
| CN110009654A (en) * | 2019-04-10 | 2019-07-12 | 大连理工大学 | Three-dimensional data dividing method based on maximum Flow Policy |
| CN110136142A (en) * | 2019-04-26 | 2019-08-16 | 微梦创科网络科技（中国）有限公司 | A kind of image cropping method, apparatus, electronic equipment |
| CN110147833A (en) * | 2019-05-09 | 2019-08-20 | 北京迈格威科技有限公司 | Facial image processing method, apparatus, system and readable storage medium storing program for executing |
| CN112001406A (en) * | 2019-05-27 | 2020-11-27 | 杭州海康威视数字技术股份有限公司 | Text region detection method and device |
| CN110189340A (en) * | 2019-06-03 | 2019-08-30 | 北京达佳互联信息技术有限公司 | Image partition method, device, electronic equipment and storage medium |
| CN110796663A (en) * | 2019-09-17 | 2020-02-14 | 北京迈格威科技有限公司 | Picture clipping method, device, equipment and storage medium |
| CN110795925A (en) * | 2019-10-12 | 2020-02-14 | 腾讯科技（深圳）有限公司 | Image-text typesetting method based on artificial intelligence, image-text typesetting device and electronic equipment |
| CN111274981A (en) * | 2020-02-03 | 2020-06-12 | 中国人民解放军国防科技大学 | Target detection network construction method and device and target detection method |
| CN112017189A (en) * | 2020-10-26 | 2020-12-01 | 腾讯科技（深圳）有限公司 | Image segmentation method and device, computer equipment and storage medium |
Non-Patent Citations (3)
| Title |
|---|
| A shape context fully convolutional neural network for segmentation and classification of cervical nuclei in Pap smear images;Elima Hussain等;《Artificial Intelligence in Medicine》;1-11 * |
| 基于深度学习的图像目标检测与分割算法的研究与应用;刘培;《中国优秀硕士学位论文全文数据库 信息科技辑》;第2019年卷(第5期);I138-1309 * |
| 陈向宇.基于航拍图像的单木提取和机载LiDAR数据的树种分类研究.《中国优秀硕士学位论文全文数据库 农业科技辑》.2020,第2020年卷(第5期),第三章. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112884787A (en) | 2021-06-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112184738B (en) | Image segmentation method, device, equipment and storage medium | |
| CN110634047B (en) | Method and device for recommending house resources, electronic equipment and storage medium | |
| CN110852258A (en) | Object detection method, device, equipment and storage medium | |
| CN111461968B (en) | Picture processing method, device, electronic equipment and computer readable medium | |
| CN111461967B (en) | Picture processing method, device, equipment and computer readable medium | |
| CN110795196A (en) | Window display method, device, terminal and storage medium | |
| CN113610034B (en) | Method and device for identifying character entities in video, storage medium and electronic equipment | |
| CN111461965B (en) | Picture processing method and device, electronic equipment and computer readable medium | |
| CN113205601A (en) | Roaming path generation method and device, storage medium and electronic equipment | |
| CN111158881B (en) | Data processing method, device, electronic equipment and computer readable storage medium | |
| CN112884787B (en) | Image clipping method and device, readable medium and electronic equipment | |
| CN110852242A (en) | Watermark identification method, device, equipment and storage medium based on multi-scale network | |
| CN111461969B (en) | Method, device, electronic equipment and computer readable medium for processing picture | |
| CN111680754B (en) | Image classification method, device, electronic equipment and computer readable storage medium | |
| CN111461964B (en) | Picture processing method, device, electronic equipment and computer readable medium | |
| CN114004229A (en) | Text recognition method and device, readable medium and electronic equipment | |
| CN113255812A (en) | Video frame detection method and device and electronic equipment | |
| CN111835917A (en) | Method, device and equipment for showing activity range and computer readable medium | |
| CN110991312A (en) | Method, apparatus, electronic device, and medium for generating detection information | |
| CN114359673B (en) | Small sample smoke detection method, device and equipment based on metric learning | |
| CN111489286B (en) | Picture processing method, device, equipment and medium | |
| CN113256659B (en) | Picture processing method and device and electronic equipment | |
| CN113283115B (en) | Image model generation method and device and electronic equipment | |
| CN111368015B (en) | Method and device for compressing map | |
| CN111310031B (en) | House source information display method, device, terminal and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |