US20190317959A1 - Sketch-based image retrieval using feedback and hierarchies - Google Patents
Sketch-based image retrieval using feedback and hierarchies Download PDFInfo
- Publication number
- US20190317959A1 US20190317959A1 US16/452,646 US201916452646A US2019317959A1 US 20190317959 A1 US20190317959 A1 US 20190317959A1 US 201916452646 A US201916452646 A US 201916452646A US 2019317959 A1 US2019317959 A1 US 2019317959A1
- Authority
- US
- United States
- Prior art keywords
- inputs
- canvas
- sketch
- user
- importance
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
- G06V30/32—Digital ink
- G06V30/36—Matching; Classification
- G06V30/387—Matching; Classification using human interaction, e.g. selection of the best displayed recognition candidate
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
- G06F16/24578—Query processing with adaptation to user needs using ranking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/53—Querying
- G06F16/532—Query formulation, e.g. graphical querying
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
- G06F16/5838—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using colour
-
- G06K9/00416—
-
- G06K9/00436—
-
- G06K9/48—
-
- G06K9/78—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/10—Image acquisition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
- G06V30/32—Digital ink
- G06V30/333—Preprocessing; Feature extraction
- G06V30/347—Sampling; Contour coding; Stroke extraction
Definitions
- the subject matter of this invention relates to sketch-based image retrieval, and more particularly to a system and method for image searching based on customizable sketches and real-time feedback.
- sketch-based retrieval provides a promising technique to search for image data.
- Sketch-based image retrieval allows a user to sketch some input, and based on the input, a set of matching images are returned.
- existing tools lack the ability to effectively interpret complex user inputs and return accurate results.
- aspects of the disclosure provides a sketch-based image retrieval system and method that utilizes real-time feedback to enhance searching and provides an intuitive way to communicate hierarchy (i.e., importance) of inputted information.
- a first aspect discloses a sketch-based retrieval system, including: an interface having a canvas for entering inputs including user generated strokes, and for extracting features associated with the inputs; a hierarchy system for assigning an importance to the inputs; a matching system for retrieving matches from a database based on extracted features and importance associated with the extracted features; a feedback system that provides real-time feedback on the canvas to suggest improved inputs for the user; and a ranking system that ranks retrieved matches for output in the interface.
- a second aspect discloses a computer program product stored on a computer readable storage medium, which when executed by a computing system, provides sketch-based retrieval, the program product including: program code for rendering a canvas for entering inputs including user generated strokes, and for extracting features associated with the inputs; program code for assigning an importance to the inputs; program code for retrieving matches from a database based on extracted features and importance associated with the extracted features; program code that provides real-time feedback on the canvas to suggest improved inputs for the user; and program code that ranks retrieved matches for output.
- a third aspect discloses a computerized method of providing sketch-based retrieval, including: rendering a canvas for receiving inputs including user generated strokes, and for extracting features associated with the inputs; assigning an importance to the inputs; retrieving matches from a database based on extracted features and importance associated with the extracted features; providing real-time feedback on the canvas to suggest improved inputs for the user; and ranking retrieved matches for output in the interface.
- FIG. 1 shows a computing system having a sketch-based retrieval system according to embodiments.
- FIG. 2 shows a flow diagram for implementing a sketch-based retrieval system according to embodiments.
- FIG. 3 shows a use case involving seismic data retrieval according to embodiments.
- FIG. 4 shows a use case involving real estate data retrieval according to embodiments.
- FIG. 5 shows a use case involving document data retrieval according to embodiments.
- Sketches are generally inputted via strokes made by a user on touch screen or gesture based input system.
- the described approach provides real-time feedback of the input to enhance the search process.
- feedback may include suggestions of a next stroke or identification of problematic strokes (i.e., strokes that result in no matches).
- Such real-time feedback will result in accelerating the search process and create more accurate results.
- the approach further provides an intuitive way to communicate an importance hierarchy of inputted data. For example, strokes drawn with greater pressure or slower speed may be interpreted as more important than other strokes. This ability to convey and interpret importance of inputted information, such as a relative importance of different strokes, allows for enhanced searching and ranking of results.
- FIG. 1 depicts a computing system 10 having a sketch-based retrieval system 18 that retrieves image-related data from a database 70 in response to input (i.e., sketches and other data) inputted via an interface device 32 by a user.
- Interface device 32 may be implemented with a touch screen tablet, laptop, smart device, or other gesture based appliance.
- Interface device 32 may be fully or partially integrated with the computing system 10 or reside as a standalone device such that some or all of the functions described are performed by either or both the interface device 32 and computing system 10 .
- Database 70 refers to any collection of image-related data, which may for example reside in a single location or be distributed among various storage systems including cloud based storage systems.
- Sketch-based retrieval system 18 generally includes an input/output manager 20 that is responsible for displaying and collecting information from the interface device 32 . This includes displaying inputs entered by the user, extracting features from the inputs and displaying results and feedback. Extracted features may for example comprise polylines or curves that may be represented as image data and/or via mathematical formulae.
- Interface device 32 generally includes a canvas 30 onto which a user can sketch images by inputting a series of strokes, e.g., with a finger, a pen, a gesture, etc. As the user makes changes to the canvas 30 , a set of ranked images 34 (or other data) are retrieved and displayed that most closely match the input.
- the canvas 30 may include one or more layers for receiving input. Different layers can include, e.g., inputted strokes, loaded images, selectable designs, search categories, etc. Loaded images may for example include photos, patterns, blueprints, seismic images, charts, plots, etc.
- a single layer is depicted in which the user has entered two circular shapes 36 and a squiggly line 38 .
- the circular shapes 36 are inputted bolder than the squiggly line 38 , indicating a higher importance.
- feedback in the form of dashed lines 40 are automatically rendered on the canvas 30 in real-time. The feedback is utilized to help the user better refine the input to achieve enhanced results.
- the system 18 may not be certain if the user is attempting to draw a bicycle or a pair of glasses.
- the user can click on one of the dashed lines to better define the input.
- the user could ignore the feedback and continue entering strokes until matching images are identified.
- Hierarchy system 22 provides a mechanism through which the user can assign a relative importance to different inputs. For example, relative importance can be assigned by the user with strokes made with greater pressure, with bolder lines, with highlighting, with coloring, via an interface that allows for manual selections, etc. Further, different layers of the canvas 30 may be indicated as more or less important. The relative importance of different inputs will be utilized by the matching system 24 and ranking system 28 when attempting to match and rank retrieved image-related data from the database 70 .
- input/output manager 20 will extract “features” associated with the inputs and matching system 24 will attempt to match the features (e.g., curves, polylines, etc.) with indexed image related data in the database 70 .
- Matching can be done on a feature level (e.g., matching circles to wheels) and/or on an entire composite image basis (e.g., matching circles and lines to a complete bicycle).
- Feedback system 26 provides real-time feedback to the user as the user enters inputs. For example, when the matching system 24 is having difficulty matching extracted features with image related data in the database 70 , feedback system 26 can highlight existing user inputs as problematic inputs that are creating ambiguities, which should either be removed or redrawn. Alternatively, feedback system 26 can generate proposed inputs (e.g., dashed lines 40 , blinking lines, colored shapes, etc.) to help guide the user in the input process. Accordingly, based on the displayed proposed inputs, the user can select a proposed input to see if the output results in a desired match.
- proposed inputs e.g., dashed lines 40 , blinking lines, colored shapes, etc.
- ranking system 28 ranks the retrieved matches and displays them to the user, e.g., as a set of ranked images 34 , e.g., as thumbnails, textual descriptions, scrollable images, etc.
- user interaction history 72 can be collected from one or more users.
- the history 72 can thereafter be analyzed and used to improve future searches and rankings. For instance, the analysis may dictate that inputting two circles indicates the user is attempting to sketch a bicycle 55% of the time, eyeglasses 40% of the time, or something else 5% of the time.
- Artificial intelligence including the use of neural networks may be utilized to facilitate the analysis and learning process.
- FIG. 2 depicts a flow diagram for implementing an embodiment of the sketch-based retrieval system 18 with reference to FIG. 1 .
- individual features or subsets of features extracted from the user inputs are first analyzed for feature based matches. After each match is made at the feature level, a composite match of the entire input (i.e., different layers, strokes, etc.) is attempted.
- the process begins at 52 with the user making input changes to the canvas 30 .
- This may involve any user action associated with the canvas, including sketching in an active layer, loading image data, adding a new layer to the canvas, etc.
- the input may include hierarchical information to assign different weights or importance to different inputs. Assigning importance to different inputs may involve implicit and explicit actions by the user. For example, implicit actions may involve: pen pressure by to rank different strokes; the time taken to draw a stroke; etc. Explicit actions may involve: manually assigning weights via an interface tool to prioritize layers or strokes; assigning thresholds for each layer or stroke to be more/less strict for matching of features with indexed images; etc.
- the input/output manager 20 is triggered to extract associated features from the canvas 30 at 54 .
- Extraction methods may for example include Extended Trace Transform (ETT) and Image Deformation Model Features (IDM), among others.
- ETT Extended Trace Transform
- IDM Image Deformation Model Features
- inputs e.g., with dashed lines, blinking lines, etc.
- the ranking system 28 ranks results at 62 using, e.g., an algorithm to match similarities between the extracted features and results.
- the ranked results may be displayed such that the user can select a best match.
- the ranking system 28 may collect and evaluate the user interaction history 72 of the extracted features, results, selections, and user identity. For instance, in the case where the user selects a second ranked item (e.g., eyeglasses) in the results instead of a first one (e.g., a bicycle), then the ranking system 28 registers this choice in the user interaction history database 72 with the extracted feature and user information. Thus, in future interactions, the ranking system 28 would consider the interaction history to better identify what the user is trying to draw and how to rank results based on past interactions.
- a second ranked item e.g., eyeglasses
- the matching system 24 determines whether the various features matched so far (i.e., the collection of inputs in the canvas) results in a composite match. If the whole collection of features is not returning a composite match, then no results are presented to the user, or a best guess of matches may be presented. If the whole collection generates at least one match, then resulting images or the like from the database 70 are presented to the user as a ranked list at 66 . The ranked list may be presented to the user in a user interface window, allowing the user to navigate and explore the results retrieved based on the current state of the canvas, including the combination of all layers composing the canvas 30 .
- FIG. 3 depicts an illustrative use case involving seismic interpretation.
- a geologist has sketched an unfamiliar geologic feature on the canvas 30 and a ranked set of results 34 are returned.
- the user can click on a result to, e.g., link to more comprehensive information such as the source of the image, textual information, etc.
- a toolbar 80 is provide which allows the user to highlight one or more inputs as more important. Additionally, in the case where inaccurate or no matches are located, the user can choose to show real-time feedback, which may for example suggest additional inputs or corrective actions to enhance the search.
- the geologist could thus use the system 18 to search for geological concepts with sketching, which is a common technique used by geoscientists to communicate complex terms.
- FIG. 4 depicts a further use case for the real estate industry.
- the user is looking for properties that have a particular configuration.
- the user could sketch the plan and further enrich the sketch by adding layers of detail, e.g., for windows, doors, neighborhoods, etc.
- the system 18 could evaluate these layers to find similar properties and return them to the user, either as retrieved image data, links, text information, etc.
- FIG. 5 depicts a further use case involving searching for a document in which only the general layout of certain pages of a book are recalled by the user.
- the user could sketch the layout of the document, e.g., drawing rectangles where the images were placed in the document pages.
- the user could specify additional layers that limited the search to a particular field of endeavor, a certain color, etc.
- the user could highlight the element as more important.
- the user can change the importance/threshold related to the submitted layers, indicating, for instance, that the layout should have a higher weight than the field of endeavor or the color.
- sketch-based retrieval system 18 may be implemented as a computer program product stored on a computer readable storage medium.
- the computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device.
- the computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing.
- a non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing.
- a computer readable storage medium is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.
- Computer readable program instructions described herein can be downloaded to respective computing/processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network.
- the network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers.
- a network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing/processing device.
- Computer readable program instructions for carrying out operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Java, Python, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the “C” programming language or similar programming languages.
- the computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server.
- the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).
- electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present invention.
- These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
- These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowchart and/or block diagram block or blocks.
- the computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
- each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s).
- the functions noted in the block may occur out of the order noted in the figures.
- two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved.
- Computing system 10 may comprise any type of computing device and for example includes at least one processor 12 , memory 21 , an input/output (I/O) 14 (e.g., one or more I/O interfaces and/or devices), and a communications pathway 16 .
- processor(s) 12 execute program code which is at least partially fixed in memory 21 . While executing program code, processor(s) 12 can process data, which can result in reading and/or writing transformed data from/to memory and/or I/O 14 for further processing.
- the pathway 16 provides a communications link between each of the components in computing system 10 .
- I/O 14 can comprise one or more human I/O devices, which enable a user to interact with computing system 10 .
- Computing system 10 may also be implemented in a distributed manner such that different components reside in different physical locations.
- the sketch-based retrieval system 18 or relevant components thereof may also be automatically or semi-automatically deployed into a computer system by sending the components to a central server or a group of central servers.
- the components are then downloaded into a target computer that will execute the components.
- the components are then either detached to a directory or loaded into a directory that executes a program that detaches the components into a directory.
- Another alternative is to send the components directly to a directory on a client computer hard drive.
- the process will select the proxy server code, determine on which computers to place the proxy servers' code, transmit the proxy server code, then install the proxy server code on the proxy computer.
- the components will be transmitted to the proxy server and then it will be stored on the proxy server.
Abstract
Description
- The subject matter of this invention relates to sketch-based image retrieval, and more particularly to a system and method for image searching based on customizable sketches and real-time feedback.
- Electronic data retrieval is common task in almost all fields of endeavor. For certain types of data retrieval, such as image searching, existing solutions often lack the necessary efficacy to be truly effective.
- For example, there may be instances when a user wants to search for an image, but does not know how to name the desired image and consequently does not know how to articulate a textual search query. In this situation, sketch-based retrieval provides a promising technique to search for image data. Sketch-based image retrieval allows a user to sketch some input, and based on the input, a set of matching images are returned. However, because this area is relatively new field, existing tools lack the ability to effectively interpret complex user inputs and return accurate results.
- Aspects of the disclosure provides a sketch-based image retrieval system and method that utilizes real-time feedback to enhance searching and provides an intuitive way to communicate hierarchy (i.e., importance) of inputted information.
- A first aspect discloses a sketch-based retrieval system, including: an interface having a canvas for entering inputs including user generated strokes, and for extracting features associated with the inputs; a hierarchy system for assigning an importance to the inputs; a matching system for retrieving matches from a database based on extracted features and importance associated with the extracted features; a feedback system that provides real-time feedback on the canvas to suggest improved inputs for the user; and a ranking system that ranks retrieved matches for output in the interface.
- A second aspect discloses a computer program product stored on a computer readable storage medium, which when executed by a computing system, provides sketch-based retrieval, the program product including: program code for rendering a canvas for entering inputs including user generated strokes, and for extracting features associated with the inputs; program code for assigning an importance to the inputs; program code for retrieving matches from a database based on extracted features and importance associated with the extracted features; program code that provides real-time feedback on the canvas to suggest improved inputs for the user; and program code that ranks retrieved matches for output.
- A third aspect discloses a computerized method of providing sketch-based retrieval, including: rendering a canvas for receiving inputs including user generated strokes, and for extracting features associated with the inputs; assigning an importance to the inputs; retrieving matches from a database based on extracted features and importance associated with the extracted features; providing real-time feedback on the canvas to suggest improved inputs for the user; and ranking retrieved matches for output in the interface.
- These and other features of this invention will be more readily understood from the following detailed description of the various aspects of the invention taken in conjunction with the accompanying drawings in which:
-
FIG. 1 shows a computing system having a sketch-based retrieval system according to embodiments. -
FIG. 2 shows a flow diagram for implementing a sketch-based retrieval system according to embodiments. -
FIG. 3 shows a use case involving seismic data retrieval according to embodiments. -
FIG. 4 shows a use case involving real estate data retrieval according to embodiments. -
FIG. 5 shows a use case involving document data retrieval according to embodiments. - The drawings are not necessarily to scale. The drawings are merely schematic representations, not intended to portray specific parameters of the invention. The drawings are intended to depict only typical embodiments of the invention, and therefore should not be considered as limiting the scope of the invention. In the drawings, like numbering represents like elements.
- Disclosed is a sketch-based image retrieval system and method in which the user inputs sketches and other information to generate matches in a database. Sketches are generally inputted via strokes made by a user on touch screen or gesture based input system. The described approach provides real-time feedback of the input to enhance the search process. In one embodiment, feedback may include suggestions of a next stroke or identification of problematic strokes (i.e., strokes that result in no matches). Such real-time feedback will result in accelerating the search process and create more accurate results. The approach further provides an intuitive way to communicate an importance hierarchy of inputted data. For example, strokes drawn with greater pressure or slower speed may be interpreted as more important than other strokes. This ability to convey and interpret importance of inputted information, such as a relative importance of different strokes, allows for enhanced searching and ranking of results.
- Referring now to the drawings,
FIG. 1 depicts acomputing system 10 having a sketch-basedretrieval system 18 that retrieves image-related data from adatabase 70 in response to input (i.e., sketches and other data) inputted via aninterface device 32 by a user.Interface device 32 may be implemented with a touch screen tablet, laptop, smart device, or other gesture based appliance.Interface device 32 may be fully or partially integrated with thecomputing system 10 or reside as a standalone device such that some or all of the functions described are performed by either or both theinterface device 32 andcomputing system 10.Database 70 refers to any collection of image-related data, which may for example reside in a single location or be distributed among various storage systems including cloud based storage systems. - Sketch-based
retrieval system 18 generally includes an input/output manager 20 that is responsible for displaying and collecting information from theinterface device 32. This includes displaying inputs entered by the user, extracting features from the inputs and displaying results and feedback. Extracted features may for example comprise polylines or curves that may be represented as image data and/or via mathematical formulae. -
Interface device 32 generally includes acanvas 30 onto which a user can sketch images by inputting a series of strokes, e.g., with a finger, a pen, a gesture, etc. As the user makes changes to thecanvas 30, a set of ranked images 34 (or other data) are retrieved and displayed that most closely match the input. Thecanvas 30 may include one or more layers for receiving input. Different layers can include, e.g., inputted strokes, loaded images, selectable designs, search categories, etc. Loaded images may for example include photos, patterns, blueprints, seismic images, charts, plots, etc. - In the
example canvas 30 shown, a single layer is depicted in which the user has entered twocircular shapes 36 and asquiggly line 38. Note that thecircular shapes 36 are inputted bolder than thesquiggly line 38, indicating a higher importance. Furthermore, in response to the user input, feedback in the form ofdashed lines 40 are automatically rendered on thecanvas 30 in real-time. The feedback is utilized to help the user better refine the input to achieve enhanced results. For example, thesystem 18 may not be certain if the user is attempting to draw a bicycle or a pair of glasses. By presenting feedback, the user can click on one of the dashed lines to better define the input. Alternatively, the user could ignore the feedback and continue entering strokes until matching images are identified. -
Hierarchy system 22 provides a mechanism through which the user can assign a relative importance to different inputs. For example, relative importance can be assigned by the user with strokes made with greater pressure, with bolder lines, with highlighting, with coloring, via an interface that allows for manual selections, etc. Further, different layers of thecanvas 30 may be indicated as more or less important. The relative importance of different inputs will be utilized by the matchingsystem 24 and rankingsystem 28 when attempting to match and rank retrieved image-related data from thedatabase 70. - Whenever new inputs are detected on the
canvas 30, input/output manager 20 will extract “features” associated with the inputs and matchingsystem 24 will attempt to match the features (e.g., curves, polylines, etc.) with indexed image related data in thedatabase 70. Matching can be done on a feature level (e.g., matching circles to wheels) and/or on an entire composite image basis (e.g., matching circles and lines to a complete bicycle). -
Feedback system 26 provides real-time feedback to the user as the user enters inputs. For example, when thematching system 24 is having difficulty matching extracted features with image related data in thedatabase 70,feedback system 26 can highlight existing user inputs as problematic inputs that are creating ambiguities, which should either be removed or redrawn. Alternatively,feedback system 26 can generate proposed inputs (e.g., dashedlines 40, blinking lines, colored shapes, etc.) to help guide the user in the input process. Accordingly, based on the displayed proposed inputs, the user can select a proposed input to see if the output results in a desired match. - Regardless, whenever an input results in a set of possible matches,
ranking system 28 ranks the retrieved matches and displays them to the user, e.g., as a set of rankedimages 34, e.g., as thumbnails, textual descriptions, scrollable images, etc. - Throughout the process,
user interaction history 72 can be collected from one or more users. Thehistory 72 can thereafter be analyzed and used to improve future searches and rankings. For instance, the analysis may dictate that inputting two circles indicates the user is attempting to sketch a bicycle 55% of the time,eyeglasses 40% of the time, or something else 5% of the time. Artificial intelligence including the use of neural networks may be utilized to facilitate the analysis and learning process. -
FIG. 2 depicts a flow diagram for implementing an embodiment of the sketch-basedretrieval system 18 with reference toFIG. 1 . In this illustrative approach, individual features or subsets of features extracted from the user inputs are first analyzed for feature based matches. After each match is made at the feature level, a composite match of the entire input (i.e., different layers, strokes, etc.) is attempted. - The process begins at 52 with the user making input changes to the
canvas 30. This may involve any user action associated with the canvas, including sketching in an active layer, loading image data, adding a new layer to the canvas, etc. As noted, the input may include hierarchical information to assign different weights or importance to different inputs. Assigning importance to different inputs may involve implicit and explicit actions by the user. For example, implicit actions may involve: pen pressure by to rank different strokes; the time taken to draw a stroke; etc. Explicit actions may involve: manually assigning weights via an interface tool to prioritize layers or strokes; assigning thresholds for each layer or stroke to be more/less strict for matching of features with indexed images; etc. - Once the
canvas 30 is changed with a new input or collection of inputs, the input/output manager 20 is triggered to extract associated features from thecanvas 30 at 54. Extraction methods may for example include Extended Trace Transform (ETT) and Image Deformation Model Features (IDM), among others. Once extracted, thematching system 24 searches for the features at 56 using indexed data indatabase 70. - At 58, a determination is made whether features were located in the database 70 (e.g., within a predetermined confidence threshold). If no, the
feedback system 26 updates thecanvas 30 with feedback at 60 and waits for further user changes at 52. For example, thefeedback system 26 may highlight the last input the user performed, i.e., a stroke added or an image loaded in thecanvas 30. The highlight can be used to inform the user that the last change resulted in a problematic input, e.g., no database matches or it created a significant ambiguity. Highlighting can be done in any manner, for instance, highlighting in red, marking the border of the stroke, using a shaded pattern to cover the stroke, etc. Thefeedback system 26 may also suggest or propose inputs (e.g., with dashed lines, blinking lines, etc.) that, if entered, would increase the likelihood or result in a match. - If extracted features were matched with image related data in the
database 70, the rankingsystem 28 ranks results at 62 using, e.g., an algorithm to match similarities between the extracted features and results. The ranked results may be displayed such that the user can select a best match. The rankingsystem 28 may collect and evaluate theuser interaction history 72 of the extracted features, results, selections, and user identity. For instance, in the case where the user selects a second ranked item (e.g., eyeglasses) in the results instead of a first one (e.g., a bicycle), then theranking system 28 registers this choice in the userinteraction history database 72 with the extracted feature and user information. Thus, in future interactions, the rankingsystem 28 would consider the interaction history to better identify what the user is trying to draw and how to rank results based on past interactions. - At 64, the
matching system 24 determines whether the various features matched so far (i.e., the collection of inputs in the canvas) results in a composite match. If the whole collection of features is not returning a composite match, then no results are presented to the user, or a best guess of matches may be presented. If the whole collection generates at least one match, then resulting images or the like from thedatabase 70 are presented to the user as a ranked list at 66. The ranked list may be presented to the user in a user interface window, allowing the user to navigate and explore the results retrieved based on the current state of the canvas, including the combination of all layers composing thecanvas 30. - If the user selects a result at 68 as a match, information associated with the match is retrieved from the
database 70 and theuser interaction history 72 is updated. -
FIG. 3 depicts an illustrative use case involving seismic interpretation. In this example, a geologist has sketched an unfamiliar geologic feature on thecanvas 30 and a ranked set ofresults 34 are returned. The user can click on a result to, e.g., link to more comprehensive information such as the source of the image, textual information, etc. Atoolbar 80 is provide which allows the user to highlight one or more inputs as more important. Additionally, in the case where inaccurate or no matches are located, the user can choose to show real-time feedback, which may for example suggest additional inputs or corrective actions to enhance the search. The geologist could thus use thesystem 18 to search for geological concepts with sketching, which is a common technique used by geoscientists to communicate complex terms. -
FIG. 4 depicts a further use case for the real estate industry. In this case, the user is looking for properties that have a particular configuration. The user could sketch the plan and further enrich the sketch by adding layers of detail, e.g., for windows, doors, neighborhoods, etc. Thesystem 18 could evaluate these layers to find similar properties and return them to the user, either as retrieved image data, links, text information, etc. -
FIG. 5 depicts a further use case involving searching for a document in which only the general layout of certain pages of a book are recalled by the user. In this case, the user could sketch the layout of the document, e.g., drawing rectangles where the images were placed in the document pages. The user could specify additional layers that limited the search to a particular field of endeavor, a certain color, etc. Further, for elements the user was more certain of, e.g., the location of a picture, the user could highlight the element as more important. Once the user receives the list of results, the user can change the importance/threshold related to the submitted layers, indicating, for instance, that the layout should have a higher weight than the field of endeavor or the color. - It is understood that sketch-based
retrieval system 18 may be implemented as a computer program product stored on a computer readable storage medium. The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. - The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.
- Computer readable program instructions described herein can be downloaded to respective computing/processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing/processing device.
- Computer readable program instructions for carrying out operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Java, Python, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present invention.
- Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.
- These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowchart and/or block diagram block or blocks.
- The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
- The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.
-
Computing system 10, as shown inFIG. 1 , may comprise any type of computing device and for example includes at least oneprocessor 12,memory 21, an input/output (I/O) 14 (e.g., one or more I/O interfaces and/or devices), and acommunications pathway 16. In general, processor(s) 12 execute program code which is at least partially fixed inmemory 21. While executing program code, processor(s) 12 can process data, which can result in reading and/or writing transformed data from/to memory and/or I/O 14 for further processing. Thepathway 16 provides a communications link between each of the components incomputing system 10. I/O 14 can comprise one or more human I/O devices, which enable a user to interact withcomputing system 10.Computing system 10 may also be implemented in a distributed manner such that different components reside in different physical locations. - Furthermore, it is understood that the sketch-based
retrieval system 18 or relevant components thereof (such as an API component, agents, etc.) may also be automatically or semi-automatically deployed into a computer system by sending the components to a central server or a group of central servers. The components are then downloaded into a target computer that will execute the components. The components are then either detached to a directory or loaded into a directory that executes a program that detaches the components into a directory. Another alternative is to send the components directly to a directory on a client computer hard drive. When there are proxy servers, the process will select the proxy server code, determine on which computers to place the proxy servers' code, transmit the proxy server code, then install the proxy server code on the proxy computer. The components will be transmitted to the proxy server and then it will be stored on the proxy server. - The foregoing description of various aspects of the invention has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed, and obviously, many modifications and variations are possible. Such modifications and variations that may be apparent to an individual in the art are included within the scope of the invention as defined by the accompanying claims.
Claims (20)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/452,646 US20190317959A1 (en) | 2017-06-06 | 2019-06-26 | Sketch-based image retrieval using feedback and hierarchies |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/614,682 US10380175B2 (en) | 2017-06-06 | 2017-06-06 | Sketch-based image retrieval using feedback and hierarchies |
US16/452,646 US20190317959A1 (en) | 2017-06-06 | 2019-06-26 | Sketch-based image retrieval using feedback and hierarchies |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/614,682 Continuation US10380175B2 (en) | 2017-06-06 | 2017-06-06 | Sketch-based image retrieval using feedback and hierarchies |
Publications (1)
Publication Number | Publication Date |
---|---|
US20190317959A1 true US20190317959A1 (en) | 2019-10-17 |
Family
ID=64460566
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/614,682 Active 2037-06-14 US10380175B2 (en) | 2017-06-06 | 2017-06-06 | Sketch-based image retrieval using feedback and hierarchies |
US16/452,646 Abandoned US20190317959A1 (en) | 2017-06-06 | 2019-06-26 | Sketch-based image retrieval using feedback and hierarchies |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/614,682 Active 2037-06-14 US10380175B2 (en) | 2017-06-06 | 2017-06-06 | Sketch-based image retrieval using feedback and hierarchies |
Country Status (1)
Country | Link |
---|---|
US (2) | US10380175B2 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20220392123A1 (en) * | 2021-06-06 | 2022-12-08 | Apple Inc. | Image composition with multiple blend modes |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11650717B2 (en) * | 2019-07-10 | 2023-05-16 | International Business Machines Corporation | Using artificial intelligence to iteratively design a user interface through progressive feedback |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20120054177A1 (en) * | 2010-08-31 | 2012-03-01 | Microsoft Corporation | Sketch-based image search |
US20120072410A1 (en) * | 2010-09-16 | 2012-03-22 | Microsoft Corporation | Image Search by Interactive Sketching and Tagging |
US20120162244A1 (en) * | 2010-12-28 | 2012-06-28 | Microsoft Corporation | Image search color sketch filtering |
WO2014198051A1 (en) * | 2013-06-14 | 2014-12-18 | Microsoft Corporation | Color sketch image searching |
US20160203194A1 (en) * | 2015-01-09 | 2016-07-14 | Samsung Electronics Co., Ltd. | User terminal for displaying image and image display method thereof |
US20170262479A1 (en) * | 2016-03-08 | 2017-09-14 | Shutterstock, Inc. | User drawing based image search |
Family Cites Families (16)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US3624619A (en) | 1969-01-17 | 1971-11-30 | Telautograph Corp | Pressure-sensing tablet |
US20070061735A1 (en) * | 1995-06-06 | 2007-03-15 | Hoffberg Steven M | Ergonomic man-machine interface incorporating adaptive pattern recognition based control system |
US5911139A (en) * | 1996-03-29 | 1999-06-08 | Virage, Inc. | Visual image database search engine which allows for different schema |
US6208355B1 (en) | 1998-04-07 | 2001-03-27 | Adobe Systems Incorporated | Sketch-based editing of curves |
US6748398B2 (en) | 2001-03-30 | 2004-06-08 | Microsoft Corporation | Relevance maximizing, iteration minimizing, relevance-feedback, content-based image retrieval (CBIR) |
US8429174B2 (en) * | 2003-01-25 | 2013-04-23 | Purdue Research Foundation | Methods, systems, and data structures for performing searches on three dimensional objects |
US20050132269A1 (en) | 2003-12-10 | 2005-06-16 | Siemens Corporate Research, Inc. | Method for retrieving image documents using hierarchy and context techniques |
US20050283752A1 (en) * | 2004-05-17 | 2005-12-22 | Renate Fruchter | DiVAS-a cross-media system for ubiquitous gesture-discourse-sketch knowledge capture and reuse |
US20060227140A1 (en) | 2005-03-21 | 2006-10-12 | Karthik Ramani | Sketch beautification |
CN100456300C (en) | 2006-10-27 | 2009-01-28 | 北京航空航天大学 | Method for searching 3D model based on 2D sketch |
US20100049629A1 (en) * | 2008-08-22 | 2010-02-25 | Nainesh B Rathod | Method of enabling a user to draw a component part as input for searching component parts in a database |
US20110193795A1 (en) | 2010-02-09 | 2011-08-11 | Yahoo! Inc. | Haptic search feature for touch screens |
US8589410B2 (en) * | 2011-10-18 | 2013-11-19 | Microsoft Corporation | Visual search using multiple visual input modalities |
US9528847B2 (en) * | 2012-10-15 | 2016-12-27 | Microsoft Technology Licensing, Llc | Pictures from sketches |
US9324102B2 (en) * | 2013-03-14 | 2016-04-26 | Ebay Inc. | System and method to retrieve relevant inventory using sketch-based query |
US9183467B2 (en) | 2013-05-03 | 2015-11-10 | Microsoft Technology Licensing, Llc | Sketch segmentation |
-
2017
- 2017-06-06 US US15/614,682 patent/US10380175B2/en active Active
-
2019
- 2019-06-26 US US16/452,646 patent/US20190317959A1/en not_active Abandoned
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20120054177A1 (en) * | 2010-08-31 | 2012-03-01 | Microsoft Corporation | Sketch-based image search |
US20120072410A1 (en) * | 2010-09-16 | 2012-03-22 | Microsoft Corporation | Image Search by Interactive Sketching and Tagging |
US20120162244A1 (en) * | 2010-12-28 | 2012-06-28 | Microsoft Corporation | Image search color sketch filtering |
WO2014198051A1 (en) * | 2013-06-14 | 2014-12-18 | Microsoft Corporation | Color sketch image searching |
US20160203194A1 (en) * | 2015-01-09 | 2016-07-14 | Samsung Electronics Co., Ltd. | User terminal for displaying image and image display method thereof |
US20170262479A1 (en) * | 2016-03-08 | 2017-09-14 | Shutterstock, Inc. | User drawing based image search |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20220392123A1 (en) * | 2021-06-06 | 2022-12-08 | Apple Inc. | Image composition with multiple blend modes |
Also Published As
Publication number | Publication date |
---|---|
US20180349415A1 (en) | 2018-12-06 |
US10380175B2 (en) | 2019-08-13 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10459995B2 (en) | Search engine for processing image search queries in multiple languages | |
RU2688271C2 (en) | Image search in natural language | |
US9690831B2 (en) | Computer-implemented system and method for visual search construction, document triage, and coverage tracking | |
Behrisch et al. | Feedback-driven interactive exploration of large multidimensional data supported by visual classifier | |
US10289727B2 (en) | Incorporation of semantic attributes within social media | |
US10417492B2 (en) | Conversion of static images into interactive maps | |
CN104380249B (en) | folding view in development environment | |
US11361002B2 (en) | Method and apparatus for recognizing entity word, and storage medium | |
JP7242994B2 (en) | Video event identification method, apparatus, electronic device and storage medium | |
CN111263943B (en) | Semantic normalization in document digitization | |
US20220253719A1 (en) | Schema augmentation system for exploratory research | |
CN112507090A (en) | Method, apparatus, device and storage medium for outputting information | |
CN110178129A (en) | The system and method for generating the semantic expressiveness of the target image for image processing operations | |
US20190317959A1 (en) | Sketch-based image retrieval using feedback and hierarchies | |
US9607216B2 (en) | Identifying updated content in images | |
US11810177B2 (en) | Clothing collocation | |
US20210271637A1 (en) | Creating descriptors for business analytics applications | |
US20220413997A1 (en) | Automated locating of gui elements during testing using multidimensional indices | |
CN114281990A (en) | Document classification method and device, electronic equipment and medium | |
Roullet et al. | Transfer learning methods for extracting, classifying and searching large collections of historical images and their captions | |
US9740796B2 (en) | Navigating a website using visual analytics and a dynamic data source | |
US10922476B1 (en) | Resource-efficient generation of visual layout information associated with network-accessible documents | |
US20240126807A1 (en) | Visual Search Determination for Text-To-Image Replacement | |
US20240070943A1 (en) | Generating image editing presets based on editing intent extracted from a digital query | |
CN115809334B (en) | Training method of event relevance classification model, text processing method and device |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: INTERNATIONAL BUSINESS MACHINES CORPORATION, NEW Y Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:ABREU DE PAULA, ROGERIO;ASHTON VITAL BRAZIL, EMILIO;DA SILVA FERREIRA, RODRIGO;AND OTHERS;REEL/FRAME:049590/0346 Effective date: 20170626 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
AS | Assignment |
Owner name: INTERNATIONAL BUSINESS MACHINES CORPORATION, NEW YORK Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE ASSIGNOR'S NAME PREVIOUSLY RECORDED AT REEL: 049590 FRAME: 0346. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT;ASSIGNORS:ABREU DE PAULA, ROGERIO;ASHTON VITAL BRAZIL, EMILIO;DA SILVA FERREIRA, RODRIGO;AND OTHERS;REEL/FRAME:053119/0621 Effective date: 20170626 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE AFTER FINAL ACTION FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: ADVISORY ACTION MAILED |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |