JP2024519504A - Reverse image search based on deep neural network (DNN) model and image feature detection model - Google Patents

Abstract

An electronic device and method for reverse image searching is provided. The electronic device receives an image. The electronic device extracts a first set of image features associated with the image via a DNN model and generates a first feature vector based on the first set of image features. The electronic device extracts a second set of image features associated with the image via an image feature detection model and generates a second feature vector based on the second set of image features. The electronic device generates a third feature vector based on a combination of the first and second feature vectors. The electronic device determines a similarity metric between the third feature vector and a fourth feature vector of each image in a set of pre-stored images and identifies the pre-stored images based on the similarity metric. The electronic device controls a display device to display information associated with the pre-stored images. (FIG. 1)

Description

CROSS REFERENCE TO RELATED APPLICATIONS/INCORPORATION BY REFERENCE
This application claims priority to U.S. Provisional Patent Application Serial No. 63/189,956, filed May 18, 2021, the contents of which are incorporated herein by reference in their entirety.

Various embodiments of the present disclosure relate to reverse image searching. In particular, various embodiments of the present disclosure relate to electronic devices and methods for reverse image searching based on deep neural network (DNN) models and image feature detection models.

Advances in information and communication technology have led to a variety of Internet-based image retrieval systems (e.g., web search engines). Traditionally, a user may upload an input image to a web search engine as a search query. In such a case, the web search engine may provide an output image set from the Internet (using a reverse image search technique). The output image set may be similar to the input image. Such a reverse image search technique may employ a machine learning model to determine an output image set that is similar to the input image. In some cases, the machine learning model may misclassify one or more objects in the input image, resulting in the output image set including undesirable or irrelevant images.

The limitations and disadvantages of conventional approaches will become apparent to one skilled in the art by comparing the described system with certain aspects of the present disclosure illustrated in the remainder of this application and with reference to the drawings.

The present invention provides an electronic device and method for reverse image search based on a deep neural network (DNN) model and an image feature detection model, substantially as illustrated and/or described in connection with at least one of the figures and more fully set forth in the claims.

These and other features and advantages of the present disclosure can be understood by considering the following detailed description of the disclosure in conjunction with the accompanying drawings, in which like reference characters refer to like elements throughout.

FIG. 1 is a block diagram illustrating an exemplary network environment for reverse image search based on a deep neural network (DNN) model and an image feature detection model, in accordance with an embodiment of the present disclosure. FIG. 1 is a block diagram illustrating an exemplary electronic device for reverse image search based on a deep neural network (DNN) model and an image feature detection model, in accordance with an embodiment of the present disclosure. FIG. 1 illustrates an example operation for reverse image search based on a deep neural network (DNN) model and an image feature detection model, according to an embodiment of the present disclosure. 1 is a flowchart illustrating an example method for reverse image search based on a deep neural network (DNN) model and an image feature detection model, according to an embodiment of the present disclosure.

The disclosed electronic device and method for performing reverse image search based on a deep neural network (DNN) model and an image feature detection model to improve the accuracy of the reverse image search can be implemented as described below. An exemplary aspect of the present disclosure provides an electronic device implementing a deep neural network (DNN) model and an image feature detection model for reverse image search. The electronic device can receive a first image (such as an image for which a user needs to search for similar images). The electronic device can extract a first set of image features associated with the received first image by the deep neural network (DNN) model, and generate a first feature vector associated with the received first image based on the extracted first image feature set. The electronic device can extract a second set of image features associated with the received first image by the image feature detection model, and generate a second feature vector associated with the received first image based on the extracted second image feature set. Examples of image feature detection models include, but are not limited to, a Scale-Invariant Feature Transform (SIFT)-based model, a Speeded-Up Robust Feature (SURF)-based model, an Oriented FAST and Rotated BRIEF (ORB)-based model, or a Fast Library for Approximate Nearest Neighbors (FLANN)-based model. The image feature detection model can extract image features that may have been partially misdetected and/or misclassified by the DNN model 108.

The electronic device may further generate a third feature vector associated with the received first image based on a combination of the generated first feature vector and the generated second feature vector. In one example, the generation of the third feature vector may be further based on application of a principal component analysis (PCA) transform to the combination of the generated first feature vector and the generated second feature vector. The electronic device may further determine a similarity metric between the generated third feature vector associated with the received first image and a fourth feature vector of each image of the pre-stored second set of images (e.g., images stored in a database). Examples of similarity metrics may include, but are not limited to, cosine distance similarity or Euclidean distance similarity. The electronic device may further identify a pre-stored third image (e.g., an image that is the same or similar to the received first image) from the pre-stored second set of images based on the determined similarity metric, and control the display device to display information associated with the identified pre-stored third image.

The disclosed electronic device can automatically generate a third feature vector associated with the received first image based on a combination of the generated first feature vector and the generated second feature vector. As a result, the third feature vector can include a first image feature set that can be determined by a DNN model and a second image feature set that can be determined by an image feature detection model. The first image feature set includes high-level image features associated with the received first image (e.g., facial features such as eyes, nose, ears, hair, etc.), and the second image feature set includes low-level image features associated with the received first image (e.g., facial edges, lines, contours). By including both the high-level image features and the low-level image features in the third feature vector, the identification of similar images can be complemented. For example, if the received first image is an image that is not well represented in the training dataset of the DNN model, the first image feature set may not be sufficient to identify images similar to the received first image from the pre-stored second image set. However, since the second image feature set may include low-level image features related to the received first image, including the second image feature set in the third feature vector may increase the accuracy of identifying images similar to the received first image from the pre-stored second image set. On the other hand, if the image quality is poor (e.g., in the case of a low-resolution, blurry image), the first image feature set (i.e., high-level image features) may not be sufficient to identify similar images. In such cases, the second image features (i.e., low-level image features) may be more useful and accurate for identifying similar images.

FIG. 1 is a block diagram illustrating an exemplary network environment for reverse image search based on a deep neural network (DNN) model and an image feature detection model, according to an embodiment of the present disclosure. FIG. 1 illustrates a network environment 100. The network environment 100 may include an electronic device 102, a server 104, and a database 106. Also illustrated is a deep neural network (DNN) model 108 and an image feature detection model 110 implemented on the server 104. As illustrated in FIG. 1, the database 106 may store a training data set 112. The electronic device 102, the server 104, and the database 106 may be communicatively coupled to each other via a communication network 114. Also illustrated is a user 116 associated with the electronic device 102. Although FIG. 1 illustrates the electronic device 102 and the server 104 as two separate devices, in some embodiments, the entire functionality of the server 104 may be incorporated into the electronic device 102 without departing from the scope of the present disclosure.

The electronic device 102 may include suitable logic, circuitry, interfaces, and/or code that may be configured to identify and display a set of images similar to a first image based on implementing the DNN model 108 and the image feature detection model 110 on the first image. Examples of the electronic device 102 may include, but are not limited to, an image search engine, a server, a personal computer, a laptop, a computer workstation, a mainframe machine, a gaming device, a virtual reality (VR)/augmented reality (AR)/mixed reality (MR) device, a smartphone, a mobile phone, a computing device, a tablet, and/or any consumer electronics (CE) device.

The DNN model 108 may be a deep convolutional neural network model that can be trained to detect a first set of image features in a first image based on an image feature detection task. The DNN model 108 may be defined by hyperparameters such as activation function(s), number of weights, cost function, regularization function, input size, and number of layers. The DNN model 108 may be referred to as a computational network or a system of artificial neurons (also called nodes). The nodes of the DNN model 108 may be arranged in multiple layers as defined by the neural network topology of the DNN model 108. The multiple layers of the DNN model 108 may include an input layer, one or more hidden layers, and an output layer. Each layer of the multiple layers may include one or more nodes (or artificial neurons). The output of every node in the input layer may be coupled to at least one node in the hidden layer(s). Similarly, the input of each hidden layer may be coupled to the output of at least one node in another layer of the DNN model 108. The output of each hidden layer may be coupled to the input of at least one node in another layer of the DNN model 108. The node(s) in the final layer may receive input from at least one hidden layer and output a result. The number of layers and the number of nodes in each layer may be determined from hyperparameters of the DNN model 108. Such hyperparameters may be set before or during training of the DNN model 108 based on the training dataset 112.

Each node of the DNN model 108 can correspond to a mathematical function (e.g., a sigmoid function or a rectified linear unit) having a parameter set that can be adjusted during training of the network. The parameter set can include, for example, weight parameters and regularization parameters. Each node can calculate an output using a mathematical function based on one or more inputs from nodes in other layer(s) (e.g., previous layer(s)) of the DNN model 108. All or some of the nodes of the DNN model 108 can correspond to the same or different mathematical functions.

Training the DNN model 108 may involve updating one or more parameters of each node of the DNN model 108 based on whether the output of the final layer for a given input (from the training dataset) matches the correct result based on the loss function of the DNN model 108. The above process may be repeated for the same or different inputs until a minimum of the loss function is achieved and the training error is minimized. Several training methods are known in the art, such as gradient descent, stochastic gradient descent, batch gradient descent, gradient boosting, and metaheuristic methods.

In some embodiments, the DNN model 108 may include electronic data that may be implemented, for example, as a software component of an application executable on the electronic device 102 or server 104. The DNN model 108 may rely on libraries, external scripts, or other logic/instructions for execution by a processing device, such as the electronic device 102 or server 110. The DNN model 108 may include computer-executable code or routines that enable a computing device, such as the electronic device 102 or server 104, to perform one or more operations to detect image features in an input image. Additionally or alternatively, the DNN model 108 may be implemented using hardware, including a processor, a microprocessor (e.g., performing or controlling one or more operations), a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC). For example, the electronic device 102 (or server 104) may include an inference accelerator chip that accelerates the computation of the DNN model 108 for image feature detection tasks. In some embodiments, the DNN model 108 may be implemented using a combination of both hardware and software. Examples of the DNN model 108 include, but are not limited to, an artificial neural network (ANN), a convolutional neural network (CNN), Regions with CNN (R-CNN), Fast R-CNN, Faster R-CNN, You Only Look Once (YOLO) network, a residual neural network (Res-Net), a feature pyramid network (FPN), a retina net, a single shot detector (SSD), and/or combinations thereof.

The image feature detection model 110 may be an image processing algorithm configured to extract image features associated with the first image. The image feature detection model 110 may be defined by hyperparameters such as, for example, the number of image features, an edge threshold, the number of weights, a cost function, an input size, and the number of layers. The hyperparameters of the image feature detection model 110 may be adjusted to move toward a global minimum of the cost function of the image feature detection model 110, and the weights may be updated accordingly. The image feature detection model 110 may include electronic data that may be implemented as a software component of an application executable on, for example, the electronic device 102 or the server 104. The image feature detection model 110 may rely on libraries, external scripts, or other logic/instructions for execution by a processing device, such as the electronic device 102 or the server 104. The image feature detection model 110 may include code and routines configured to enable a computing device, such as the electronic device 102 or the server 104, to perform one or more operations, such as extracting a set of image features associated with the first image. Additionally or alternatively, the image feature detection model 110 may be implemented using hardware, including a processor, a microprocessor (e.g., performing or controlling one or more operations), a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC). Alternatively, in some embodiments, the image feature detection model 110 may be implemented using a combination of hardware and software. Examples of the image feature detection model 110 may include, but are not limited to, a scale invariant feature transform (SIFT)-based model, a speed-up robust features (SURF)-based model, an oriented FAST and rotated BRIEF (ORB)-based model, or a fast library for approximate nearest neighbors (FLANN)-based model.

The server 104 may include suitable logic, circuitry, interfaces, and/or code that may be configured to store the DNN model 108 and the image feature detection model 110. The server 104 may use the DNN model 108 to generate a first feature vector associated with a first image and the image feature detection model 110 to generate a second feature vector associated with the first image. The server 104 may further store a machine learning model different from the DNN model 108 and the image feature detection model 110. The stored machine learning model may be configured to determine a first weight associated with the generated first feature vector and a second weight associated with the generated second feature vector. In an exemplary embodiment, the server 104 is implemented as a cloud server and may perform operations through web applications, cloud applications, HTTP requests, repository operations, file transfers, and the like. Other implementations of the server 104 may include, but are not limited to, a database server, a file server, a web server, an application server, a mainframe server, or a cloud computing server.

In at least one embodiment, the server 104 can be implemented as multiple distributed cloud-based resources using techniques known to those of skill in the art. Those skilled in the art will appreciate that the scope of the present disclosure may not be limited to the implementation of the server 104 and the electronic device 102 as two separate entities. In some embodiments, the functionality of the server 104 may be incorporated in whole or at least in part into the electronic device 102 without departing from the scope of the present disclosure.

The database 106 may include suitable logic, interfaces, and/or code that may be configured to store a training data set 112 for the DNN model 108. The training data set 112 may include a pre-stored set of training images and a pre-defined tag assigned to each image of the pre-stored set of training images. The pre-defined tag assigned to a particular training image may include a label corresponding to an image feature that may have been pre-determined for the particular training image. The DNN model 108 may be pre-trained for an image feature detection task based on the training data set 112. In some embodiments, the database 106 may be further configured to store a second pre-stored set of images. The database 106 may be a relational or non-relational database. Also, in some cases, the database 106 may be stored on a server, such as a cloud server (e.g., the server 104), or cached and stored on the electronic device 102. Additionally or alternatively, database 106 may be implemented using hardware, including a processor, a microprocessor (e.g., performing or controlling one or more operations), a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC). In some other cases, database 106 may be implemented using a combination of hardware and software.

The communication network 114 may include a communication medium that allows the electronic device 102, the server 104, and the database 106 to communicate with each other. Examples of the communication network 114 may include, but are not limited to, the Internet, a cloud network, a Long Term Evolution (LTE) network, a (wireless local area network) WLAN, a local area network (LAN), a telephone line (POTS), and/or a metropolitan area network (MAN). The various devices in the network environment 100 may be configured to connect to the communication network 114 according to various wired and wireless communication protocols. Examples of such wired and wireless communication protocols include, but are not limited to, at least one of the following: Transmission Control Protocol and Internet Protocol (TCP/IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), ZigBee, EDGE, IEEE 802.11, Light Fidelity (Li-Fi), 802.16, IEEE 802.11s, IEEE 802.11g, multi-hop communication, wireless access point (AP), device-to-device communication, cellular communication protocol, or Bluetooth (BT) communication protocol, or a combination thereof.

In operation, the electronic device 102 may initiate a reverse image search query. In some embodiments, the reverse image search may be initiated based on user input received via a display device (shown in FIG. 2). The electronic device 102 may be configured to receive a first image as an image search query at the initiation of the reverse image search. For example, the first image may correspond to an image uploaded through an I/O device (shown in FIG. 2) of the electronic device 102 based on user input. The first image may relate to a still image with fixed foreground or background objects, or an image extracted from a video. The electronic device 102 may be configured to extract a first set of image features associated with the received first image by the DNN model 108. The electronic device 102 may be configured to extract a second set of image features associated with the received first image by the image feature detection model 110. Details of the first and second image feature sets are shown, for example, in FIG. 3. Examples of the image feature detection model 110 include, but are not limited to, a scale invariant feature transform (SIFT)-based model, a speed-up robust features (SURF)-based model, an oriented FAST and rotated BRIEF (ORB)-based model, or a fast library for approximate nearest neighbors (FLANN)-based model.

The electronic device 102 may be further configured to generate a first feature vector associated with the received first image based on the extracted first image feature set. The electronic device 102 may be further configured to generate a second feature vector associated with the received first image based on the extracted second image feature set. The first feature vector associated with the received first image may be a vector that may include information about the first image feature set, and the second feature vector associated with the received first image may be a vector that may include information about the second image feature set. The electronic device 102 may be further configured to generate a third feature vector associated with the received first image based on a combination of the generated first feature vector and the generated second feature vector. The third feature vector includes, but is not limited to, the generated first feature vector and the generated second vector. In an embodiment, the generation of the third feature vector may be further based on application of a principal component analysis (PCA) transform to the combination of the generated first feature vector and the generated second feature vector. Details of the generation of the third feature vector are described, for example, in FIG. 3.

The electronic device 102 may be configured to determine a similarity metric between the generated third feature vector associated with the received first image and the fourth feature vector of each image of the pre-stored second set of images (e.g., images stored in the database 106). Examples of similarity metrics may include, but are not limited to, cosine distance similarity or Euclidean distance similarity. The electronic device 102 may be configured to identify a pre-stored third image (e.g., an image that is the same or similar to the received first image) from the pre-stored second set of images based on the determined similarity metric. The electronic device 102 may be further configured to control a display device to display information related to the identified pre-stored third image. Details of the determination of the similarity metric and the identification of the pre-stored third image are further described, for example, in FIG. 3.

2 is a block diagram illustrating an exemplary electronic device for reverse image search based on a deep neural network (DNN) model and an image feature detection model, according to an embodiment of the present disclosure. The description of FIG. 2 is provided with reference to the elements of FIG. 1. FIG. 2 illustrates a block diagram 200 of an electronic device 102. The electronic device 102 may include a circuit 202, a memory 204, an input/output (I/O) device 206, and a network interface 208. The I/O device 206 may further include a display device 210. The network interface 208 may connect the electronic device 102 to the server 104 and the database 106 via the communication network 114.

The circuitry 202 may include suitable logic, circuits, and/or interfaces that may be configured to execute program instructions associated with different operations performed by the electronic device 102. The circuitry 202 may include one or more specialized processing units that may be implemented as independent processors. In some embodiments, the one or more specialized processing units may be implemented as an integrated processor or processors that may be configured to collectively perform the functions of the one or more specialized processing units. The circuitry 202 may be implemented based on multiple processor technologies known in the art. Example implementations of the circuitry 202 may be an X86-based processor, a graphic processing unit (GPU), a reduced instruction set computing (RISC) processor, an application specific integrated circuit (ASIC) processor, a complex instruction set computing (CISC) processor, a microcontroller, a central processing unit (CPU), and/or other control circuitry.

The memory 204 may include suitable logic, circuitry, interfaces, and/or code that may be configured to store program instructions executed by the circuit 202. In at least one embodiment, the memory 204 may be configured to store the DNN model 108 and the image feature detection model 110. The memory 204 may be configured to store one or more of, but is not limited to, a similarity metric, a machine learning model different from the DNN model 108 and the image feature detection model 110 (e.g., the machine learning model 316 of FIG. 3), a first weight associated with the generated first feature vector, and a second weight associated with the generated second feature vector. In an embodiment, the memory 204 may store the first image and the identified pre-stored third image. Examples of implementations of memory 204 include, but are not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), hard disk drive (HDD), solid-state drive (SSD), CPU cache, and/or secure digital (SD) cards.

The I/O device 206 may include suitable logic, circuitry, interfaces, and/or code that may be configured to receive input and provide output based on the received input. The I/O device 206 may include various input and output devices that may be configured to communicate with the circuit 202. In one example, the electronic device 102 may receive (via the I/O device 206) a user input that includes a reverse image search query. The reverse image search query may include a first image. In another example, the electronic device 102 may receive (via the I/O device 206) a user input that includes a first weight associated with the generated first feature vector and a second weight associated with the generated second feature vector. The electronic device 102 may control the I/O device 206 to output the identified pre-stored third image. Examples of the I/O device 206 may include, but are not limited to, a touch screen, a keyboard, a mouse, a joystick, a display device (e.g., the display device 210), a microphone, or a speaker.

The display device 210 may include suitable logic, circuitry, and interfaces that may be configured to display the output of the electronic device 102. The display device 210 may be utilized to display information related to the identified pre-stored third image. In some embodiments, the display device 210 may be an externally coupled display device associated with the electronic device 102. The display device 210 may be a touch screen that allows the user 116 to provide user input via the display device 210. The touch screen may be at least one of a resistive touch screen, a capacitive touch screen, a thermal touch screen, or any other touch screen that may be used to provide input to the display device 210. The display device 210 may be implemented through a number of known technologies, such as, but not limited to, at least one of a liquid crystal display (LCD) display, a light emitting diode (LED) display, a plasma display, or an organic LED (OLED) display technology, or other display device.

The network interface 208 may include suitable logic, circuitry, interfaces, and/or code that may be configured to facilitate communication between the electronic device 102, the server 104, and the database 106 over the communications network 114. The network interface 208 may be implemented to support wired or wireless communication of the electronic device 102 to and from the communications network 112 using a variety of known technologies. The network interface 208 may include, but is not limited to, an antenna, a radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a coder-decoder (CODEC) chipset, a subscriber identity module (SIM) card, or a local buffer circuit.

The network interface 208 may be configured to communicate with a network such as the Internet, an intranet, a wireless network, a cellular telephone network, a wireless local area network (LAN) or a metropolitan area network (MAN) via wired communication, wireless communication or a combination thereof. The wireless communication may be configured to use one or more of a number of communication standards, protocols, and technologies, such as Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Long Term Evolution (LTE), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (WiFi) (such as IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, or IEEE 802.11n), Voice over Internet Protocol (VoIP), Light Fidelity (Li-Fi), Worldwide Interoperability for Microwave Access (Wi-MAX), protocols for email, instant messaging, and short message service (SMS).

The operation of circuitry 202 is further described, for example, in FIGS. 3 and 4. It should be noted that electronic device 102 shown in FIG. 2 may also include various other components or systems. Descriptions of other components or systems of electronic device 102 are omitted from this disclosure for the sake of brevity.

FIG. 3 illustrates an exemplary operation for reverse image search based on a deep neural network (DNN) model and an image feature detection model, according to an embodiment of the present disclosure. FIG. 3 is described with reference to elements of FIGS. 1 and 2. FIG. 3 illustrates a block diagram 300 illustrating exemplary operations 302-314 for reverse image search based on a DNN model 108 and an image feature detection model 110. The exemplary operations may be performed by any computer system, such as, for example, the electronic device 102 of FIG. 1 or the circuit 202 of FIG. 2.

At 302, a first image may be received. In an embodiment, the circuit 202 may be configured to receive a first image. For example, a first image 302A may be received. The first image 302A may be received from a data source, such as a persistent storage device (such as memory 204) on the electronic device 102, an image capture device, a cloud server, or a combination thereof. The first image 302A may include an object of interest for which the user 116 may require similar or identical image results using a reverse image search. Alternatively, the first image 302A may correspond to an image from a series of images of a first video. The circuit 202 may be configured to extract the first image 302A from the first video. The first video may correspond to a video including an object of interest for which the user 116 may require similar or identical video results using a reverse image search. The first image 302A may represent an image with a fixed foreground or background. For example, as shown, the first image 302A may represent a scene from a movie (e.g., an image of Spider-Man as an object of interest, as shown in FIG. 3).

After receiving the first image 302A, the circuit 202 can input the received first image 302A to the DNN model 108 and the image feature detection model 110 for image feature extraction. The circuit 202 can use the DNN model 108 to extract a first set of image features associated with the first image 302A, for example, as described at 304. Additionally, the circuit 202 can use the image feature detection model 110 to extract a second set of image features associated with the first image 302A, for example, as described at 306. Operations 304 and 306 can be performed in any order without departing from the scope of this disclosure.

At 304, a first image feature may be extracted. In one embodiment, the circuit 202 may be configured to extract a first set of image features associated with the received first image 302A by a deep neural network (DNN) model (such as the DNN model 108). The extracted first set of image features may correspond to unique features associated with one or more objects in the received first image 302A. The DNN model 108 may be pre-trained for an image feature extraction task based on a training data set 112 of a pre-stored set of training images to which pre-defined tags have been assigned. The pre-defined tags assigned to a particular training image may include labels corresponding to image features that may have been pre-determined for the particular training image. The circuit 202 may provide the first image 302A as an input to the DNN model 108 and receive the first set of image features (i.e., a set of image features associated with the first image 302A) as an output from the DNN model 108 based on an image feature detection task performed by the DNN model 108 on the first image 302A.

In some embodiments, the first set of extracted image features may include information required to classify each included object into a particular object class. Examples of the first set of extracted image features may include, but are not limited to, shape, texture, color, and other high-level image features. For example, as shown in FIG. 3, the extracted first image features 304A associated with the first image 302A may include color shown as a shade of gray on the object of interest (such as Spiderman's face). For example, if the face of a person, such as Spiderman or other person/character, is the object of interest in the first image 302A, the first set of image features 304A may include eye shape, ear shape, nose shape, and shape/texture of other high-level facial details of the person/character. The detailed implementation of the extraction of the first set of image features by the DNN model 108 is believed to be well known to those skilled in the art, and therefore a detailed description of such extraction of the first set of image features is omitted from this disclosure for the sake of brevity.

The circuit 202 may be configured to generate a first feature vector associated with the received first image 302A based on the extracted first image feature set. Such a first feature vector may also be referred to as a unique first image feature set. The generated first feature vector may include a plurality of vector elements, each of which may correspond to an image feature from the extracted first image feature set. Each vector element of the first feature vector may store a value that may correspond to a particular first image feature from the first image feature set. For example, if the received first image 302A is a high definition image (e.g., an image of "1024x1024" pixels), the first feature vector may be a "1x2048" vector having 2048 vector elements. The i-th element of the first feature vector may represent the value of the i-th first image feature.

At 306, a second set of image features may be extracted. In some embodiments, the circuit 202 may be configured to extract a second set of image features associated with the received first image 302A by an image feature detection model (such as the image feature detection model 110). The extracted second set of image features may correspond to certain unique features associated with one or more objects contained in the received first image 302A. In some embodiments, the second set of image features may be image features that may have been misdetected or remain undetected by the DNN model 108 (e.g., at 304). In some embodiments, the extracted second set of image features may include information required to optimally classify each object (in the first image 302A) into a particular object class. Examples of the second set of image features may include, but are not limited to, edges, lines, contours, and other low-level image features. Examples of the image feature detection model 110 may include, but are not limited to, a scale-invariant feature transform (SIFT)-based model, a speed-up robust features (SURF)-based model, an oriented FAST and rotated BRIEF (ORB)-based model, or a fast library for approximate nearest neighbors (FLANN)-based model. Detailed implementations of these exemplary methods are believed to be well known to those skilled in the art, and therefore detailed descriptions of such methods are omitted from this disclosure for the sake of brevity. For example, as shown in FIG. 3, a second image feature set 306A associated with a first image 302A represents a second image feature set extracted based on a SIFT-based model, and a second image feature set 306B associated with a first image 302A represents a second image feature set extracted based on a SURF-based model. For example, if the face of a person, such as Spider-Man or any other person/character in the first image 302A, is the object of interest, the second image feature set 306A (or the second image feature set 306B) may include eye edges and contours, ear edges and contours, nose edges and contours, and other low level facial details of the person/character.

The circuit 202 may further be configured to generate a second feature vector associated with the received first image 302A based on the extracted second image feature set. Such a second feature vector may also be referred to as a unique second image feature set. The generated second feature vector may include a plurality of vector elements, each of which may correspond to an image feature from the extracted second image feature set. Each vector element of the second feature vector may store a value that may correspond to a particular second image feature from the second image feature set. For example, if the received first image 302A is a high definition image (e.g., an image of "1024x1024" pixels), the second feature vector may be a "1x2048" vector having 2048 vector elements. The i-th element of the second feature vector may represent the value of the i-th second image feature.

At 308, the feature vectors can be combined. In some embodiments, the circuit 202 can be configured to generate a third feature vector associated with the received first image 302A based on combining the generated first feature vector and the generated second feature vector. In some embodiments, the circuit 202 can be configured to automatically combine the generated first feature vector and the generated second feature vector. For example, if the received first image 302A is a high definition image (e.g., an image of "1024 x 1024" pixels), the first feature vector can be a "1 x 2048" vector having 2048 vector elements, and the second feature vector can also be a "1 x 2048" vector having 2048 vector elements. In such a case, the third feature vector can be a "1 x 4096" vector having 4096 vector elements.

In one embodiment, the circuit 202 can be configured to determine a first weight associated with the first generated feature vector and a second weight associated with the second generated feature vector by a machine learning model 316 (i.e., a machine learning model different from the DNN model 108 and the image feature detection model 110). The machine learning model 316 can be a regression model that can be trained based on a set of similarly sized feature vectors, where each vector can be tagged with a user-defined weight value. The machine learning model 316 can be trained based on a feature vector weight assignment task, where the machine learning model 316 can receive two similarly sized feature vector sets as inputs and output weights for each of the two similarly sized feature vector sets. The machine learning model 316 can be defined by hyperparameters, such as the number of weights, the cost function, the input size, and the number of layers. The hyperparameters of the machine learning model 316 can be adjusted to move toward a global minimum of the cost function of the machine learning model 316, and the weights can be updated accordingly. The machine learning model 316 can be trained to output weight values for an input set after several epochs of training based on feature information in a training data set. This output can indicate a weight value for each input in the input set (e.g., the first feature vector and the second feature vector).

The machine learning model 316 may include electronic data that may be implemented, for example, as a software component of an application executable on the electronic device 102. The machine learning model 316 may rely on libraries, external scripts, or other logic/instructions for execution by a computing device including a processor, such as the circuit 202. The machine learning model 316 may include code and routines configured to enable a computing device including a processor, such as the circuit 202, to perform one or more operations for determining a first weight associated with a first feature vector and determining a second weight associated with a second feature vector. Additionally or alternatively, the machine learning model 316 may be implemented using hardware, including a processor, a microprocessor (e.g., performing or controlling one or more operations), a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC). Alternatively, in some embodiments, the machine learning model 316 may be implemented using a combination of hardware and software.

Each of the first weight associated with the generated first feature vector and the second weight associated with the generated second feature vector may indicate a likelihood of the reliability of the respective feature vector for identifying an object of interest in an image and thus identifying similar images that may contain the object of interest. The first weight and the second weight may specify a confidence value for the reliability (with a probability value between 0 and 1). Thus, a more reliable feature vector may have a higher weight value. For example, if the received first image 302A is a high-resolution image and the extracted first image feature set is more precise than the extracted second image feature set, the generated first feature vector may have a higher reliability than the generated second feature vector. In such an example, the first weight associated with the generated first feature vector may have a higher weight value (e.g., a value of 0.6) compared to the second weight associated with the generated second feature vector (e.g., a value of 0.4). In contrast, if the received first image 302A is a low-resolution image and the extracted second image feature set is more precise than the extracted first image feature set, the generated second feature vector may have a higher reliability than the generated first feature vector. In such an example, the first weight associated with the generated first feature vector may have a lower weight value (e.g., a 0.4 value) compared to the second weight associated with the generated second feature vector (e.g., a 0.6 value). Furthermore, if the received first image 302A is a medium resolution image (e.g., a standard resolution image), the first weight associated with the generated first feature vector may have an equal weight value (e.g., 0.5) compared to the second weight associated with the generated second feature vector (e.g., a 0.5 value). The circuit 202 may be further configured to combine the generated first feature vector and the generated second feature vector based on the determined first weight and the determined second weight, and further generate a third feature vector based on the combination.

In one embodiment, the circuit 202 can be configured to receive a user input including a first weight associated with the generated first feature vector and a second weight associated with the generated second feature vector. In one example, the received user input can indicate the first weight associated with the generated first feature vector as "0.4", such that the circuit 202 can be configured to determine the second weight associated with the generated second feature vector as "0.6". In another example, the received user input can indicate the first weight associated with the generated first feature vector as "0.5" and the second weight associated with the generated second feature vector as "0.5". The circuit 202 can be further configured to combine the generated first feature vector and the generated second feature vector based on the received user input, and further generate a third feature vector based on the combination.

In an embodiment, the circuit 202 can be configured to classify the received first image 302A by the DNN model 108 to a first image tag from a set of image tags associated with the DNN model 108. The first image tag can specify an image tag to which the received first image 302A can belong. For example, the received first image 302A can have an image tag such as a Spider-Man character. The circuit 202 can be further configured to determine a first image count associated with the first image tag in a training dataset (such as the training dataset 112) associated with the DNN model 108. For example, the circuit 202 can compare the first image tag to image tags of pre-stored training images from a set of pre-stored training images in the training dataset 112. If the image tag of the pre-stored training image in the training dataset 112 matches the first image tag, the circuit 202 increments the first image count associated with the first image tag by one. Similarly, the circuit 202 may determine a first image count based on a comparison of the image tag of each of the pre-stored training images in the training dataset 112 with the first image tag. The circuit 202 may be further configured to determine a first weight associated with the generated first feature vector and a second weight associated with the generated second feature vector based on the determined first image count associated with the first image tag. For example, if the first image count in the training dataset 112 is higher than a certain threshold (e.g., a threshold count or percentage of all images in the training dataset 112), the first weight associated with the generated first feature vector may have a high weight value compared to the second weight associated with the generated second feature vector. In contrast, if the first image count in the training dataset 112 is lower than a threshold or nominal value (e.g., a value that is easily negligible, such as a few hundred images in a training dataset 112 of one million images), the first weight associated with the generated first feature vector may have a low weight value compared to the second weight associated with the generated second feature vector. The circuit 202 may be configured to combine the generated first feature vector and the generated second feature vector based on the determined first weight and the determined second weight to generate a third feature vector, and further generate a third feature vector based on the combination.

In some embodiments, the circuit 202 may be configured to determine an image quality score associated with the received first image 302A based on at least one of the extracted first set of image features or the extracted second set of image features. The image quality score may indicate a qualitative value associated with the fidelity of the received first image 302A. A higher image quality score may indicate a higher fidelity of the received first image 302A. The image quality score may correspond to, but is not limited to, sharpness, noise, dynamic range, tone reproduction, contrast, saturation, distortion, vignetting, exposure accuracy, chromatic aberration, lens flare, color moire, or artifacts associated with the received first image 302A. Sharpness may correspond to details associated with image features associated with the received first image 302A. For example, if the pixel count or focus of the received first image 302A is high, the sharpness of the received first image 302A may be high. Noise can correspond to disturbances in the received first image 302A, such as undesired variations at the pixel level in the received first image 302A. Dynamic range can correspond to the amount of tonal difference between the lightest and darkest shades of light captured in the received first image 302A. Tonal reproduction can correspond to the correlation between the amount of light captured in the received first image 302A and the amount of light to which the received first image 302A is exposed. Contrast can correspond to the amount of color variation in the received first image 302A. Saturation can correspond to the intensity of colors in the received first image 302A. Distortion can correspond to undesired pixel variations in the received first image 302A. Vignetting can correspond to darkening, reduced sharpness, or reduced saturation from the corners of the received first image 302A compared to the center of the received first image 302A. Exposure accuracy can correspond to capturing the received first image 302A at an optimal brightness. Chromatic aberration may correspond to color distortions in the received first image 302A. Lens flare may correspond to the response of the image capture device to bright light. Color moiré may correspond to repetitive color stripes appearing in the received first image 302A. Artifacts associated with the received first image 302A may correspond to any virtual objects that may be present in the received first image 302A.

The circuit 202 may further be configured to determine a first weight associated with the generated first feature vector and a second weight associated with the generated second feature vector based on the determined image quality score. For example, if the determined image quality score is high, the first weight associated with the generated first feature vector may be assigned a higher weight value than the second weight associated with the generated second feature vector. In contrast, if the determined image quality score is low or nominal, the first weight associated with the generated first feature vector may be assigned a lower weight value than the second weight associated with the generated second feature vector. In an embodiment, if the image quality score is above a threshold, the determined first weight may be higher than the determined second weight. The threshold may include image quality scores such as, for example, "0.4", "0.6", and "0.8". In an embodiment, the circuit 202 may be configured to receive a user input to set the image quality score threshold. In another embodiment, the circuit 202 may be configured to automatically set the image quality score threshold. The circuitry 202 may be configured to combine the generated first feature vector and the generated second feature vector based on the determined first weight and the determined second weight. The circuitry 202 may then be further configured to generate a third feature vector based on the combination of the generated first feature vector and the generated second feature vector.

At 310, the dimensionality can be reduced. In an embodiment, the circuit 202 can be configured to reduce the dimensionality of the generated third feature vector. In some embodiments, the circuit 202 can resize (or compress) the generated third feature vector to match the size of the input layer of the feature extractor and pass the resized generated third feature vector to the input layer of the feature extractor. This can reduce undesirable or repetitive information from the generated third feature vector. The generation of the third feature vector can be further based on the application of a Principal Component Analysis (PCA) transform to the combination of the generated first feature vector and the generated second feature vector. For example, if the generated third vector is a "1x4096" vector having 4096 vector elements, after application of the PCA transform, the generated third vector can be reduced to a "1x256" vector having 256 vector elements. The detailed implementation of the PCA transform is believed to be well known to those skilled in the art, and therefore a detailed description of such a transform is omitted from this disclosure for the sake of brevity.

At 312, a similarity metric may be determined. In an embodiment, the circuit 202 may be configured to determine a similarity metric between the generated third feature vector associated with the received first image 302A and a fourth feature vector of each image of the pre-stored second set of images. The pre-stored second set of images may be stored in the database 106. In an embodiment, the circuit 202 may be configured to generate a fourth feature vector of each image of the pre-stored second set of images. For example, the circuit 202 may apply the DNN model 108, the image feature detection model 110, or a combination of both to each image of the pre-stored second set of images to generate a fourth feature vector of each pre-stored second image. In another example, the fourth feature vector of each respective pre-stored second image may be pre-determined and pre-stored in the database 106 with the respective pre-stored second image. The similarity metric may correspond to a similarity measure for determining images from the pre-stored second set of images that are similar to the received first image 302A. In such a case, the generated third feature vector associated with the received first image 302A can be compared to the fourth feature vector of each image of the pre-stored second set of images based on the determined similarity metric to identify similar images. Examples of similarity metrics can include, but are not limited to, cosine distance similarity or Euclidean distance similarity. In cosine distance similarity, a cosine distance between the generated third feature vector associated with the received first image 302A and the fourth feature vector of each image of the pre-stored second set of images can be determined. For example, if the fourth feature vector of a particular pre-stored second image has a small cosine distance to the generated third vector, the particular pre-stored second image can be identified as one of the images similar to the received first image 302A.

At 314, similar images can be identified. In an embodiment, the circuit 202 can be configured to identify the pre-stored third image as a similar image from the pre-stored second set of images based on the determined similarity metric. For example, the circuit 202 can compare the generated third feature vector associated with the received first image 302A to the fourth feature vector of each image of the pre-stored second set of images based on the similarity metric. If it is determined that the fourth feature vector of the particular pre-stored second image matches the generated third feature vector associated with the received first image 302A based on the similarity metric, the circuit 202 can identify the particular pre-stored second image from the pre-stored second set of images as a pre-stored third image (i.e., a similar image).

The circuit 202 may further be configured to control a display device (such as the display device 210) to display information related to the identified pre-stored third image. The information related to the identified pre-stored third image may include information such as, but not limited to, the pre-stored third image itself, metadata related to the pre-stored third image, a feature map between the third feature vector and the fourth feature vector, a file size of the pre-stored third image, a storage location associated with the pre-stored third image, or a file download path associated with the pre-stored third image. In an embodiment, the identified pre-stored third image may correspond to a pre-stored second video. The pre-stored second video may be associated with the first video. For example, the pre-stored third image may be one of the image frames from a set of image frames in the pre-stored second video. In an embodiment, the first image 302A may be extracted from the first video. The pre-stored third image can be related or similar to the received first image 302A, and the pre-stored second video can be related or similar to the first video.

3 shows information 314A related to an identified pre-stored third image that can be identified based on the DNN model 108 and the image feature detection model 110 (e.g., a SIFT-based model). The information 314A can include a feature map between a generated third feature vector related to the received first image 302A and a fourth feature vector related to the identified pre-stored third image. ...

As described above, the disclosed electronic device 102 can automatically generate a third feature vector associated with the received first image 302A based on a combination of the generated first feature vector and the generated second feature vector. As a result, the third feature vector can include a first image feature set that can be determined by the DNN model 108 and a second image feature set that can be determined by the image feature detection model 110. The first image feature set can include high-level image features (e.g., facial features such as eyes, nose, ears, hair, etc.) associated with the received first image 302A (e.g., an image of a person's face), and the second image feature set can include low-level image features (e.g., points, edges, lines, contours, or basic objects and shapes of a face). Including both the high-level image features and the low-level image features in the third feature vector can complement each other in identifying similar images. In some scenarios, the first image feature set may not detect and extract all features that may be present in the received first image 302A. For example, some features may be misdetected or remain undetected by the DNN model 108. For example, if the received first image 302A is an image that is not well represented in the training data set 112 of the DNN model 108, the first image feature set may not be sufficient to identify images similar to the received first image from the pre-stored second image set. However, since the second image feature set (i.e., the second image feature set determined by the image feature detection model 110) may include low-level image features related to the received first image 302A, the second image feature set may be included in the third feature vector to further improve the accuracy of identifying images similar to the received first image 302A from the pre-stored second image set. For example, if the image quality is poor (e.g., in the case of a blurry image with low resolution), the first image feature set (i.e., high-level image features) may not be sufficient to identify similar images. In such a case, the second image features (i.e., low-level image features) may be more useful and accurate for identifying similar images.

FIG. 4 is a flowchart illustrating an exemplary method for reverse image search based on a deep neural network (DNN) model and an image feature detection model, according to an embodiment of the present disclosure. FIG. 4 is described with reference to elements of FIGS. 1, 2, and 3. FIG. 4 illustrates a flowchart 400. The method illustrated in flowchart 400 may be performed by any computer system, such as electronic device 102 or circuit 202. The method may start at 402 and proceed to 404.

At 404, a first image (e.g., first image 302A) can be received. In one or more embodiments, circuitry 202 can be configured to receive first image 302A. Receiving first image 302A is further described, for example, in FIG. 3 (at 302).

At 406, a first set of image features (e.g., first image feature set 304A) associated with the received first image (e.g., first image 302A) can be extracted by a deep neural network (DNN) model (e.g., DNN model 108). In one or more embodiments, the circuit 202 can be configured to extract the first set of image features 304A associated with the received first image 302A by the DNN model 108. Extraction of the first set of image features 304A is further described, for example, in FIG. 3 (at 304).

At 408, a first feature vector associated with the received first image 302A may be generated based on the extracted set of first image features 304A. The circuit 202 may be configured to generate a first feature vector associated with the received first image 302A based on the extracted set of first image features 304A. Generation of the first feature vector is further described, for example, in FIG. 3 (at 304).

At 410, a second set of image features (e.g., second set of image features 306A) associated with the received first image 302A can be extracted by an image feature detection model (e.g., image feature detection model 110). In one or more embodiments, the circuit 202 can be configured to extract the second set of image features 306A associated with the received first image 302A by the image feature detection model 110. The image feature detection model 110 includes at least one of a scale invariant feature transform (SIFT)-based model, a speeded up robust features (SURF)-based model, an oriented FAST and rotated BRIEF (ORB)-based model, or a fast library for approximate nearest neighbors (FLANN)-based model. Extraction of the second set of image features 306A is further described, for example, in FIG. 3 (at 306).

At 412, a second feature vector associated with the received first image 302A may be generated based on the extracted set of second image features 306A. In one or more embodiments, the circuit 202 may be configured to generate a second feature vector associated with the received first image 302A based on the extracted set of second image features 306A. Generation of the second feature vector is further described, for example, in FIG. 3 (at 306).

At 414, a third feature vector associated with the received first image 302A may be generated based on a combination of the generated first feature vector and the generated second feature vector. In one or more embodiments, the circuit 202 may be configured to generate a third feature vector associated with the received first image 302A based on a combination of the generated first feature vector and the generated second feature vector. The generation of the third feature vector may be further based on application of a principal component analysis (PCA) transform to the combination of the generated first feature vector and the generated second feature vector. The generation of the third feature vector is further described, for example, in FIG. 3 (at 308).

At 416, a similarity metric may be determined between the generated third feature vector associated with the received first image 302A and the fourth feature vector of each image of the pre-stored second set of images. In one or more embodiments, the circuit 202 may be configured to determine a similarity metric between the generated third feature vector associated with the received first image 302A and the fourth feature vector of each image of the pre-stored second set of images. In one example, the similarity metric may include, but is not limited to, at least one of cosine distance similarity or Euclidean distance similarity. Determining the similarity metric is further described, for example, in FIG. 3 (at 312).

At 418, a pre-stored third image (e.g., a pre-stored third image) may be identified from the set of pre-stored second images based on the determined similarity metric. In one or more embodiments, the circuit 202 may be configured to identify the pre-stored third image from the set of pre-stored second images based on the determined similarity metric. Identifying the pre-stored third image is further described, for example, in FIG. 3 (at 314).

At 420, a display device (such as display device 210) may be controlled to display information associated with the identified pre-stored third image. Circuitry 202 may be configured to control display device 210 to display information associated with the identified pre-stored third image. Control of display device 210 is further described, for example, in FIG. 3 (at 314). Control proceeds to an end.

Although flowchart 400 is illustrated as discrete operations such as 404, 406, 408, 410, 412, 416, 418, and 420, the disclosure is not so limited. Thus, in some embodiments, such discrete operations may be further divided into additional operations, combined into fewer operations, or eliminated, depending on the particular implementation, without departing from the essence of the disclosed embodiments.

Various embodiments of the present disclosure may provide a non-transitory computer-readable medium and/or storage medium having instructions executable by a machine and/or a computer (such as electronic device 102). The instructions may cause the machine and/or computer to perform operations including receiving a first image (such as first image 302A). The operations may further include extracting a first set of image features (such as first image feature set 304A) associated with the received first image 302A by a deep neural network (DNN) model (such as DNN model 108). The operations may further include generating a first feature vector associated with the received first image 302A based on the extracted first image features 304A. The operations may further include extracting a second set of image features (such as second image feature set 306A) associated with the received first image 302A by an image feature detection model (such as image feature detection model 110). The operations may further include generating a second feature vector associated with the received first image 302A based on the extracted second image features 304A. The operations may further include generating a third feature vector associated with the received first image 302A based on a combination of the generated first feature vector and the generated second feature vector. The operations may further include determining a similarity metric between the generated third feature vector associated with the received first image and a fourth feature vector of each image of the pre-stored second set of images. The operations may further include identifying a pre-stored third image from the pre-stored second set of images based on the determined similarity metric. The operations may further include controlling a display device (such as display device 210) to display information associated with the identified pre-stored third image.

An exemplary aspect of the present disclosure may provide an electronic device (such as the electronic device 102 of FIG. 1 ) including a circuit (such as the circuit 202). The circuit 202 may be configured to receive a first image 302A. The circuit 202 may be configured to extract a first set of image features (such as the first image feature set 304A) associated with the received first image 302A by a deep neural network (DNN) model 108. The circuit 202 may be configured to generate a first feature vector associated with the received first image 302A based on the extracted first image features 304A. The circuit 202 may be configured to extract a second set of image features (such as the second image feature set 306A) associated with the received first image 302A by an image feature detection model 110. The circuit 202 may be configured to generate a second feature vector associated with the received first image 302A based on the extracted second image features 306A. The circuit 202 may be configured to generate a third feature vector associated with the received first image 302A based on a combination of the generated first feature vector and the generated second feature vector. The circuit 202 may be configured to determine a similarity metric between the generated third feature vector associated with the received first image 302A and a fourth feature vector of each image of the pre-stored second set of images. The circuit 202 may be configured to identify a pre-stored third image from the pre-stored second set of images based on the determined similarity metric. The circuit 202 may be configured to control the display device 210 to display information associated with the identified pre-stored third image.

According to an embodiment, the image feature detection model 110 may include at least one of, but is not limited to, a scale invariant feature transform (SIFT) based model, a speeded up robust features (SURF) based model, an oriented FAST and rotated BRIEF (ORB) based model, or a fast library for approximate nearest neighbors (FLANN) based model.

According to an embodiment, the generation of the third feature vector may be further based on applying a Principal Component Analysis (PCA) transform to a combination of the generated first feature vector and the generated second feature vector. According to an embodiment, the similarity metric may include at least one of, but is not limited to, cosine distance similarity or Euclidean distance similarity.

According to an embodiment, the circuit 202 may be further configured to determine a first weight associated with the generated first feature vector and a second weight associated with the generated second feature vector by a machine learning model (such as the machine learning model 316) different from the DNN model 108 and the image feature detection model 110. The circuit 202 may be further configured to combine the generated first feature vector and the generated second feature vector based on the determined first weight and the determined second weight. The circuit 202 may be configured to generate a third feature vector based on the combination of the generated first feature vector and the generated second feature vector.

According to an embodiment, the circuit 202 may be further configured to receive a user input including a first weight associated with the generated first feature vector and a second weight associated with the generated second feature vector. The circuit 202 may be further configured to combine the generated first feature vector and the generated second feature vector based on the received user input. The circuit 202 may be configured to generate a third feature vector based on the combination of the generated first feature vector and the generated second feature vector.

According to an embodiment, the circuit 202 may be further configured to classify, by the DNN model 108, the received first image 302A to a first image tag from a set of image tags associated with the DNN model 108. The circuit 202 may be configured to determine a first image count associated with the first image tag in a training data set (such as the training data set 112) associated with the DNN model 108. The circuit 202 may be further configured to determine a first weight associated with the generated first feature vector and a second weight associated with the generated second feature vector based on the determined first count of images associated with the first image tag. The circuit 202 may be further configured to combine the generated first feature vector and the generated second feature vector based on the determined first weight and the determined second weight. The circuit 202 may be configured to generate a third feature vector based on the combination of the generated first feature vector and the generated second feature vector.

According to an embodiment, the circuit 202 may be configured to determine an image quality score associated with the received first image 302A based on at least one of the extracted first image feature set 304A or the extracted second image feature set 306A. The circuit 202 may be further configured to determine a first weight associated with the generated first feature vector and a second weight associated with the generated second feature vector based on the determined image quality score. The circuit 202 may be further configured to combine the generated first feature vector and the generated second feature vector based on the determined first weight and the determined second weight. The circuit 202 may be configured to generate a third feature vector based on the combination of the generated first feature vector and the generated second feature vector. According to an embodiment, the image quality score may correspond to at least one of, but not limited to, sharpness, noise, dynamic range, tone reproduction, contrast, saturation, distortion, vignetting, exposure accuracy, chromatic aberration, lens flare, color moiré, or artifacts associated with the received first image 302A.

According to an embodiment, the circuitry 202 may be further configured to extract the first image 302A from the first video and an identified pre-stored third image that may correspond to a pre-stored second video. The pre-stored second video may be related to the first video.

The present disclosure can be implemented in hardware or in a combination of hardware and software. The present disclosure can be implemented in a centralized manner in at least one computer system, or in a distributed manner where different elements can be distributed across several interconnected computer systems. Any computer system or other device adapted to perform the methods described herein can be suitable. The combination of hardware and software can be a general-purpose computer system including a computer program that, when loaded and executed, can control a computer system to perform the methods described herein. The present disclosure can be implemented in hardware, including as part of an integrated circuit that also performs other functions.

The present disclosure may also be embodied in a computer program product, including all features enabling the implementation of the methods described herein, which is capable of executing these methods when loaded into a computer system. A computer program in this context means any expression in any language, code or notation of a set of instructions intended to cause a system capable of processing information to execute a particular function, either directly or after a) conversion into another language, code or notation, b) reproduction in a different content form, or both.

Although the present disclosure has been described with reference to several embodiments, those skilled in the art will recognize that various modifications can be made and equivalents substituted without departing from the scope of the present disclosure. Additionally, many modifications can be made to adapt a particular situation or material to the teachings of the present disclosure without departing from its scope. Therefore, it is not intended that the present disclosure be limited to the particular embodiments disclosed, but rather, it is intended to include all embodiments falling within the scope of the appended claims.

102 Electronic device 104 Server 106 Database 108 Deep Neural Network (DNN) model 110 Image feature detection model 112 Training dataset 114 Communication network 116 User

Claims (20)

1. An electronic device comprising:
Receiving a first image;
extracting a first set of image features associated with the received first image using a deep neural network (DNN) model;
generating a first feature vector associated with the received first image based on the extracted set of first image features;
extracting a second set of image features associated with the received first image using an image feature detection model;
generating a second feature vector associated with the received first image based on the extracted set of second image features;
generating a third feature vector associated with the received first image based on a combination of the generated first feature vector and the generated second feature vector;
determining a similarity metric between the generated third feature vector associated with the received first image and a fourth feature vector of each image of a second set of pre-stored images;
identifying a third pre-stored image from the second set of pre-stored images based on the determined similarity metric;
controlling a display device to display information related to the identified pre-stored third image.
The circuit is configured to
1. An electronic device comprising: The image feature detection model includes at least one of a Scale Invariant Feature Transform (SIFT) based model, a Speed Up Robust Features (SURF) based model, an Oriented FAST and Rotated BRIEF (ORB) based model, or a Fast Library for Approximate Nearest Neighbors (FLANN) based model.
2. The electronic device of claim 1. the generation of the third feature vector is further based on applying a principal component analysis (PCA) transform to the combination of the generated first feature vector and the generated second feature vector.
2. The electronic device of claim 1. the similarity metric includes at least one of a cosine distance similarity or a Euclidean distance similarity;
2. The electronic device of claim 1. The circuit comprises:
determining a first weight associated with the generated first feature vector and a second weight associated with the generated second feature vector by a machine learning model distinct from the DNN model and the image feature detection model;
combining the generated first feature vector and the generated second feature vector based on the determined first weight and the determined second weight;
generating the third feature vector based on the combination of the generated first feature vector and the generated second feature vector;
The electronic device of claim 1 , further configured to: The circuit comprises:
receiving user input including a first weight associated with the generated first feature vector and a second weight associated with the generated second feature vector;
combining the generated first feature vector and the generated second feature vector based on the received user input;
generating the third feature vector based on the combination of the generated first feature vector and the generated second feature vector;
The electronic device of claim 1 , further configured to: The circuit comprises:
classifying, with the DNN model, the received first image into a first image tag from a set of image tags associated with the DNN model;
determining a first image count associated with the first image tag in a training data set associated with the DNN model;
determining a first weight associated with the generated first feature vector and a second weight associated with the generated second feature vector based on the determined first image count associated with the first image tag;
combining the generated first feature vector and the generated second feature vector based on the determined first weight and the determined second weight;
generating the third feature vector based on the combination of the generated first feature vector and the generated second feature vector;
The electronic device of claim 1 , further configured to: The circuit comprises:
determining an image quality score associated with the received first image based on at least one of the extracted first set of image features or the extracted second set of image features;
determining a first weight associated with the generated first feature vector and a second weight associated with the generated second feature vector based on the determined image quality score;
combining the generated first feature vector and the generated second feature vector based on the determined first weight and the determined second weight;
generating the third feature vector based on the combination of the generated first feature vector and the generated second feature vector;
The electronic device of claim 1 , further configured to: the image quality score corresponds to at least one of sharpness, noise, dynamic range, tone reproduction, contrast, color saturation, distortion, vignetting, exposure accuracy, chromatic aberration, lens flare, color moiré, or artifacts associated with the received first image;
9. The electronic device of claim 8. the circuitry is further configured to extract the first image from a first video and the identified pre-stored third image corresponding to a pre-stored second video related to the first video.
2. The electronic device of claim 1. In an electronic device,
Receiving a first image;
extracting a first set of image features associated with the received first image using a deep neural network (DNN) model;
generating a first feature vector associated with the received first image based on the extracted set of first image features;
extracting a second set of image features associated with the received first image using an image feature detection model;
generating a second feature vector associated with the received first image based on the extracted set of second image features;
generating a third feature vector associated with the received first image based on a combination of the generated first feature vector and the generated second feature vector;
determining a similarity metric between the generated third feature vector associated with the received first image and a fourth feature vector of each image of a second set of pre-stored images;
identifying a third pre-stored image from the second set of pre-stored images based on the determined similarity metric;
controlling a display device to display information related to the identified pre-stored third image;
The method according to claim 1, further comprising: The image feature detection model includes at least one of a Scale Invariant Feature Transform (SIFT) based model, a Speed Up Robust Features (SURF) based model, an Oriented FAST and Rotated BRIEF (ORB) based model, or a Fast Library for Approximate Nearest Neighbors (FLANN) based model.
The method of claim 11. the generation of the third feature vector is further based on applying a principal component analysis (PCA) transform to the combination of the generated first feature vector and the generated second feature vector.
The method of claim 11. the similarity metric includes at least one of a cosine distance similarity or a Euclidean distance similarity;
The method of claim 11. determining a first weight associated with the generated first feature vector and a second weight associated with the generated second feature vector by a machine learning model distinct from the DNN model and the image feature detection model;
combining the generated first feature vector and the generated second feature vector based on the determined first weight and the determined second weight;
generating the third feature vector based on the combination of the generated first feature vector and the generated second feature vector;
The method of claim 11 further comprising: receiving user input including a first weight associated with the generated first feature vector and a second weight associated with the generated second feature vector;
combining the generated first feature vector and the generated second feature vector based on the received user input;
generating the third feature vector based on the combination of the generated first feature vector and the generated second feature vector;
The method of claim 11 further comprising: classifying, by the DNN model, the received first image into a first image tag from a set of image tags associated with the DNN model;
determining a first image count associated with the first image tag in a training data set associated with the DNN model;
determining a first weight associated with the generated first feature vector and a second weight associated with the generated second feature vector based on the determined first image count associated with the first image tag;
combining the generated first feature vector and the generated second feature vector based on the determined first weight and the determined second weight;
generating the third feature vector based on the combination of the generated first feature vector and the generated second feature vector;
The method of claim 11 further comprising: determining an image quality score associated with the received first image based on at least one of the extracted first set of image features or the extracted second set of image features;
determining a first weight associated with the generated first feature vector and a second weight associated with the generated second feature vector based on the determined image quality score;
combining the generated first feature vector and the generated second feature vector based on the determined first weight and the determined second weight;
generating the third feature vector based on the combination of the generated first feature vector and the generated second feature vector;
The method of claim 11 further comprising: the image quality score corresponds to at least one of sharpness, noise, dynamic range, tone reproduction, contrast, color saturation, distortion, vignetting, exposure accuracy, chromatic aberration, lens flare, color moiré, or artifacts associated with the received first image;
20. The method of claim 18. A non-transitory computer-readable medium having computer-executable instructions stored thereon, the computer-executable instructions, when executed by an electronic device,
Receiving a first image;
extracting a first set of image features associated with the received first image using a deep neural network (DNN) model;
generating a first feature vector associated with the received first image based on the extracted set of first image features;
extracting a second set of image features associated with the received first image using an image feature detection model;
generating a second feature vector associated with the received first image based on the extracted set of second image features;
generating a third feature vector associated with the received first image based on a combination of the generated first feature vector and the generated second feature vector;
determining a similarity metric between the generated third feature vector associated with the received first image and a fourth feature vector of each image of a second set of pre-stored images;
identifying a third pre-stored image from the second set of pre-stored images based on the determined similarity metric;
controlling a display device to display information related to the identified pre-stored third image;
23. A non-transitory computer-readable medium for causing the electronic device to perform operations including:

Images