CN111950464B - Image retrieval method, server and scanning pen - Google Patents

Image retrieval method, server and scanning pen Download PDF

Info

Publication number
CN111950464B
CN111950464B CN202010813084.6A CN202010813084A CN111950464B CN 111950464 B CN111950464 B CN 111950464B CN 202010813084 A CN202010813084 A CN 202010813084A CN 111950464 B CN111950464 B CN 111950464B
Authority
CN
China
Prior art keywords
image
retrieval
current
candidate set
candidate
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202010813084.6A
Other languages
Chinese (zh)
Other versions
CN111950464A (en
Inventor
刘庆升
吴玉胜
储德宝
刘丛刚
王晓斐
王田
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Anhui Toycloud Technology Co Ltd
Original Assignee
Anhui Toycloud Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Anhui Toycloud Technology Co Ltd filed Critical Anhui Toycloud Technology Co Ltd
Priority to CN202010813084.6A priority Critical patent/CN111950464B/en
Publication of CN111950464A publication Critical patent/CN111950464A/en
Application granted granted Critical
Publication of CN111950464B publication Critical patent/CN111950464B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/10—Terrestrial scenes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
    • H04N1/04—Scanning arrangements, i.e. arrangements for the displacement of active reading or reproducing elements relative to the original or reproducing medium, or vice versa
    • H04N1/10—Scanning arrangements, i.e. arrangements for the displacement of active reading or reproducing elements relative to the original or reproducing medium, or vice versa using flat picture-bearing surfaces
    • H04N1/107—Scanning arrangements, i.e. arrangements for the displacement of active reading or reproducing elements relative to the original or reproducing medium, or vice versa using flat picture-bearing surfaces with manual scanning
    • H04N1/1078—Scanning arrangements, i.e. arrangements for the displacement of active reading or reproducing elements relative to the original or reproducing medium, or vice versa using flat picture-bearing surfaces with manual scanning by moving the scanned medium

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Library & Information Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Signal Processing (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Processing Or Creating Images (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The embodiment of the invention provides an image retrieval method, a server and a scanning pen, wherein the method comprises the following steps: receiving a current local image of an image to be retrieved, which is acquired by a scanning pen; matching the current local image with each candidate image in the previous candidate set to obtain a current candidate set; and based on the current candidate set, generating a retrieval result and returning the scanning pen to trigger the scanning pen to acquire a next local image of the image to be retrieved when the retrieval result indicates that the retrieval is not finished, and updating the next local image into a current local image. The method, the server and the scanning pen provided by the embodiment of the invention realize the image retrieval based on the scanning pen, and the scanning pen with the image retrieval function can meet the intelligent reading requirements of users at any time and any place due to the portable characteristic of the scanning pen.

Description

Image retrieval method, server and scanning pen
Technical Field
The invention relates to the technical field of electronic equipment, in particular to an image retrieval method, a server and a scanning pen.
Background
The intelligent reading equipment can shoot the page images of the entity book by using the camera carried by the user when the user reads the entity book, and the page images of the entity book are retrieved and identified, so that multimedia contents such as voice, video and the like corresponding to the current page of the entity book are provided for the user, and the reading experience of the user is enriched.
The current intelligent reading equipment has larger volume and poor portability, and cannot meet the reading requirements of users at any time and any place.
Disclosure of Invention
The embodiment of the invention provides an image retrieval method, a server and a scanning pen, which are used for solving the defect of poor portability of intelligent reading equipment in the prior art.
In a first aspect, an embodiment of the present invention provides an image retrieval method, including:
receiving a current local image of an image to be retrieved, which is acquired by a scanning pen;
matching the current local image with each candidate image in the previous candidate set to obtain a current candidate set;
and generating a retrieval result and returning the scanning pen based on the current candidate set so as to trigger the scanning pen to acquire a next local image of the image to be retrieved when the retrieval result indicates that the retrieval is not finished, and updating the next local image into a current local image.
Optionally, the matching the current local image with each candidate image in the previous candidate set to obtain the current candidate set specifically includes:
if receiving a retrieval session identifier sent by the scanning pen, matching the current local image with each candidate image in a previous candidate set corresponding to the retrieval session identifier to obtain a current candidate set corresponding to the retrieval session identifier;
and otherwise, generating a retrieval session identifier, returning to the scanning pen, and matching the current local image with each candidate image in a preset retrieval set to obtain a current candidate set corresponding to the retrieval session identifier.
Optionally, the matching the current local image with each candidate image in the previous candidate set to obtain the current candidate set specifically includes:
determining local image characteristics of the current local image;
and if the local image features are traversed in the candidate image features of any candidate image in the last candidate set, adding any candidate image into the current candidate set.
Optionally, the generating a retrieval result and returning the scan pen based on the current candidate set specifically includes:
if the number of the candidate images contained in the current candidate set is 1, generating retrieval completion information serving as the retrieval result and returning the retrieval result to the scanning pen;
if the number of the candidate images contained in the current candidate set is 0, generating search failure information serving as the search result and returning the search result to the scanning pen;
otherwise, based on the number of candidate images contained in the current candidate set, generating a retrieval progress as the retrieval result and returning the retrieval result to the scanning pen.
Optionally, if the number of candidate images included in the current candidate set is 1, generating retrieval completion information and returning the retrieval completion information to the scan pen specifically includes:
if the number of the candidate images contained in the current candidate set is 1 and the candidate images contained in the current candidate set are book cover images, generating retrieval completion information carrying the book information of the book cover images and returning the retrieval completion information to the scanning pen so as to trigger the scanning pen to start content page scanning;
and if the number of the candidate images contained in the current candidate set is 1 and the candidate images contained in the current candidate set are content page images, generating retrieval completion information of the media information carrying the content page images and returning the retrieval completion information to the scanning pen so as to trigger the scanning pen to display the media information.
In a second aspect, an embodiment of the present invention provides an image retrieval method, including:
acquiring a current local image of an image to be retrieved;
sending the current local image to a server side so that the server side can match the current local image with each candidate image in a previous candidate set to obtain a current candidate set, and generating a retrieval result based on the current candidate set;
and receiving the retrieval result returned by the server, and acquiring the next local image of the image to be retrieved when the retrieval result indicates that the retrieval is not finished.
Optionally, the sending the current local image to a server specifically includes:
if the current local image is the first local image of the image to be retrieved, sending the current local image to a server, and receiving a retrieval session identifier returned by the server;
otherwise, the current local image and the retrieval session identifier are sent to the server, so that the server matches the current local image with each candidate image in a last candidate set corresponding to the retrieval session identifier to obtain a current candidate set corresponding to the retrieval session identifier.
Optionally, the receiving the search result returned by the server further includes:
if the retrieval result is retrieval failure information, prompting a user to re-collect the current local image of the image to be retrieved;
if the retrieval result is retrieval completion information which carries the bibliographic information of the bibliographic cover image obtained by retrieving the image to be retrieved, prompting a user to collect a local image of the bibliographic information corresponding to the content page of the entity book;
and if the retrieval result is retrieval completion information which carries the media information of the content page image obtained by retrieving the image to be retrieved, displaying the media information.
In a third aspect, an embodiment of the present invention provides a server, including:
the image receiving unit is used for receiving a current local image of an image to be retrieved, which is acquired by the scanning pen;
the local matching unit is used for matching the current local image with each candidate image in the previous candidate set to obtain a current candidate set;
and the result feedback unit is used for generating a retrieval result and returning the scanning pen based on the current candidate set so as to trigger the scanning pen to acquire a next local image of the image to be retrieved when the retrieval result indicates that the retrieval is not finished, and update the next local image into the current local image.
In a fourth aspect, an embodiment of the present invention provides a scan pen, including:
the image acquisition unit is used for acquiring a current local image of the image to be retrieved;
the image sending unit is used for sending the current local image to a server so that the server can match the current local image with each candidate image in a previous candidate set to obtain a current candidate set, and a retrieval result is generated based on the current candidate set;
and the result receiving unit is used for receiving the retrieval result returned by the server and acquiring the next local image of the image to be retrieved when the retrieval result indicates that the retrieval is not finished.
In a fifth aspect, an embodiment of the present invention provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the processor implements the steps of the image retrieval method provided in the first aspect or the second aspect.
In a sixth aspect, embodiments of the present invention provide a non-transitory computer readable storage medium, on which a computer program is stored, which when executed by a processor, implements the steps of the image retrieval method as provided in the first or second aspect.
According to the image retrieval method, the server and the scanning pen provided by the embodiment of the invention, the scanning pen carries out local scanning on an image to be retrieved and carries out matching interaction with the server on the local image, so that the image retrieval based on the scanning pen is realized, the multimedia contents such as voice, video and the like corresponding to the current page can be triggered by the result obtained by the retrieval, and due to the portable characteristic of the scanning pen, the scanning pen with the image retrieval function can meet the intelligent reading requirement of a user at any time and any place; in addition, each matching is performed on the basis of the candidate set obtained by the last matching, so that the scanning speed can be increased in a plurality of matching processes, and the image retrieval precision can be improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.
Fig. 1 is a schematic flowchart of an image retrieval method according to an embodiment of the present invention;
fig. 2 is a schematic flowchart of a local image matching method according to an embodiment of the present invention;
fig. 3 is a schematic flowchart of a search result feedback method according to an embodiment of the present invention;
FIG. 4 is a flowchart illustrating an image retrieval method according to another embodiment of the present invention;
FIG. 5 is a flowchart illustrating a cover image retrieval method according to an embodiment of the present invention;
FIG. 6 is a flowchart illustrating a content page image retrieval method according to an embodiment of the present invention;
fig. 7 is a schematic structural diagram of a server according to an embodiment of the present invention;
FIG. 8 is a schematic structural diagram of a wand according to an embodiment of the present invention;
fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be obtained by a person skilled in the art without inventive step based on the embodiments of the present invention, are within the scope of protection of the present invention.
The scanning pen is also named as a miniature scanner or a hand-scraping scanning pen, and is a handheld electronic device. When the pen point of the scanning pen is used for scanning characters on books and periodicals and newspapers, the operations such as Recognition, storage, editing and the like can be carried out on the characters on the books and periodicals and the newspapers through a built-in Optical Character Recognition (OCR) module. Aiming at the problem that the existing intelligent reading equipment is poor in portability, the embodiment of the invention provides an image retrieval method based on a scanning pen by relying on the scanning function of the scanning pen.
Fig. 1 is a schematic flow chart of an image retrieval method according to an embodiment of the present invention, and as shown in fig. 1, an execution subject of the method is a server, and the method includes:
and step 110, receiving a current local image of the image to be retrieved, which is acquired by the scanning pen.
Specifically, the scanning pen carries a scanning head with an image acquisition function, when image retrieval is required to be performed through the scanning pen, a user can hold the scanning pen to scan the image to be retrieved along a straight line, and in the process, the scanning head of the scanning pen acquires partial images in the image to be retrieved to serve as current partial images.
After acquiring the current local image, the scanning pen sends the current local image to the server and requests the server to perform image retrieval based on the current local image. The server-side correspondingly receives the current local image.
And step 120, matching the current local image with each candidate image in the previous candidate set to obtain a current candidate set.
Specifically, after receiving the current local image, the server may match the current local image with each candidate image in the previous candidate set, retain the candidate images that are successfully matched, and delete the candidate images that are not successfully matched, thereby obtaining the current candidate set. The previous candidate set is a set of candidate images matched with the previous local image, and the current candidate set is a set of candidate images matched with the current local image on the basis of the previous local image.
Before performing step 120, the server may preset a preset search set, where all stored candidate images may be used for image search. After receiving the first local image, the server matches the first local image with all candidate images in a preset retrieval set, and takes a set formed by all candidate images matched with the first local image as a first candidate set. On the basis, each time a new local image is received, the local image can be matched with the candidate image in the previous candidate set, so that the scale of the candidate set is gradually reduced until one candidate image remains in the candidate set, and the remaining candidate image can be directly used as the retrieval result of the image to be retrieved.
And step 130, based on the current candidate set, generating a retrieval result and returning to the scanning pen to trigger the scanning pen to acquire a next local image of the image to be retrieved when the retrieval result indicates that the retrieval is not completed, and updating the next local image into the current local image.
Specifically, after the current candidate set is obtained, the current retrieval progress for the image to be retrieved can be analyzed according to the number of candidate images in the current candidate set, so as to generate a retrieval result, and the retrieval result is fed back to the scanning pen. The search result herein may specifically indicate completion of the search, incomplete search, or failure of the search, and the search result may specifically be a search progress between 0 and 100% when the indication indicates incomplete search.
After the scanning pen receives the current retrieval result, if the retrieval result indicates that the retrieval is not finished, the user can be prompted to continuously hold the scanning pen to scan in other areas of the image to be retrieved along straight lines, and the scanning pen can acquire partial images of other areas in the image to be retrieved in the process to serve as a next partial image. Since the image retrieval is not completed, the next local image can be updated to the current local image and sent to the server, so as to start the next local image retrieval.
And step 110 to step 130 are executed in a circulating manner, a scanning pen can be used for carrying out local image scanning for a plurality of times on the same image to be retrieved, and the local images obtained by each scanning are further matched on the basis of the candidate set obtained by the last matching, so that the image retrieval range is gradually reduced until the image retrieval for the image to be retrieved is completed.
According to the method provided by the embodiment of the invention, the image to be retrieved is locally scanned by the scanning pen, and the image to be retrieved is matched and interacted with the local image of the server side, so that the image retrieval based on the scanning pen is realized, and the multimedia contents such as voice, video and the like corresponding to the current page can be triggered by the result obtained by the retrieval. Due to the portable characteristic of the scanning pen, the scanning pen with the image retrieval function can meet the intelligent reading requirement of a user at any time and any place; in addition, each matching is performed on the basis of the candidate set obtained by the last matching, so that the scanning speed can be increased in a plurality of matching processes, and the image retrieval precision can be improved.
Based on the above embodiment, step 120 specifically includes:
if receiving a retrieval session identifier sent by a scanning pen, matching the current local image with each candidate image in a previous candidate set corresponding to the retrieval session identifier to obtain a current candidate set corresponding to the retrieval session identifier;
and otherwise, generating a retrieval session identifier, returning to the scanning pen, and matching the current local image with each candidate image in the preset retrieval set to obtain a current candidate set corresponding to the retrieval session identifier.
Specifically, in the image retrieval process, multiple interactions exist between the scanning pen and the server, and the process of retrieving one image can be taken as one retrieval session. Specifically, in the field of physical book identification, the identification of the content page may be performed in two stages, namely, a cover image retrieval and a content page image retrieval, and at this time, the cover image retrieval and the content page image retrieval may be respectively used as a retrieval session, and the cover image retrieval and the content page image retrieval may also be integrally used as a retrieval session.
In the interaction process, the scanning pen sends the current local image to the server and simultaneously sends the retrieval session identification of the current session to the server. Here, the retrieval session identification is the session ID of the current retrieval session.
If the server receives the current local image and the retrieval session identifier at the same time, the server can search the last candidate set generated after the last local image matching of the current retrieval session according to the retrieval session identifier. After the last candidate set is found, the current local image and each candidate image in the last candidate set can be respectively matched, the candidate images which are successfully matched are reserved, and the candidate images which are failed to be matched are deleted, so that the current candidate set corresponding to the retrieval session identifier is obtained.
If the server side does not receive the retrieval session identification while receiving the current local image, the server side indicates that the current local image is the first local image of the image retrieval, and the retrieval session identification is not generated in the image retrieval. The server can directly generate the retrieval session identifier corresponding to the image retrieval, and return the newly generated retrieval session identifier to the scanning pen. Because the current local image is the first local image and the corresponding last candidate set does not exist, the current local image can be directly matched with each candidate image in the preset retrieval set, the candidate images which are successfully matched are reserved, and the candidate images which are failed in matching are deleted, so that the current candidate set corresponding to the newly generated retrieval session identifier is obtained. After the scanning pen obtains the newly generated retrieval session identifier, when the next partial image of the image retrieval is aimed at, the scanning pen can carry the retrieval session identifier so that the server can search the current candidate set obtained by the matching.
The method provided by the embodiment of the invention considers the characteristic that the image retrieval by the scanning pen needs multiple interactions, and ensures the smooth realization of the multiple interactions by setting the retrieval session identifier.
The conventional image matching method usually calculates the similarity between the vector representation of the image to be matched and the vector representation of the candidate image, and judges whether the image to be matched and the candidate image are matched according to the similarity. However, when a scanning pen is used for image scanning, only a part of the image to be matched can be obtained by single scanning, and the local image cannot actually reflect the overall characteristics of the image to be matched, so that the matching method cannot be applied to an image retrieval method based on the scanning pen.
To solve this problem, based on any of the above embodiments, fig. 2 is a schematic flow chart of the local image matching method provided by the embodiment of the present invention, as shown in fig. 2, step 120 specifically includes:
and step 121, determining local image characteristics of the current local image.
Step 122, if the local image feature is traversed in the candidate image feature of any candidate image in the previous candidate set, adding the candidate image into the current candidate set.
Specifically, after receiving the current partial image, it is first necessary to determine the image feature of the current partial image as the partial image feature. Here, the local image feature may be a feature of the current local image in each dimension such as color, texture, shape, and the like, and the local image feature may be obtained by inputting the current local image into a pre-trained image feature extraction model, or may be obtained by directly encoding a value of each pixel point in the current local image to obtain a color feature as the local image feature. Each candidate image corresponds to a candidate image feature, and the candidate image features of the candidate images may be obtained based on the same feature extraction manner as the local image features of the current local image, which is not specifically limited in the embodiment of the present invention.
When the current local image is matched with any candidate image in the previous candidate set, the candidate image characteristics of the candidate image can be traversed, if the local image characteristics are obtained through traversal, the current local image is possibly a part of the candidate image, the current local image is matched with the candidate image, and the candidate image is added into the current candidate set; if the local image features cannot be obtained through traversal, the current local image is not part of the candidate image, the current local image is not matched with the candidate image, and the candidate image is not added into the current candidate set.
According to the method provided by the embodiment of the invention, the local image features are traversed in the candidate image features, so that the matching of the local image and the whole image is realized.
Based on any of the above embodiments, fig. 3 is a schematic flowchart of a search result feedback method provided by an embodiment of the present invention, and as shown in fig. 3, in step 130, the generating a search result and returning to the wand based on the current candidate set specifically includes:
step 131, if the number of candidate images included in the current candidate set is 1, generating retrieval completion information as a retrieval result and returning the retrieval result to the scanning pen;
step 132, if the number of candidate images contained in the current candidate set is 0, generating search failure information as a search result and returning the search result to the scanning pen;
and step 133, otherwise, based on the number of candidate images included in the current candidate set, generating a retrieval progress as a retrieval result and returning the retrieval result to the scanning pen.
Specifically, after the current candidate set is obtained, the search result for the image to be searched may be determined according to the number of candidate images included in the current candidate set:
if the number of the candidate images in the current candidate set is 1, that is, only one candidate image is currently matched with each local image of the image to be retrieved, the candidate image, that is, the image obtained by retrieving the image to be retrieved can be determined, at this time, the retrieval is determined to be completed, and the retrieval completion information is returned to the scanning pen as the retrieval result. The retrieval completion information may indicate that the retrieval is completed, and the retrieval completion information may further include related information of the retrieved image, for example, related information of an entity book indicated by the image, or content information included in the image, or multimedia voice associated with the image, which is not specifically limited in this embodiment of the present invention.
If the number of the candidate images in the current candidate set is 0, that is, no candidate image matching with each local image of the image to be retrieved exists, that is, the candidate image matching with the image to be retrieved cannot be retrieved, and the retrieval fails. At this time, the search failure information can be generated and returned to the scanning pen as the search result, and after receiving the search failure information, the scanning pen can prompt the user that the current search is failed and wait for the user to determine whether to perform image search again.
If the number of candidate images included in the current candidate set is greater than 1, that is, a plurality of candidate images matched with each local image of the image to be retrieved currently exist, and image retrieval is still in progress, at this time, the retrieval progress can be evaluated based on the number of candidate images included in the current candidate set, and the retrieval progress is returned to the scanning pen as a retrieval result. For example, if the number of candidate images in the preset search set is 1000 and the number of candidate images in the current candidate set is 50, 1-50/1000=95% may be calculated as the current search progress. After receiving the retrieval progress, the scanning pen can display or broadcast the retrieval progress to the user and prompt the user to execute next local scanning.
Based on any of the above embodiments, step 131 specifically includes:
if the number of the candidate images contained in the current candidate set is 1 and the candidate images contained in the current candidate set are book cover images, generating retrieval completion information carrying the book information of the book cover images and returning the retrieval completion information to the scanning pen so as to trigger the scanning pen to start content page scanning;
and if the number of the candidate images in the current candidate set is 1 and the candidate images in the current candidate set are the content page images, generating retrieval completion information of the media information carrying the content page images and returning the retrieval completion information to the scanning pen so as to trigger the scanning pen to display the media information.
Specifically, when the image retrieval method based on the scanning pen is applied to intelligent reading of an entity book, image retrieval needs to be specifically divided into two stages, namely front cover image retrieval and content page image retrieval. When the number of candidate images included in the current candidate set is 1, it may be determined which stage the current search is in according to the type of candidate image obtained by the search at this time, so as to further determine the next execution action:
if the candidate image obtained by retrieval is the book cover image, the retrieved information returned to the scanning pen can carry the book information of the book cover image, the scanning pen can display the book information carried in the retrieved information to the user after receiving the retrieved information, and the user judges whether the image retrieval of the cover page is correct or not and further executes the image retrieval of the content page of the book. The bibliographic information may specifically be a title, a book version, an author, and the like.
If the candidate image obtained by searching is the content page image, the search completion information returned to the scanning pen can carry media information preset for the content page image, and the scanning pen can display the media information carried in the search completion information to a user after receiving the search completion information, so that intelligent reading based on the scanning pen is realized. The media information herein may specifically include voice, sound effect, etc. corresponding to the content page image, and may also include text content, video, etc. corresponding to the content page image.
Based on any of the above embodiments, fig. 4 is a schematic flowchart of an image retrieval method according to another embodiment of the present invention, as shown in fig. 4, an execution subject of the method is a scan pen, and the method includes:
step 410, collecting the current local image of the image to be retrieved.
Specifically, the scanning pen carries a scanning head with an image acquisition function, when image retrieval is required to be performed through the scanning pen, a user can hold the scanning pen to scan the image to be retrieved along a straight line, and the scanning head of the scanning pen acquires partial images in the image to be retrieved as the current partial images in the process.
Step 420, sending the current local image to the server, so that the server matches the current local image with each candidate image in the previous candidate set to obtain a current candidate set, and generating a retrieval result based on the current candidate set.
Specifically, after acquiring the current local image, the scanning pen sends the current local image to the server, and requests the server to perform image retrieval based on the current local image.
After receiving the current local image, the server may match the current local image with each candidate image in the previous candidate set, retain the candidate images successfully matched, and delete the candidate images failed in matching, thereby obtaining the current candidate set. The previous candidate set is a set of candidate images matching the previous local image, and the current candidate set is a set of candidate images matching the current local image on the basis of matching the previous local image.
Before this, the server may preset a preset retrieval set, where all stored candidate images may be used for image retrieval. After receiving the first local image, the server matches the first local image with all candidate images in a preset retrieval set, and takes a set formed by all candidate images matched with the first local image as a first candidate set. On the basis, each time a new local image is received, the new local image can be matched with the candidate image in the previous candidate set, so that the scale of the candidate set is gradually reduced until one candidate image remains in the candidate set, and the remaining candidate image can be directly used as the retrieval result of the image to be retrieved.
After the server side obtains the current candidate set, the server side can analyze the current retrieval progress aiming at the image to be retrieved according to the number of the candidate images in the current candidate set, further generate a retrieval result, and feed the retrieval result back to the scanning pen. The search result herein may specifically indicate that the search is completed, the search is not completed, or the search is failed, wherein the search result indicating that the search is not completed may be represented as a search progress of 0 to 100%.
And step 430, receiving a retrieval result returned by the server, and acquiring a next local image of the image to be retrieved when the retrieval result indicates that the retrieval is not completed.
Specifically, after the scanning pen receives the current retrieval result, if the retrieval result is that the retrieval is not completed, the user may be prompted to continue holding the scanning pen to scan along a straight line in other areas of the image to be retrieved, and the scanning pen may acquire partial images of other areas in the image to be retrieved in the process as a next partial image. Since the image retrieval is not completed, the next local image can be updated to the current local image and sent to the server, so as to start the next local image retrieval.
And step 410 to step 430 are executed in a circulating manner, the scanning pen can be used for scanning the local images of the same image to be retrieved for a plurality of times, and the server side is used for further matching the local images obtained by each scanning on the basis of the candidate set obtained by the last matching, so that the image retrieval range is gradually reduced until the retrieval result of the image to be retrieved is obtained.
According to the method provided by the embodiment of the invention, the image to be retrieved is locally scanned by the scanning pen, and the image to be retrieved is matched and interacted with the server side aiming at the local image, so that the image retrieval based on the scanning pen is realized, and the multimedia contents such as voice, video and the like corresponding to the current page can be triggered by the retrieval result. Due to the portable characteristic of the scanning pen, the scanning pen with the image retrieval function can meet the intelligent reading requirement of a user at any time and any place; in addition, each matching is performed on the basis of the candidate set obtained by the last matching, so that the scanning speed can be increased in a plurality of matching processes, and the image retrieval precision is improved.
Based on any of the foregoing embodiments, in step 420, the sending the current local image to the server specifically includes:
if the current local image is the first local image of the image to be retrieved, sending the current local image to the server, and receiving a retrieval session identifier returned by the server;
otherwise, the current local image and the retrieval session identifier are sent to the server side, so that the server side matches the current local image with each candidate image in the last candidate set corresponding to the retrieval session identifier, and the current candidate set corresponding to the retrieval session identifier is obtained.
Specifically, in the image retrieval process, multiple interactions exist between the scanning pen and the server, and the process of retrieving one image can be used as one retrieval session. In the interaction process, the scanning pen sends the current local image to the server and simultaneously sends the retrieval conversation identification of the current conversation to the server. Here, the retrieval session identification is the session ID of the current retrieval session.
Further, before the scan pen sends the current local image to the server, it needs to determine whether the current local image is the first local image of the image to be retrieved:
if the local image is the first local image, the current local image is the local image which is firstly sent to the server side in the retrieval session of the image retrieval, before the current local image is sent, the server side does not generate a retrieval session identifier aiming at the current retrieval session, and the scanning pen does not have the retrieval session identifier corresponding to the current retrieval session, so that the scanning pen only sends the current local image to the server side, the server side does not receive the retrieval session identifier while receiving the current local image, namely the current local image is the first local image of the image retrieval, the server side can directly generate the retrieval session identifier corresponding to the image retrieval, and the newly generated retrieval session identifier is returned to the scanning pen. Because the current local image is the first local image and the corresponding last candidate set does not exist, the current local image can be directly matched with each candidate image in the preset retrieval set, the candidate images which are successfully matched are reserved, and the candidate images which are failed in matching are deleted, so that the current candidate set corresponding to the newly generated retrieval session identifier is obtained.
If the image is not the first local image, the scanning pen is already provided with the retrieval session identification generated by the server end aiming at the retrieval session of the image retrieval before. And the scanning pen can send the retrieval session identifier I to the server side while sending the current local image. If the server receives the current local image and the retrieval session identifier at the same time, the server can search the last candidate set generated after the last local image matching of the current retrieval session according to the retrieval session identifier. After the last candidate set is found, the current local image can be matched with each candidate image in the last candidate set respectively, the candidate images which are successfully matched are reserved, and the candidate images which are unsuccessfully matched are deleted, so that the current candidate set corresponding to the retrieval session identifier is obtained.
The method provided by the embodiment of the invention considers the characteristic that the image retrieval by the scanning pen needs multiple interactions, and ensures the smooth realization of the multiple interactions by setting the retrieval session identifier.
Based on any of the above embodiments, step 430 receives the retrieval result returned by the server, and then further includes:
if the retrieval result is retrieval failure information, prompting a user to re-acquire a current local image of the image to be retrieved;
if the retrieval result is retrieval completion information which carries the bibliographic information of the bibliographic cover image obtained by retrieving the image to be retrieved, prompting a user to collect a local image of the bibliographic information corresponding to the content page of the entity book;
and if the retrieval result is retrieval completion information which carries the media information of the content page image obtained by retrieving the image to be retrieved, displaying the media information.
Specifically, after receiving a retrieval result returned by the server, the scan pen needs to execute corresponding operations according to different retrieval results:
if the retrieval result is retrieval failure information, the server side is indicated to fail to retrieve the candidate image matched with the image to be retrieved, at the moment, the scanning pen can prompt the user that the current retrieval is failed, wait for the user to determine whether to perform image retrieval again, and if the user determines to perform image retrieval again, re-acquire the current local image of the image to be retrieved;
if the retrieval result is retrieval completion information, for the field of intelligent reading of the entity book, the retrieval completion information may carry bibliographic information for completing retrieval of the cover page or media information for completing retrieval of the content page, and two specific situations can be analyzed:
if the retrieved information carries the bibliographic information of the bibliographic cover image obtained by retrieving the image to be retrieved, the candidate image obtained by retrieving the image to be retrieved by the server is the bibliographic cover image, the scanning pen can display the bibliographic information to the user, and the user judges whether the image retrieval of the cover page is correct or not and whether the image retrieval of the content page of the bibliographic is further executed or not. The bibliographic information may specifically be a title, a book version, an author, and the like.
If the media information of the content page image obtained by searching the image to be searched is carried in the searching completion information, the candidate image obtained by searching the server side is the content page image, and the scanning pen can display the media information to the user, so that the intelligent reading based on the scanning pen is realized. The media information herein may specifically include voice, sound effect, etc. corresponding to the content page image, and may also include text content, video, etc. corresponding to the content page image.
Based on any of the above embodiments, step 410 specifically includes:
acquiring an image sequence obtained by current scanning;
determining the overlapping part of any two adjacent images in the image sequence based on the image characteristics of the two adjacent images;
and splicing the two adjacent images based on the overlapped parts of the two adjacent images.
Specifically, when image retrieval is required to be performed through the scanning pen, a user can hold the scanning pen to scan an image to be retrieved along a straight line, and in the process, the scanning head of the scanning pen continuously performs image acquisition, so that an image sequence formed by a plurality of images in sequence is obtained.
In consideration of the situation of high-speed scanning, an overlapped part exists between two continuously acquired frames of images, so that the two frames of images can be respectively subjected to feature extraction aiming at any two adjacent frames of images in an image sequence, and the positions of feature pixels in the two frames of images are identified as the image features of the two frames of images. On the basis, the positions of the characteristic pixels in the two frames of images can be compared, the overlapped part of the two frames of images can be found according to the relative positions of the characteristic pixels, and then the two frames of images are spliced. And repeating the steps to splice all the images in the image sequence into transverse images according to the scanning direction, and taking the spliced images as the current local images obtained by the current scanning.
Based on any of the embodiments, in the intelligent reading scene of the physical book, the front cover identification stage and the content page identification stage can be divided based on the image retrieval of the scanning pen. When the entity book is the picture book, the image retrieval aiming at the picture book can be realized through the picture book recognition function of the scanning pen, and in order to distinguish the picture book recognition function from the picture book recognition function, the user can set the current mode of the scanning pen as the picture book recognition mode through a touch screen, a key or a voice interaction interface of the scanning pen in advance.
Fig. 5 is a schematic flow chart of the cover image retrieval method provided in the embodiment of the present invention, and as shown in fig. 5, in the book drawing identification mode, the scan pen first enters the cover identification stage to prompt the user to scan the cover to be retrieved, in this process, the user holds the scan pen to scan the cover to be retrieved along a straight line, and the scan pen acquires a partial image of the cover to be retrieved as a current partial image.
After the current local image is obtained, the scanning pen judges whether a retrieval session identifier for the retrieval session exists or not, and if yes, the current local image and the retrieval session identifier I are sent to the server side; and if the partial image does not exist, namely the current partial image is the first partial image of the book cover to be retrieved, directly sending the current partial image to the server.
After receiving the current local image, the server judges whether a retrieval session identifier is received together:
if the current local image is received and the retrieval session identifier is also received, the last candidate set generated after the last local image matching of the current retrieval session can be searched according to the retrieval session identifier. After the last candidate set is found, respectively matching the current local image with each candidate image in the last candidate set, reserving the candidate images which are successfully matched, and deleting the candidate images which are failed to be matched, so as to obtain the current candidate set corresponding to the retrieval session identifier;
if the retrieval session identification is not received while the current local image is received, the server side can directly generate the retrieval session identification corresponding to the image retrieval, and the newly generated retrieval session identification is returned to the scanning pen. Because the current local image is the first local image and the corresponding last candidate set does not exist, the current local image can be directly matched with each candidate image in the preset retrieval set, the candidate images which are successfully matched are reserved, and the candidate images which are failed to be matched are deleted, so that the current candidate set corresponding to the newly generated retrieval session identifier is obtained. Note that, the preset search image herein stores a cover image of each of the sketches.
After the current candidate set is obtained, the server may determine, according to the number of candidate images included in the current candidate set, a retrieval result for the image to be retrieved:
and if the number of the candidate images in the current candidate set is 1, the server determines that the retrieval is finished, and returns the retrieval finished information serving as a retrieval result to the scanning pen. The retrieval completion information can carry the bibliographic information of the bibliographic cover image;
if the number of the candidate images contained in the current candidate set is 0, the server determines that the retrieval is failed, and the retrieval failure information is used as a retrieval result and returned to the scanning pen;
if the number of candidate images included in the current candidate set is greater than 1 and image retrieval is still in progress, the server may evaluate the retrieval progress based on the number of candidate images included in the current candidate set, and return the retrieval progress as a retrieval result to the scan pen.
After the scanning pen receives the retrieval result returned by the server, corresponding operations can be executed according to different retrieval results:
if the retrieval result is retrieval failure information, the scanning pen can prompt the user that the current retrieval is failed, wait for the user to determine whether to perform image retrieval again, and if the user determines to perform image retrieval again, re-acquire the current local image of the image to be retrieved;
if the retrieval result is retrieval completion information, the scanning pen can display the bibliography information carried in the retrieval completion information to a user, and the user judges whether the image retrieval of the cover page is correct or not and whether the image retrieval of the content page of the bibliography is further executed or not;
if the retrieval result is the retrieval progress, the scanning pen can show the retrieval progress to prompt the user to continuously hold the scanning pen to scan in other areas of the image to be retrieved along a straight line, the scanning pen can acquire partial images of the other areas in the image to be retrieved in the process and serve as a next partial image, the next partial image is updated to be the current partial image and is sent to the server, and therefore the next partial image retrieval aiming at the cover is started until the cover retrieval is completed.
Fig. 6 is a schematic flow chart of the content page image retrieval method according to the embodiment of the present invention, and as shown in fig. 6, in the book drawing identification mode, after the book drawing cover is determined, the scanning pen enters the content page identification stage to prompt the user to scan the content page to be retrieved. In the process, a user holds the scanning pen to scan the contents page of the picture book to be retrieved along a straight line, and the scanning pen acquires partial images of the contents page of the picture book as current partial images. It should be noted that, the determination of the cover of the textbook here may be implemented by scanning the cover of the textbook with the scan pen and performing cover retrieval in cooperation with the server, or may be set by the user directly through a touch screen, a key or a voice interaction interface of the scan pen, which is not specifically limited in this embodiment of the present invention.
After the current local image is obtained, the scanning pen can directly send the current local image and the first retrieval session identifier of the current retrieval session to the server side.
After receiving the current local image and the retrieval session identifier, the server can determine a previous candidate set through the retrieval session identifier, and if the previous candidate set contains the candidate content page images, the server matches the current local image with the candidate content page images contained in the previous candidate set; and if the last candidate set does not contain the candidate content page images, matching all candidate images corresponding to the book to which the content pages to be retrieved belong with the current local images as a preset retrieval set. And after matching is completed, obtaining a current candidate set.
After the current candidate set is obtained, the server may determine, according to the number of candidate images included in the current candidate set, a retrieval result for the image to be retrieved:
and if the number of the candidate images in the current candidate set is 1, the server determines that the retrieval is finished, and returns the retrieval finished information serving as a retrieval result to the scanning pen. The retrieval completion information can carry the media information of the content page image;
if the number of the candidate images contained in the current candidate set is 0, the server side determines that the retrieval is failed, and returns the retrieval failure information serving as a retrieval result to the scanning pen;
if the number of candidate images included in the current candidate set is greater than 1 and image retrieval is still in progress, the server may evaluate the retrieval progress based on the number of candidate images included in the current candidate set, and return the retrieval progress as a retrieval result to the scan pen.
After the scanning pen receives the retrieval result returned by the server, corresponding operations can be executed according to different retrieval results:
if the retrieval result is retrieval failure information, the scanning pen can prompt the user that the current retrieval is failed, wait for the user to determine whether to perform image retrieval again, and if the user determines to perform image retrieval again, re-collect the current local image of the image to be retrieved;
if the retrieval result is retrieval completion information, the scanning pen can display the media information carried in the retrieval completion information to the user;
if the retrieval result is the retrieval progress, the scanning pen can show the retrieval progress to prompt the user to continuously hold the scanning pen to scan in other areas of the image to be retrieved along straight lines, the scanning pen can acquire partial images of the other areas in the image to be retrieved in the process to serve as next partial images, the next partial images are updated to be current partial images and sent to the server, and therefore next partial image retrieval aiming at the content page is started until the content page retrieval is completed.
Based on any of the above embodiments, fig. 7 is a schematic structural diagram of a server according to an embodiment of the present invention, as shown in fig. 7, the server includes an image receiving unit 710, a local matching unit 720, and a result feedback unit 730;
the image receiving unit 710 is configured to receive a current local image of an image to be retrieved, which is acquired by a scanning pen;
the local matching unit 720 is configured to match the current local image with each candidate image in the previous candidate set to obtain a current candidate set;
the result feedback unit 730 is configured to generate a retrieval result based on the current candidate set and return the wand, so as to trigger the wand to acquire a next local image of the image to be retrieved when the retrieval result indicates that the retrieval is not completed, and update the next local image to a current local image.
The server provided by the embodiment of the invention matches the local image obtained by matching and scanning the image to be retrieved by the scanning pen, so that the image retrieval based on the scanning pen is realized, the result obtained by the retrieval can trigger multimedia contents such as voice, video and the like corresponding to the current page, and due to the portable characteristic of the scanning pen, the scanning pen with the image retrieval function can meet the intelligent reading requirement of a user at any time and any place; in addition, each matching is performed on the basis of the candidate set obtained by the last matching, so that the scanning speed can be increased in a plurality of matching processes, and the image retrieval precision can be improved.
Based on any of the above embodiments, the local matching unit 720 is specifically configured to:
if receiving a retrieval session identifier sent by the scanning pen, matching the current local image with each candidate image in a previous candidate set corresponding to the retrieval session identifier to obtain a current candidate set corresponding to the retrieval session identifier;
and otherwise, generating a retrieval session identifier, returning to the scanning pen, and matching the current local image with each candidate image in a preset retrieval set to obtain a current candidate set corresponding to the retrieval session identifier.
Based on any of the above embodiments, the local matching unit 720 is specifically configured to:
determining local image characteristics of the current local image;
and if the local image features are traversed in the candidate image features of any candidate image in the last candidate set, adding any candidate image into the current candidate set.
Based on any of the above embodiments, the result feedback unit 730 includes:
a retrieval completion information feedback subunit, configured to generate retrieval completion information as the retrieval result and return the retrieval result to the scanning pen if the number of candidate images included in the current candidate set is 1;
a retrieval failure information feedback subunit, configured to generate, if the number of candidate images included in the current candidate set is 0, retrieval failure information serving as the retrieval result and returned to the stylus;
and the retrieval progress feedback subunit is used for generating the retrieval progress as the retrieval result and returning the retrieval result to the scanning pen based on the number of the candidate images contained in the current candidate set if the current candidate set does not contain the candidate images.
Based on any of the embodiments above, the retrieval completion information feedback subunit is specifically configured to:
if the number of the candidate images contained in the current candidate set is 1 and the candidate images contained in the current candidate set are book cover images, generating retrieval completion information carrying the book information of the book cover images and returning the retrieval completion information to the scanning pen so as to trigger the scanning pen to start content page scanning;
and if the number of the candidate images in the current candidate set is 1 and the candidate images in the current candidate set are content page images, generating retrieval completion information carrying the media information of the content page images and returning the retrieval completion information to the scanning pen so as to trigger the scanning pen to display the media information.
Based on any of the embodiments, fig. 8 is a schematic structural diagram of a wand provided in an embodiment of the present invention, and as shown in fig. 8, the wand includes an image acquisition unit 810, an image sending unit 820, and a result receiving unit 830;
the image acquisition unit 810 is configured to acquire a current local image of an image to be retrieved;
the image sending unit 820 is configured to send the current local image to a server, so that the server matches the current local image with each candidate image in a previous candidate set to obtain a current candidate set, and generates a retrieval result based on the current candidate set;
the result receiving unit 830 is configured to receive the retrieval result returned by the server, and acquire a next partial image of the image to be retrieved when the retrieval result indicates that the retrieval is not completed.
The scanning pen provided by the embodiment of the invention carries out local scanning on the image to be retrieved and carries out matching interaction with the server aiming at the local image, thereby realizing the image retrieval based on the scanning pen, the result obtained by the retrieval can trigger multimedia contents such as voice, video and the like corresponding to the current page, and the scanning pen with the image retrieval function can meet the intelligent reading requirement of a user at any time and any place due to the portable characteristic of the scanning pen; in addition, each matching is performed on the basis of the candidate set obtained by the last matching, so that the scanning speed can be increased in a plurality of matching processes, and the image retrieval precision can be improved.
Based on any of the above embodiments, the image sending unit 820 is specifically configured to:
if the current local image is the first local image of the image to be retrieved, sending the current local image to a server and receiving a retrieval session identifier returned by the server;
otherwise, the current local image and the retrieval session identifier are sent to the server, so that the server matches the current local image with each candidate image in a last candidate set corresponding to the retrieval session identifier to obtain a current candidate set corresponding to the retrieval session identifier.
Based on any of the above embodiments, the result receiving unit 830 is further configured to:
if the retrieval result is retrieval failure information, prompting a user to re-acquire a current local image of the image to be retrieved;
if the retrieval result is retrieval completion information which carries the bibliographic information of the bibliographic cover image obtained by retrieving the image to be retrieved, prompting a user to acquire a local image of the content page of the entity book corresponding to the bibliographic information;
and if the retrieval result is retrieval completion information which carries the media information of the content page image obtained by retrieving the image to be retrieved, displaying the media information.
Fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present invention, and as shown in fig. 9, the electronic device may include: a processor (processor) 910, a communication Interface (Communications Interface) 920, a memory (memory) 930, and a communication bus 940, wherein the processor 910, the communication Interface 920, and the memory 930 communicate with each other via the communication bus 940. Processor 910 may invoke logical commands in memory 930 to perform the following method:
receiving a current local image of an image to be retrieved, which is acquired by a scanning pen;
matching the current local image with each candidate image in the previous candidate set to obtain a current candidate set;
and generating a retrieval result and returning the scanning pen based on the current candidate set so as to trigger the scanning pen to acquire a next local image of the image to be retrieved when the retrieval result indicates that the retrieval is not finished, and updating the next local image into a current local image.
In addition, processor 910 may also invoke logical commands in memory 930 to perform the following method:
acquiring a current local image of an image to be retrieved;
sending the current local image to a server side so that the server side can match the current local image with each candidate image in a previous candidate set to obtain a current candidate set, and generating a retrieval result based on the current candidate set;
and receiving the retrieval result returned by the server, and acquiring the next partial image of the image to be retrieved when the retrieval result indicates that the retrieval is not finished.
In addition, the logic commands in the memory 930 may be implemented in the form of software functional units and stored in a computer readable storage medium when the logic commands are sold or used as independent products. Based on such understanding, the technical solution of the present invention or a part thereof which substantially contributes to the prior art may be embodied in the form of a software product, which is stored in a storage medium and includes several commands for enabling a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
Embodiments of the present invention further provide a non-transitory computer-readable storage medium, on which a computer program is stored, where the computer program is implemented to perform the method provided in the foregoing embodiments when executed by a processor, and the method includes:
receiving a current local image of an image to be retrieved, which is acquired by a scanning pen;
matching the current local image with each candidate image in the previous candidate set to obtain a current candidate set;
and generating a retrieval result and returning the scanning pen based on the current candidate set so as to trigger the scanning pen to acquire a next local image of the image to be retrieved when the retrieval result indicates that the retrieval is not finished, and updating the next local image into a current local image.
Embodiments of the present invention further provide a non-transitory computer-readable storage medium, on which a computer program is stored, where the computer program is implemented to perform the method provided in the foregoing embodiments when executed by a processor, for example, the method includes:
acquiring a current local image of an image to be retrieved;
sending the current local image to a server side so that the server side can match the current local image with each candidate image in a previous candidate set to obtain a current candidate set, and generating a retrieval result based on the current candidate set;
and receiving the retrieval result returned by the server, and acquiring the next partial image of the image to be retrieved when the retrieval result indicates that the retrieval is not finished.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment may be implemented by software plus a necessary general hardware platform, and may also be implemented by hardware. Based on such understanding, the technical solutions in essence or contributing to the prior art may be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., and includes several commands for enabling a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method according to the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims (10)

1. An image retrieval method, comprising:
receiving a current local image of an image to be retrieved, which is acquired by a scanning pen;
matching the current local image with each candidate image in the previous candidate set to obtain a current candidate set;
based on the current candidate set, generating a retrieval result and returning the scanning pen to trigger the scanning pen to acquire a next local image of the image to be retrieved when the retrieval result indicates that the retrieval is not finished, and updating the next local image into a current local image;
generating a search result based on the current candidate set, comprising:
analyzing the current retrieval progress aiming at the image to be retrieved according to the number of the candidate images in the current candidate set, and further generating a retrieval result, wherein the retrieval result specifically indicates that the retrieval is finished, the retrieval is not finished or the retrieval is failed;
and the next local image is obtained by the scanning of the other areas in the image to be retrieved by holding the scanning pen by the user.
2. The image retrieval method of claim 1, wherein the matching the current local image with each candidate image in a previous candidate set to obtain a current candidate set specifically comprises:
if receiving a retrieval session identifier sent by the scanning pen, matching the current local image with each candidate image in a previous candidate set corresponding to the retrieval session identifier to obtain a current candidate set corresponding to the retrieval session identifier;
and otherwise, generating a retrieval session identifier, returning to the scanning pen, and matching the current local image with each candidate image in a preset retrieval set to obtain a current candidate set corresponding to the retrieval session identifier.
3. The image retrieval method of claim 1, wherein the matching the current local image with each candidate image in a previous candidate set to obtain a current candidate set specifically comprises:
determining local image characteristics of the current local image;
and if the local image features are traversed in the candidate image features of any candidate image in the last candidate set, adding any candidate image into the current candidate set.
4. The image retrieval method of claim 1, wherein the generating a retrieval result and returning the wand based on the current candidate set specifically comprises:
if the number of the candidate images in the current candidate set is 1, generating retrieval completion information serving as the retrieval result and returning the retrieval result to the scanning pen;
if the number of the candidate images contained in the current candidate set is 0, generating search failure information serving as the search result and returning the search result to the scanning pen;
otherwise, based on the number of candidate images contained in the current candidate set, generating a retrieval progress as the retrieval result and returning the retrieval result to the scanning pen.
5. The image retrieval method according to claim 4, wherein if the number of candidate images included in the current candidate set is 1, generating retrieval completion information to return to the stylus pen specifically includes:
if the number of the candidate images contained in the current candidate set is 1 and the candidate images contained in the current candidate set are book cover images, generating retrieval completion information carrying the book information of the book cover images and returning the retrieval completion information to the scanning pen so as to trigger the scanning pen to start content page scanning;
and if the number of the candidate images contained in the current candidate set is 1 and the candidate images contained in the current candidate set are content page images, generating retrieval completion information of the media information carrying the content page images and returning the retrieval completion information to the scanning pen so as to trigger the scanning pen to display the media information.
6. An image retrieval method, comprising:
acquiring a current local image of an image to be retrieved;
sending the current local image to a server side so that the server side can match the current local image with each candidate image in a previous candidate set to obtain a current candidate set, and generating a retrieval result based on the current candidate set;
receiving the retrieval result returned by the server, and acquiring the next partial image of the image to be retrieved when the retrieval result indicates that the retrieval is not finished;
generating a search result based on the current candidate set, comprising:
analyzing the current retrieval progress aiming at the image to be retrieved according to the number of the candidate images in the current candidate set, and further generating a retrieval result, wherein the retrieval result specifically indicates that the retrieval is finished, the retrieval is not finished or the retrieval is failed;
and the next local image is obtained by scanning other areas in the image to be retrieved by holding a scanning pen by a user.
7. The image retrieval method according to claim 6, wherein the sending the current local image to a server specifically includes:
if the current local image is the first local image of the image to be retrieved, sending the current local image to a server, and receiving a retrieval session identifier returned by the server;
otherwise, the current local image and the retrieval session identifier are sent to the server, so that the server matches the current local image with each candidate image in a last candidate set corresponding to the retrieval session identifier, and a current candidate set corresponding to the retrieval session identifier is obtained.
8. The image retrieval method of claim 6, wherein the receiving the retrieval result returned by the server further comprises:
if the retrieval result is retrieval failure information, prompting a user to re-collect the current local image of the image to be retrieved;
if the retrieval result is retrieval completion information which carries the bibliographic information of the bibliographic cover image obtained by retrieving the image to be retrieved, prompting a user to acquire a local image of the content page of the entity book corresponding to the bibliographic information;
and if the retrieval result is retrieval completion information which carries the media information of the content page image obtained by retrieving the image to be retrieved, displaying the media information.
9. A server, comprising:
the image receiving unit is used for receiving a current local image of an image to be retrieved, which is acquired by the scanning pen;
the local matching unit is used for matching the current local image with each candidate image in the previous candidate set to obtain a current candidate set;
a result feedback unit, configured to generate a retrieval result and return to the wand based on the current candidate set, so as to trigger the wand to acquire a next partial image of the image to be retrieved when the retrieval result indicates that retrieval is not completed, and update the next partial image to a current partial image;
generating a search result based on the current candidate set, comprising:
analyzing the current retrieval progress aiming at the image to be retrieved according to the number of the candidate images in the current candidate set, and further generating a retrieval result, wherein the retrieval result specifically indicates that the retrieval is finished, the retrieval is not finished or the retrieval is failed;
and the next local image is obtained by scanning other areas in the image to be retrieved by holding the scanning pen by the user.
10. A wand, comprising:
the image acquisition unit is used for acquiring a current local image of the image to be retrieved;
the image sending unit is used for sending the current local image to a server so that the server can match the current local image with each candidate image in a previous candidate set to obtain a current candidate set, and a retrieval result is generated based on the current candidate set;
the result receiving unit is used for receiving the retrieval result returned by the server and acquiring the next partial image of the image to be retrieved when the retrieval result indicates that the retrieval is not finished;
generating a search result based on the current candidate set, including:
analyzing the current retrieval progress aiming at the image to be retrieved according to the number of the candidate images in the current candidate set, and further generating a retrieval result, wherein the retrieval result specifically indicates that the retrieval is finished, the retrieval is not finished or the retrieval is failed;
and the next local image is obtained by scanning other areas in the image to be retrieved by holding the scanning pen by the user.
CN202010813084.6A 2020-08-13 2020-08-13 Image retrieval method, server and scanning pen Active CN111950464B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010813084.6A CN111950464B (en) 2020-08-13 2020-08-13 Image retrieval method, server and scanning pen

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010813084.6A CN111950464B (en) 2020-08-13 2020-08-13 Image retrieval method, server and scanning pen

Publications (2)

Publication Number Publication Date
CN111950464A CN111950464A (en) 2020-11-17
CN111950464B true CN111950464B (en) 2023-01-24

Family

ID=73341862

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010813084.6A Active CN111950464B (en) 2020-08-13 2020-08-13 Image retrieval method, server and scanning pen

Country Status (1)

Country Link
CN (1) CN111950464B (en)

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111522986A (en) * 2020-04-23 2020-08-11 北京百度网讯科技有限公司 Image retrieval method, apparatus, device and medium

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7747050B2 (en) * 2005-11-23 2010-06-29 General Electric Company System and method for linking current and previous images based on anatomy
CN102231188A (en) * 2011-07-05 2011-11-02 上海合合信息科技发展有限公司 Business card identifying method combining character identification with image matching
CN102495998B (en) * 2011-11-10 2013-11-06 西安电子科技大学 Static object detection method based on visual selective attention computation module
CN102542058B (en) * 2011-12-29 2013-04-03 天津大学 Hierarchical landmark identification method integrating global visual characteristics and local visual characteristics
CN103970775A (en) * 2013-01-31 2014-08-06 山东财经大学 Object spatial position relationship-based medical image retrieval method
CN103517041B (en) * 2013-09-29 2016-04-27 北京理工大学 Based on real time panoramic method for supervising and the device of polyphaser rotation sweep
CN108664526B (en) * 2017-04-01 2022-04-29 华为技术有限公司 Retrieval method and device
CN110458855B (en) * 2019-07-08 2022-04-05 安徽淘云科技股份有限公司 Image extraction method and related product

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111522986A (en) * 2020-04-23 2020-08-11 北京百度网讯科技有限公司 Image retrieval method, apparatus, device and medium

Also Published As

Publication number Publication date
CN111950464A (en) 2020-11-17

Similar Documents

Publication Publication Date Title
US8418055B2 (en) Identifying a document by performing spectral analysis on the contents of the document
CN110740389B (en) Video positioning method, video positioning device, computer readable medium and electronic equipment
CN111276149B (en) Voice recognition method, device, equipment and readable storage medium
CN106407358A (en) Image searching method and device and mobile terminal
CN108121987B (en) Information processing method and electronic equipment
US9659224B1 (en) Merging optical character recognized text from frames of image data
CN112235632A (en) Video processing method and device and server
CN108133209B (en) Target area searching method and device in text recognition
CN112866577B (en) Image processing method and device, computer readable medium and electronic equipment
CN109344275B (en) Image identification-based resource acquisition device and method
CN109697242A (en) It takes pictures and searches topic method, apparatus, storage medium and calculate equipment
CN111950464B (en) Image retrieval method, server and scanning pen
CN108108143B (en) Recording playback method, mobile terminal and device with storage function
CN111542817A (en) Information processing device, video search method, generation method, and program
CN115174506A (en) Session information processing method, device, readable storage medium and computer equipment
CN111582281B (en) Picture display optimization method and device, electronic equipment and storage medium
CN111405194A (en) Image processing method and device
CN112232342A (en) Method for setting theme of scanning pen
CN110673727A (en) AR remote assistance method and system
EP4125026A1 (en) Product identification assistance techniques in an electronic marketplace application
CN111582264B (en) Method, device and system for accurate frame questions, electronic equipment and storage medium
CN112764601B (en) Information display method and device and electronic equipment
JP3486168B2 (en) Search system, filing system, recording medium
CN113900602B (en) Intelligent printing method and system for automatically eliminating target object filling information
CN110196956B (en) User head portrait generation method and device, electronic equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
CB02 Change of applicant information

Address after: 230088 China (Anhui) pilot Free Trade Zone, Hefei, Anhui province 6 / F and 23 / F, scientific research building, building 2, zone a, China sound Valley, No. 3333 Xiyou Road, high tech Zone, Hefei

Applicant after: Anhui taoyun Technology Co.,Ltd.

Address before: 230031 9th floor, building 1, tianyuandike science and Technology Park, 66 Qianshui East Road, high tech Zone, Hefei City, Anhui Province

Applicant before: ANHUI TAOYUN TECHNOLOGY Co.,Ltd.

CB02 Change of applicant information
GR01 Patent grant
GR01 Patent grant