CN111582038A - Method and device for searching article, storage medium and mobile robot - Google Patents
Method and device for searching article, storage medium and mobile robot Download PDFInfo
- Publication number
- CN111582038A CN111582038A CN202010280152.7A CN202010280152A CN111582038A CN 111582038 A CN111582038 A CN 111582038A CN 202010280152 A CN202010280152 A CN 202010280152A CN 111582038 A CN111582038 A CN 111582038A
- Authority
- CN
- China
- Prior art keywords
- article
- image
- current position
- item
- searching
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/5866—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
- G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
- G06V10/751—Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
Abstract
A method, a device, a storage medium and a mobile robot for searching articles are provided, which comprises the following steps: when an instruction for searching for a target article is received, searching for the target article in a pre-constructed article database according to the identification of the target article in the instruction and determining the current position of the target article; feeding back the determined current position, or moving to the determined current position to find the target item through target detection. The method and the device can automatically search for the articles for the user.
Description
Technical Field
The present disclosure relates to computer technologies, and in particular, to a method and an apparatus for searching for an object, a storage medium, and a mobile robot.
Background
In daily life, each user has a lot of common articles, and the articles can be frequently found, for example, a mobile phone is used, and almost each user has experience of finding the mobile phone, which brings great inconvenience to daily life.
Disclosure of Invention
The application provides a method and a device for searching articles, a storage medium and a mobile robot, which can achieve the purpose of automatically searching articles for a user.
The application provides a method for searching for an object, which comprises the steps of searching for the object in a pre-constructed object database according to the identification of the object in an instruction and determining the current position of the object when the instruction for searching for the object is received; feeding back the determined current position, or moving to the determined current position to search the target object through target detection; the pre-constructed article database stores attribute information of at least one article, wherein the attribute information of each article comprises an identifier, an original image and position information of the article; the position information of each article is acquired through the following modes: and shooting images of all positions of the designated place, and recording the position as the position information of the article when the image shot at one position is matched with the original image of the article.
Compared with the related art, the method and the device have the advantages that the article database is constructed in advance, the basic information of various articles is stored, the designated place is shot regularly or in real time, the position of the article appearing in the designated place is recorded and stored in the article database constructed according to the scheme, and when the user needs to search, the specific position is automatically provided for the user or the user is searched.
In an exemplary embodiment, the method matches a shot image with an original image of an article to determine the position of the article, generates 128-dimensional digital features for the intercepted subgraph, matches the 128-dimensional digital features generated by the original image of each article of the same type in the article database respectively, and calculates the similarity between the intercepted subgraph and the original image of each article of the same type, so that the accuracy of article identification is higher.
Additional features and advantages of the application will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the application. Other advantages of the present application may be realized and attained by the instrumentalities and combinations particularly pointed out in the specification and the drawings.
Drawings
The accompanying drawings are included to provide an understanding of the present disclosure and are incorporated in and constitute a part of this specification, illustrate embodiments of the disclosure and together with the examples serve to explain the principles of the disclosure and not to limit the disclosure.
FIG. 1 is a diagram illustrating an article search method according to an embodiment of the present disclosure;
fig. 2 is a schematic diagram illustrating an application of the article search method to a mobile robot according to the embodiment of the present application.
Detailed Description
The present application describes embodiments, but the description is illustrative rather than limiting and it will be apparent to those of ordinary skill in the art that many more embodiments and implementations are possible within the scope of the embodiments described herein. Although many possible combinations of features are shown in the drawings and discussed in the detailed description, many other combinations of the disclosed features are possible. Any feature or element of any embodiment may be used in combination with or instead of any other feature or element in any other embodiment, unless expressly limited otherwise.
The present application includes and contemplates combinations of features and elements known to those of ordinary skill in the art. The embodiments, features and elements disclosed in this application may also be combined with any conventional features or elements to form a unique inventive concept as defined by the claims. Any feature or element of any embodiment may also be combined with features or elements from other inventive aspects to form yet another unique inventive aspect, as defined by the claims. Thus, it should be understood that any of the features shown and/or discussed in this application may be implemented alone or in any suitable combination. Accordingly, the embodiments are not limited except as by the appended claims and their equivalents. Furthermore, various modifications and changes may be made within the scope of the appended claims.
Further, in describing representative embodiments, the specification may have presented the method and/or process as a particular sequence of steps. However, to the extent that the method or process does not rely on the particular order of steps set forth herein, the method or process should not be limited to the particular sequence of steps described. Other orders of steps are possible as will be understood by those of ordinary skill in the art. Therefore, the particular order of the steps set forth in the specification should not be construed as limitations on the claims. Further, the claims directed to the method and/or process should not be limited to the performance of their steps in the order written, and one skilled in the art can readily appreciate that the sequences may be varied and still remain within the spirit and scope of the embodiments of the present application.
As shown in fig. 1, a method for finding an article according to an embodiment of the present application includes the following operations:
s1, when receiving an instruction for searching for a target item, searching for the target item in a pre-constructed item database according to the identification of the target item in the instruction and determining the current position of the target item;
the pre-constructed article database stores attribute information of at least one article, wherein the attribute information of each article comprises an identifier, an original image and position information of the article; the position information of each article is acquired through the following modes: and shooting images of all positions of the appointed place, and recording the position as the current position of the article when the image shot at one position is matched with the original image of the article.
S2, feeding back the determined current position, or moving to the determined current position to search the target item through target detection;
in an exemplary embodiment, the items in the pre-constructed item database may be "mobile phone", "key", "glasses", "remote controller", etc. items that the user often needs to find. The identification of the item includes information such as the item's ID, name, affiliate, etc. The original image of each article can be an image provided by the user in advance or an image which is shot by the searching device at a specified place and confirmed by the user. The designated location may be the user's residence, etc. The instruction can be an operation instruction, a language instruction and the like.
In one exemplary embodiment, the current location of the item refers to the last acquired location of the item in the three-dimensional scene of the designated place; and the three-dimensional scene is constructed by shooting point cloud information of the appointed place and adopting a visual algorithm. Specifically, a three-dimensional scene is constructed in real time by adopting a depth camera to shoot point cloud information of a surrounding environment in real time And running a visual SLAM (simultaneous Localization And mapping) algorithm on a robot operating system, And the three-dimensional scene comprises (x, y, z) coordinates of all indoor machine visual visible article composition points.
For example, the user first enters and stores identification information and basic information such as an original image in the attribute information of each article. After a user inputs basic information of an article, a searching device carries out real-time or regular image shooting on each position in a home, and when the image shot at one position contains one or more pre-stored articles, such as 'keys', 'glasses', 'mobile phones' and the like, the position is correspondingly stored in attribute information of the one or more articles in a pre-constructed article database.
When images of various positions in a home are shot in real time or periodically, if the searching equipment is a movable robot, the searching equipment starts to move in a specified place, surrounding objects are continuously sensed through a depth camera and a monocular camera in the moving process, for picture information shot by the monocular camera, whether articles such as 'keys', 'glasses', 'mobile phones' and the like appear in a shot picture or not is detected in real time through a target detection algorithm YOLO9000, the target detection algorithm is pre-trained to detect 9000 common articles, and when the specified articles are found in one position, the position is recorded.
When a user needs to search for a certain article, such as "search for the mobile phone with the third page", the manual input of the user can be received on a user interaction interface, or the voice recognition system is adopted to detect the language information of the user, after two identification information of the "third page" and the "mobile phone" are recognized, the identification information is matched with the identification of each article in the pre-constructed article database, the mobile phone with the third page in the pre-constructed article database is confirmed to be needed to be searched for by the user, and then the current position of the "mobile phone with the third page" is determined according to the attribute information of the mobile phone with the third page, so that the mobile phone with the third page is provided for the user to search for or. If the current position of the 'Zhang III mobile phone' is recorded to be on a certain place of a residence such as a sofa, the position is prompted to a user for searching, or the searching device moves to the sofa and sends a prompting message.
In an exemplary embodiment, the matching of the image taken at the one location with the original image of the article comprises the following operations:
s11, when an article of a preset type is detected in the image shot at one position, a sub-image containing the article is intercepted from the image;
s12, generating 128-dimensional digital features for the intercepted subgraph, respectively matching the 128-dimensional digital features generated by the original pictures of various articles of the same type in the article database, and calculating the similarity between the intercepted subgraph and the original pictures of various articles of the same type;
and S13, when the similarity of the intercepted subgraph and an original picture of the article exceeds a preset threshold value, determining that the image shot at the position is matched with the original image of the article.
In one exemplary embodiment, a 128-dimensional digital feature SIFT algorithm generation is generated for the truncated subgraph.
Illustratively, if the similarity of the truncated subgraph and two or more items exceeds a threshold, it is determined that the subgraph matches the original image of the item with the highest similarity. For example, in the above example of searching for "mobile phone with third image", when it is detected that there is a mobile phone in the image, and the similarity between the mobile phone with third image and the mobile phone with fourth image exceeds 0.8, the object with the highest similarity is regarded as the object matched with the image. The cosine distance can be used to calculate the degree of similarity between the captured image and the original image, e.g., between 0 and 1, with 0 being completely different and 1 being completely the same.
And when the matching is successful, the constructed three-dimensional scene information is registered by using a pose estimation algorithm (such as an ICP algorithm) to determine the 6D posture (including three-dimensional coordinates and three-dimensional rotation information) of the article, and the three-dimensional position coordinate information and the current time of the article are stored in a pre-constructed article database.
In one exemplary embodiment, the method further comprises the operations of:
s3, periodically updating the current position in the attribute information of each item in the item database;
and S4, respectively storing the current position of each item before updating as the historical position of the item.
Periodically shooting each position of a designated place, and matching the shot image with an original image of an article in a database; when the matching of one image and the original image of one article is successful, updating the shooting position of the image to the current position of the article; for an item that is not successfully matched with the image of any location, the current location may be updated to be empty (i.e., the designated location does not find the item this time).
In one exemplary embodiment, the method further comprises the operations of:
s5, when the target object is not searched at the determined current position, shooting images of each position of the appointed place, and respectively matching the shot images with the original images of the target object;
and S6, when the image of a position is matched with the original image of the target item, taking the position as the current position of the target item.
For example, in the above example of finding "mobile phone with three pages", if a mobile phone with three pages is not found at the current position, the mobile phone with three pages is photographed at each position of the specified place, and the actual position of the mobile phone with three pages is confirmed by recognizing the mobile phone with three pages for each photographed picture.
In one exemplary embodiment, the method further comprises the operations of:
and S7, when the target item is not found at the determined current position, counting the historical position distribution of the target item in a specified historical period, and determining the historical position with the maximum probability as the current position of the target item for searching.
For example, as an example of searching for "zhang san mobile phone" as described above, if, according to the time information in the database, it is counted that the location of zhang san mobile phone within one month includes: [3,3,3],[1,1,1],[2,2,2],[3,3,3]. When the searching device fails to find the mobile phone (possibly blocked by other articles such as books), the most probable position can be given by three pieces of mobile phone historical position information: [3,3,3].
The application also provides an article searching device, which comprises a processor and a memory, wherein the memory is stored with a program for searching articles; the processor is used for reading the program for searching for the article and executing the operation method of any one of the above items.
The present application also provides a computer storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method of operation of any one of the above.
As shown in fig. 2, a mobile robot includes a motion control system for performing image photographing; the image acquisition equipment is used for shooting images; one or more processors, memory; and one or more programs in the memory and configured to be executed by the one or more processors, the programs including instructions for performing any of the operations described above.
In one exemplary embodiment, the mobile robot is configured to include a visual slam (simultaneous Localization And mapping) algorithm running on the robot operating system to construct a three-dimensional scene in the room in real time that includes (x, y, z) coordinates of all machine-visually visible item composition points in the room.
In one exemplary embodiment, the mobile robot is configured to include a visual recognition algorithm running on the robot operating system for recognizing the item pictures in the built item database.
In one exemplary embodiment, the mobile robot is configured to include a pose estimation algorithm running on the robot operating system for identifying the location of the target item.
In one exemplary embodiment, the mobile robot is configured to include a speech recognition system that recognizes a user's speech.
In one exemplary embodiment, the mobile robot is configured to include a laser sensor for identifying its distance from objects within a specified venue.
In one exemplary embodiment, the image acquisition device comprises a depth camera for capturing point cloud information of the surrounding environment; and the monocular camera is used for shooting images of all positions of the specified place.
It will be understood by those of ordinary skill in the art that all or some of the steps of the methods, systems, functional modules/units in the devices disclosed above may be implemented as software, firmware, hardware, and suitable combinations thereof. In a hardware implementation, the division between functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be performed by several physical components in cooperation. Some or all of the components may be implemented as software executed by a processor, such as a digital signal processor or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). The term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, as is well known to those of ordinary skill in the art. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, Digital Versatile Disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed by a computer. In addition, communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media as known to those skilled in the art.
Claims (10)
1. A method for locating an item, comprising:
when an instruction for searching for a target article is received, searching for the target article in a pre-constructed article database according to the identification of the target article in the instruction and determining the current position of the target article;
feeding back the determined current position, or moving to the determined current position to search the target object through target detection;
the pre-constructed article database stores attribute information of at least one article, wherein the attribute information of each article comprises an identifier, an original image and position information of the article;
wherein, the current position of each article is respectively obtained by the following modes:
and shooting images of all positions of the appointed place, and recording the position as the current position of the article when the image shot at one position is matched with the original image of the article.
2. The method of claim 1, wherein the current position of the item is a last acquired position of the item in a three-dimensional scene of a specified location; the three-dimensional scene is constructed by shooting point cloud information of the appointed place and adopting a visual algorithm.
3. The method of claim 1, wherein matching the captured image at the one location with the original image of the item comprises:
when an article of a preset type is detected in an image shot at one position, a sub-image containing the article is intercepted from the image;
generating 128-dimensional digital features for the intercepted subgraph, respectively matching the 128-dimensional digital features generated by the original pictures of various articles of the same type in the article database, and calculating the similarity between the intercepted subgraph and the original pictures of various articles of the same type;
and when the similarity between the intercepted subgraph and an original image of the article exceeds a preset threshold, determining that the image shot at the position is matched with the original image of the article.
4. The method of claim 1, further comprising:
periodically updating the current position in the attribute information of each article in the article database;
and respectively storing the current position of each item before updating as the historical position of the item.
5. The method of claim 1, further comprising:
when the target item is not searched at the determined current position, the following operations are carried out:
shooting images of all positions of the appointed place, and respectively matching the shot images with original images of the target object;
and when the image of one position is matched with the original image of the target object, taking the position as the current position of the target object.
6. The method of claim 4, further comprising:
and when the target object is not found at the determined current position, counting the historical position distribution of the target object in a specified historical period, and determining the historical position with the maximum probability as the current position of the target object for searching.
7. An article searching device comprises a processor and a memory, and is characterized in that the memory stores a program for searching articles; the processor is used for reading the program for finding the article and executing the method of any one of claims 1-6.
8. A computer storage medium on which a computer program is stored, the computer program being executable by a processor to implement a method according to any one of claims 1-6.
9. A mobile robot, comprising:
a motion control system for taking an image;
the image acquisition equipment is used for shooting images;
one or more processors, memory; and one or more programs in the memory and configured to be executed by the one or more processors, the programs comprising instructions for performing the operations of claims 1-6.
10. The mobile robot of claim 9, wherein the image capture device comprises a depth camera for capturing point cloud information of the surrounding environment; and the monocular camera is used for shooting images of all positions of the specified place.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010280152.7A CN111582038A (en) | 2020-04-10 | 2020-04-10 | Method and device for searching article, storage medium and mobile robot |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010280152.7A CN111582038A (en) | 2020-04-10 | 2020-04-10 | Method and device for searching article, storage medium and mobile robot |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111582038A true CN111582038A (en) | 2020-08-25 |
Family
ID=72113035
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010280152.7A Withdrawn CN111582038A (en) | 2020-04-10 | 2020-04-10 | Method and device for searching article, storage medium and mobile robot |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111582038A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112818750A (en) * | 2021-01-05 | 2021-05-18 | 青岛海尔科技有限公司 | Article positioning method and device, readable storage medium and electronic device |
| CN113111213A (en) * | 2021-04-23 | 2021-07-13 | 大同市巴什卡机械制造有限公司 | Method and device for automatically storing articles in storage system |
| CN113269828A (en) * | 2021-04-25 | 2021-08-17 | 青岛海尔空调器有限总公司 | Article searching method and device, air conditioning equipment and storage medium |
| CN113743856A (en) * | 2021-01-05 | 2021-12-03 | 北京京东乾石科技有限公司 | Article sorting method and device, and storage medium |
-
2020
- 2020-04-10 CN CN202010280152.7A patent/CN111582038A/en not_active Withdrawn
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112818750A (en) * | 2021-01-05 | 2021-05-18 | 青岛海尔科技有限公司 | Article positioning method and device, readable storage medium and electronic device |
| CN113743856A (en) * | 2021-01-05 | 2021-12-03 | 北京京东乾石科技有限公司 | Article sorting method and device, and storage medium |
| CN113111213A (en) * | 2021-04-23 | 2021-07-13 | 大同市巴什卡机械制造有限公司 | Method and device for automatically storing articles in storage system |
| CN113111213B (en) * | 2021-04-23 | 2023-07-21 | 大同市巴什卡机械制造有限公司 | Automatic article storage method and device for warehouse system |
| CN113269828A (en) * | 2021-04-25 | 2021-08-17 | 青岛海尔空调器有限总公司 | Article searching method and device, air conditioning equipment and storage medium |
| WO2022227526A1 (en) * | 2021-04-25 | 2022-11-03 | 青岛海尔空调器有限总公司 | Item search method and apparatus, air-conditioning device, and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111582038A (en) | Method and device for searching article, storage medium and mobile robot | |
| CN107990899B (en) | Positioning method and system based on SLAM | |
| CN104487915B (en) | Maintain the continuity of amplification | |
| US9298976B2 (en) | Method, apparatus and computer readable recording medium for managing a reference face database to improve face recognition performance under a restricted memory environment | |
| US9805331B2 (en) | Smartphone-based asset management system | |
| CN109791608A (en) | Mapping abstract and localization | |
| CN109993045B (en) | Article searching method, article searching device, article searching system and machine-readable storage medium | |
| JP5662670B2 (en) | Image processing apparatus, image processing method, and program | |
| CN110533694B (en) | Image processing method, device, terminal and storage medium | |
| JP2016201093A (en) | Image processing apparatus and image processing method | |
| CN110969644B (en) | Personnel track tracking method, device and system | |
| US20140049600A1 (en) | Method and system for improving surveillance of ptz cameras | |
| JP6868061B2 (en) | Person tracking methods, devices, equipment and storage media | |
| CN103428537A (en) | Video processing method and video processing device | |
| KR20150127503A (en) | Service providing system and method for recognizing object, apparatus and computer readable medium having computer program recorded therefor | |
| JP6687199B2 (en) | Product shelf position registration program and information processing device | |
| CN114529621B (en) | Household type graph generation method and device, electronic equipment and medium | |
| CN108985131B (en) | Target identification method and image processing equipment | |
| KR101990823B1 (en) | Method, program, and apparatus for estimating location and managing identification of target using facial information | |
| CN112396654A (en) | Method and device for determining pose of tracking object in image tracking process | |
| CN110084187B (en) | Position identification method, device, equipment and storage medium based on computer vision | |
| JP2011223250A (en) | Photographing assisting apparatus, photographing assisting method, and program used therewith | |
| CN113610967B (en) | Three-dimensional point detection method, three-dimensional point detection device, electronic equipment and storage medium | |
| WO2020108618A1 (en) | Object prompting method, apparatus, device, and computer-readable medium | |
| CN110880187A (en) | Camera position information determining method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20200825 |