WO2014177751A1 - Method and apparatus for finding a location - Google Patents

Method and apparatus for finding a location Download PDF

Info

Publication number
WO2014177751A1
WO2014177751A1 PCT/FI2013/050477 FI2013050477W WO2014177751A1 WO 2014177751 A1 WO2014177751 A1 WO 2014177751A1 FI 2013050477 W FI2013050477 W FI 2013050477W WO 2014177751 A1 WO2014177751 A1 WO 2014177751A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
matching
refdata1
model
reference data
Prior art date
Application number
PCT/FI2013/050477
Other languages
French (fr)
Inventor
Timo J TOIVANEN
Jarmo Saari
Original Assignee
Nokia Corporation
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nokia Corporation filed Critical Nokia Corporation
Priority to PCT/FI2013/050477 priority Critical patent/WO2014177751A1/en
Publication of WO2014177751A1 publication Critical patent/WO2014177751A1/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G06T7/75—Determining position or orientation of objects or cameras using feature-based methods involving models
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/10—Image acquisition
    • G06V10/16—Image acquisition using multiple overlapping images; Image stitching

Definitions

  • consumer products may be sold in a self-service shop, where a customer may personally collect one or more tangible items e.g. to a shopping cart or a basket.
  • the items put on sale may be displayed e.g. on shelves, cabinets, freezer compartments and/or pallets.
  • a single shop may comprise e.g. more than one million products. Finding a particular product may be time- consuming. The number of salespersons in the shop may be low. Even if the customer could find a salesperson and ask about the location of the product that he wishes to buy, the verbal instructions given by the salesperson for finding the location of the product may be inaccurate.
  • Some embodiments may relate to an apparatus for finding the location of a product. Some embodiments may relate to a method for finding the location of a product. Some embodiments may relate to a computer program for finding the location of a product. Some embodiments may relate to a computer program product storing a computer program for finding the location of a product. Some embodiments may relate to means for finding the location of a product.
  • a method comprising:
  • an apparatus comprising at least one processor, a memory including computer program code, the memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform at least the following:
  • a computer program comprising computer program code configured to, when executed on at least one processor, cause an apparatus or a system to:
  • a computer program product embodied on a non-transitory computer readable medium, comprising computer program code configured to, when executed on at least one processor, cause an apparatus or a system to:
  • a means for seeking an item comprising:
  • a portable seeking device may be used to assist in finding a desired item e.g. in a shop.
  • the user may carry the seeking device e.g. when moving in the shop.
  • the user may wish to find an item, which may be called e.g. as a desired item.
  • the desired item may be represented by a model .
  • the seeking device may comprise a user interface for selecting a model .
  • the seeking device may comprise an image sensor for capturing one or more image frames.
  • the seeking device may be configured to analyze an image frame in order to determine whether the image frame contains a matching portion, which matches with the selected model.
  • the seeking device may comprise one or more data processors for determining whether a candidate portion of an image frame is a matching image portion. An item matching with the model may be called as a matching item.
  • the seeking device may be configured to indicate the location of a matching item by analyzing the captured image frames.
  • the seeking device may be configured to indicate the location of a matching item based on the position of a matching image portion.
  • the seeking device may comprise a user interface for providing indication about the location of a matching item.
  • the image sensor and the user interface may be implemented in the same portable device.
  • the user may have previously selected a model item or a model image, and he may wish to find at least one item, which substantially matches with said model item or model image.
  • the model item or a model image may be represented by a model, which can be stored in a computer memory.
  • the model may comprise data specifying one or more characteristic features, which may be used to differentiate the model item or a model image from other items.
  • Searching a desired product in the shop may be time-consuming. Finding the desired product (i.e. the desired item) by bare eyes may be difficult, because a high number of other items and advertisements may be simultaneously competing to attract the customer's attention. It may often happen that the desired product is located e.g. on a bottom shelf, wherein other items have been arranged on the most prominent places in order to promote sales of said other items.
  • the desired item may belong to a group, which comprises e.g. more than one thousand items having different visual appearance.
  • the items may be divided in several groups, which are arranged on different shelves. Finding the desired product may require walking a long distance between shelves of the shop. Using the seeking device may substantially reduce the time needed to find the location of the matching item.
  • a person may use a portable seeking device to help in finding the desired product in the shop faster.
  • the user may substantially reduce the time needed to find a matching item from a group of items arranged on e.g. a shelf, cabinet, freezer compartment and/or pallet.
  • Using the seeking device may facilitate finding the desired product from a group of items e.g. when the visual appearance of other items of the group is very similar to the desired product.
  • the group may comprise e.g. more than one thousand products arranged on one or more shelves such that the products have substantially identical size and shape.
  • the seeking device may find the desired product based on a very small visual difference.
  • the desired product may have e.g.
  • the seeking device may assist in finding the location of a desired product in a situation where searching the desired product by bare eyes would represent a boring monotonous work.
  • the seeking device may assist in finding the location of a desired product in a situation where probability of finding the desired product by bare eyes would be low.
  • Using the seeking device may make it possible that the user can perform one or more additional tasks during seeking the desired item.
  • the user may concentrate on studying other products while the seeking device is automatically seeking the desired product.
  • the user may use an internet browser or watch a video while the seeking device is seeking the desired product.
  • a location database of a shop may comprise item-specific location data, which associates a location with an identification code of each item.
  • the database may often contain erroneous data e.g. because the personnel may have re-arranged the items to new locations without updating the database.
  • the seeking device it is not necessary to rely on data stored previously stored in the location database.
  • Using the seeking device may facilitate finding the desired product e.g.
  • the seeking device may facilitate finding the desired product e.g. when the user cannot verbally describe the desired product.
  • Using the seeking device may facilitate finding the desired product e.g. when the number of personnel in the shop is so low.
  • Using the seeking device may facilitate finding the desired product e.g. when the personnel is not familiar with the locations of the products.
  • Using the seeking device may facilitate finding the desired product e.g. when the user is hearing- impaired or cannot speak the language of the personnel.
  • the seeking device may be arranged to pinpoint the location of the desired product.
  • Fig. 1 a shows, in a side view, a group of items
  • Fig. 1 b shows, in a three-dimensional view, a person using a seeking device
  • Fig. 2a shows, by way of example, a model
  • Fig. 2b shows, by way of example, reference data representing a model, shows, by way of example, enabling a seeking mode by using a user interface
  • Fig. 4 shows, by way of example, functional units of the seeking device
  • Fig. 5 shows, by way of example, capturing an image of several items
  • Fig. 6a shows, by way of example, indicating the position of a matching image portion
  • Fig. 6b shows, by way of example, indicating the position of a matching image portion
  • Fig. 6c shows, by way of example, indicating the position of a matching image portion
  • Fig. 6d shows, by way of example, indicating the position of a matching image portion
  • Fig. 7a shows, by way of example, method steps for seeking a matching item, by using real-time imaging
  • Fig. 7b shows, by way of example, method steps for seeking a matching item, by using image frames retrieved from a memory
  • Fig. 8a shows, by way of example, several image frames captured at consecutive times
  • Fig. 8b shows, by way of example, a combined image frame formed by stitching several image frames together
  • Fig. 9a shows, by way of example, information provided by a user interface
  • Fig. 9b shows, by way of example, information provided by a user interface
  • Fig. 10a shows, by way of example, a system for assisting in the seeking
  • Fig. 10b shows, by way of example, a seeking device
  • Fig. 10c shows, by way of example, a server for assisting in the seeking.
  • a plurality of tangible items P1 , P2, P3, P4, P21 , ..., P31 , ..., P41 , ... may be arranged in one or more groups G1 , G2, G3, G4 so that a person C1 may see the items.
  • the items may be put on display e.g. in a shop SHOP1 .
  • the shop SHOP1 may be e.g. a supermarket, market, mart, store, or mall.
  • the items may be put on display on market square.
  • the items may be put on display e.g. in a building.
  • the person C1 may have previously selected a model REF1 , and he may wish to find at least one item, which substantially matches the model REF1 .
  • An item which substantially matches the model REF1 may be called as a matching item PM1 .
  • the person C1 does not know the location x,y,z, of the matching item PM1 when the search begins.
  • the model REF1 may have one or more characteristic visual features, e.g. one or more graphical symbols (e.g. a star), and/or a certain geometrical shape (e.g. rounded corners).
  • the model REF1 may comprise one or more markings (e.g. "A14").
  • the model REF1 may have a marking which represents e.g. a product code and/or a trade name.
  • the matching item PM1 may belong e.g. to a group G1 , which comprises more than 10 items.
  • the matching item PM1 may belong e.g. to a group G1 , which comprises more than 100 items.
  • the matching item PM1 may belong e.g. to a group G1 , which comprises more than 1000 items.
  • the group G1 may comprise only one matching item PM1 , or the group G1 may comprise several matching items PM1 , wherein the group may further comprise several other items such that the visual appearance of the other items is different from the visual appearance of the matching item PM1 .
  • the group G1 may comprise one or more matching items PM1 , which have a first visual appearance, and the group G1 may comprise one or more other items, which have a second visual appearance.
  • a group (e.g. the group G2) does not comprise any matching items PM1 .
  • the person C1 may use a seeking device 500 to assist in finding the location x,y,z of a matching item PM1 .
  • the seeking device 500 may comprise a camera, which is arranged to capture one or more image frames F1 , F2 (see e.g. Fig. 5).
  • the image frame F1 , F2 may be analyzed in order to determine whether the image frame comprises an image portion, which matches with the model REF1 .
  • the seeking device 500 may be arranged:
  • the seeking device 500 may be arranged to e.g. display an arrow symbol IND2, which at least approximately indicates the location of the matching item PM1 .
  • Using the seeking device 500 may optionally comprise moving the seeking device 500 with respect to the group G1 e.g. along a path PTH1 so that image data may be gathered from a wide area.
  • Moving the seeking device 500 may comprise changing the position of the seeking device 500 with respect to the group G1 and/or changing the orientation of the seeking device 500 with respect to the group G1 .
  • the seeking device 500 does not need to be moved.
  • SX, SY, SZ denote orthogonal directions.
  • the location may be expressed e.g. by using position coordinates x,y,z.
  • the coordinate x may denote a coordinate specifying a position in the direction SX.
  • the coordinate y may denote a coordinate specifying a position in the direction SY.
  • the coordinate z may denote a coordinate specifying a position in the direction SZ.
  • the location may be defined e.g. with respect to an origin ORIG1 .
  • the origin ORIG1 may be defined e.g. with respect to a visually prominent feature LND1 .
  • the visually prominent feature may also be called e.g. as a landmark feature LND1 .
  • the visually prominent feature LND1 may be e.g.
  • the origin ORIG1 may coincide with the visually prominent feature LND1 .
  • the model REF1 may have one or more characteristic visual features, e.g. a graphical symbol REFSYM1 , a shape REFSHAPE1 , and/or a marking REFID1 .
  • the marking REFID1 may represent e.g. trade name or a product code.
  • the model REF1 may be represented by the reference data REFDATA1 .
  • the reference data REFDATA1 may comprise e.g. an image representing a model REF1 .
  • the reference data REFDATA1 may comprise e.g. a (digital) image of a model item.
  • the reference data REFDATA1 may comprise e.g. an image of a two-dimensional model image, wherein the model image may be e.g. a graphical symbol appearing on a web page.
  • the model image may be e.g. a graphical symbol, which has been printed on a product.
  • the reference data REFDATA1 may comprise compressed data relevant for image recognition. Determining whether a portion of an image frame is a matching image portion or not may comprise:
  • Determining whether a portion of an image frame is a matching image portion or not may comprise:
  • the threshold value may be adjusted e.g. if the device 500 provides erroneous indications, and/or if no matching portion is found.
  • the seeking method may comprise adjusting the threshold value.
  • the device 500 may be configured to adjust the threshold value based on input provided by a user C1 .
  • the reference data REFDATA1 may represent a three- dimensional model REF1 .
  • the reference data REFDATA1 may comprise e.g. two or more images of a three dimensional model item taken from different angles. A first image may be an image of a first side of the model item, and a second image may be an image of a second side of the model item.
  • the reference data REFDATA1 may comprise a numerical model representing the three-dimensional shape of a model REF1 .
  • the reference data REFDATA1 may comprise e.g. a video sequence, which comprises several images representing the model REF1 from different angles.
  • the model REF1 may be represented e.g. a point cloud, which may define a three-dimensional reference shape.
  • the reference data REFDATA1 representing the model REF1 may comprise several reference images RIMG1 , RIMG2, RIMG3 of a three-dimensional model REF1 , wherein each of said images RIMG1 , RIMG2, RIMG3 may represent a different view direction specified by a different view angle CM , ⁇ 3 ⁇ 4, ⁇ 3 ⁇ 4.
  • the model REF1 may be located at an origin ORIG3 in a coordinate system defined by the (orthogonal) directions SX', SY, SZ'.
  • the reference image RIMG1 may be an image of the model REF1 when viewed from a view point VP1 .
  • the image RIMG2 may be an image of the model REF1 when viewed from a view point VP2.
  • the image RIMG3 may be an image of the model REF1 when viewed from a view point VP3.
  • the orientation and the location of the model REF1 may be fixed in the coordinate system (SX', SY, SZ').
  • the viewpoint VP1 and the origin ORIG3 may define a view line VLIN1 .
  • the viewpoint VP2 and the origin ORIG3 may define a view line VLIN2.
  • the viewpoint VP3 and the origin ORIG3 may define a view line VLIN3.
  • the angle between the view line VLIN1 and the direction SX' may be specified by the view angle CM
  • the angle between the view line VLIN2 and the direction SX' may be specified by the view angle ⁇ 3 ⁇ 4
  • the angle between the view line VLIN3 and the direction SX' may be specified by the view angle ⁇ 3 ⁇ 4.
  • Two or more images RIMG1 , RIMG2, RIMG3 may be simultaneously stored in a memory.
  • Reference data REFDATA1 which comprises two or more images RIMG1 , RIMG2, RIMG3 may be stored in a memory.
  • a point cloud representing the model REF1 may be stored in a memory, and the images RIMG1 , RIMG2, RIMG3 may be generated from the point cloud according to a view angle.
  • the view angle may be varied in order to generate two or more different images RIMG1 , RIMG2, RIMG3.
  • One or more of the images RIMG1 , RIMG2, RIMG3 may be rotated, magnified and/or stretched in a selected direction to provide one or more reference images, which may be used for determining a matching portion.
  • the reference data REFDATA1 may comprise a first image RIMG1 of the model REF1 when viewed from a first view direction CM, and a second image RIMG2 of the model REF1 when viewed from a second view direction ⁇ 3 ⁇ 4, which is different from the first view direction CM .
  • This may facilitate finding the matching item PM1 a situation where the candidate items of the group G1 may have arbitrary orientation (i.e. random or unknown orientation) with respect to the seeking device 500 during capturing image frames F1 , F2.
  • the reference data REFDATA1 may comprise e.g. an image RIMG1 of a first side of a model, and also an image RIMG2 of a second side of said model.
  • the method for indicating the position of the desired item PM1 may comprise:
  • first reference image RIMG1 represents an image of the model taken from a first view point VP1
  • second reference image RIMG2 represents an image of the model taken from a second different view point VP2.
  • the model REF1 may be selected e.g. from among candidate models stored in a database.
  • the database may be stored e.g. in a memory contained in the seeking device 500.
  • the database may be stored e.g. in an internet server 1240 (See Fig. 10a, 10c).
  • the reference data REFDATA1 associated with the model REF1 may be retrieved from the database.
  • the database may be accessed from a remote location.
  • the reference data REFDATA1 may be retrieved e.g. from a server by using the Internet.
  • a remote database stored in a server and a local database stored in a portable device 500 may be synchronized.
  • Reference data REFDATA1 may be added to the database e.g. by a manufacturer of a product, by a shopkeeper, by an internet community and/or by the user C1 .
  • the method for indicating the position of the desired item may comprise:
  • the image sensor 100 of the seeking device 500 may capture an image of a model item.
  • the reference data REFDATA1 may be determined from the image, and the reference data REFDATA1 may be stored in a memory of the seeking device 500 and/or in a database to be used as the reference data REFDATA1 representing the model REF1 .
  • the model REF1 may be selected e.g. by using a user interface UIF1 of the seeking device 500.
  • the user C1 may select the model REF1 e.g. from among several candidate models displayed on a web site.
  • the user interface UIF1 may comprise e.g. a display, which is configured to display an image of the selected model REF1 .
  • the user interface UIF1 may comprise e.g. a key KEY1 for enabling operation in a seeking mode.
  • the user C1 may enable operation in a seeking mode e.g. by touching the key KEY1 with his hand H1 .
  • the key KEY1 may be touched with a finger.
  • the key KEY1 may be optionally associated with displayed symbol, e.g. with a displayed word "SEEK".
  • the user interface UIF1 may optionally comprise a second key KEY2 e.g. for enabling some other functionality of the device 500.
  • the key KEY2 may be e.g. a home button, and touching the key KEY2 may enable displaying an initial menu or a home menu.
  • the seeking device 500 may comprise an image sensor 100 for providing image data SDATA1 , a memory MEM1 for storing a model data REFDATA1 , and an image analysis unit 400 for comparing the model data REFDATA1 with the image data SDATA1 .
  • the image sensor 100 may comprise e.g. a two-dimensional array of light- detecting detector pixels, which cover the active area 101 of the image sensor 100.
  • the image sensor 100 may convert the optical image IMG1 into a digital image.
  • the image sensor 100 may comprise e.g. a CMOS array or a CCD array.
  • the seeking device 500 may comprise imaging optics 200, which may be arranged to form an optical image IMG1 on the image sensor 100.
  • the imaging optics 200 may form the optical image IMG1 by focusing light LBX reflected from items P1 , P2, P3, P4 to the image sensor 100.
  • the seeking device 500 may comprise a memory MEM2 for storing the image data SDATA1 .
  • the image data SDATA1 may comprise one or more image frames F1 , F2.
  • the image frames F1 , F2 may also be called e.g. as a digital images.
  • the seeking device 500 may comprise a user interface UIF1 for receiving instructions from a user C1 and/or for providing information to the user C1 .
  • the seeking device 500 may comprise a control unit CNT1 for controlling operation of the device 500.
  • the control unit CNT1 may comprise one or more data processors for controlling operation of the device 500.
  • the control unit CNT1 may comprise the image analysis unit 400.
  • the image analysis unit 400 may be a part of the control unit CNT1 .
  • the control unit CNT1 and the image analysis unit 400 may be implemented on the same semiconductor chip.
  • the seeking device 500 may comprise a memory MEM3, which may store computer program PROG1 for executing the steps of the seeking method.
  • the computer program PROG1 may comprise computer program code configured to, when executed on at least one processor, cause an apparatus 500 or a system 1000 to:
  • - provide a selective indication IND1 , IND2 about the location of a matching item based on the position (u1 ,v1 ) of the matching image portion POR1 .
  • a computer program product embodied on a non-transitory computer readable medium may comprise the computer program PROG1 .
  • the seeking device 500 may comprise a communication unit RXTX1 arranged to receive data from a web (e.g. Internet), local area network and/or mobile communications network.
  • the communication unit RXTX1 may comprise e.g. a radio frequency receiver, a radio frequency transmitter, and an antenna.
  • reference data REFDATA1 may be received by using the communication unit RXTXL
  • the imaging optics 200 may be arranged to form an optical image IMG1 of several items P1 , P2, P3 on the image sensor 100.
  • the combination of the imaging optics 200 and the image sensor 100 may have a field of view CAMVIEW1 .
  • the image sensor 100 may convert the optical image IMG1 into a digital image F1 .
  • the image sensor 100 may provide image data SDATA1 , which may comprise an image frame F1 .
  • the image frame F1 may comprise sub- images SUB1 , SUB2, SUB3, SUB4 of the items P1 , P2, P3, P4.
  • the sub-images may also be called e.g. as candidate portions of the image frame.
  • the sub-image SUB1 may be a candidate portion SUB1 .
  • the sub-image SUB2 may be a candidate portion SUB2.
  • the sub-image SUB3 may be a candidate portion SUB3.
  • the items P1 , P2, P3, P4 may be called as candidate items.
  • the candidate items P1 , P2, P3, P4 may comprise one or more characteristic graphical features (for example markings A01 , A02, A03..., A78, a symbol resembling a rectangle, a star or the Sun).
  • the candidate items P1 , P2, P3, P4 may comprise a graphical symbol (e.g. the symbol %) to indicate additional information, e.g. to indicate a discounted price.
  • the image analysis unit 400 may be configured to compare the model REF1 with the image frame F1 in order to determine whether at least one sub-image SUB1 , SUB2, SUB3, SUB4 of the image frame F1 matches with the model REF1 .
  • the matching sub image SUB3 may be classified to be a matching portion POR1 .
  • the sub-image of an item P3 is classified to be a matching portion POR1 , then said item P3 may be classified to be matching item PM1 .
  • a candidate item P3 may be classified to be a matching item PM1 e.g. when the sub-image SUB3 of said candidate item P3 is detected to match with the reference data REFDATA1 .
  • the image analysis unit 400 may be configured to determine the position of the matching portion POR1 .
  • the position of the matching portion POR1 may be defined e.g. by using position coordinates u1 , v1 .
  • the position of the matching portion POR1 may be expressed e.g. with respect to an origin ORIG2.
  • the origin ORIG2 may be e.g. at a corner of the image frame F1 , at the center of the image frame F1 , or at a visually prominent feature ("landmark") appearing in the image frame F1 .
  • the image frame F1 may have a width umax and a height vmax.
  • the width umax and the height vmax may be expressed e.g. as a number of pixels.
  • the width umax may be 2560 pixels, and the height may be 1600 pixels.
  • the center of the optics 200 and the center of the image sensor 100 may together define a nominal aiming line AIM1 of the seeking device 500.
  • the center of the optics 200 and the position of a predetermined pixel of the image sensor 100 may together define a nominal aiming line AIM1 of the seeking device 500.
  • the predetermined pixel may be selected such that the nominal aiming line AIM1 is e.g. substantially perpendicular to a display screen of the device 500.
  • the nominal aiming line AIM1 may have a fixed orientation with respect to a housing of the portable seeking device 500 so that the aiming line AIM1 may be moved together with the portable seeking device 500.
  • the seeking device may comprise an image sensor 100 for capturing the one or more image frames F1 , F2.
  • the image sensor 100 may be arranged to capture an image frame F1 by capturing a still image.
  • the image sensor may be arranged to capture several image frames F1 , F2 by capturing several still images.
  • the image sensor 100 may be arranged to capture an image frame F1 by capturing a video sequence.
  • the image sensor 100 may be arranged to capture several image frames F1 , F2 by capturing a video sequence.
  • the seeking device 500 may comprise:
  • an image sensor 100 configured to capture an image frame F1 , which comprises sub-images SUB1 , SUB2 of items P1 , P2,
  • At least one processor CNT1 , 400 configured to determine whether a portion of an image frame F1 is a matching image portion POR1 by comparing the sub- images SUB1 , SUB2 with the reference data REFDATA1 , and to detect the position u1 , v1 of the matching image portion, and
  • the seeking device 500 may be configured to:
  • the seeking device 500 may be configured to
  • a candidate portion SUB1 is a matching portion POR1 when the candidate portion SUB1 matches with at least one of the first reference image RIMG1 and the second reference image RIMG2, wherein the first reference image RIMG1 represents an image of the model REF1 taken from a first view point VP1 , and the second reference image RIMG2 represents an image of the model REF1 taken from a second different view point VP2.
  • a first image frame F1 taken from a first view point may comprise the sub-image SUB3
  • a second image frame F2 taken from a second different view point may comprise an additional sub-image.
  • the sub image SUB3 of a candidate item P3 may be determined to be a matching portion POR1 when the sub-image SUB3 matches with the first reference image RIMG1 , and if an additional sub image of said candidate item P3 matches with the second reference image RIMG1 .
  • the user interface UIF1 of the device 500 may be configured to provide a selective indicator IND1 , which selectively indicates the location of the matching item PM1 .
  • the user interface UIF1 may be configured to display e.g. an altered image FT obtained by locally altering a portion of the image frame F1 , which contains the matching portion POR1 .
  • the user interface UIF1 may be configured to display an altered image obtained by locally altering a portion POR1 of the image frame F1 .
  • the user interface UIF1 may be arranged to display an altered image FT, which comprises at least a part of an image frame F1 , and a selective indicator IND1 , which is dependent on the selection of the model REF1 . At least one portion of said altered image FT may be dependent on the selection of the model REF1 so that selection of the model has an effect on the visual appearance of the altered image F1 ⁇
  • the selective indicator IND1 may highlight (i.e. emphasize) the position of the matching portion POR1 with respect to the other portions of the displayed image.
  • the selective indicator IND1 may be e.g. a borderline, which surrounds the matching portion POR1 .
  • the selective indicator IND1 may be e.g. an arrow symbol, which points at the matching portion POR1 .
  • the selective indicator IND1 may be e.g. a crosshair pattern, which coincides with the matching portion POR1 .
  • the brightness and/or color of the matching portion POR1 may be altered with respect to the original matching portion POR1 and with respect to an adjacent portion of the displayed image such that the altered matching portion POR1 may itself act as a selective indicator IND1 .
  • the brightness and/or color of a surrounding portion surrounding the matching portion POR1 may be altered such that said surrounding portion may act as a selective indicator IND1 .
  • the selective indicator IND1 may indicate the location of the matching item PM1 such that said selective indicator IND1 is dependent on the selection of the model REF1 .
  • the matching portion POR1 of the original unaltered image frame F1 is not considered to be a selective indicator IND1 .
  • the seeking device 500 may be configured to display an image, which contains the selective indicator IND1 , wherein the selective indicator IND1 may be dependent on the selection of the model REF1 such that selecting a second different model instead of a first model may cause a visual change in the displayed image.
  • the seeking device 500 may be configured to display a first altered image, which contains a selective indicator indicating the location of a first item matching with a first model, and to display a second altered image, which contains a selective indicator indicating the location of a second item matching with a second model, wherein the device 500 may be configured to generate the first altered image by altering a first image frame when the first model has been selected, and to generate the second altered image by altering a second image frame when the second model has been selected, wherein the first altered image may be substantially different from the second altered image also in a situation where the first image frame is identical to the second image frame.
  • the selective indicator IND1 may also indicate that the matching portion POR1 is found, in addition to indicating the location of the matching item PM1 .
  • displaying the borderline of Fig. 6a may indicate that that a candidate portion is detected to match with the reference data REFDATA1 of the model REF1 .
  • an additional indicator may also be provided to indicate that a matching portion POR1 is found.
  • the device may be configured to generate an audio signal and/or a flashing signal when a candidate portion is detected to match with the reference data REFDATA1 of the model REF1 .
  • the user interface UIF1 may optionally display additional information INFO2, INFO3. Additional information INFO2 may e.g. specify that the device 500 is operating in a seeking mode (For example, an acronym LSA may be displayed to indicate "Location Seeking Assistant"). Additional information INFO3 may e.g. specify that the displayed image is a realtime image.
  • the user interface UIF1 may be arranged to display a selective indicator IND2, which points at the position of the matching portion POR1 .
  • the indicator IND2 pointing at the position of the matching portion POR1 may be called as a directional indicator.
  • the user interface UIF1 may be configured to display a directional indicator IND2 based on the position u1 ,v1 of the matching portion POR1 .
  • the directional indicator IND2 may be an arrow symbol pointing at the position of the matching portion POR1 .
  • the directional indicator IND2 may be e.g. a spot, which is arranged to move on the display such that the movement of the spot indicates a direction.
  • the selective indicator IND2 may indicate the position of the matching portion POR1 with respect to a sub-image SUBLND1 of a visually prominent feature LND1 .
  • the user interface UIF1 may be arranged to indicate the position of the matching portion POR1 with respect to a sub-image SUBLND1 of a visually prominent feature LND1 .
  • the visually prominent feature LND1 may be e.g. corner or edge of the group G1 (see Fig. 5).
  • the device 500 may be configured to classify a candidate portion of an image frame as a sub-image SUBLND1 of a visually prominent feature LND1 e.g. when the brightness and/or color of the candidate portion substantially deviates from an adjacent portion, which covers e.g. more than 20% of the image frame. Thus, it may be easy for the user C1 to visually locate said visually prominent feature LND1 .
  • a part of an original image F4 and the indicator IND2 may together form an altered image F4'.
  • the matching portion POR1 does not need to be displayed.
  • the matching portion POR1 does not need to be displayed simultaneously with the indicator IND2.
  • the user interface UIF1 may be arranged to display a selective indicator IND2, which indicates the location of the matching item PM1 with respect to an aiming line AIM1 (Fig. 5).
  • the selective indicator IND2 may be e.g. an arrow symbol, whose orientation depends on the position of the matching portion POR1 .
  • the user interface may be arranged to indicate the position of the matching item PM1 with respect to a nominal aiming line AIM1 of the device 500.
  • the aiming line AIM1 may e.g. coincide with the optical axis of the imaging optics 200.
  • the aiming line AIM1 may be e.g. substantially perpendicular to the surface of a display screen of the seeking device 500.
  • the user interface UIF1 may be configured to display information INFO1 , which specifies the model REF1 selected by the user C1 .
  • the information INFO1 may be an image of the model REF1 .
  • the user interface UIF1 may be configured to display an indicator IND2, which points at the location of the matching portion POR1 .
  • the user interface UIF1 may be configured to display a selective indicator IND1 , which selectively indicates the position of the portion POR1 .
  • the user interface UIF1 may display an embedded selective indicator IND1 , which highlights the position of the matching portion POR1 .
  • step 7a shows method steps for seeking a matching item PM1 by using an image frame F1 , which is captured after the model REF1 has been selected.
  • the image frame F1 may represent the most accurate and up-to-date information which is available regarding the location of the matching item PM1 .
  • reference data REFDATA1 associated with the selected model REF1 may be stored in a memory MEM1 .
  • the reference data REFDATA1 may be obtained from a server 1240 (Fig. 10a, 10c), and the reference data REFDATA1 may be communicated to the seeking device 500 via the Internet.
  • an image frame F1 representing a group G1 of items P1 , P2, P3 may be captured by using the image sensor 100 of the seeking device 500.
  • the image frame F1 may be captured by using the image sensor 100 after the model REF1 has been selected.
  • the image frame F1 may be captured by using the image sensor 100 after the reference data REFDATA1 has been communicated to the seeking device 500.
  • the image analysis unit 400 may determine whether at least one portion of the image frame F1 matches with the model REF1 . If a matching portion is not found, a next image frame may be analyzed. In step 840, the image analysis unit 400 may determine the position u1 , v1 of the matching portion POR1 .
  • the user interface UIF1 may provide a selective indicator IND1 , which indicates the location of the matching item PM1 .
  • the user interface UIF1 may display an altered image, which contains a selective indicator dependent on the selection of the model.
  • Fig. 7b shows method steps for seeking a matching item PM1 by using one or more previously captured image frames.
  • the user interface UIF1 may rapidly indicate the location of the matching item PM1 without a need to capture additional image frames.
  • the user C1 does not need to move around in the shop in order to capture additional image frames.
  • the information contained in the image frame F1 may sometimes be more erroneous when compared with the case of Fig. 7a. For example, someone may have removed a last matching item PM1 from the group G1 after the image frame F1 was captured.
  • the seeking device 500 would indicate the previous location of the matching item PM1 based on information derived from the image frame F1 , said indication may be erroneous because the matching item PM1 is not any more at the indicated location.
  • reference data REFDATA1 associated with the selected model REF1 may be stored in a memory MEM1 .
  • the reference data REFDATA1 may be obtained from a server 1240 (Fig. 10a, 10c), and the reference data REFDATA1 may be communicated to the seeking device 500 via the Internet.
  • an image frame F1 representing a group G1 of items P1 , P2, P3 may be retrieved from a memory MEM2.
  • the image frame F1 may have been previously captured by using the image sensor 100 of the seeking device 500.
  • the image frame F1 may have been captured by using the image sensor 100 before the model REF1 has been selected.
  • the image frame F1 may have been captured by using the image sensor 100 before the reference data REFDATA1 has been communicated (COM1 ) to the seeking device 500.
  • the image analysis unit 400 may determine whether at least one portion of the image frame F1 matches with the model REF1 . If a matching portion is not found, a next image frame may be captured.
  • the image analysis unit 400 may determine the position u1 , v1 of the matching portion POR1 .
  • the user interface UIF1 may provide a selective indicator IND1 , which indicates the location of the matching item PM1 .
  • the user interface UIF1 may display an altered image, which contains a selective indicator, which is dependent on the selection of the model.
  • Fig. 8a shows several (i.e. two or more) image frames F1 , F2, F3, F4 captured by the image sensor 100 at different times t1 , t2, t3, t4.
  • Each image frame may be associated with a different time stamp (e.g. 19:20:01 ).
  • the temporal difference between the times may be selected such that at least one common feature appears in two or more consecutive image frames.
  • the feature SUB4 may appear in the image frames F1 and F2
  • the feature SUB5 may appear in the image frames F2 and F3
  • the feature SUBLND1 may appear in the image frames F3, F4.
  • the image sensor 100 may be configured to capture a video sequence, which comprises two or more image frames F1 , F2, F3, F4.
  • the seeking device 500 may be configured to combine image data of two or more image frames F1 , F2, F3, F4 to form a combined image frame COMBI1 .
  • the image frame COMBI1 may be e.g. a panorama image.
  • the image frame COMBI1 may be a panorama image, which shows e.g. all items of a shelf or which shows items of several shelves.
  • the seeking device 500 may be configured to form the combined image frame COMBI1 by stitching image data from two or more image frames F1 , F2.
  • the common features SUB4, SUB5, SUBLND1 may be used to provide reference positions for the stitching operation.
  • the seeking device 500 may be configured to use the combined image frame COMBI1 e.g. for determining the position of the matching portion POR1 with respect to a second feature SUBLND1 , even though the matching portion POR1 and the second feature SUBLND1 did not originally appear in the same image frame.
  • the matching portion POR1 appears in the image frames F1 and F2 but not in the image frame F3.
  • the second feature e.g. visually prominent feature
  • Fig. 9a shows, by way of example, an image displayed on a screen of the user interface UIF1 .
  • the displayed image may contain e.g. information INFO2, which specifies that the device is operating in the seeking mode.
  • the displayed image may contain e.g. information INFO3, which specifies that the displayed image represents real time situation.
  • the displayed image may contain information INFO4, which specifies that a matching portion POR1 is found at an image frame (F2), which was captured earlier.
  • the matching portion POR1 may be found at an image frame, which was captured one second earlier.
  • the device 500 may be configured to provide an indication that indicates that a matching portion POR1 is found.
  • the user interface UIF1 may be configured to display the earlier image frame F2 or an altered earlier image frame F2'.
  • the user interface UIF1 may comprise a touch screen.
  • the image frame F2' may be displayed e.g. by touching a key KEY3 or KEY4.
  • the user interface UIF1 may be configured to directly display an altered image frame F2' when the key KEY3 has been touched.
  • the altered image frame F2' may comprise a selective indicator IND1 , which shows the position of the matching portion POR1 .
  • the user interface UIF1 may be configured to display a video sequence when the key KEY4 has been touched. Touching the key KEY4 may enable showing one or more previous image frames in temporally reversed order.
  • the video sequence may show the positional relationship of the matching portion POR1 with respect to a second feature (e.g. with respect to the sub-image SUBIMG1 of the visually prominent feature LND1 ).
  • Fig. 9b shows an altered image frame F2', which comprises a selective indicator IND1 and a portion of the image frame F2.
  • Information INFO2 may specify that the device is operating in the seeking mode.
  • Information INFO3, may specify that the displayed image represents an earlier situation.
  • Information INFO4 may specify that the matching portion POR1 is found in an image frame (e.g. in F2), which was captured earlier, e.g. one second earlier.
  • Touching a key KEY4 may enable playing the video sequence in temporally reversed order.
  • Touching a key KEY5 may enable playing the video sequence in the normal order (i.e. forward).
  • Touching a key KEY4 may enable playing the image frames of a video sequence in temporally reversed order (i.e. backwards).
  • Playing the video sequence in the normal order and/or in the reverse order may make it easier for the user C1 to perceive the relative position of the matching item PM1 with respect to another item. Playing the video sequence in the normal order and/or in the reverse order may make it easier for the user C1 to perceive the relative position of the matching item PM1 with respect to the visually prominent feature LND1 .
  • Fig. 10a shows, by way of example, a distributed system 1000 for assisting in seeking a matching item.
  • the system 1000 may comprise a plurality of devices arranged to communicate with each other.
  • the system 1000 may comprise a seeking device 500.
  • the seeking device 500 may be portable.
  • the seeking device 500 may comprise a user interface UIF1 for providing selective indication IND1 about the location of the matching item.
  • the seeking device 500 may comprise an image sensor 100 for capturing an image frame F1 representing a group G1 of items P1 , P2, P3.
  • the image sensor 100 may receive light LBX reflected and/or emitted from the items P1 , P2, P3.
  • the seeking device 500 or a server 1240, 1241 , 1242 may comprise one or more data processors configured to compare the model REF1 with image data SDATA1 .
  • the system 1000 may comprise end-user devices such as one or more portable devices 500, mobile phones or smart phones 1251 , Internet access devices (Internet tablets), personal computers 1260, a display or an image projector 1261 (e.g. a television), and/or a video player 1262.
  • a mobile phone, a smart phone, an Internet access device, or a personal computer may comprise an image sensor for providing reference data REFDATA1 , which may represent a model REF1 .
  • a server, a mobile phone, a smart phone, an Internet access device, or a personal computer may be arranged to distribute reference data REFDATA1 representing a model REF1 .
  • Distribution and/or storing reference data REFDATA1 may be implemented in the network service framework with one or more servers 1240, 1241 , 1242 and one or more user devices.
  • the different devices of the system 1000 may be connected via a fixed network 1210 such as the Internet or a local area network (LAN).
  • the devices may be connected via a mobile communication network 1220 such as the Global System for Mobile communications (GSM) network, 3rd Generation (3G) network, 3.5th Generation (3.5G) network, 4th Generation (4G) network, Wireless Local Area Network (WLAN), Bluetooth®, or other contemporary and future networks.
  • GSM Global System for Mobile communications
  • 3G 3rd Generation
  • 3.5G 3.5th Generation
  • 4G 4th Generation
  • WLAN Wireless Local Area Network
  • Bluetooth® Wireless Local Area Network
  • a network (1210 and/or 1220) may comprise network elements such as routers and switches to handle data (not shown).
  • a network may comprise communication interfaces such as one or more base stations 1230 and 1231 to provide access for the different devices to the network.
  • the base stations 1230, 1231 may themselves be connected to the mobile communications network 1220 via a fixed connection 1276 and/or via a wireless connection 1277.
  • There may be a number of servers connected to the network.
  • a server 1240 for providing a network service such as a social media service may be connected to the network 1210.
  • a second server 1241 for providing a network service may be connected to the network 1210.
  • a server 1242 for providing a network service may be connected to the mobile communications network 1220.
  • the servers 1240, 1241 , 1242 may be arranged such that they make up the Internet with the communication elements residing in the network 1210.
  • the devices 500, 1251 , 1260, 1261 , 1262 can also be made of multiple parts.
  • One or more devices may be connected to the networks 1210, 1220 via a wireless connection 1273.
  • Communication COM1 between a device 500 and a second device of the system 1000 may be fixed and/or wireless.
  • One or more devices may be connected to the networks 1210, 1220 via communication connections such as a fixed connection 1270, 1271 , 1272 and 1280.
  • One or more devices may be connected to the Internet via a wireless connection 1273.
  • One or more devices may be connected to the mobile network 1220 via a fixed connection 1275.
  • a device 500, 1251 may be connected to the mobile network 1220 via a wireless connection COM1 , 1279 and/or 1282.
  • the connections 1271 to 1282 may be implemented by means of communication interfaces at the respective ends of the communication connection.
  • a user device 500, 1251 or 1260 may also act as web service server, just like the various network devices 1240, 1241 and 1242. The functions of this web service server may be distributed across multiple devices.
  • Application elements and libraries may be implemented as software components residing on one device. Alternatively, the software components may be distributed across several devices. The software components may be distributed across several devices so as to form a cloud.
  • the reference data REFDATA1 may be obtained from social media.
  • the reference data REFDATA1 may be obtained by using a social media service.
  • the reference data REFDATA1 of a model REF1 may be received from a social media service.
  • the model REF1 may be selected by using a social media service.
  • the social media service may also be called e.g. as a social networking service.
  • the social media service may be implemented e.g. on one or more servers 1240.
  • the seeking device 500 may be carried by user C1 .
  • a second person may be associated with the user by a link in the social media service.
  • the second person may be associated with one or more models REF1 based on information provided by the social media service.
  • the second person may e.g. inform that he is using a specific model REF1 .
  • the second person may e.g. recommend a specific model REF1 .
  • the model REF1 may be selected from among one or more candidate models listed in the social media service.
  • the user interface UIF1 of the seeking device 500 may be e.g.
  • the reference data REFDATA1 of the model may be subsequently retrieved from the social media service.
  • the social media service may provide a link to a second data distribution service, which may also be implemented on one or more servers.
  • the reference data REFDATA1 of the model may also be retrieved from the second data distribution service.
  • the reference data REFDATA1 may be communicated to the seeking device e.g. via the communication unit RXTX1 of the device 500.
  • a social media service may be a web-based online service, which may provide a platform for a first person (e.g. the user C1 of the device 500) to:
  • the social media service may provide pictures of the other persons.
  • the social media service may provide pictures of models associated with the other persons.
  • the profile may contain information specifying e.g. age, location, and/or interests of the first person.
  • the profile may contain information specifying e.g. age, location, and/or interests of the second person.
  • the profile may contain information specifying one or more models REF1 associated with the second person.
  • the term public profile or a semi-public means that two or more other persons may view the profile of the first person and view the first list of connections.
  • the social networks may have an access control to enable and/or disable viewing the profile of the first person and the first list of connections.
  • the access control may utilize e.g. password.
  • the method for indicating the position of the desired item may comprise obtaining the reference data from browser cache.
  • the method may comprise obtaining the reference data from browsing history data.
  • the user C1 or a second person may visit one or more web pages (e.g. in the Internet) by using a web browser.
  • One or more of the web pages may contain one or more images of a model.
  • the web browser may be implemented in a browsing device.
  • the browsing device may be e.g.
  • the device 500 may be configured to store browsing history data in a memory.
  • the browsing history data may also be called e.g. as web search history data.
  • the browsing history data may comprise REFDATA1 associated with a model.
  • the device 500 may be configured to store reference data REFDATA1 associated with a model in a memory.
  • the device 500 may comprise said memory.
  • the reference data REFDATA1 associated with the model may be subsequently retrieved from the memory e.g. in a situation where connection to the Internet is not available.
  • Fig. 10b shows a portable seeking device 500, which may be configured to capture an image frame F1 representing a group G1 of items, and to provide a selective indication IND1 of the location of the matching item PM1 .
  • the device 500 may be e.g.
  • the device 500 may comprise an image sensor 100 providing image data SDATA1 .
  • the device 500 may comprise a communication unit RXTX1 , a user interface UIF1 , a memory MEM1 for storing reference data REFDATA1 representing the selected model REF1 , one or more processors CNT1 , 400, a memory MEM2 for storing an image frame F1 representing a group G1 of items, a memory MEM3 for storing computer program PROG1 .
  • the device 500 may comprise a communication unit RXTX1 for receiving reference data REFDATA1 wirelessly e.g. from the Internet 1210, and/or from a mobile telephone network 1220.
  • the portable device 500 may comprise a communication module RXTX1 for receiving the reference data REFDATA via the Internet based on a selection of the model REF1 .
  • the units and functionalities of the device 500 may be implemented as described e.g. in the context of Figs. 1 b, 2a, 2b, 3, 4, 5, 6a, 6b, 6c, 6d, 7a, 7b, 8a, 8b, 9a, 9b and/or 10a.
  • the device 500 may optionally comprise one or more microphones 1258 for converting sound waves into audio signals.
  • the device 500 may further comprise one or more speakers 1255 for reproducing audio signals.
  • a microphone 1258 may be used e.g. to implement a mobile phone functionality.
  • the location of the matching item PM1 may be indicated by an audio signal.
  • the user interface UIF1 may be arranged to provide an audio signal based on the position u1 ,v1 of the matching portion POR1 .
  • the seeking device 500 may be configured to detect and indicate when the aiming line AIM1 (Fig. 5) is moving closer to (or farther away from) the location of the matching item PM1 .
  • a frequency of an audio signal may depend on the distance between the location of the matching item PM1 and the aiming line AIM1 .
  • the user may manually move and/or change the orientation of the seeking device 500 so that he can find the location of the matching item PM1 by listening to the variation of an audio signal provided by the seeking device 500.
  • the seeking device 500 may be configured to play a predetermined tone or song when the aiming line AIM1 is getting closer to the location of the matching item PM1 .
  • the seeking device 500 may be configured to provide a voice indication about the location of the matching item PM1 e.g. by playing pre-recorded verbal instructions or by using a speech synthesizer. For example, the device 500 may play a voice signal "move to the left".
  • the seeking device 500 may be configured to provide a haptic indication about the location of the matching item PM1 .
  • the user interface UIF1 may be arranged to provide a haptic indicator based on the position u1 ,v1 of the matching portion POR1 .
  • the seeking device 500 may be configured to e.g. vibrate when the aiming line AIM1 is getting closer to the location of the matching item PM1 . In an embodiment, the seeking device 500 may be configured to vibrate when the aiming line AIM1 is moving farther away from the location of the matching item PM1 .
  • Fig. 10c shows a server 1240, which may comprise a memory 1245 (MEM), one or more processors 1246, 1247 (PROC), and computer program code 1248 (PROGRAM) residing in the memory 1245 for implementing, for example, a service for distributing reference data REFDATA1 .
  • Reference data REFDATA1 for several candidate models may be stored in a memory MEM4 of the server 1240.
  • the server 1240 may comprise a memory 1245, one or more processors 1246, 1247, and computer program code 1248 residing in the memory 1245 for implementing, for example, a social networking service.
  • the model REF1 may be selected by the user C1 of the device 500, i.e. by a person who is carrying the device 500 during capturing the image frames F1 , F2.
  • the user may search suitable candidate models e.g. by using a web browser.
  • the device 500 and/or a server 1240 may be configured to determine or propose one or models REF1 from a web search history of the user C1 .
  • the device 500 and/or a server 1240 may be configured to determine or propose one or models REF1 based on the detected location of the device 500 and/or based on the clock time. In an embodiment, the device 500 and/or a server 1240 may be configured to determine or propose one or models REF1 based on a proposal made by a shopkeeper.
  • the device 500 and/or a server 1240 may be configured to determine or propose one or models REF1 based on a recommendation presented in a social media network.
  • the device 500 and/or a server 1240 may be configured to determine or propose one or models REF1 based on age of the user, based on family status of the user and/or based on a bonus card.
  • the reference data REFDATA1 associated with a model may be retrieved from a database e.g. based on a barcode scanned by using the seeking device 500.
  • identification data identifying all items collected by the user C1 e.g. to a shopping cart
  • the seeking device 500 may be configured to communicate the list to a point of sale terminal of the shop. This may reduce the time, which will be needed at the point of sales terminal.
  • the seeking device 500 or a server 1240 may receive a list of items previously collected (and bought) by the user C1 or by another person.
  • the seeking device 500 or server 1240 may select one or more models based on said list.
  • the seeking device 500 or server 1240 may be configured to retrieve reference data REFDATA1 associated with said models.
  • the seeking device 500 may be configured to seek items, which match with said models.
  • the seeking device 500 may be configured to simultaneously seek items, which match two or more different models REF1 .
  • the seeking device 500 may store a "shopping list", which defines two or more different models REF1 .
  • the device 500 may be configured to compare sub-images of a captured image frame with first reference data REFDATA1 representing a first model REF1 , and to compare the sub-images SUB1 , SUB2 with second reference data representing a second different model.
  • the image processing apparatus 500 may comprise a user interface UIF1 for receiving commands from a user and/or for providing information to the user.
  • a model REF1 may be selected by using the user interface UIF1 .
  • the user interface UIF1 may comprise e.g. a touch screen for visually displaying information and for receiving commands from a user.
  • the user interface UIF1 may comprise hardware, e.g. a display, keypad and/or a touch screen.
  • the user interface may comprise a display screen for viewing graphical elements displayed on the screen.
  • the user interface UIF1 may comprise a software application e.g. for displaying various different virtual keys on a touch screen.
  • the user interface UIF1 may comprise one or more virtual keys for receiving instructions from the user. The virtual keys may be implemented on the touch screen.
  • the user interface UIF1 may comprise one or more push buttons for receiving instructions from the user.
  • the user interface UIF1 may comprise a keypad for receiving instructions from the user.
  • the user interface UIF1 may comprise a touchpad, a keypad, a mouse and/or a joystick for receiving instructions from the user.
  • the user interface UIF1 may comprise a microphone and a speech recognition unit for receiving instructions, which are spoken aloud.
  • the user interface UIF1 may be implemented e.g. in a portable device, e.g. in a smart phone.
  • a user may receive information via the interface UIF1 .
  • the user may control operation of the apparatus 500 by giving commands via the user interface UIF1 .
  • the user may input information by touching the touch screen with a touching member.
  • the touching member may be e.g. a finger or a stylus.
  • Touching the touch screen may refer to actual physical contact between the touching member and the screen. Touching the touch screen may also mean bringing the touching member close to the screen so that the distance between the finger H1 and the screen is smaller than a predetermined distance (e.g. smaller than 1 % of the width of the touch screen).
  • the seeking device 500 may comprise two or more image sensors 100 to capture images of a candidate item from different angles.
  • the device 500 may comprise a first image sensor and a second image sensor. Using several image sensors may e.g. increase rate of capturing the image frames from different viewpoints.
  • shape information about the three- dimensional shape of a candidate item may be determined from a first image of the candidate item taken from a first view point, and from a second image of the candidate item taken from a second view point (without moving the candidate item).
  • said shape information determined from the image data SDATA1 may be compared with reference data REFDATA1 of the model REF1 in order to determine whether a portion of the first image is a matching portion.
  • the shape information of the candidate item may be compared with the shape information of the model REF1 in order to determine whether a portion of the first image is a matching portion.
  • the shape information of the candidate item may be compared with the shape information of the model REF1 in order to determine whether the candidate item is a matching item PM1 .
  • the first image representing the first view angle and the second image representing the second view angle may be obtained from image data SDATA1 , which is provided by a single image sensor.
  • the first image may be captured when the device 500 is at a first location with respect to the candidate item, and the second image may be captured when the device is at a second location.
  • the second location may be displaced with respect to the first location e.g. in a direction, which is transverse to the aiming line AIM1 .
  • the optics 200 of the device 500 may be arranged to simultaneously form the two images on a single image sensor 100.

Abstract

A method for seeking a desired item from a group of items may comprise: -providing reference data (REFDATA1) representing a model(REF1), -capturing one or more image frames (F1, F2) of candidate items (P1, P2, P3) by using an image sensor (100), -determining whether a candidate portion (SUB1, SUB2, SUB3) of an image frame (F1) is a matching image portion (POR1) by comparing the candidate portion (SUB3) with the reference data (REFDATA1), and -providing a selective indication (IND1, IND2) about the location of a matching item (PM1) based on the position (u1,v1) of the matching image portion(POR1).

Description

METHOD AND APPARATUS FOR FINDING A LOCATION FIELD Various embodiments relate to assisting in seeking the location of a product. BACKGROUND
Referring to Fig. 1 a, consumer products may be sold in a self-service shop, where a customer may personally collect one or more tangible items e.g. to a shopping cart or a basket. The items put on sale may be displayed e.g. on shelves, cabinets, freezer compartments and/or pallets. A single shop may comprise e.g. more than one million products. Finding a particular product may be time- consuming. The number of salespersons in the shop may be low. Even if the customer could find a salesperson and ask about the location of the product that he wishes to buy, the verbal instructions given by the salesperson for finding the location of the product may be inaccurate.
SUMMARY
Some embodiments may relate to an apparatus for finding the location of a product. Some embodiments may relate to a method for finding the location of a product. Some embodiments may relate to a computer program for finding the location of a product. Some embodiments may relate to a computer program product storing a computer program for finding the location of a product. Some embodiments may relate to means for finding the location of a product.
According to a first aspect, there is provided a method comprising:
- providing reference data representing a model,
- capturing one or more image frames of candidate items by using an image sensor,
- determining whether a candidate portion of an image frame is a matching image portion by comparing the candidate portion with the reference data, and
- providing a selective indication about the location of a matching item based on the position of the matching image portion. According to a second aspect, there is provided an apparatus comprising at least one processor, a memory including computer program code, the memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform at least the following:
- providing reference data representing a model,
- capturing one or more image frames of candidate items by using an image sensor,
- determining whether a candidate portion of an image frame is a matching image portion by comparing the candidate portion with the reference data, and
- providing a selective indication about the location of a matching item based on the position of the matching image portion.
According to a third aspect, there is provided a computer program comprising computer program code configured to, when executed on at least one processor, cause an apparatus or a system to:
- provide reference data representing a model,
- capture one or more image frames of candidate items by using an image sensor,
- determine whether a candidate portion of an image frame is a matching image portion by comparing the candidate portion with the reference data, and
- provide a selective indication about the location of a matching item based on the position of the matching image portion.
According to a fourth aspect, there is provided a computer program product embodied on a non-transitory computer readable medium, comprising computer program code configured to, when executed on at least one processor, cause an apparatus or a system to:
- provide reference data representing a model,
- capture one or more image frames of candidate items by using an image sensor,
- determine whether a candidate portion of an image frame is a matching image portion by comparing the candidate portion with the reference data, and
- provide a selective indication about the location of a matching item based on the position of the matching image portion.
According to a fifth aspect, there is provided a means for seeking an item, the means for seeking comprising:
- means for providing reference data representing a model, - means for capturing one or more image frames of candidate items by using an image sensor,
- means for determining whether a candidate portion of an image frame is a matching image portion by comparing the candidate portion with the reference data, and
- means for providing a selective indication about the location of a matching item based on the position of the matching image portion.
A portable seeking device may be used to assist in finding a desired item e.g. in a shop. The user may carry the seeking device e.g. when moving in the shop. The user may wish to find an item, which may be called e.g. as a desired item. The desired item may be represented by a model . The seeking device may comprise a user interface for selecting a model . The seeking device may comprise an image sensor for capturing one or more image frames. The seeking device may be configured to analyze an image frame in order to determine whether the image frame contains a matching portion, which matches with the selected model. The seeking device may comprise one or more data processors for determining whether a candidate portion of an image frame is a matching image portion. An item matching with the model may be called as a matching item. The seeking device may be configured to indicate the location of a matching item by analyzing the captured image frames. The seeking device may be configured to indicate the location of a matching item based on the position of a matching image portion. The seeking device may comprise a user interface for providing indication about the location of a matching item. In an embodiment, the image sensor and the user interface may be implemented in the same portable device.
The user may have previously selected a model item or a model image, and he may wish to find at least one item, which substantially matches with said model item or model image. The model item or a model image may be represented by a model, which can be stored in a computer memory. The model may comprise data specifying one or more characteristic features, which may be used to differentiate the model item or a model image from other items.
Searching a desired product in the shop may be time-consuming. Finding the desired product (i.e. the desired item) by bare eyes may be difficult, because a high number of other items and advertisements may be simultaneously competing to attract the customer's attention. It may often happen that the desired product is located e.g. on a bottom shelf, wherein other items have been arranged on the most prominent places in order to promote sales of said other items. For example, the desired item may belong to a group, which comprises e.g. more than one thousand items having different visual appearance. The items may be divided in several groups, which are arranged on different shelves. Finding the desired product may require walking a long distance between shelves of the shop. Using the seeking device may substantially reduce the time needed to find the location of the matching item. A person (i.e. the user) may use a portable seeking device to help in finding the desired product in the shop faster. By using the seeking device, the user may substantially reduce the time needed to find a matching item from a group of items arranged on e.g. a shelf, cabinet, freezer compartment and/or pallet. Using the seeking device may facilitate finding the desired product from a group of items e.g. when the visual appearance of other items of the group is very similar to the desired product. The group may comprise e.g. more than one thousand products arranged on one or more shelves such that the products have substantially identical size and shape. In an embodiment, the seeking device may find the desired product based on a very small visual difference. The desired product may have e.g. a specific graphical symbol and/or a specific combination of colors, which is different from the symbols or color combinations of the other products. The seeking device may assist in finding the location of a desired product in a situation where searching the desired product by bare eyes would represent a boring monotonous work. The seeking device may assist in finding the location of a desired product in a situation where probability of finding the desired product by bare eyes would be low.
Using the seeking device may make it possible that the user can perform one or more additional tasks during seeking the desired item. For example, the user may concentrate on studying other products while the seeking device is automatically seeking the desired product. For example, the user may use an internet browser or watch a video while the seeking device is seeking the desired product. A location database of a shop may comprise item-specific location data, which associates a location with an identification code of each item. However, the database may often contain erroneous data e.g. because the personnel may have re-arranged the items to new locations without updating the database. When using the seeking device to assist in finding the desired product, it is not necessary to rely on data stored previously stored in the location database. Using the seeking device may facilitate finding the desired product e.g. when the user has a poor vision (e.g. myopia, hyperopia, cataracts). Using the seeking device may facilitate finding the desired product e.g. when the user cannot verbally describe the desired product. Using the seeking device may facilitate finding the desired product e.g. when the number of personnel in the shop is so low. Using the seeking device may facilitate finding the desired product e.g. when the personnel is not familiar with the locations of the products. Using the seeking device may facilitate finding the desired product e.g. when the user is hearing- impaired or cannot speak the language of the personnel. In an embodiment, the seeking device may be arranged to pinpoint the location of the desired product.
BRIEF DESCRIPTION OF THE DRAWINGS In the following examples, various embodiments will be described in more detail with reference to the appended drawings of example embodiments, in which
Fig. 1 a shows, in a side view, a group of items, Fig. 1 b shows, in a three-dimensional view, a person using a seeking device,
Fig. 2a shows, by way of example, a model,
Fig. 2b shows, by way of example, reference data representing a model, shows, by way of example, enabling a seeking mode by using a user interface,
Fig. 4 shows, by way of example, functional units of the seeking device, Fig. 5 shows, by way of example, capturing an image of several items, Fig. 6a shows, by way of example, indicating the position of a matching image portion,
Fig. 6b shows, by way of example, indicating the position of a matching image portion,
Fig. 6c shows, by way of example, indicating the position of a matching image portion, Fig. 6d shows, by way of example, indicating the position of a matching image portion,
Fig. 7a shows, by way of example, method steps for seeking a matching item, by using real-time imaging,
Fig. 7b shows, by way of example, method steps for seeking a matching item, by using image frames retrieved from a memory,
Fig. 8a shows, by way of example, several image frames captured at consecutive times,
Fig. 8b shows, by way of example, a combined image frame formed by stitching several image frames together, Fig. 9a shows, by way of example, information provided by a user interface,
Fig. 9b shows, by way of example, information provided by a user interface,
Fig. 10a shows, by way of example, a system for assisting in the seeking,
Fig. 10b shows, by way of example, a seeking device, and
Fig. 10c shows, by way of example, a server for assisting in the seeking. DETAILED DESCRIPTION
Referring to Fig. 1 b, a plurality of tangible items P1 , P2, P3, P4, P21 , ..., P31 , ..., P41 , ... may be arranged in one or more groups G1 , G2, G3, G4 so that a person C1 may see the items. The items may be put on display e.g. in a shop SHOP1 . The shop SHOP1 may be e.g. a supermarket, market, mart, store, or mall. The items may be put on display on market square. The items may be put on display e.g. in a building. The person C1 may have previously selected a model REF1 , and he may wish to find at least one item, which substantially matches the model REF1 . An item which substantially matches the model REF1 may be called as a matching item PM1 . In an embodiment, the person C1 does not know the location x,y,z, of the matching item PM1 when the search begins.
The model REF1 may have one or more characteristic visual features, e.g. one or more graphical symbols (e.g. a star), and/or a certain geometrical shape (e.g. rounded corners). The model REF1 may comprise one or more markings (e.g. "A14"). The model REF1 may have a marking which represents e.g. a product code and/or a trade name.
The matching item PM1 may belong e.g. to a group G1 , which comprises more than 10 items. The matching item PM1 may belong e.g. to a group G1 , which comprises more than 100 items. The matching item PM1 may belong e.g. to a group G1 , which comprises more than 1000 items. The group G1 may comprise only one matching item PM1 , or the group G1 may comprise several matching items PM1 , wherein the group may further comprise several other items such that the visual appearance of the other items is different from the visual appearance of the matching item PM1 . The group G1 may comprise one or more matching items PM1 , which have a first visual appearance, and the group G1 may comprise one or more other items, which have a second visual appearance.
In an embodiment, a group (e.g. the group G2) does not comprise any matching items PM1 .
The person C1 may use a seeking device 500 to assist in finding the location x,y,z of a matching item PM1 . The seeking device 500 may comprise a camera, which is arranged to capture one or more image frames F1 , F2 (see e.g. Fig. 5). The image frame F1 , F2 may be analyzed in order to determine whether the image frame comprises an image portion, which matches with the model REF1 . The seeking device 500 may be arranged:
- to capture an image frame F1 of a group G1 of items P1 , P2, P3, P4,
- to compare the captured image frame F1 with the model REF1 in order to determine whether a portion of the image frame is a matching portion, which matches with the model REF1 , and
- to indicate the location of the matching item PM1 based on the position of the matching portion.
The seeking device 500 may be arranged to e.g. display an arrow symbol IND2, which at least approximately indicates the location of the matching item PM1 .
Using the seeking device 500 may optionally comprise moving the seeking device 500 with respect to the group G1 e.g. along a path PTH1 so that image data may be gathered from a wide area. Moving the seeking device 500 may comprise changing the position of the seeking device 500 with respect to the group G1 and/or changing the orientation of the seeking device 500 with respect to the group G1 . In an embodiment, the seeking device 500 does not need to be moved.
SX, SY, SZ denote orthogonal directions. The location may be expressed e.g. by using position coordinates x,y,z. The coordinate x may denote a coordinate specifying a position in the direction SX. The coordinate y may denote a coordinate specifying a position in the direction SY. The coordinate z may denote a coordinate specifying a position in the direction SZ. The location may be defined e.g. with respect to an origin ORIG1 . The origin ORIG1 may be defined e.g. with respect to a visually prominent feature LND1 . The visually prominent feature may also be called e.g. as a landmark feature LND1 . The visually prominent feature LND1 may be e.g. a corner of a shelf and/or an edge of a group G1 of items. In an embodiment, the origin ORIG1 may coincide with the visually prominent feature LND1 . Referring to Fig. 2a, the model REF1 may have one or more characteristic visual features, e.g. a graphical symbol REFSYM1 , a shape REFSHAPE1 , and/or a marking REFID1 . The marking REFID1 may represent e.g. trade name or a product code.
Information about the visual features may be contained in reference data REFDATA1 associated with the model REF1 . The model REF1 may be represented by the reference data REFDATA1 . The reference data REFDATA1 may comprise e.g. an image representing a model REF1 . The reference data REFDATA1 may comprise e.g. a (digital) image of a model item. The reference data REFDATA1 may comprise e.g. an image of a two-dimensional model image, wherein the model image may be e.g. a graphical symbol appearing on a web page. The model image may be e.g. a graphical symbol, which has been printed on a product. The reference data REFDATA1 may comprise compressed data relevant for image recognition. Determining whether a portion of an image frame is a matching image portion or not may comprise:
- determining a matching index value, which depends on the number of those characteristic visual features of the model REF1 , which are detected to substantially match with said portion of the image frame,
- comparing the matching index value with a threshold value, and
- classifying the portion of the image frame as a matching image portion when the matching index value exceeds the threshold value.
Determining whether a portion of an image frame is a matching image portion or not may comprise:
- using an image recognition algorithm, which provides an index value indicative of the degree of matching between the portion of the image frame and the model REF1 ,
- comparing the index value with a threshold value, and
- classifying the portion of the image frame as a matching image portion when the index value exceeds the threshold value.
If needed, the threshold value may be adjusted e.g. if the device 500 provides erroneous indications, and/or if no matching portion is found. The seeking method may comprise adjusting the threshold value. The device 500 may be configured to adjust the threshold value based on input provided by a user C1 . Referring to Fig. 2b, the reference data REFDATA1 may represent a three- dimensional model REF1 . The reference data REFDATA1 may comprise e.g. two or more images of a three dimensional model item taken from different angles. A first image may be an image of a first side of the model item, and a second image may be an image of a second side of the model item. The reference data REFDATA1 may comprise a numerical model representing the three-dimensional shape of a model REF1 . The reference data REFDATA1 may comprise e.g. a video sequence, which comprises several images representing the model REF1 from different angles. The model REF1 may be represented e.g. a point cloud, which may define a three-dimensional reference shape. The reference data REFDATA1 representing the model REF1 may comprise several reference images RIMG1 , RIMG2, RIMG3 of a three-dimensional model REF1 , wherein each of said images RIMG1 , RIMG2, RIMG3 may represent a different view direction specified by a different view angle CM , <¾, <¾.
The model REF1 may be located at an origin ORIG3 in a coordinate system defined by the (orthogonal) directions SX', SY, SZ'. The reference image RIMG1 may be an image of the model REF1 when viewed from a view point VP1 . The image RIMG2 may be an image of the model REF1 when viewed from a view point VP2. The image RIMG3 may be an image of the model REF1 when viewed from a view point VP3. The orientation and the location of the model REF1 may be fixed in the coordinate system (SX', SY, SZ'). The viewpoint VP1 and the origin ORIG3 may define a view line VLIN1 . The viewpoint VP2 and the origin ORIG3 may define a view line VLIN2. The viewpoint VP3 and the origin ORIG3 may define a view line VLIN3. The angle between the view line VLIN1 and the direction SX' may be specified by the view angle CM, the angle between the view line VLIN2 and the direction SX' may be specified by the view angle <¾ the angle between the view line VLIN3 and the direction SX' may be specified by the view angle <¾. Two or more images RIMG1 , RIMG2, RIMG3 may be simultaneously stored in a memory. Reference data REFDATA1 , which comprises two or more images RIMG1 , RIMG2, RIMG3 may be stored in a memory.
In an embodiment, a point cloud representing the model REF1 may be stored in a memory, and the images RIMG1 , RIMG2, RIMG3 may be generated from the point cloud according to a view angle. The view angle may be varied in order to generate two or more different images RIMG1 , RIMG2, RIMG3. Thus, it is not necessary to simultaneously store the images RIMG1 , RIMG2, RIMG3 in a memory.
One or more of the images RIMG1 , RIMG2, RIMG3 may be rotated, magnified and/or stretched in a selected direction to provide one or more reference images, which may be used for determining a matching portion.
The reference data REFDATA1 may comprise a first image RIMG1 of the model REF1 when viewed from a first view direction CM, and a second image RIMG2 of the model REF1 when viewed from a second view direction <¾, which is different from the first view direction CM . This may facilitate finding the matching item PM1 a situation where the candidate items of the group G1 may have arbitrary orientation (i.e. random or unknown orientation) with respect to the seeking device 500 during capturing image frames F1 , F2. The reference data REFDATA1 may comprise e.g. an image RIMG1 of a first side of a model, and also an image RIMG2 of a second side of said model.
The method for indicating the position of the desired item PM1 may comprise:
- determining a first reference image RIMG1 from the reference data REFDATA1 , - determining a second reference image RIMG2 from the reference data
REFDATA1 ,
- determining whether a candidate portion SUB1 , SUB2, SUB3 (see Fig. 5) of an image frame matches F1 with the first reference image RIMG1 ,
- determining whether a candidate portion SUB1 , SUB2, SUB3 of an image frame matches F1 with the second reference image RIMG1 , and
- determining that the candidate portion SUB3 is a matching portion POR1 when the candidate portion SUB3 matches with at least one of the first reference image RIMG1 and the second reference image RIMG2,
wherein the first reference image RIMG1 represents an image of the model taken from a first view point VP1 , and the second reference image RIMG2 represents an image of the model taken from a second different view point VP2.
The model REF1 may be selected e.g. from among candidate models stored in a database. The database may be stored e.g. in a memory contained in the seeking device 500. The database may be stored e.g. in an internet server 1240 (See Fig. 10a, 10c). The reference data REFDATA1 associated with the model REF1 may be retrieved from the database. In an embodiment, the database may be accessed from a remote location. The reference data REFDATA1 may be retrieved e.g. from a server by using the Internet. A remote database stored in a server and a local database stored in a portable device 500 may be synchronized. Reference data REFDATA1 may be added to the database e.g. by a manufacturer of a product, by a shopkeeper, by an internet community and/or by the user C1 .
The method for indicating the position of the desired item may comprise:
- selecting the model, and
- receiving the reference data of the model from a server via the Internet.
In an embodiment, the image sensor 100 of the seeking device 500 may capture an image of a model item. The reference data REFDATA1 may be determined from the image, and the reference data REFDATA1 may be stored in a memory of the seeking device 500 and/or in a database to be used as the reference data REFDATA1 representing the model REF1 .
Referring to Fig. 3, the model REF1 may be selected e.g. by using a user interface UIF1 of the seeking device 500. The user C1 may select the model REF1 e.g. from among several candidate models displayed on a web site. The user interface UIF1 may comprise e.g. a display, which is configured to display an image of the selected model REF1 . The user interface UIF1 may comprise e.g. a key KEY1 for enabling operation in a seeking mode. In an embodiment, the user C1 may enable operation in a seeking mode e.g. by touching the key KEY1 with his hand H1 . In particular, the key KEY1 may be touched with a finger. The key KEY1 may be optionally associated with displayed symbol, e.g. with a displayed word "SEEK".
The user interface UIF1 may optionally comprise a second key KEY2 e.g. for enabling some other functionality of the device 500. In particular, the key KEY2 may be e.g. a home button, and touching the key KEY2 may enable displaying an initial menu or a home menu.
Referring to Fig. 4, the seeking device 500 may comprise an image sensor 100 for providing image data SDATA1 , a memory MEM1 for storing a model data REFDATA1 , and an image analysis unit 400 for comparing the model data REFDATA1 with the image data SDATA1 . The image sensor 100 may comprise e.g. a two-dimensional array of light- detecting detector pixels, which cover the active area 101 of the image sensor 100. The image sensor 100 may convert the optical image IMG1 into a digital image. The image sensor 100 may comprise e.g. a CMOS array or a CCD array.
The seeking device 500 may comprise imaging optics 200, which may be arranged to form an optical image IMG1 on the image sensor 100. The imaging optics 200 may form the optical image IMG1 by focusing light LBX reflected from items P1 , P2, P3, P4 to the image sensor 100.
The seeking device 500 may comprise a memory MEM2 for storing the image data SDATA1 . The image data SDATA1 may comprise one or more image frames F1 , F2. The image frames F1 , F2 may also be called e.g. as a digital images.
The seeking device 500 may comprise a user interface UIF1 for receiving instructions from a user C1 and/or for providing information to the user C1 .
The seeking device 500 may comprise a control unit CNT1 for controlling operation of the device 500. The control unit CNT1 may comprise one or more data processors for controlling operation of the device 500. In an embodiment, the control unit CNT1 may comprise the image analysis unit 400. The image analysis unit 400 may be a part of the control unit CNT1 . In an embodiment, the control unit CNT1 and the image analysis unit 400 may be implemented on the same semiconductor chip.
The seeking device 500 may comprise a memory MEM3, which may store computer program PROG1 for executing the steps of the seeking method. The computer program PROG1 may comprise computer program code configured to, when executed on at least one processor, cause an apparatus 500 or a system 1000 to:
- provide reference data REFDATA1 representing a model REF1 ,
- capture one or more image frames F1 , F2 of candidate items P1 , P2, P3 by using an image sensor 100, - determine whether a candidate portion SUB1 , SUB2, SUB3 of an image frame F1 is a matching image portion POR1 by comparing the candidate portion SUB3 with the reference data REFDATA1 , and
- provide a selective indication IND1 , IND2 about the location of a matching item based on the position (u1 ,v1 ) of the matching image portion POR1 .
A computer program product embodied on a non-transitory computer readable medium (MEM3) may comprise the computer program PROG1 . The seeking device 500 may comprise a communication unit RXTX1 arranged to receive data from a web (e.g. Internet), local area network and/or mobile communications network. The communication unit RXTX1 may comprise e.g. a radio frequency receiver, a radio frequency transmitter, and an antenna. For example reference data REFDATA1 may be received by using the communication unit RXTXL
Referring to Fig. 5, the imaging optics 200 may be arranged to form an optical image IMG1 of several items P1 , P2, P3 on the image sensor 100. The combination of the imaging optics 200 and the image sensor 100 may have a field of view CAMVIEW1 . The image sensor 100 may convert the optical image IMG1 into a digital image F1 . The image sensor 100 may provide image data SDATA1 , which may comprise an image frame F1 . The image frame F1 may comprise sub- images SUB1 , SUB2, SUB3, SUB4 of the items P1 , P2, P3, P4. The sub-images may also be called e.g. as candidate portions of the image frame. The sub-image SUB1 may be a candidate portion SUB1 . The sub-image SUB2 may be a candidate portion SUB2. The sub-image SUB3 may be a candidate portion SUB3. The items P1 , P2, P3, P4 may be called as candidate items.
The candidate items P1 , P2, P3, P4 may comprise one or more characteristic graphical features (for example markings A01 , A02, A03..., A78, a symbol resembling a rectangle, a star or the Sun). The candidate items P1 , P2, P3, P4 may comprise a graphical symbol (e.g. the symbol %) to indicate additional information, e.g. to indicate a discounted price. The image analysis unit 400 may be configured to compare the model REF1 with the image frame F1 in order to determine whether at least one sub-image SUB1 , SUB2, SUB3, SUB4 of the image frame F1 matches with the model REF1 . When the image analysis unit 400 detects that a sub-image SUB3 substantially matches with the model REF1 , then the matching sub image SUB3 may be classified to be a matching portion POR1 . When the sub-image of an item P3 is classified to be a matching portion POR1 , then said item P3 may be classified to be matching item PM1 . In other words, a candidate item P3 may be classified to be a matching item PM1 e.g. when the sub-image SUB3 of said candidate item P3 is detected to match with the reference data REFDATA1 .
The image analysis unit 400 may be configured to determine the position of the matching portion POR1 . The position of the matching portion POR1 may be defined e.g. by using position coordinates u1 , v1 . The position of the matching portion POR1 may be expressed e.g. with respect to an origin ORIG2. The origin ORIG2 may be e.g. at a corner of the image frame F1 , at the center of the image frame F1 , or at a visually prominent feature ("landmark") appearing in the image frame F1 . The image frame F1 may have a width umax and a height vmax. The width umax and the height vmax may be expressed e.g. as a number of pixels. For example the width umax may be 2560 pixels, and the height may be 1600 pixels. The position coordinates u1 , v1 may be expressed e.g. as a number of pixels (e.g. u1 =1000 pixels) or a fractional numbers (e.g. u1 = 39% of the width umax).
The center of the optics 200 and the center of the image sensor 100 may together define a nominal aiming line AIM1 of the seeking device 500. The center of the optics 200 and the position of a predetermined pixel of the image sensor 100 may together define a nominal aiming line AIM1 of the seeking device 500. The predetermined pixel may be selected such that the nominal aiming line AIM1 is e.g. substantially perpendicular to a display screen of the device 500. The nominal aiming line AIM1 may have a fixed orientation with respect to a housing of the portable seeking device 500 so that the aiming line AIM1 may be moved together with the portable seeking device 500.
The seeking device may comprise an image sensor 100 for capturing the one or more image frames F1 , F2. The image sensor 100 may be arranged to capture an image frame F1 by capturing a still image. The image sensor may be arranged to capture several image frames F1 , F2 by capturing several still images. The image sensor 100 may be arranged to capture an image frame F1 by capturing a video sequence. The image sensor 100 may be arranged to capture several image frames F1 , F2 by capturing a video sequence. The seeking device 500 may comprise:
- a memory MEM1 for storing reference data REFDATA1 , which represents a model REF1 ,
- an image sensor 100 configured to capture an image frame F1 , which comprises sub-images SUB1 , SUB2 of items P1 , P2,
- at least one processor CNT1 , 400 configured to determine whether a portion of an image frame F1 is a matching image portion POR1 by comparing the sub- images SUB1 , SUB2 with the reference data REFDATA1 , and to detect the position u1 , v1 of the matching image portion, and
- a user interface UIF1 configured to provide a selective indication IND1 , IND2 about the location (x,y,z) of a matching item PM1 based on the detected position u1 , v1 of the matching image portion POR1 . The seeking device 500 may be configured to:
- provide reference data REFDATA1 representing a model REF1 ,
- capture one or more image frames F1 , F2 of candidate items P1 , P2, P3 by using an image sensor 100,
- determining whether a candidate portion SUB1 of an image frame F1 is a matching image portion POR1 by comparing the candidate portion SUB1 with the reference data REFDATA1 , and
- providing a selective indication IND1 , IND2 about the location of a matching item PM1 based on the position u1 , v1 of the matching image portion POR1 . In case of a three-dimensional model REF1 (Fig. 2b), the seeking device 500 may be configured to
- determine a first reference image RIMG1 from the reference data REFDATA1 of the model REF1 ,
- determine a second reference image RIMG2 from the reference data REFDATA1 ,
- determine whether a candidate portion SUB1 , SUB2 of an image frame F1 matches with the first reference image RIMG1 ,
- determine whether a candidate portion SUB1 , SUB2 of an image frame F1 , F2 matches with the second reference image RIMG2, and
- determine that a candidate portion SUB1 is a matching portion POR1 when the candidate portion SUB1 matches with at least one of the first reference image RIMG1 and the second reference image RIMG2, wherein the first reference image RIMG1 represents an image of the model REF1 taken from a first view point VP1 , and the second reference image RIMG2 represents an image of the model REF1 taken from a second different view point VP2.
In an embodiment, a first image frame F1 taken from a first view point may comprise the sub-image SUB3, and a second image frame F2 taken from a second different view point may comprise an additional sub-image. The sub image SUB3 of a candidate item P3 may be determined to be a matching portion POR1 when the sub-image SUB3 matches with the first reference image RIMG1 , and if an additional sub image of said candidate item P3 matches with the second reference image RIMG1 .
Referring to Fig. 6a, the user interface UIF1 of the device 500 may be configured to provide a selective indicator IND1 , which selectively indicates the location of the matching item PM1 . The user interface UIF1 may be configured to display e.g. an altered image FT obtained by locally altering a portion of the image frame F1 , which contains the matching portion POR1 . The user interface UIF1 may be configured to display an altered image obtained by locally altering a portion POR1 of the image frame F1 . For example, the user interface UIF1 may be arranged to display an altered image FT, which comprises at least a part of an image frame F1 , and a selective indicator IND1 , which is dependent on the selection of the model REF1 . At least one portion of said altered image FT may be dependent on the selection of the model REF1 so that selection of the model has an effect on the visual appearance of the altered image F1 \
The selective indicator IND1 may highlight (i.e. emphasize) the position of the matching portion POR1 with respect to the other portions of the displayed image. The selective indicator IND1 may be e.g. a borderline, which surrounds the matching portion POR1 . The selective indicator IND1 may be e.g. an arrow symbol, which points at the matching portion POR1 . The selective indicator IND1 may be e.g. a crosshair pattern, which coincides with the matching portion POR1 . The brightness and/or color of the matching portion POR1 may be altered with respect to the original matching portion POR1 and with respect to an adjacent portion of the displayed image such that the altered matching portion POR1 may itself act as a selective indicator IND1 . The brightness and/or color of a surrounding portion surrounding the matching portion POR1 may be altered such that said surrounding portion may act as a selective indicator IND1 .
The selective indicator IND1 may indicate the location of the matching item PM1 such that said selective indicator IND1 is dependent on the selection of the model REF1 . The matching portion POR1 of the original unaltered image frame F1 is not considered to be a selective indicator IND1 .
For example, the seeking device 500 may be configured to display an image, which contains the selective indicator IND1 , wherein the selective indicator IND1 may be dependent on the selection of the model REF1 such that selecting a second different model instead of a first model may cause a visual change in the displayed image. The seeking device 500 may be configured to display a first altered image, which contains a selective indicator indicating the location of a first item matching with a first model, and to display a second altered image, which contains a selective indicator indicating the location of a second item matching with a second model, wherein the device 500 may be configured to generate the first altered image by altering a first image frame when the first model has been selected, and to generate the second altered image by altering a second image frame when the second model has been selected, wherein the first altered image may be substantially different from the second altered image also in a situation where the first image frame is identical to the second image frame.
The selective indicator IND1 may also indicate that the matching portion POR1 is found, in addition to indicating the location of the matching item PM1 . For example, displaying the borderline of Fig. 6a may indicate that that a candidate portion is detected to match with the reference data REFDATA1 of the model REF1 .
In an embodiment, an additional indicator may also be provided to indicate that a matching portion POR1 is found. For example, the device may be configured to generate an audio signal and/or a flashing signal when a candidate portion is detected to match with the reference data REFDATA1 of the model REF1 . The user interface UIF1 may optionally display additional information INFO2, INFO3. Additional information INFO2 may e.g. specify that the device 500 is operating in a seeking mode (For example, an acronym LSA may be displayed to indicate "Location Seeking Assistant"). Additional information INFO3 may e.g. specify that the displayed image is a realtime image.
Referring to Fig. 6b, the user interface UIF1 may be arranged to display a selective indicator IND2, which points at the position of the matching portion POR1 . The indicator IND2 pointing at the position of the matching portion POR1 may be called as a directional indicator. The user interface UIF1 may be configured to display a directional indicator IND2 based on the position u1 ,v1 of the matching portion POR1 . In particular, the directional indicator IND2 may be an arrow symbol pointing at the position of the matching portion POR1 . The directional indicator IND2 may be e.g. a spot, which is arranged to move on the display such that the movement of the spot indicates a direction. The selective indicator IND2 may indicate the position of the matching portion POR1 with respect to a sub-image SUBLND1 of a visually prominent feature LND1 . The user interface UIF1 may be arranged to indicate the position of the matching portion POR1 with respect to a sub-image SUBLND1 of a visually prominent feature LND1 . The visually prominent feature LND1 may be e.g. corner or edge of the group G1 (see Fig. 5). The device 500 may be configured to classify a candidate portion of an image frame as a sub-image SUBLND1 of a visually prominent feature LND1 e.g. when the brightness and/or color of the candidate portion substantially deviates from an adjacent portion, which covers e.g. more than 20% of the image frame. Thus, it may be easy for the user C1 to visually locate said visually prominent feature LND1 .
A part of an original image F4 and the indicator IND2 may together form an altered image F4'. The matching portion POR1 does not need to be displayed. The matching portion POR1 does not need to be displayed simultaneously with the indicator IND2.
Referring to Fig. 6c, the user interface UIF1 may be arranged to display a selective indicator IND2, which indicates the location of the matching item PM1 with respect to an aiming line AIM1 (Fig. 5). The selective indicator IND2 may be e.g. an arrow symbol, whose orientation depends on the position of the matching portion POR1 . The user interface may be arranged to indicate the position of the matching item PM1 with respect to a nominal aiming line AIM1 of the device 500. The aiming line AIM1 may e.g. coincide with the optical axis of the imaging optics 200. The aiming line AIM1 may be e.g. substantially perpendicular to the surface of a display screen of the seeking device 500.
Referring to Fig. 6d, the user interface UIF1 may be configured to display information INFO1 , which specifies the model REF1 selected by the user C1 . In particular, the information INFO1 may be an image of the model REF1 . The user interface UIF1 may be configured to display an indicator IND2, which points at the location of the matching portion POR1 . The user interface UIF1 may be configured to display a selective indicator IND1 , which selectively indicates the position of the portion POR1 . The user interface UIF1 may display an embedded selective indicator IND1 , which highlights the position of the matching portion POR1 . Fig. 7a shows method steps for seeking a matching item PM1 by using an image frame F1 , which is captured after the model REF1 has been selected. In this case, the image frame F1 may represent the most accurate and up-to-date information which is available regarding the location of the matching item PM1 . In step 810, reference data REFDATA1 associated with the selected model REF1 may be stored in a memory MEM1 . In particular, the reference data REFDATA1 may be obtained from a server 1240 (Fig. 10a, 10c), and the reference data REFDATA1 may be communicated to the seeking device 500 via the Internet. In step 820, an image frame F1 representing a group G1 of items P1 , P2, P3 may be captured by using the image sensor 100 of the seeking device 500.
The image frame F1 may be captured by using the image sensor 100 after the model REF1 has been selected. The image frame F1 may be captured by using the image sensor 100 after the reference data REFDATA1 has been communicated to the seeking device 500.
In step 830, the image analysis unit 400 may determine whether at least one portion of the image frame F1 matches with the model REF1 . If a matching portion is not found, a next image frame may be analyzed. In step 840, the image analysis unit 400 may determine the position u1 , v1 of the matching portion POR1 .
In step 850, the user interface UIF1 may provide a selective indicator IND1 , which indicates the location of the matching item PM1 . In particular, the user interface UIF1 may display an altered image, which contains a selective indicator dependent on the selection of the model.
Fig. 7b shows method steps for seeking a matching item PM1 by using one or more previously captured image frames.
Thus, the user interface UIF1 may rapidly indicate the location of the matching item PM1 without a need to capture additional image frames. In particular, the user C1 does not need to move around in the shop in order to capture additional image frames. However, the information contained in the image frame F1 may sometimes be more erroneous when compared with the case of Fig. 7a. For example, someone may have removed a last matching item PM1 from the group G1 after the image frame F1 was captured. Thus, if the seeking device 500 would indicate the previous location of the matching item PM1 based on information derived from the image frame F1 , said indication may be erroneous because the matching item PM1 is not any more at the indicated location.
In step 810, reference data REFDATA1 associated with the selected model REF1 may be stored in a memory MEM1 . In particular, the reference data REFDATA1 may be obtained from a server 1240 (Fig. 10a, 10c), and the reference data REFDATA1 may be communicated to the seeking device 500 via the Internet.
In step 825, an image frame F1 representing a group G1 of items P1 , P2, P3 may be retrieved from a memory MEM2. The image frame F1 may have been previously captured by using the image sensor 100 of the seeking device 500. The image frame F1 may have been captured by using the image sensor 100 before the model REF1 has been selected. The image frame F1 may have been captured by using the image sensor 100 before the reference data REFDATA1 has been communicated (COM1 ) to the seeking device 500. In step 830, the image analysis unit 400 may determine whether at least one portion of the image frame F1 matches with the model REF1 . If a matching portion is not found, a next image frame may be captured. In step 840, the image analysis unit 400 may determine the position u1 , v1 of the matching portion POR1 .
In step 850, the user interface UIF1 may provide a selective indicator IND1 , which indicates the location of the matching item PM1 . In particular, the user interface UIF1 may display an altered image, which contains a selective indicator, which is dependent on the selection of the model.
Fig. 8a shows several (i.e. two or more) image frames F1 , F2, F3, F4 captured by the image sensor 100 at different times t1 , t2, t3, t4. Each image frame may be associated with a different time stamp (e.g. 19:20:01 ). The temporal difference between the times may be selected such that at least one common feature appears in two or more consecutive image frames. For example, the feature SUB4 may appear in the image frames F1 and F2, the feature SUB5 may appear in the image frames F2 and F3, and the feature SUBLND1 may appear in the image frames F3, F4.
In an embodiment, the image sensor 100 may be configured to capture a video sequence, which comprises two or more image frames F1 , F2, F3, F4. Referring to Fig. 8b, the seeking device 500 may be configured to combine image data of two or more image frames F1 , F2, F3, F4 to form a combined image frame COMBI1 . The image frame COMBI1 may be e.g. a panorama image. The image frame COMBI1 may be a panorama image, which shows e.g. all items of a shelf or which shows items of several shelves.
The seeking device 500 may be configured to form the combined image frame COMBI1 by stitching image data from two or more image frames F1 , F2. The common features SUB4, SUB5, SUBLND1 may be used to provide reference positions for the stitching operation.
The seeking device 500 may be configured to use the combined image frame COMBI1 e.g. for determining the position of the matching portion POR1 with respect to a second feature SUBLND1 , even though the matching portion POR1 and the second feature SUBLND1 did not originally appear in the same image frame. The matching portion POR1 appears in the image frames F1 and F2 but not in the image frame F3. The second feature (e.g. visually prominent feature) appears in the image frame F3 but not in the image frames F1 and F2.
Fig. 9a shows, by way of example, an image displayed on a screen of the user interface UIF1 . The displayed image may contain e.g. information INFO2, which specifies that the device is operating in the seeking mode. The displayed image may contain e.g. information INFO3, which specifies that the displayed image represents real time situation. The displayed image may contain information INFO4, which specifies that a matching portion POR1 is found at an image frame (F2), which was captured earlier. For example, the matching portion POR1 may be found at an image frame, which was captured one second earlier. The device 500 may be configured to provide an indication that indicates that a matching portion POR1 is found.
The user interface UIF1 may be configured to display the earlier image frame F2 or an altered earlier image frame F2'. The user interface UIF1 may comprise a touch screen. The image frame F2' may be displayed e.g. by touching a key KEY3 or KEY4. The user interface UIF1 may be configured to directly display an altered image frame F2' when the key KEY3 has been touched. The altered image frame F2' may comprise a selective indicator IND1 , which shows the position of the matching portion POR1 .
The user interface UIF1 may be configured to display a video sequence when the key KEY4 has been touched. Touching the key KEY4 may enable showing one or more previous image frames in temporally reversed order. The video sequence may show the positional relationship of the matching portion POR1 with respect to a second feature (e.g. with respect to the sub-image SUBIMG1 of the visually prominent feature LND1 ).
Fig. 9b shows an altered image frame F2', which comprises a selective indicator IND1 and a portion of the image frame F2. Information INFO2 may specify that the device is operating in the seeking mode. Information INFO3, may specify that the displayed image represents an earlier situation. Information INFO4 may specify that the matching portion POR1 is found in an image frame (e.g. in F2), which was captured earlier, e.g. one second earlier. Touching a key KEY4 may enable playing the video sequence in temporally reversed order. Touching a key KEY5 may enable playing the video sequence in the normal order (i.e. forward). Touching a key KEY4 may enable playing the image frames of a video sequence in temporally reversed order (i.e. backwards). Playing the video sequence in the normal order and/or in the reverse order may make it easier for the user C1 to perceive the relative position of the matching item PM1 with respect to another item. Playing the video sequence in the normal order and/or in the reverse order may make it easier for the user C1 to perceive the relative position of the matching item PM1 with respect to the visually prominent feature LND1 .
Fig. 10a shows, by way of example, a distributed system 1000 for assisting in seeking a matching item. The system 1000 may comprise a plurality of devices arranged to communicate with each other. The system 1000 may comprise a seeking device 500. The seeking device 500 may be portable. The seeking device 500 may comprise a user interface UIF1 for providing selective indication IND1 about the location of the matching item. The seeking device 500 may comprise an image sensor 100 for capturing an image frame F1 representing a group G1 of items P1 , P2, P3. The image sensor 100 may receive light LBX reflected and/or emitted from the items P1 , P2, P3. The seeking device 500 or a server 1240, 1241 , 1242 may comprise one or more data processors configured to compare the model REF1 with image data SDATA1 .
The system 1000 may comprise end-user devices such as one or more portable devices 500, mobile phones or smart phones 1251 , Internet access devices (Internet tablets), personal computers 1260, a display or an image projector 1261 (e.g. a television), and/or a video player 1262. A mobile phone, a smart phone, an Internet access device, or a personal computer may comprise an image sensor for providing reference data REFDATA1 , which may represent a model REF1 . A server, a mobile phone, a smart phone, an Internet access device, or a personal computer may be arranged to distribute reference data REFDATA1 representing a model REF1 . Distribution and/or storing reference data REFDATA1 may be implemented in the network service framework with one or more servers 1240, 1241 , 1242 and one or more user devices. As shown in the example of Fig. 10a, the different devices of the system 1000 may be connected via a fixed network 1210 such as the Internet or a local area network (LAN). The devices may be connected via a mobile communication network 1220 such as the Global System for Mobile communications (GSM) network, 3rd Generation (3G) network, 3.5th Generation (3.5G) network, 4th Generation (4G) network, Wireless Local Area Network (WLAN), Bluetooth®, or other contemporary and future networks. Different networks may be connected to each other by means of a communication interface 1280. A network (1210 and/or 1220) may comprise network elements such as routers and switches to handle data (not shown). A network may comprise communication interfaces such as one or more base stations 1230 and 1231 to provide access for the different devices to the network. The base stations 1230, 1231 may themselves be connected to the mobile communications network 1220 via a fixed connection 1276 and/or via a wireless connection 1277. There may be a number of servers connected to the network. For example, a server 1240 for providing a network service such as a social media service may be connected to the network 1210. A second server 1241 for providing a network service may be connected to the network 1210. A server 1242 for providing a network service may be connected to the mobile communications network 1220. Some of the above devices, for example the servers 1240, 1241 , 1242 may be arranged such that they make up the Internet with the communication elements residing in the network 1210. The devices 500, 1251 , 1260, 1261 , 1262 can also be made of multiple parts. One or more devices may be connected to the networks 1210, 1220 via a wireless connection 1273. Communication COM1 between a device 500 and a second device of the system 1000 may be fixed and/or wireless. One or more devices may be connected to the networks 1210, 1220 via communication connections such as a fixed connection 1270, 1271 , 1272 and 1280. One or more devices may be connected to the Internet via a wireless connection 1273. One or more devices may be connected to the mobile network 1220 via a fixed connection 1275. A device 500, 1251 may be connected to the mobile network 1220 via a wireless connection COM1 , 1279 and/or 1282. The connections 1271 to 1282 may be implemented by means of communication interfaces at the respective ends of the communication connection. A user device 500, 1251 or 1260 may also act as web service server, just like the various network devices 1240, 1241 and 1242. The functions of this web service server may be distributed across multiple devices. Application elements and libraries may be implemented as software components residing on one device. Alternatively, the software components may be distributed across several devices. The software components may be distributed across several devices so as to form a cloud. In an embodiment, the reference data REFDATA1 may be obtained from social media. The reference data REFDATA1 may be obtained by using a social media service. The reference data REFDATA1 of a model REF1 may be received from a social media service. The model REF1 may be selected by using a social media service.
The social media service may also be called e.g. as a social networking service. The social media service may be implemented e.g. on one or more servers 1240. The seeking device 500 may be carried by user C1 . A second person may be associated with the user by a link in the social media service. The second person may be associated with one or more models REF1 based on information provided by the social media service. The second person may e.g. inform that he is using a specific model REF1 . The second person may e.g. recommend a specific model REF1 . The model REF1 may be selected from among one or more candidate models listed in the social media service. The user interface UIF1 of the seeking device 500 may be e.g. configured to display one or more models provided by a social media service, and the user may select a model e.g. by touching a touch screen of the user interface UIF1 . The reference data REFDATA1 of the model may be subsequently retrieved from the social media service. The social media service may provide a link to a second data distribution service, which may also be implemented on one or more servers. The reference data REFDATA1 of the model may also be retrieved from the second data distribution service. The reference data REFDATA1 may be communicated to the seeking device e.g. via the communication unit RXTX1 of the device 500.
A social media service may be a web-based online service, which may provide a platform for a first person (e.g. the user C1 of the device 500) to:
- construct a public profile or a semi-public profile,
- provide a first list of two or more other persons that the first person shares a connection with, and
- view further list of connections of the other persons.
The social media service may provide pictures of the other persons. The social media service may provide pictures of models associated with the other persons. The profile may contain information specifying e.g. age, location, and/or interests of the first person. The profile may contain information specifying e.g. age, location, and/or interests of the second person. The profile may contain information specifying one or more models REF1 associated with the second person.
The term public profile or a semi-public means that two or more other persons may view the profile of the first person and view the first list of connections. To protect user privacy, the social networks may have an access control to enable and/or disable viewing the profile of the first person and the first list of connections. The access control may utilize e.g. password. In an embodiment, the method for indicating the position of the desired item may comprise obtaining the reference data from browser cache. The method may comprise obtaining the reference data from browsing history data. The user C1 or a second person may visit one or more web pages (e.g. in the Internet) by using a web browser. One or more of the web pages may contain one or more images of a model. The web browser may be implemented in a browsing device. The browsing device may be e.g. the seeking device 500. The device 500 may be configured to store browsing history data in a memory. The browsing history data may also be called e.g. as web search history data. The browsing history data may comprise REFDATA1 associated with a model. The device 500 may be configured to store reference data REFDATA1 associated with a model in a memory. The device 500 may comprise said memory. The reference data REFDATA1 associated with the model may be subsequently retrieved from the memory e.g. in a situation where connection to the Internet is not available. Fig. 10b shows a portable seeking device 500, which may be configured to capture an image frame F1 representing a group G1 of items, and to provide a selective indication IND1 of the location of the matching item PM1 . The device 500 may be e.g. a mobile phone, a smartphone, a communicator, a portable computer, a camera, or a personal digital assistant (PDA). The device 500 may comprise an image sensor 100 providing image data SDATA1 . The device 500 may comprise a communication unit RXTX1 , a user interface UIF1 , a memory MEM1 for storing reference data REFDATA1 representing the selected model REF1 , one or more processors CNT1 , 400, a memory MEM2 for storing an image frame F1 representing a group G1 of items, a memory MEM3 for storing computer program PROG1 . The device 500 may comprise a communication unit RXTX1 for receiving reference data REFDATA1 wirelessly e.g. from the Internet 1210, and/or from a mobile telephone network 1220. The portable device 500 may comprise a communication module RXTX1 for receiving the reference data REFDATA via the Internet based on a selection of the model REF1 .
The units and functionalities of the device 500 may be implemented as described e.g. in the context of Figs. 1 b, 2a, 2b, 3, 4, 5, 6a, 6b, 6c, 6d, 7a, 7b, 8a, 8b, 9a, 9b and/or 10a.
The device 500 may optionally comprise one or more microphones 1258 for converting sound waves into audio signals. The device 500 may further comprise one or more speakers 1255 for reproducing audio signals. A microphone 1258 may be used e.g. to implement a mobile phone functionality.
In an embodiment, the location of the matching item PM1 may be indicated by an audio signal. The user interface UIF1 may be arranged to provide an audio signal based on the position u1 ,v1 of the matching portion POR1 . The seeking device 500 may be configured to detect and indicate when the aiming line AIM1 (Fig. 5) is moving closer to (or farther away from) the location of the matching item PM1 . For example, a frequency of an audio signal may depend on the distance between the location of the matching item PM1 and the aiming line AIM1 . The user may manually move and/or change the orientation of the seeking device 500 so that he can find the location of the matching item PM1 by listening to the variation of an audio signal provided by the seeking device 500. In an embodiment, the seeking device 500 may be configured to play a predetermined tone or song when the aiming line AIM1 is getting closer to the location of the matching item PM1 . The seeking device 500 may be configured to provide a voice indication about the location of the matching item PM1 e.g. by playing pre-recorded verbal instructions or by using a speech synthesizer. For example, the device 500 may play a voice signal "move to the left". In an embodiment, the seeking device 500 may be configured to provide a haptic indication about the location of the matching item PM1 . The user interface UIF1 may be arranged to provide a haptic indicator based on the position u1 ,v1 of the matching portion POR1 . In an embodiment, the seeking device 500 may be configured to e.g. vibrate when the aiming line AIM1 is getting closer to the location of the matching item PM1 . In an embodiment, the seeking device 500 may be configured to vibrate when the aiming line AIM1 is moving farther away from the location of the matching item PM1 . Fig. 10c shows a server 1240, which may comprise a memory 1245 (MEM), one or more processors 1246, 1247 (PROC), and computer program code 1248 (PROGRAM) residing in the memory 1245 for implementing, for example, a service for distributing reference data REFDATA1 . Reference data REFDATA1 for several candidate models may be stored in a memory MEM4 of the server 1240.
The server 1240 may comprise a memory 1245, one or more processors 1246, 1247, and computer program code 1248 residing in the memory 1245 for implementing, for example, a social networking service.
In an embodiment, the model REF1 may be selected by the user C1 of the device 500, i.e. by a person who is carrying the device 500 during capturing the image frames F1 , F2. The user may search suitable candidate models e.g. by using a web browser.
In an embodiment, the device 500 and/or a server 1240 may be configured to determine or propose one or models REF1 from a web search history of the user C1 .
In an embodiment, the device 500 and/or a server 1240 may be configured to determine or propose one or models REF1 based on the detected location of the device 500 and/or based on the clock time. In an embodiment, the device 500 and/or a server 1240 may be configured to determine or propose one or models REF1 based on a proposal made by a shopkeeper.
In an embodiment, the device 500 and/or a server 1240 may be configured to determine or propose one or models REF1 based on a recommendation presented in a social media network.
In an embodiment, the device 500 and/or a server 1240 may be configured to determine or propose one or models REF1 based on age of the user, based on family status of the user and/or based on a bonus card. In an embodiment, the reference data REFDATA1 associated with a model may be retrieved from a database e.g. based on a barcode scanned by using the seeking device 500. In an embodiment, identification data identifying all items collected by the user C1 (e.g. to a shopping cart) may be stored in a memory, in order to provide a list of items collected by the user. The seeking device 500 may be configured to communicate the list to a point of sale terminal of the shop. This may reduce the time, which will be needed at the point of sales terminal.
In an embodiment, the seeking device 500 or a server 1240 may receive a list of items previously collected (and bought) by the user C1 or by another person. The seeking device 500 or server 1240 may select one or more models based on said list. The seeking device 500 or server 1240 may be configured to retrieve reference data REFDATA1 associated with said models. During a subsequent visit to the shop SHOP1 , the seeking device 500 may be configured to seek items, which match with said models.
In an embodiment, the seeking device 500 may be configured to simultaneously seek items, which match two or more different models REF1 . The seeking device 500 may store a "shopping list", which defines two or more different models REF1 . The device 500 may be configured to compare sub-images of a captured image frame with first reference data REFDATA1 representing a first model REF1 , and to compare the sub-images SUB1 , SUB2 with second reference data representing a second different model.
The image processing apparatus 500 may comprise a user interface UIF1 for receiving commands from a user and/or for providing information to the user. A model REF1 may be selected by using the user interface UIF1 . The user interface UIF1 may comprise e.g. a touch screen for visually displaying information and for receiving commands from a user. The user interface UIF1 may comprise hardware, e.g. a display, keypad and/or a touch screen. The user interface may comprise a display screen for viewing graphical elements displayed on the screen. The user interface UIF1 may comprise a software application e.g. for displaying various different virtual keys on a touch screen. The user interface UIF1 may comprise one or more virtual keys for receiving instructions from the user. The virtual keys may be implemented on the touch screen. The user interface UIF1 may comprise one or more push buttons for receiving instructions from the user. The user interface UIF1 may comprise a keypad for receiving instructions from the user. The user interface UIF1 may comprise a touchpad, a keypad, a mouse and/or a joystick for receiving instructions from the user. In an embodiment, the user interface UIF1 may comprise a microphone and a speech recognition unit for receiving instructions, which are spoken aloud. The user interface UIF1 may be implemented e.g. in a portable device, e.g. in a smart phone. A user may receive information via the interface UIF1 . The user may control operation of the apparatus 500 by giving commands via the user interface UIF1 . When using a touch screen, the user may input information by touching the touch screen with a touching member. The touching member may be e.g. a finger or a stylus. Touching the touch screen may refer to actual physical contact between the touching member and the screen. Touching the touch screen may also mean bringing the touching member close to the screen so that the distance between the finger H1 and the screen is smaller than a predetermined distance (e.g. smaller than 1 % of the width of the touch screen).
In an embodiment, the seeking device 500 may comprise two or more image sensors 100 to capture images of a candidate item from different angles. The device 500 may comprise a first image sensor and a second image sensor. Using several image sensors may e.g. increase rate of capturing the image frames from different viewpoints. In an embodiment, shape information about the three- dimensional shape of a candidate item may be determined from a first image of the candidate item taken from a first view point, and from a second image of the candidate item taken from a second view point (without moving the candidate item). In an embodiment, said shape information determined from the image data SDATA1 may be compared with reference data REFDATA1 of the model REF1 in order to determine whether a portion of the first image is a matching portion. The shape information of the candidate item may be compared with the shape information of the model REF1 in order to determine whether a portion of the first image is a matching portion. The shape information of the candidate item may be compared with the shape information of the model REF1 in order to determine whether the candidate item is a matching item PM1 . In an embodiment, the first image representing the first view angle and the second image representing the second view angle may be obtained from image data SDATA1 , which is provided by a single image sensor. The first image may be captured when the device 500 is at a first location with respect to the candidate item, and the second image may be captured when the device is at a second location. The second location may be displaced with respect to the first location e.g. in a direction, which is transverse to the aiming line AIM1 . In an embodiment, the optics 200 of the device 500 may be arranged to simultaneously form the two images on a single image sensor 100.
For the person skilled in the art, it will be clear that modifications and variations of the devices and the methods according to the present invention are perceivable. The figures are schematic. The particular embodiments described above with reference to the accompanying drawings are illustrative only and not meant to limit the scope of the invention, which is defined by the appended claims.

Claims

1 . A method comprising:
- providing reference data (REFDATA1 ) representing a model (REF1 ),
- capturing one or more image frames (F1 , F2) of candidate items (P1 , P2, P3) by using an image sensor (100),
- determining whether a candidate portion (SUB1 , SUB2, SUB3) of an image frame (F1 ) is a matching image portion (POR1 ) by comparing the candidate portion (SUB3) with the reference data (REFDATA1 ), and
- providing a selective indication (IND1 , IND2) about the location of a matching item (PM1 ) based on the position (u1 ,v1 ) of the matching image portion (POR1 ).
2. The method of claim 1 , comprising providing an altered image (FV) by locally altering a portion of an image frame (F1 ), and displaying the altered image (FV) on a display (UIF1 ).
3. The method of claim 1 or 2, comprising displaying a directional indicator (IND2) based on the position (u1 ,v1 ) of the matching portion (POR1 ).
4. The method according to any of the claims 1 to 3, comprising indicating the position (u1 ,v1 ) of the matching portion (POR1 ) with respect to a sub-image (SUBLND1 ) of a visually prominent feature (LND1 ).
5. The method according to any of the claims 1 to 4, comprising indicating the position (y,z) of the matching item (PM1 ) with respect to a nominal aiming line
(AIM1 ) of the image sensor (100).
6. The method according to any of the claims 1 to 5, comprising providing an audio signal and/or a haptic indicator based on the position (u1 ,v1 ) of the matching portion (PM1 ).
7. The method according to any of the claims 1 to 6, comprising:
- determining a first reference image (RIMG1 ) from the reference data (REFDATA1 ),
- determining a second reference image (RIMG2) from the reference data (REFDATA1 ), - determining whether a candidate portion (SUB1 ,SUB2, SUB3) of an image frame (F1 ) matches with the first reference image (RIMG1 ),
- determining whether a candidate portion (SUB1 ,SUB2, SUB3) of an image frame (F1 ) matches with the second reference image (RIMG2), and
- determining that the candidate portion (SUB3) is a matching portion (POR1 ) when the candidate portion (SUB3) matches with at least one of the first reference image (RIMG1 ) and the second reference image (RIMG2),
wherein the first reference image (RIMG1 ) represents an image of the model (REF1 ) taken from a first view point (VP1 ), and the second reference image (RIMG2) represents an image of the model (REF1 ) taken from a second different view point (VP2).
8. The method according to any of the claims 1 to 7, comprising obtaining the reference data (REFDATA1 ) from the Internet.
9. The method of claim 8, wherein the reference data (REFDATA1 ) is obtained from a web based online store.
10. The method of claim 8, wherein the reference data (REFDATA1 ) is obtained from social media.
1 1 . The method of claim 8, wherein the reference data (REFDATA1 ) is obtained from a browser cache.
12. An apparatus (500, 1000) comprising at least one processor (CNT1 ), a memory (MEM3) including computer program code (PROG1 ), the memory (MEM3) and the computer program code (PROG1 ) being configured to, with the at least one processor (CNT1 ), cause the apparatus (500, 1000) to perform at least the following:
- providing reference data (REFDATA1 ) representing a model (REF1 ),
- capturing one or more image frames (F1 , F2) of candidate items (P1 , P2, P3) by using an image sensor (100),
- determining whether a candidate portion (SUB3) of an image frame (F1 ) is a matching image portion (POR1 ) by comparing the candidate portion (SUB3) with the reference data (REFDATA1 ), and
- providing a selective indication (IND1 , IND2) about the location of a matching item (PM1 ) based on the position (u1 , v1 ) of the matching image portion (POR1 ).
13. The apparatus (500, 1000) of claim 12, wherein the apparatus (500, 1000) is configured to provide an altered image (F1 ') by locally altering a portion of an image frame (F1 ), and to display the altered image (F1 ') on a display (UIF1 ).
14. The apparatus (500, 1000) of claim 13, wherein the apparatus (500, 1000) is configured to display a directional indicator (IND2) based on the position (u1 ,v1 ) of the matching portion (POR1 ).
15. The apparatus (500, 1000) according to any of the claims 12 to 14, wherein the apparatus (500, 1000) is configured to indicate the position of the matching item (PM1 ) with respect to a nominal aiming line (AIM1 ) of the image sensor (100).
PCT/FI2013/050477 2013-04-29 2013-04-29 Method and apparatus for finding a location WO2014177751A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/FI2013/050477 WO2014177751A1 (en) 2013-04-29 2013-04-29 Method and apparatus for finding a location

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/FI2013/050477 WO2014177751A1 (en) 2013-04-29 2013-04-29 Method and apparatus for finding a location

Publications (1)

Publication Number Publication Date
WO2014177751A1 true WO2014177751A1 (en) 2014-11-06

Family

ID=51843202

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/FI2013/050477 WO2014177751A1 (en) 2013-04-29 2013-04-29 Method and apparatus for finding a location

Country Status (1)

Country Link
WO (1) WO2014177751A1 (en)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060171564A1 (en) * 2003-07-10 2006-08-03 James Simon Autonomous wide-angle license plate recognition
EP2157545A1 (en) * 2008-08-19 2010-02-24 Sony Computer Entertainment Europe Limited Entertainment device, system and method
US20120062596A1 (en) * 2010-09-14 2012-03-15 International Business Machines Corporation Providing augmented reality information
US20120233003A1 (en) * 2011-03-08 2012-09-13 Bank Of America Corporation Providing retail shopping assistance
US20120229657A1 (en) * 2011-03-08 2012-09-13 Bank Of America Corporation Providing data associated with relationships between individuals and images
WO2013137789A1 (en) * 2012-03-15 2013-09-19 Sca Hygiene Products Ab Method for assisting in locating an item in a storage location

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060171564A1 (en) * 2003-07-10 2006-08-03 James Simon Autonomous wide-angle license plate recognition
EP2157545A1 (en) * 2008-08-19 2010-02-24 Sony Computer Entertainment Europe Limited Entertainment device, system and method
US20120062596A1 (en) * 2010-09-14 2012-03-15 International Business Machines Corporation Providing augmented reality information
US20120233003A1 (en) * 2011-03-08 2012-09-13 Bank Of America Corporation Providing retail shopping assistance
US20120229657A1 (en) * 2011-03-08 2012-09-13 Bank Of America Corporation Providing data associated with relationships between individuals and images
WO2013137789A1 (en) * 2012-03-15 2013-09-19 Sca Hygiene Products Ab Method for assisting in locating an item in a storage location

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
WEST HENRIETTA: "Smarter Supermarket Way Finding", IP.COM JOURNAL, 17 August 2011 (2011-08-17), NY, US, pages 1 - 4 *
ZHU WEI ET AL.: "Design of the PromoPad: an augmented reality shopping assistant", 12TH AMERICA S CONFERENCE ON INFORMATION SYSTEMS, 4 August 2006 (2006-08-04), ACAPULCO, MEXICO, pages 1 - 16, Retrieved from the Internet <URL:http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.85.2629&rep=rep1&type=pdf> *

Similar Documents

Publication Publication Date Title
US10839605B2 (en) Sharing links in an augmented reality environment
US9342930B1 (en) Information aggregation for recognized locations
CN102722517B (en) Enhanced information for viewer-selected video object
US8180396B2 (en) User augmented reality for camera-enabled mobile devices
US20190333478A1 (en) Adaptive fiducials for image match recognition and tracking
US10146316B2 (en) Method and apparatus for disambiguating a plurality of targets
JP2017130978A (en) Information acquisition system, portable information device, server, and program
JP2010152884A (en) Image recognition algorithm, method of identifying target image using the same, and method of selecting data for transmission to portable electronic device
JP2016192767A (en) Imaging device, information achieving system and program
CN105934760A (en) Adaptable image search with computer vision assistance
CN109189986B (en) Information recommendation method and device, electronic equipment and readable storage medium
JP5395920B2 (en) Search device, search method, search program, and computer-readable recording medium storing the program
CN105069089B (en) Picture detection method and device
JP2010519656A (en) Remote object recognition
CN103686344A (en) Enhanced video system and method
CN105980975B (en) Information processing apparatus, information processing method, and program
US11593635B2 (en) Information processing device, method of processing information, and method of providing information
US9058660B2 (en) Feature searching based on feature quality information
CN104025615A (en) Interactive streaming video
CN111491187A (en) Video recommendation method, device, equipment and storage medium
US10664887B2 (en) System and method for associating sensibility words with physical product characteristics based on user attributes and displaying product images on a coordinate system
CN112020709A (en) Visual menu
CN106980612A (en) Information recommendation system and method
US20140019378A1 (en) Evaluation system, method, and computer-readable recording medium
US10123094B1 (en) Location-based movie identification systems and methods

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 13883374

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 13883374

Country of ref document: EP

Kind code of ref document: A1