US20150193909A1 - Method and device for panoramic image processing - Google Patents

Method and device for panoramic image processing Download PDF

Info

Publication number
US20150193909A1
US20150193909A1 US14/151,011 US201414151011A US2015193909A1 US 20150193909 A1 US20150193909 A1 US 20150193909A1 US 201414151011 A US201414151011 A US 201414151011A US 2015193909 A1 US2015193909 A1 US 2015193909A1
Authority
US
United States
Prior art keywords
image
floating
overlapping
panoramic
images
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US14/151,011
Inventor
Hilit MAAYAN
Daniel Shimon COHEN
Jacob GREENSHPAN
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Trax Technology Solutions Pte Ltd
Original Assignee
Trax Technology Solutions Pte Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Trax Technology Solutions Pte Ltd filed Critical Trax Technology Solutions Pte Ltd
Priority to US14/151,011 priority Critical patent/US20150193909A1/en
Assigned to TRAX TECHNOLOGY SOLUTIONS PTE LTD. reassignment TRAX TECHNOLOGY SOLUTIONS PTE LTD. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: COHEN, Daniel Shimon, GREENSHPAN, Jacob, MAAYAN, HILIT
Priority to PCT/IL2014/051139 priority patent/WO2015104705A1/en
Publication of US20150193909A1 publication Critical patent/US20150193909A1/en
Priority to US15/187,276 priority patent/US10122915B2/en
Assigned to PACIFIC WESTERN BANK reassignment PACIFIC WESTERN BANK SECURITY INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: TRAX TECHNOLOGY SOLUTIONS PTE LTD.
Assigned to TRAX TECHNOLOGY SOLUTIONS PTE LTD. reassignment TRAX TECHNOLOGY SOLUTIONS PTE LTD. RELEASE BY SECURED PARTY (SEE DOCUMENT FOR DETAILS). Assignors: PACIFIC WESTERN BANK
Abandoned legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00Geometric image transformation in the plane of the image
    • G06T3/0006Affine transformations
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60Control of cameras or camera modules
    • H04N23/64Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00Geometric image transformation in the plane of the image
    • G06T3/20Linear translation of a whole image or part thereof, e.g. panning
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00Geometric image transformation in the plane of the image
    • G06T3/40Scaling the whole image or part thereof
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60Control of cameras or camera modules
    • H04N23/698Control of cameras or camera modules for achieving an enlarged field of view, e.g. panoramic image capture
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00Details of television systems
    • H04N5/222Studio circuitry; Studio devices; Studio equipment
    • H04N5/262Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/265Mixing

Definitions

  • the present disclosure relates to the field of digital imaging. More particularly, the present disclosure relates to methods and devices useful in the field of panoramic photography.
  • panoramic images are generally acquired by pivoting an imaging device at a given location.
  • panoramic images can also be acquired by sweeping the imaging device in a given direction.
  • the imaging device can be translated substantially parallel to an object plane and to the ground in order to acquire a panoramic scene.
  • specific challenges are encountered related to maintaining the 6 degrees of freedom of the imaging device in accordance with an expected trajectory.
  • additional constraints arise when using panoramic photography in a retail store environment to scan a retail unit.
  • the present disclosure provides a method useful in panoramic photography which notably facilitates the positioning of the imaging device by a user.
  • scanning or “sweeping” may refer to translating an imaging device along a scanning direction while acquiring images with the imaging device.
  • a frame rate of acquisition suitable for the present disclosure may be of around 2 frames per second. It is noted that the scanning may comprise several stages and may contain one or more horizontal and/or vertical stages.
  • stream of images may refer to an ordered set of images i.e. to a sequence of images.
  • image prior to a given image in the context of a stream of images may be understood—unless mentioned otherwise—as referring to the image directly prior to the given image in the stream. It is further noted that the stream of images may be directly provided by an image sensor or may alternatively be pre-processed so as to result from a selection and/or a filtering of images in the series of images acquired by the image sensor.
  • a panoramic image is concurrently calculated and displayed in a preview window.
  • overlapping images may be understood as referring to images which at least present a partial overlap i.e. at least a part of one of the overlapping image is found in the other overlapping image.
  • scaling factor may refer to a ratio between a width of a floating panoramic image as displayed in the preview window and a width of a floating panoramic image as computed. For example, if the computed a floating panoramic image has a width of 1000 pixels and is displayed in a cell of the preview window of 150 pixels, the scaling factor may be of 15%.
  • inner slice may be used herein to refer to a slice of an image taken within (inside) the image i.e. an inner portion/cut of an image along a thickness of the image.
  • outer slice (or “peripheral slice”) may be used in contrast to refer to a slice of an image along the thickness of the image which extends until an end of the image i.e. the outer slice reach three edges of the image.
  • the present disclosure provides an image processing method comprising: receiving a stream of overlapping images; obtaining a corresponding stream of floating panoramic images by computing in real time, for each received overlapping image, an associated floating panoramic image using said overlapping image and a floating panoramic image associated with an overlapping image received prior to said overlapping image; displaying in real time with a predetermined scaling factor the stream of floating panoramic images in a preview window of a display unit, wherein the displaying comprises shifting the floating panoramic images relative to the preview window when the floating panoramic images scaled according to the predetermined scaling factor extend beyond the preview window so that at least a working portion of the overlapping image associated with the floating panoramic image being displayed is visible in the preview window.
  • an initial floating panoramic image is derived from a first image of the stream of overlapping images.
  • an amplitude of the shifting further enables to leave a blank area of a predetermined size in the preview window thereby enabling displaying at least one subsequent floating panoramic image using the same shifting amplitude.
  • the stream of overlapping images is acquired by moving an imaging sensor along a predetermined direction.
  • the received stream of overlapping images result from a selection of acquired overlapping images acquired by moving an imaging sensor along a predetermined direction.
  • the computing comprises cropping a margin around the received overlapping images.
  • obtaining an initial floating panoramic image comprises downsampling the first image of the stream of overlapping images and the real time computing of the floating panoramic image associated with each overlapping image comprises stitching a downsampled version of said overlapping image and the floating panoramic image associated with an overlapping image received prior to said overlapping image.
  • the stitching is performed by appending an inner slice of the overlapping image at an edge of the floating panoramic image associated with the overlapping image received prior to said overlapping image.
  • the stitching is performed by superimposing an outer slice of the overlapping image at an inner stitching portion of the floating panoramic image associated with the overlapping image received prior to said overlapping image.
  • the displaying of a floating panoramic image associated with a given overlapping image comprises: revealing instantaneously an overlying portion of the floating panoramic image, wherein the overlying portion is analogous to a corresponding portion of a floating panoramic image associated with an overlapping image received prior to said given overlapping image; and revealing progressively an additional portion of the floating panoramic image.
  • the method further comprises displaying in real time the overlapping image associated with the floating panoramic image.
  • the method further comprises displaying a speed indicator indicative of a speed of an imaging sensor providing the stream of overlapping images.
  • the method further comprises displaying a distance indicator indicative of a distance from an imaging sensor providing the stream of overlapping images to a forefront of a scene imaged by the imaging sensor.
  • the method further comprises displaying a scan direction indicator indicative of a predetermined scan direction.
  • the method further comprises displaying in the preview window a target indicator indicative of an expected display of a subsequent floating panoramic image.
  • the method further comprises displaying a drift indicator indicative of a drift between an actual trajectory of an imaging sensor providing the stream of overlapping images and an expected trajectory of said imaging sensor.
  • the method further comprises displaying assisting lines bounding a position of the drift indicator so as to facilitate a visual assessment of the drift.
  • the preview window is superimposed on the display of the overlapping image.
  • the preview window has predetermined dimensions.
  • the preview window is divided into one or more rows of cells.
  • the present disclosure provides a method of scanning a retail unit comprising: moving an imaging sensor along a predetermined direction while acquiring a stream of overlapping images of the retail unit; for each acquired overlapping image, computing in real time a floating panoramic image associated with said overlapping image using said overlapping image and a floating panoramic image associated with an overlapping image acquired prior to said overlapping image, wherein an initial floating panoramic image is derived from a first image of the stream of overlapping images, thereby obtaining a stream of floating panoramic images; displaying in real time with a predetermined scaling factor the stream of floating panoramic images in a preview window of a display unit, wherein the displaying comprises shifting the floating panoramic images relative to the preview window when the floating panoramic images scaled according to the predetermined scaling factor extend beyond the preview window so that at least a working portion of the overlapping image associated with the floating panoramic image being displayed is visible in the preview window.
  • the overlapping images are received from an image sensor of a handheld electronic device, the computing is run on a processing unit of the handheld electronic device and the display unit is comprised in said handheld electronic device.
  • the present disclosure provides a computer program product implemented on a non-transitory computer usable medium having computer readable program code embodied therein to cause the computer to perform an image processing method comprising: receiving a stream of overlapping images; obtaining a corresponding stream of floating panoramic images by computing in real time, for each received overlapping image, an associated floating panoramic image using said overlapping image and a floating panoramic image associated with an overlapping image received prior to said overlapping image; displaying in real time with a predetermined scaling factor the stream of floating panoramic images in a preview window of a display unit, wherein the displaying comprises shifting the floating panoramic images relative to the preview window when the floating panoramic images scaled according to the predetermined scaling factor extend beyond the preview window so that at least a working portion of the overlapping image associated with the floating panoramic image being displayed is visible in the preview window.
  • a handheld electronic device comprising: memory; an image sensor; a display communicatively coupled to the memory; and a processing unit communicatively coupled to the memory, display and image sensor, wherein the memory includes instructions for causing the processing unit to perform an image processing method comprising: receiving a stream of overlapping images; obtaining a corresponding stream of floating panoramic images by computing in real time, for each received overlapping image, an associated floating panoramic image using said overlapping image and a floating panoramic image associated with an overlapping image received prior to said overlapping image; displaying in real time with a predetermined scaling factor the stream of floating panoramic images in a preview window of the display, wherein the displaying comprises shifting the floating panoramic images relative to the preview window when the floating panoramic images scaled according to the predetermined scaling factor extend beyond the preview window so that at least a working portion of the overlapping image associated with the floating panoramic image being displayed is visible in the preview window.
  • FIG. 1 is a block diagram illustrating schematically an electronic device according to embodiments of the present disclosure.
  • FIG. 2 is a flow chart illustrating steps of a method of image processing according to embodiments of the present disclosure.
  • FIG. 3 is a diagram illustrating a step of computing floating panoramic images according to some embodiments of the present disclosure.
  • FIG. 4 illustrates a display screen of the electronic device displaying an image and a floating panoramic image associated to said image in a preview window.
  • FIG. 5 illustrates a display implementing a shifting function in some embodiments of the present disclosure.
  • FIG. 6 illustrates a step of displaying a floating panoramic image according to embodiments of the present disclosure.
  • FIGS. 7A-7E illustrate advantageous features of a method according to the present disclosure.
  • FIG. 7A illustrates a preliminary step prior to a scanning step according to some embodiments of the present disclosure.
  • FIG. 7B illustrates an attitude indicator according to some embodiments of the present disclosure.
  • FIG. 7C illustrates a speed indicator according to some embodiments of the present disclosure.
  • FIG. 7D illustrates a target indicator and a scanning indicator in a preview window according to embodiments of the present disclosure.
  • FIG. 7E illustrates a drift indicator in the preview window according to some embodiments of the present disclosure.
  • FIG. 1 illustrates a simplified functional block diagram of an electronic device according to embodiments of the present disclosure.
  • the electronic device may include a display 10 , a processor 20 , an imaging sensor 30 , memory 40 and a position sensor 50 .
  • the processor 20 may be any suitable programmable control device and may control the operation of many functions, such as the generation and or processing of image as well as other functions performed by the electronic device.
  • the processor 20 may drive the display (display screen) 10 and may receive user inputs from a user interface.
  • the display screen 10 may be a touch screen capable of receiving user inputs.
  • the memory 40 may store software for implementing various functions of the electronic device including software for implementing the image processing method according to the present disclosure.
  • the memory 40 may also store media such as images and video files.
  • the memory 40 may include one or more storage medium tangibly recording image data and program instructions, including for example a hard-drive, permanent memory and semi permanent memory or cache memory.
  • Program instructions may comprise a software implementation encoded in any desired language.
  • the imaging sensor 30 may be a camera with a predetermined field of view. The camera may either be used in a video mode in which a stream of images is acquired upon command of the user or in a photographic mode in which a single image is acquired upon command of the user.
  • the position sensor 50 may facilitate panorama processing.
  • the position sensor 50 may include a gyroscope enabling calculation of a rotational change of the electronic device from image to image.
  • the position sensor 50 may also be able to determine an acceleration and/or a speed of the electronic device according to three linear axes.
  • FIG. 2 illustrates generally steps of a method of image processing according to embodiments of the present disclosure.
  • the method may be implemented by a processor of a handheld electronic device as previously described.
  • a stream of overlapping images may be received.
  • the stream of overlapping images may stem from the scanning of an object plane using the camera of the electronic device previously described.
  • the camera may be swept along a scanning axis parallel to a horizontal plane (ground) and to the object plane.
  • the object plane may be a forefront of a retail unit (shelving unit) presenting retail items.
  • the user may adjust the camera to be perpendicular to the floor, for example with the assistance of the attitude indicator described hereinafter, and parallel to shelves of the retail unit, for example with assistance of horizontal lines displayed.
  • the stream of overlapping images may be displayed in real time on a display screen of the handheld electronic device to form a live video.
  • a stream of floating panoramic images (stitched images) corresponding to the stream of overlapping images may be computed in real time upon receiving of said overlapping images.
  • the floating panoramic images may be computed recursively.
  • An initial floating panoramic image may be derived from a first image of the stream of overlapping images.
  • the initial floating panoramic image may be a downsampled version of the first image.
  • the initial floating panoramic may be a downsampled cropped version of the first image.
  • the cropping of the first image may be performed to crop out the outer border (margins) of the first image. For example, 1 to 5% of the filed of view captured by the first image may be cropped out of the first image.
  • the cropping may be further performed on each overlapping images.
  • cropping the overlapping images may enable avoiding that small misalignments during the scanning cause black areas (without information) on the floating panoramic images.
  • the floating panoramic images corresponding to a given subsequent overlapping image may be computed by stitching a downsampled (and optionally cropped) version of the given overlapping image with the floating panoramic image computed for the overlapping image directly prior to the given overlapping image.
  • the computing may also involve image registration techniques in order to assess a transformation between subsequent overlapping images.
  • the transformation may be computed on downsampled (and optionally cropped) versions of subsequent overlapping images).
  • the transformation may be computed on specific portions of the subsequent downsampled (and optionally cropped) versions of subsequent overlapping images.
  • the registration techniques used are classical in the field of panorama processing and are therefore not described herein for the sake of conciseness.
  • the stitching may be performed between an inner slice of a downsampled (and optionally cropped) version of the given overlapping image and an edge of the floating panoramic image associated with the overlapping image received prior to said given overlapping image.
  • a specific inner slice of a downsampled (and optionally cropped) version of a subsequent overlapping image is appended at an edge of the floating panoramic image computed at a prior iteration. This slice by slice stitching enables avoiding backward updating of the floating panoramic images.
  • the stitching may be performed by superimposing an outer (peripheral) slice of a downsampled (and optionally cropped) version of the given overlapping image on the floating panoramic image associated with the overlapping image received prior to said given overlapping image.
  • the superimposing may be performed at an inner stitching portion of the floating panoramic image associated with the overlapping image received prior to said given overlapping image.
  • Said inner stitching portion may belong to the overlapping image received prior to said overlapping image.
  • the inner stitching portion may be representative of a central portion of the overlapping image received prior to said overlapping image.
  • the stitching is performed by superimposing an outer (peripheral) slice of the subsequent overlapping image on the prior floating panoramic image, wherein the outer slice is stitched at an inner portion of the prior floating panoramic image.
  • this stitching option provides the possibility of displaying completely the first frame and, when the user reaches the end of the first shelf, an edge of the frame can fit the edge of the preview window. Furthermore, this stitching option enable time saving for the user and the Applicant has found that it is also more intuitive because the stitched image progresses according to what the user faces and sees on the screen.
  • the stream of floating panoramic images may be displayed in real time with a predetermined (constant) scaling factor.
  • the displaying may be performed in a preview window 15 of a display screen 10 of a handheld electronic device as previously described.
  • the preview window 15 may have predetermined maximum dimensions.
  • the preview window may be divided into a plurality of cells distributed according to one or more rows. The width of the cells may be predetermined. The height of the cells of a first row of cells may be determined so that one cell contains a single frame (image) with the same aspect ratio as the acquired/received overlapping images.
  • the height of the cells of the first row may be determined so that one cell contains a single frame with the same aspect ratio as the acquired/received overlapping images after margins crop.
  • the preview window 15 may be centered with respect to the display screen 10 and may be positioned proximate to the bottom of the display screen 10 .
  • the preview window 15 may be a rectangle placed on top of the live video. Indeed, as shown on FIG. 4 , the last received/acquired overlapping image of the stream may be displayed in real time on the display screen to form the live video.
  • the preview window 15 may provide blank areas when the floating panorama does not extend beyond its dimensions and the blank areas may be semi-transparent.
  • display of the floating panoramic image in the preview window may be carried out in an opaque manner.
  • a following cell may be highlighted. The subsequent floating panoramic images may then progress into the highlighted cell.
  • the Applicant has found that a user may be particularly confused when performing scanning of long amplitude at short distance of objects which resemble each other. Therefore, the Applicant provides hereby a displaying which maintains a scaling factor of the floating panoramic image into a preview window and also provides a shifting function of the floating panoramic image when the floating panoramic image extends beyond the dimensions of the preview window.
  • the displaying comprises shifting the floating panoramic images relative to the preview window when the floating panoramic images scaled according to the predetermined scaling factor extend beyond the preview window so that at least a working portion of the overlapping image associated with the floating panoramic image being displayed is visible in the preview window.
  • the working portion may comprise a central portion of the overlapping image associated with the floating panoramic image being displayed and/or a fronto-parallel portion of the overlapping image associated with the displayed floating panoramic image.
  • the fronto-parallel portion may be a portion of the overlapping image representative of a region of the object plane extending slightly around a line perpendicular to the object plane and passing by the camera when the camera acquires said overlapping image.
  • the fronto-parallel region extends of around ⁇ 5 degrees around the perpendicular line.
  • an amplitude of the shifting may be constant, for example a cell size.
  • the shifting may provide a blank area of a predetermined size in the preview window thereby enabling displaying at least one subsequent floating panoramic image using the same shifting amplitude i.e. so that the working portion for the subsequent floating panoramic image be visible without modifying the shifting amplitude.
  • the floating panoramic image may be shifted sufficiently for being able to display one or more subsequent floating panoramic image without modifying the shifting amplitude.
  • the shifting enables to make a possibly concealed specific portion of the floating panoramic image visible on the display screen and with a resolution that enables the user to apprehend it.
  • displaying the working portion is of particular importance for enabling the user to position the camera for appropriately performing long amplitude scanning at short distance of objects which resemble each other.
  • the shifting may be implemented smoothly so as to improve user comfort.
  • the present disclosure may comprise another type of shifting of the floating panoramic image for fitting the extremity of the preview window with the extremity of the floating panoramic image corresponding to the end of the scanning according to said first direction.
  • the user may actuate a dedicated icon to mark the end of the scanning (end of a shelf).
  • Images resulting from a scanning according to another direction may further be acquired and stitched to the floating panoramic images previously computed.
  • the resulting floating panoramic images may be displayed in the preview window. For example the resulting floating panoramic images may also fill a second row of cells of the preview window.
  • an undo function may be available to the user upon command, for example by pressing an undo icon item on the display screen.
  • the undo function may enable the user to cancel the last stitching which occurred.
  • the undo function may also enable the user to cancel a predetermined amount of stitching which previously occurred.
  • the undo function may for example be advantageous in cases of obstruction of the camera by any unwanted objects.
  • another type of shifting may occur to optimize the occupancy of the preview window. This enables to reallocate the preview window to a portion of the floating panoramic image which had been expelled from the preview window.
  • the displaying of a floating panoramic image associated with a given overlapping image may comprises revealing instantaneously an overlying portion B of the floating panoramic image, wherein the overlying portion is analogous to a corresponding portion A of a floating panoramic image associated with an overlapping image received prior to said given overlapping image.
  • the displaying of the floating panoramic image may comprise revealing progressively an additional portion C of the floating panoramic image. A rate for progressively revealing the additional portion C may be proportional to the scanning speed.
  • the present disclosure may also comprise monitoring and displaying in real time an indication related to a distance from the electronic device (camera) to the object plane.
  • the distance may be derived from the position sensor of the electronic device and/or using image processing.
  • a distance indicator may be displayed to warn the user that the distance is within and/or out of an acceptable range.
  • a preliminary step including distance calibration may be performed prior to scanning of the retail unit. For example, when imaging a known retail item (or at least an item from a category with known proportions), the distance to the object plane may be calibrated by matching the imaged retail item with a template image 61 of said retail item stored in the memory of the electronic device and displayed during the preliminary step together with an image preview of the retail unit.
  • the template image 61 may only be a representation of the silhouette of the retail item.
  • a generic template may be used. The Applicant has found that in order to optimize the scanning, a product should approximately occupy 30% of the field of view of the imaging sensor ( ⁇ 5% or 10%). Further, modification of the distance to the object plane may be monitored by using the position sensor and/or image processing and displaying of the distance indicator may be performed in real time. For example, the distance indicator may be superimposed on the overlapping image display. Further, as illustrated, the preliminary step may also include a definition of a scanning direction by using a specific icon of a user interface (i.e. left-to-right or right-to-left with respect to the user holding the camera).
  • the present disclosure may also comprise monitoring in real time an attitude of the camera with respect to a reference attitude.
  • the attitude may be monitored using the position sensor of the electronic device and/or by using image processing.
  • the reference attitude may be an attitude of the camera when acquiring a first image of the stream of images.
  • the rotational change with respect to the reference attitude may be monitored in order to keep the camera facing the object plane.
  • An attitude indicator may be displayed in real time to warn the user that the camera's attitude is out of an acceptable range.
  • the attitude indicator may be superimposed on the overlapping image display.
  • the attitude indicator may be displayed as a geometrical shape distorted in accordance with the rotational change between a current attitude and the reference attitude.
  • the geometrical shape may be a rectangle 62 enveloping a periphery of the display screen.
  • the distortion of the rectangle may be a rotation according to the measured rotational change followed by a projection.
  • the rectangle may be displayed with a predetermined color different to that shown when the rotational change is acceptable so as to attract the user's attention.
  • the present disclosure may also comprise monitoring a scanning speed of the camera using the position sensor of the electronic device and/or image processing to assess motion blur.
  • a speed indicator 63 may be displayed in real time to warn the user that the scanning speed is within and/or out of an acceptable range.
  • the present disclosure may also comprise displaying in real time a scanning direction indicator 64 and a target indicator 65 indicative of the predetermined scanning direction so as to improve user comfort.
  • the scanning direction indicator 64 (arrow) and the target indicator 65 (circle) may be displayed as an over layer (superimposed) in the preview window in which the floating panoramic image is to be displayed.
  • the scanning direction indicator 64 and the target indicator 65 may be updated so as to progressively follow the filling of the preview window by the floating panoramic images. For example, when the preview window is divided into cells, when the floating panoramic image reaches an extremity of a cell, the target indicator 65 may be swept to the following cell.
  • the target indicator 65 may be semi transparent.
  • the scanning direction indicator 64 may follow the target indicator 65 . Following to a shifting of one cell size, the scanning direction indicator may shift back at the start of the last cell of the row of cells currently modified.
  • the present disclosure may also comprise displaying in real time a drift indicator indicative of a vertical drift of the electronic device during the scanning.
  • the term vertical refers herein to a direction perpendicular to the scanning direction and parallel to the object plane.
  • the vertical drift may be derived from the position sensor of the electronic device and/or using image processing.
  • the drift indicator may be combined with the scanning direction indicator and the drift indication may be provided by modifying a relative position of the scanning direction indicator and target indicator. For example, this can be done by misaligning the center of the target indicator 65 with the head of the arrow 64 .
  • assisting lines may be displayed, wherein a relative position of the drift indicator with respect to the assisting lines indicates the drift level.
  • system can be implemented, at least partly, as a suitably programmed computer.
  • the presently disclosed subject matter contemplates a computer program being readable by a computer for executing the disclosed method.
  • the presently disclosed subject matter further contemplates a machine-readable memory tangibly embodying a program of instructions executable by the machine for executing the disclosed method.

Abstract

The present disclosure provides an image processing method comprising: receiving a stream of overlapping images; obtaining a corresponding stream of floating panoramic images by computing in real time, for each received overlapping image, an associated floating panoramic image using said overlapping image and a floating panoramic image associated with an overlapping image received prior to said overlapping image; displaying in real time with a predetermined scaling factor the stream of floating panoramic images in a preview window of a display unit, wherein the displaying comprises shifting the floating panoramic images relative to the preview window when the floating panoramic images scaled according to the predetermined scaling factor extend beyond the preview window so that at least a working portion of the overlapping image associated with the floating panoramic image being displayed is visible in the preview window.

Description

    TECHNOLOGICAL FIELD
  • The present disclosure relates to the field of digital imaging. More particularly, the present disclosure relates to methods and devices useful in the field of panoramic photography.
  • BACKGROUND
  • Conventional panoramic images are generally acquired by pivoting an imaging device at a given location. However, panoramic images can also be acquired by sweeping the imaging device in a given direction. For example, the imaging device can be translated substantially parallel to an object plane and to the ground in order to acquire a panoramic scene. In the latter panoramic scenario, specific challenges are encountered related to maintaining the 6 degrees of freedom of the imaging device in accordance with an expected trajectory. Furthermore, additional constraints arise when using panoramic photography in a retail store environment to scan a retail unit.
  • GENERAL DESCRIPTION
  • The present disclosure provides a method useful in panoramic photography which notably facilitates the positioning of the imaging device by a user.
  • In the present application, the following terms and their derivatives may be understood in light of the below explanations.
  • The terms “scanning” or “sweeping” may refer to translating an imaging device along a scanning direction while acquiring images with the imaging device. A frame rate of acquisition suitable for the present disclosure may be of around 2 frames per second. It is noted that the scanning may comprise several stages and may contain one or more horizontal and/or vertical stages.
  • The term “stream of images” may refer to an ordered set of images i.e. to a sequence of images. The term “image prior to a given image” in the context of a stream of images may be understood—unless mentioned otherwise—as referring to the image directly prior to the given image in the stream. It is further noted that the stream of images may be directly provided by an image sensor or may alternatively be pre-processed so as to result from a selection and/or a filtering of images in the series of images acquired by the image sensor.
  • The term “real time” may be used herein to refer to substantially simultaneous operations. In the present disclosure, upon acquiring by an image sensor and receiving of (data representative of) the image by the processor, a panoramic image is concurrently calculated and displayed in a preview window.
  • The term “overlapping images” may be understood as referring to images which at least present a partial overlap i.e. at least a part of one of the overlapping image is found in the other overlapping image.
  • The term “scaling factor” may refer to a ratio between a width of a floating panoramic image as displayed in the preview window and a width of a floating panoramic image as computed. For example, if the computed a floating panoramic image has a width of 1000 pixels and is displayed in a cell of the preview window of 150 pixels, the scaling factor may be of 15%.
  • The term “inner slice” may be used herein to refer to a slice of an image taken within (inside) the image i.e. an inner portion/cut of an image along a thickness of the image. The term “outer slice” (or “peripheral slice”) may be used in contrast to refer to a slice of an image along the thickness of the image which extends until an end of the image i.e. the outer slice reach three edges of the image.
  • Therefore, the present disclosure provides an image processing method comprising: receiving a stream of overlapping images; obtaining a corresponding stream of floating panoramic images by computing in real time, for each received overlapping image, an associated floating panoramic image using said overlapping image and a floating panoramic image associated with an overlapping image received prior to said overlapping image; displaying in real time with a predetermined scaling factor the stream of floating panoramic images in a preview window of a display unit, wherein the displaying comprises shifting the floating panoramic images relative to the preview window when the floating panoramic images scaled according to the predetermined scaling factor extend beyond the preview window so that at least a working portion of the overlapping image associated with the floating panoramic image being displayed is visible in the preview window.
  • In some embodiments, an initial floating panoramic image is derived from a first image of the stream of overlapping images.
  • In some embodiments, an amplitude of the shifting further enables to leave a blank area of a predetermined size in the preview window thereby enabling displaying at least one subsequent floating panoramic image using the same shifting amplitude.
  • In some embodiments, the stream of overlapping images is acquired by moving an imaging sensor along a predetermined direction.
  • In some embodiments, the received stream of overlapping images result from a selection of acquired overlapping images acquired by moving an imaging sensor along a predetermined direction.
  • In some embodiments, the computing comprises cropping a margin around the received overlapping images.
  • In some embodiments, obtaining an initial floating panoramic image comprises downsampling the first image of the stream of overlapping images and the real time computing of the floating panoramic image associated with each overlapping image comprises stitching a downsampled version of said overlapping image and the floating panoramic image associated with an overlapping image received prior to said overlapping image.
  • In some embodiments, the stitching is performed by appending an inner slice of the overlapping image at an edge of the floating panoramic image associated with the overlapping image received prior to said overlapping image.
  • In some embodiments, the stitching is performed by superimposing an outer slice of the overlapping image at an inner stitching portion of the floating panoramic image associated with the overlapping image received prior to said overlapping image.
  • In some embodiments, the displaying of a floating panoramic image associated with a given overlapping image comprises: revealing instantaneously an overlying portion of the floating panoramic image, wherein the overlying portion is analogous to a corresponding portion of a floating panoramic image associated with an overlapping image received prior to said given overlapping image; and revealing progressively an additional portion of the floating panoramic image.
  • In some embodiments, the method further comprises displaying in real time the overlapping image associated with the floating panoramic image.
  • In some embodiments, the method further comprises displaying a speed indicator indicative of a speed of an imaging sensor providing the stream of overlapping images.
  • In some embodiments, the method further comprises displaying a distance indicator indicative of a distance from an imaging sensor providing the stream of overlapping images to a forefront of a scene imaged by the imaging sensor.
  • In some embodiments, the method further comprises displaying a scan direction indicator indicative of a predetermined scan direction.
  • In some embodiments, the method further comprises displaying in the preview window a target indicator indicative of an expected display of a subsequent floating panoramic image.
  • In some embodiments, the method further comprises displaying a drift indicator indicative of a drift between an actual trajectory of an imaging sensor providing the stream of overlapping images and an expected trajectory of said imaging sensor.
  • In some embodiments, the method further comprises displaying assisting lines bounding a position of the drift indicator so as to facilitate a visual assessment of the drift.
  • In some embodiments, the preview window is superimposed on the display of the overlapping image.
  • In some embodiments, the preview window has predetermined dimensions.
  • In some embodiments, the preview window is divided into one or more rows of cells.
  • In another aspect, the present disclosure provides a method of scanning a retail unit comprising: moving an imaging sensor along a predetermined direction while acquiring a stream of overlapping images of the retail unit; for each acquired overlapping image, computing in real time a floating panoramic image associated with said overlapping image using said overlapping image and a floating panoramic image associated with an overlapping image acquired prior to said overlapping image, wherein an initial floating panoramic image is derived from a first image of the stream of overlapping images, thereby obtaining a stream of floating panoramic images; displaying in real time with a predetermined scaling factor the stream of floating panoramic images in a preview window of a display unit, wherein the displaying comprises shifting the floating panoramic images relative to the preview window when the floating panoramic images scaled according to the predetermined scaling factor extend beyond the preview window so that at least a working portion of the overlapping image associated with the floating panoramic image being displayed is visible in the preview window.
  • In some embodiments, the overlapping images are received from an image sensor of a handheld electronic device, the computing is run on a processing unit of the handheld electronic device and the display unit is comprised in said handheld electronic device.
  • In another aspect, the present disclosure provides a computer program product implemented on a non-transitory computer usable medium having computer readable program code embodied therein to cause the computer to perform an image processing method comprising: receiving a stream of overlapping images; obtaining a corresponding stream of floating panoramic images by computing in real time, for each received overlapping image, an associated floating panoramic image using said overlapping image and a floating panoramic image associated with an overlapping image received prior to said overlapping image; displaying in real time with a predetermined scaling factor the stream of floating panoramic images in a preview window of a display unit, wherein the displaying comprises shifting the floating panoramic images relative to the preview window when the floating panoramic images scaled according to the predetermined scaling factor extend beyond the preview window so that at least a working portion of the overlapping image associated with the floating panoramic image being displayed is visible in the preview window.
  • In another aspect, the present disclosure provides A handheld electronic device comprising: memory; an image sensor; a display communicatively coupled to the memory; and a processing unit communicatively coupled to the memory, display and image sensor, wherein the memory includes instructions for causing the processing unit to perform an image processing method comprising: receiving a stream of overlapping images; obtaining a corresponding stream of floating panoramic images by computing in real time, for each received overlapping image, an associated floating panoramic image using said overlapping image and a floating panoramic image associated with an overlapping image received prior to said overlapping image; displaying in real time with a predetermined scaling factor the stream of floating panoramic images in a preview window of the display, wherein the displaying comprises shifting the floating panoramic images relative to the preview window when the floating panoramic images scaled according to the predetermined scaling factor extend beyond the preview window so that at least a working portion of the overlapping image associated with the floating panoramic image being displayed is visible in the preview window.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • In order to better understand the subject matter that is disclosed herein and to exemplify how it may be carried out in practice, embodiments will now be described, by way of non-limiting example only, with reference to the accompanying drawings, in which:
  • FIG. 1 is a block diagram illustrating schematically an electronic device according to embodiments of the present disclosure.
  • FIG. 2 is a flow chart illustrating steps of a method of image processing according to embodiments of the present disclosure.
  • FIG. 3 is a diagram illustrating a step of computing floating panoramic images according to some embodiments of the present disclosure.
  • FIG. 4 illustrates a display screen of the electronic device displaying an image and a floating panoramic image associated to said image in a preview window.
  • FIG. 5 illustrates a display implementing a shifting function in some embodiments of the present disclosure.
  • FIG. 6 illustrates a step of displaying a floating panoramic image according to embodiments of the present disclosure.
  • FIGS. 7A-7E illustrate advantageous features of a method according to the present disclosure. FIG. 7A illustrates a preliminary step prior to a scanning step according to some embodiments of the present disclosure. FIG. 7B illustrates an attitude indicator according to some embodiments of the present disclosure. FIG. 7C illustrates a speed indicator according to some embodiments of the present disclosure. FIG. 7D illustrates a target indicator and a scanning indicator in a preview window according to embodiments of the present disclosure. FIG. 7E illustrates a drift indicator in the preview window according to some embodiments of the present disclosure.
  • DETAILED DESCRIPTION OF EMBODIMENTS
  • In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the subject matter. However, it will be understood by those skilled in the art that some examples of the subject matter may be practiced without these specific details. In other instances, well-known methods, procedures and components have not been described in detail so as not to obscure the description.
  • As used herein, the phrase “for example,” “such as”, “for instance” and variants thereof describe non-limiting examples of the subject matter.
  • Reference in the specification to “one example”, “some examples”, “another example”, “other examples, “one instance”, “some instances”, “another instance”, “other instances”, “one case”, “some cases”, “another case”, “other cases” or variants thereof means that a particular described feature, structure or characteristic is included in at least one example of the subject matter, but the appearance of the same term does not necessarily refer to the same example.
  • It should be appreciated that certain features, structures and/or characteristics disclosed herein, which are, for clarity, described in the context of separate examples, may also be provided in combination in a single example. Conversely, various features, structures and/or characteristics disclosed herein, which are, for brevity, described in the context of a single example, may also be provided separately or in any suitable sub-combination.
  • Unless specifically stated otherwise, as apparent from the following discussions, it is appreciated that throughout the specification discussions utilizing terms such as “generating”, “determining”, “providing”, “receiving”, “using”, “computing”, “transmitting”, “performing”, or the like, may refer to the action(s) and/or process(es) of any combination of software, hardware and/or firmware. For example, these terms may refer in some cases to the action(s) and/or process(es) of a programmable machine, that manipulates and/or transforms data represented as physical, such as electronic quantities, within the programmable machine's registers and/or memories into other data similarly represented as physical quantities within the programmable machine's memories, registers and/or other such information storage, transmission and/or display element(s).
  • FIG. 1 illustrates a simplified functional block diagram of an electronic device according to embodiments of the present disclosure. The electronic device may include a display 10, a processor 20, an imaging sensor 30, memory 40 and a position sensor 50. The processor 20 may be any suitable programmable control device and may control the operation of many functions, such as the generation and or processing of image as well as other functions performed by the electronic device. The processor 20 may drive the display (display screen) 10 and may receive user inputs from a user interface. The display screen 10 may be a touch screen capable of receiving user inputs. The memory 40 may store software for implementing various functions of the electronic device including software for implementing the image processing method according to the present disclosure. The memory 40 may also store media such as images and video files. The memory 40 may include one or more storage medium tangibly recording image data and program instructions, including for example a hard-drive, permanent memory and semi permanent memory or cache memory. Program instructions may comprise a software implementation encoded in any desired language. The imaging sensor 30 may be a camera with a predetermined field of view. The camera may either be used in a video mode in which a stream of images is acquired upon command of the user or in a photographic mode in which a single image is acquired upon command of the user. The position sensor 50 may facilitate panorama processing. The position sensor 50 may include a gyroscope enabling calculation of a rotational change of the electronic device from image to image. The position sensor 50 may also be able to determine an acceleration and/or a speed of the electronic device according to three linear axes.
  • FIG. 2 illustrates generally steps of a method of image processing according to embodiments of the present disclosure. The method may be implemented by a processor of a handheld electronic device as previously described. In a first step S100, a stream of overlapping images may be received. The stream of overlapping images may stem from the scanning of an object plane using the camera of the electronic device previously described. The camera may be swept along a scanning axis parallel to a horizontal plane (ground) and to the object plane. The object plane may be a forefront of a retail unit (shelving unit) presenting retail items. The user may adjust the camera to be perpendicular to the floor, for example with the assistance of the attitude indicator described hereinafter, and parallel to shelves of the retail unit, for example with assistance of horizontal lines displayed. In some embodiments, the stream of overlapping images may be displayed in real time on a display screen of the handheld electronic device to form a live video.
  • In a second step S200, a stream of floating panoramic images (stitched images) corresponding to the stream of overlapping images may be computed in real time upon receiving of said overlapping images. The floating panoramic images may be computed recursively. An initial floating panoramic image may be derived from a first image of the stream of overlapping images. In some embodiments, the initial floating panoramic image may be a downsampled version of the first image. In some embodiments, the initial floating panoramic may be a downsampled cropped version of the first image. The cropping of the first image may be performed to crop out the outer border (margins) of the first image. For example, 1 to 5% of the filed of view captured by the first image may be cropped out of the first image. For example, 2%, 3% or 4% may be acceptable values. It is noted that the cropping may be further performed on each overlapping images. In particular, cropping the overlapping images may enable avoiding that small misalignments during the scanning cause black areas (without information) on the floating panoramic images. The floating panoramic images corresponding to a given subsequent overlapping image may be computed by stitching a downsampled (and optionally cropped) version of the given overlapping image with the floating panoramic image computed for the overlapping image directly prior to the given overlapping image. The computing may also involve image registration techniques in order to assess a transformation between subsequent overlapping images. In some embodiments, the transformation may be computed on downsampled (and optionally cropped) versions of subsequent overlapping images). In some embodiments, the transformation may be computed on specific portions of the subsequent downsampled (and optionally cropped) versions of subsequent overlapping images. The registration techniques used are classical in the field of panorama processing and are therefore not described herein for the sake of conciseness. In some embodiments, the stitching may be performed between an inner slice of a downsampled (and optionally cropped) version of the given overlapping image and an edge of the floating panoramic image associated with the overlapping image received prior to said given overlapping image. In other words, a specific inner slice of a downsampled (and optionally cropped) version of a subsequent overlapping image is appended at an edge of the floating panoramic image computed at a prior iteration. This slice by slice stitching enables avoiding backward updating of the floating panoramic images. In an alternative embodiment illustrated in FIG. 3, the stitching may be performed by superimposing an outer (peripheral) slice of a downsampled (and optionally cropped) version of the given overlapping image on the floating panoramic image associated with the overlapping image received prior to said given overlapping image. The superimposing may be performed at an inner stitching portion of the floating panoramic image associated with the overlapping image received prior to said given overlapping image. Said inner stitching portion may belong to the overlapping image received prior to said overlapping image. In some embodiments, the inner stitching portion may be representative of a central portion of the overlapping image received prior to said overlapping image. FIG. 3 represents three downsampled versions of overlapping images I1-I3 acquired by scanning a camera along a predetermined axis X and the corresponding panoramic images P1-P3 computed according to the second alternative. As can be seen, the stitching is performed by superimposing an outer (peripheral) slice of the subsequent overlapping image on the prior floating panoramic image, wherein the outer slice is stitched at an inner portion of the prior floating panoramic image. The Applicant has surprisingly noticed that this stitching option, which incurs backward updating of the stitched image and was therefore not intuitively previously developed, when displayed to the user, enables the user to improve the positioning of the handheld electronic device thereby enabling him to perform the scanning more efficiently. In particular, this stitching option provides the possibility of displaying completely the first frame and, when the user reaches the end of the first shelf, an edge of the frame can fit the edge of the preview window. Furthermore, this stitching option enable time saving for the user and the Applicant has found that it is also more intuitive because the stitched image progresses according to what the user faces and sees on the screen.
  • In a third step S300, the stream of floating panoramic images may be displayed in real time with a predetermined (constant) scaling factor. As illustrated in FIG. 4, the displaying may be performed in a preview window 15 of a display screen 10 of a handheld electronic device as previously described. The preview window 15 may have predetermined maximum dimensions. In some embodiments, the preview window may be divided into a plurality of cells distributed according to one or more rows. The width of the cells may be predetermined. The height of the cells of a first row of cells may be determined so that one cell contains a single frame (image) with the same aspect ratio as the acquired/received overlapping images. In some embodiments in which the margins (borders) are cropped, the height of the cells of the first row may be determined so that one cell contains a single frame with the same aspect ratio as the acquired/received overlapping images after margins crop. The preview window 15 may be centered with respect to the display screen 10 and may be positioned proximate to the bottom of the display screen 10. In some embodiments, the preview window 15 may be a rectangle placed on top of the live video. Indeed, as shown on FIG. 4, the last received/acquired overlapping image of the stream may be displayed in real time on the display screen to form the live video. The preview window 15 may provide blank areas when the floating panorama does not extend beyond its dimensions and the blank areas may be semi-transparent. In contrast, display of the floating panoramic image in the preview window may be carried out in an opaque manner. In some embodiments, when the floating panoramic image reaches an extremity of a cell, a following cell may be highlighted. The subsequent floating panoramic images may then progress into the highlighted cell.
  • The Applicant has found that a user may be particularly confused when performing scanning of long amplitude at short distance of objects which resemble each other. Therefore, the Applicant provides hereby a displaying which maintains a scaling factor of the floating panoramic image into a preview window and also provides a shifting function of the floating panoramic image when the floating panoramic image extends beyond the dimensions of the preview window. Thus, the displaying comprises shifting the floating panoramic images relative to the preview window when the floating panoramic images scaled according to the predetermined scaling factor extend beyond the preview window so that at least a working portion of the overlapping image associated with the floating panoramic image being displayed is visible in the preview window. The working portion may comprise a central portion of the overlapping image associated with the floating panoramic image being displayed and/or a fronto-parallel portion of the overlapping image associated with the displayed floating panoramic image. The fronto-parallel portion may be a portion of the overlapping image representative of a region of the object plane extending slightly around a line perpendicular to the object plane and passing by the camera when the camera acquires said overlapping image. In some embodiments, the fronto-parallel region extends of around ±5 degrees around the perpendicular line. In some embodiments, an amplitude of the shifting may be constant, for example a cell size. In some embodiments, the shifting may provide a blank area of a predetermined size in the preview window thereby enabling displaying at least one subsequent floating panoramic image using the same shifting amplitude i.e. so that the working portion for the subsequent floating panoramic image be visible without modifying the shifting amplitude. In other words, the floating panoramic image may be shifted sufficiently for being able to display one or more subsequent floating panoramic image without modifying the shifting amplitude. The shifting enables to make a possibly concealed specific portion of the floating panoramic image visible on the display screen and with a resolution that enables the user to apprehend it. The Applicant has found that displaying the working portion is of particular importance for enabling the user to position the camera for appropriately performing long amplitude scanning at short distance of objects which resemble each other. In some embodiments, the shifting may be implemented smoothly so as to improve user comfort.
  • For example, as illustrated on FIG. 5, in a left-to-right scan with a preview window consisting of three cells, once the floating panoramic image reaches an end of the last cell (see stitched image step #N+2), the floating panoramic image is shifted by one cell to the left i.e.:
      • The floating panoramic image information that appeared in the first cell (extreme left) disappears;
      • The floating panoramic image information that appeared in the second cell now appears in the first cell;
      • The floating panoramic image information that appeared in the third cell (extreme right) now appears in the second cell
      • The third cell is emptied.
  • In some embodiments, when the scanning according to a first direction ends, the present disclosure may comprise another type of shifting of the floating panoramic image for fitting the extremity of the preview window with the extremity of the floating panoramic image corresponding to the end of the scanning according to said first direction. The user may actuate a dedicated icon to mark the end of the scanning (end of a shelf). Images resulting from a scanning according to another direction (vertical) may further be acquired and stitched to the floating panoramic images previously computed. The resulting floating panoramic images may be displayed in the preview window. For example the resulting floating panoramic images may also fill a second row of cells of the preview window.
  • In some embodiments, an undo function may be available to the user upon command, for example by pressing an undo icon item on the display screen. The undo function may enable the user to cancel the last stitching which occurred. The undo function may also enable the user to cancel a predetermined amount of stitching which previously occurred. The undo function may for example be advantageous in cases of obstruction of the camera by any unwanted objects. When undo operation is performed, another type of shifting may occur to optimize the occupancy of the preview window. This enables to reallocate the preview window to a portion of the floating panoramic image which had been expelled from the preview window.
  • Furthermore, as illustrated on FIG. 6, the displaying of a floating panoramic image associated with a given overlapping image may comprises revealing instantaneously an overlying portion B of the floating panoramic image, wherein the overlying portion is analogous to a corresponding portion A of a floating panoramic image associated with an overlapping image received prior to said given overlapping image. Further, the displaying of the floating panoramic image may comprise revealing progressively an additional portion C of the floating panoramic image. A rate for progressively revealing the additional portion C may be proportional to the scanning speed.
  • Furthermore, the present disclosure may also comprise monitoring and displaying in real time an indication related to a distance from the electronic device (camera) to the object plane. The distance may be derived from the position sensor of the electronic device and/or using image processing. A distance indicator may be displayed to warn the user that the distance is within and/or out of an acceptable range. In some embodiments, as shown in FIG. 7A, a preliminary step including distance calibration may be performed prior to scanning of the retail unit. For example, when imaging a known retail item (or at least an item from a category with known proportions), the distance to the object plane may be calibrated by matching the imaged retail item with a template image 61 of said retail item stored in the memory of the electronic device and displayed during the preliminary step together with an image preview of the retail unit. When the dimensions of the retail item in the template image coincide with the dimensions of the imaged retail item, the distance to the object plane may be considered as satisfactory. The template image 61 may only be a representation of the silhouette of the retail item. In some embodiments, when the retail item or retail item category is unknown, a generic template may be used. The Applicant has found that in order to optimize the scanning, a product should approximately occupy 30% of the field of view of the imaging sensor (±5% or 10%). Further, modification of the distance to the object plane may be monitored by using the position sensor and/or image processing and displaying of the distance indicator may be performed in real time. For example, the distance indicator may be superimposed on the overlapping image display. Further, as illustrated, the preliminary step may also include a definition of a scanning direction by using a specific icon of a user interface (i.e. left-to-right or right-to-left with respect to the user holding the camera).
  • Further, the present disclosure may also comprise monitoring in real time an attitude of the camera with respect to a reference attitude. The attitude may be monitored using the position sensor of the electronic device and/or by using image processing. The reference attitude may be an attitude of the camera when acquiring a first image of the stream of images. The rotational change with respect to the reference attitude may be monitored in order to keep the camera facing the object plane. An attitude indicator may be displayed in real time to warn the user that the camera's attitude is out of an acceptable range. For example, the attitude indicator may be superimposed on the overlapping image display. In some embodiments, instead of displaying the Euler angles as a series of digits, the attitude indicator may be displayed as a geometrical shape distorted in accordance with the rotational change between a current attitude and the reference attitude. In some embodiments illustrated on FIG. 7B, the geometrical shape may be a rectangle 62 enveloping a periphery of the display screen. The distortion of the rectangle may be a rotation according to the measured rotational change followed by a projection. When the rotational change is out of an acceptable range, the rectangle may be displayed with a predetermined color different to that shown when the rotational change is acceptable so as to attract the user's attention.
  • Furthermore, the present disclosure may also comprise monitoring a scanning speed of the camera using the position sensor of the electronic device and/or image processing to assess motion blur. As shown on FIG. 7C, a speed indicator 63 may be displayed in real time to warn the user that the scanning speed is within and/or out of an acceptable range.
  • Furthermore, as shown in FIG. 7D, the present disclosure may also comprise displaying in real time a scanning direction indicator 64 and a target indicator 65 indicative of the predetermined scanning direction so as to improve user comfort. Advantageously, the scanning direction indicator 64 (arrow) and the target indicator 65 (circle) may be displayed as an over layer (superimposed) in the preview window in which the floating panoramic image is to be displayed. The scanning direction indicator 64 and the target indicator 65 may be updated so as to progressively follow the filling of the preview window by the floating panoramic images. For example, when the preview window is divided into cells, when the floating panoramic image reaches an extremity of a cell, the target indicator 65 may be swept to the following cell. The target indicator 65 may be semi transparent. The scanning direction indicator 64 may follow the target indicator 65. Following to a shifting of one cell size, the scanning direction indicator may shift back at the start of the last cell of the row of cells currently modified.
  • Additionally, the present disclosure may also comprise displaying in real time a drift indicator indicative of a vertical drift of the electronic device during the scanning. The term vertical refers herein to a direction perpendicular to the scanning direction and parallel to the object plane. The vertical drift may be derived from the position sensor of the electronic device and/or using image processing. As shown in FIG. 7E, in some embodiments the drift indicator may be combined with the scanning direction indicator and the drift indication may be provided by modifying a relative position of the scanning direction indicator and target indicator. For example, this can be done by misaligning the center of the target indicator 65 with the head of the arrow 64. In order to assess a drift level, assisting lines may be displayed, wherein a relative position of the drift indicator with respect to the assisting lines indicates the drift level.
  • While certain features of the invention have been illustrated and described herein, many modifications, substitutions, changes, and equivalents will now occur to those of ordinary skill in the art. It is, therefore, to be understood that the appended claims are intended to cover all such modifications and changes as fall within the true spirit of the invention.
  • It will be appreciated that the embodiments described above are cited by way of example, and various features thereof and combinations of these features can be varied and modified.
  • While various embodiments have been shown and described, it will be understood that there is no intent to limit the invention by such disclosure, but rather, it is intended to cover all modifications and alternate constructions falling within the scope of the invention, as defined in the appended claims.
  • It will also be understood that the system according to the presently disclosed subject matter can be implemented, at least partly, as a suitably programmed computer. Likewise, the presently disclosed subject matter contemplates a computer program being readable by a computer for executing the disclosed method. The presently disclosed subject matter further contemplates a machine-readable memory tangibly embodying a program of instructions executable by the machine for executing the disclosed method.

Claims (24)

1. An image processing method comprising:
receiving a stream of overlapping images;
obtaining a corresponding stream of floating panoramic images by computing in real time, for each received overlapping image, an associated floating panoramic image using said overlapping image and a floating panoramic image associated with an overlapping image received prior to said overlapping image;
displaying in real time with a predetermined scaling factor the stream of floating panoramic images in a preview window of a display unit,
wherein the displaying comprises shifting the floating panoramic images relative to the preview window when the floating panoramic images scaled according to the predetermined scaling factor extend beyond the preview window so that at least a working portion of the overlapping image associated with the floating panoramic image being displayed is visible in the preview window.
2. The method according to claim 1, wherein an initial floating panoramic image is derived from a first image of the stream of overlapping images.
3. The method according to claim 1, wherein an amplitude of the shifting further enables to leave a blank area of a predetermined size in the preview window thereby enabling displaying at least one subsequent floating panoramic image using the same shifting amplitude.
4. The method according to claim 1, wherein the stream of overlapping images is acquired by moving an imaging sensor along a predetermined direction.
5. The method according to claim 1, wherein the received stream of overlapping images result from a selection of acquired overlapping images acquired by moving an imaging sensor along a predetermined direction.
6. The method according to claim 1, wherein the computing comprises cropping a margin around the received overlapping images.
7. The method according to claim 1, wherein obtaining an initial floating panoramic image comprises downsampling the first image of the stream of overlapping images and the real time computing of the floating panoramic image associated with each overlapping image comprises stitching a downsampled version of said overlapping image and the floating panoramic image associated with an overlapping image received prior to said overlapping image.
8. The method according to claim 7, wherein the stitching is performed by appending an inner slice of the overlapping image at an edge of the floating panoramic image associated with the overlapping image received prior to said overlapping image.
9. The method according to claim 7, wherein the stitching is performed by superimposing an outer slice of the overlapping image at an inner stitching portion of the floating panoramic image associated with the overlapping image received prior to said overlapping image.
10. The method according to claim 9, wherein the displaying of a floating panoramic image associated with a given overlapping image comprises:
(a) revealing instantaneously an overlying portion of the floating panoramic image, wherein the overlying portion is analogous to a corresponding portion of a floating panoramic image associated with an overlapping image received prior to said given overlapping image; and
(b) revealing progressively an additional portion of the floating panoramic image.
11. The method according to claim 1, further comprising displaying in real time the overlapping image associated with the floating panoramic image.
12. The method according to claim 1, further comprising displaying a speed indicator indicative of a speed of an imaging sensor providing the stream of overlapping images.
13. The method according to claim 1, further comprising displaying a distance indicator indicative of a distance from an imaging sensor providing the stream of overlapping images to a forefront of a scene imaged by the imaging sensor.
14. The method according to claim 1, further comprising displaying a scan direction indicator indicative of a predetermined scan direction.
15. The method according to claim 1, further comprising displaying in the preview window a target indicator indicative of an expected display of a subsequent floating panoramic image.
16. The method according to claim 1, further comprising displaying a drift indicator indicative of a drift between an actual trajectory of an imaging sensor providing the stream of overlapping images and an expected trajectory of said imaging sensor.
17. The method according to claim 16, further comprising displaying assisting lines bounding a position of the drift indicator so as to facilitate a visual assessment of the drift.
18. The method according to claim 11, wherein the preview window is superimposed on the display of the overlapping image.
19. The method according to claim 1, wherein the preview window has predetermined dimensions.
20. The method according to claim 1, wherein the preview window is divided into one or more rows of cells.
21. A method of scanning a retail unit comprising:
moving an imaging sensor along a predetermined direction while acquiring a stream of overlapping images of the retail unit;
for each acquired overlapping image, computing in real time a floating panoramic image associated with said overlapping image using said overlapping image and a floating panoramic image associated with an overlapping image acquired prior to said overlapping image, wherein an initial floating panoramic image is derived from a first image of the stream of overlapping images, thereby obtaining a stream of floating panoramic images;
displaying in real time with a predetermined scaling factor the stream of floating panoramic images in a preview window of a display unit,
wherein the displaying comprises shifting the floating panoramic images relative to the preview window when the floating panoramic images scaled according to the predetermined scaling factor extend beyond the preview window so that at least a working portion of the overlapping image associated with the floating panoramic image being displayed is visible in the preview window.
22. The method according to claim 1, wherein the overlapping images are received from an image sensor of a handheld electronic device, the computing is run on a processing unit of the handheld electronic device and the display unit is comprised in said handheld electronic device.
23. A computer program product implemented on a non-transitory computer usable medium having computer readable program code embodied therein to cause the computer to perform an image processing method comprising:
receiving a stream of overlapping images;
obtaining a corresponding stream of floating panoramic images by computing in real time, for each received overlapping image, an associated floating panoramic image using said overlapping image and a floating panoramic image associated with an overlapping image received prior to said overlapping image;
displaying in real time with a predetermined scaling factor the stream of floating panoramic images in a preview window of a display unit,
wherein the displaying comprises shifting the floating panoramic images relative to the preview window when the floating panoramic images scaled according to the predetermined scaling factor extend beyond the preview window so that at least a working portion of the overlapping image associated with the floating panoramic image being displayed is visible in the preview window.
24. A handheld electronic device comprising:
(a) memory;
(b) an image sensor;
(c) a display communicatively coupled to the memory; and
(d) a processing unit communicatively coupled to the memory, display and image sensor, wherein the memory includes instructions for causing the processing unit to perform an image processing method comprising:
receiving a stream of overlapping images;
obtaining a corresponding stream of floating panoramic images by computing in real time, for each received overlapping image, an associated floating panoramic image using said overlapping image and a floating panoramic image associated with an overlapping image received prior to said overlapping image;
displaying in real time with a predetermined scaling factor the stream of floating panoramic images in a preview window of the display,
wherein the displaying comprises shifting the floating panoramic images relative to the preview window when the floating panoramic images scaled according to the predetermined scaling factor extend beyond the preview window so that at least a working portion of the overlapping image associated with the floating panoramic image being displayed is visible in the preview window.
US14/151,011 2014-01-09 2014-01-09 Method and device for panoramic image processing Abandoned US20150193909A1 (en)

Priority Applications (3)

Application Number Priority Date Filing Date Title
US14/151,011 US20150193909A1 (en) 2014-01-09 2014-01-09 Method and device for panoramic image processing
PCT/IL2014/051139 WO2015104705A1 (en) 2014-01-09 2014-12-31 Method and device for panoramic image processing
US15/187,276 US10122915B2 (en) 2014-01-09 2016-06-20 Method and device for panoramic image processing

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US14/151,011 US20150193909A1 (en) 2014-01-09 2014-01-09 Method and device for panoramic image processing

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US15/187,276 Continuation US10122915B2 (en) 2014-01-09 2016-06-20 Method and device for panoramic image processing

Publications (1)

Publication Number Publication Date
US20150193909A1 true US20150193909A1 (en) 2015-07-09

Family

ID=53495584

Family Applications (2)

Application Number Title Priority Date Filing Date
US14/151,011 Abandoned US20150193909A1 (en) 2014-01-09 2014-01-09 Method and device for panoramic image processing
US15/187,276 Active US10122915B2 (en) 2014-01-09 2016-06-20 Method and device for panoramic image processing

Family Applications After (1)

Application Number Title Priority Date Filing Date
US15/187,276 Active US10122915B2 (en) 2014-01-09 2016-06-20 Method and device for panoramic image processing

Country Status (2)

Country Link
US (2) US20150193909A1 (en)
WO (1) WO2015104705A1 (en)

Cited By (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160112676A1 (en) * 2014-03-20 2016-04-21 Hangzhou Hikvision Digital Technology Co., Ltd. Method and system for video stitching
US20160119540A1 (en) * 2014-10-23 2016-04-28 Xerox Corporation Model-based plane-like panorama and retail applications
US20170046868A1 (en) * 2015-08-14 2017-02-16 Samsung Electronics Co., Ltd. Method and apparatus for constructing three dimensional model of object
US10368662B2 (en) 2013-05-05 2019-08-06 Trax Technology Solutions Pte Ltd. System and method of monitoring retail units
US10387996B2 (en) 2014-02-02 2019-08-20 Trax Technology Solutions Pte Ltd. System and method for panoramic image processing
US10402777B2 (en) 2014-06-18 2019-09-03 Trax Technology Solutions Pte Ltd. Method and a system for object recognition
US10482574B2 (en) * 2016-07-06 2019-11-19 Gopro, Inc. Systems and methods for multi-resolution image stitching
US10552933B1 (en) 2015-05-20 2020-02-04 Digimarc Corporation Image processing methods and arrangements useful in automated store shelf inspections
US10565548B2 (en) * 2016-03-29 2020-02-18 Bossa Nova Robotics Ip, Inc. Planogram assisted inventory system and method
US11107238B2 (en) * 2019-12-13 2021-08-31 Zebra Technologies Corporation Method, system and apparatus for detecting item facings
WO2022187354A1 (en) * 2021-03-04 2022-09-09 Focal Systems, Inc. On-shelf image based barcode reader for inventory management system
US11463700B2 (en) * 2018-01-03 2022-10-04 Huawei Technologies Co., Ltd. Video picture processing method and apparatus
CN115314750A (en) * 2022-08-10 2022-11-08 润博全景文旅科技有限公司 Video playing method, device and equipment
WO2023125362A1 (en) * 2021-12-29 2023-07-06 影石创新科技股份有限公司 Image display method and apparatus, and electronic device
US11774842B2 (en) 2019-08-16 2023-10-03 Bossa Nova Robotics Ip, Inc. Systems and methods for image capture and shelf content detection
US11842321B1 (en) * 2021-03-17 2023-12-12 Amazon Technologies, Inc. Image-based detection of planogram product spaces

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018149476A1 (en) 2017-02-14 2018-08-23 Hp Indigo B.V. Image processing apparatus
KR101810671B1 (en) * 2017-03-07 2018-01-25 링크플로우 주식회사 Method for generating direction information of omnidirectional image and device for performing the method
US11768030B2 (en) 2017-08-10 2023-09-26 Cooler Screens Inc. Smart movable closure system for cooling cabinet
US10769666B2 (en) 2017-08-10 2020-09-08 Cooler Screens Inc. Intelligent marketing and advertising platform
US11763252B2 (en) 2017-08-10 2023-09-19 Cooler Screens Inc. Intelligent marketing and advertising platform
US11698219B2 (en) 2017-08-10 2023-07-11 Cooler Screens Inc. Smart movable closure system for cooling cabinet
CN113344959B (en) * 2021-08-06 2021-11-09 山东捷瑞数字科技股份有限公司 Residual material analysis processing method and device and material conveying system

Family Cites Families (29)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7620909B2 (en) 1999-05-12 2009-11-17 Imove Inc. Interactive image seamer for panoramic images
US6618511B1 (en) * 1999-12-31 2003-09-09 Stmicroelectronics, Inc. Perspective correction for panoramic digital camera with remote processing
DE60215330T2 (en) 2001-04-20 2007-09-06 Camera Dynamics Inc. STABILIZED HIGH PERFORMANCE CAMERA HEAD WITH CAMERA POSITION DETECTION
US7467061B2 (en) 2004-05-14 2008-12-16 Canon Kabushiki Kaisha Information processing method and apparatus for finding position and orientation of targeted object
US7424218B2 (en) * 2005-07-28 2008-09-09 Microsoft Corporation Real-time preview for panoramic images
US7460730B2 (en) * 2005-08-04 2008-12-02 Microsoft Corporation Video registration and image sequence stitching
US20070070233A1 (en) 2005-09-28 2007-03-29 Patterson Raul D System and method for correlating captured images with their site locations on maps
US20070081081A1 (en) * 2005-10-07 2007-04-12 Cheng Brett A Automated multi-frame image capture for panorama stitching using motion sensor
US20100171826A1 (en) * 2006-04-12 2010-07-08 Store Eyes, Inc. Method for measuring retail display and compliance
KR100790890B1 (en) 2006-09-27 2008-01-02 삼성전자주식회사 Apparatus and method for generating panorama image
US8515207B2 (en) 2007-05-25 2013-08-20 Google Inc. Annotations in panoramic images, and applications thereof
US9135491B2 (en) * 2007-08-31 2015-09-15 Accenture Global Services Limited Digital point-of-sale analyzer
US20090192921A1 (en) * 2008-01-24 2009-07-30 Michael Alan Hicks Methods and apparatus to survey a retail environment
US8355042B2 (en) 2008-10-16 2013-01-15 Spatial Cam Llc Controller in a camera for creating a panoramic image
JP5508401B2 (en) 2008-06-05 2014-05-28 コーニンクレッカ フィリップス エヌ ヴェ Ultrasound imaging of extended field of view by guided EFOV scanning
US8351894B2 (en) 2009-02-27 2013-01-08 Research In Motion Limited Mobile wireless communications device with orientation sensing and related methods
JP5235798B2 (en) * 2009-06-22 2013-07-10 富士フイルム株式会社 Imaging apparatus and control method thereof
JP5558956B2 (en) 2010-07-29 2014-07-23 キヤノン株式会社 Imaging apparatus and control method thereof
JP5413344B2 (en) * 2010-09-27 2014-02-12 カシオ計算機株式会社 Imaging apparatus, image composition method, and program
US20120293607A1 (en) 2011-05-17 2012-11-22 Apple Inc. Panorama Processing
US8970665B2 (en) 2011-05-25 2015-03-03 Microsoft Corporation Orientation-based generation of panoramic fields
US9785898B2 (en) * 2011-06-20 2017-10-10 Hi-Tech Solutions Ltd. System and method for identifying retail products and determining retail product arrangements
US8559766B2 (en) 2011-08-16 2013-10-15 iParse, LLC Automatic image capture
US9129277B2 (en) * 2011-08-30 2015-09-08 Digimarc Corporation Methods and arrangements for identifying objects
US8908903B2 (en) * 2011-08-31 2014-12-09 The Nielsen Company (Us), Llc Image recognition to support shelf auditing for consumer research
EP2791860A4 (en) * 2011-12-16 2015-05-27 Thomas Michael Cozad Jr Systems and methods for managing product location information
US8704904B2 (en) 2011-12-23 2014-04-22 H4 Engineering, Inc. Portable system for high quality video recording
US8882662B2 (en) 2012-06-27 2014-11-11 Camplex, Inc. Interface for viewing video from cameras on a surgical visualization system
US9489765B2 (en) * 2013-11-18 2016-11-08 Nant Holdings Ip, Llc Silhouette-based object and texture alignment, systems and methods

Cited By (21)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10368662B2 (en) 2013-05-05 2019-08-06 Trax Technology Solutions Pte Ltd. System and method of monitoring retail units
US10387996B2 (en) 2014-02-02 2019-08-20 Trax Technology Solutions Pte Ltd. System and method for panoramic image processing
US20160112676A1 (en) * 2014-03-20 2016-04-21 Hangzhou Hikvision Digital Technology Co., Ltd. Method and system for video stitching
US9961305B2 (en) * 2014-03-20 2018-05-01 Hangzhou Hikvision Digital Technology Co., Ltd. Method and system for video stitching
US10402777B2 (en) 2014-06-18 2019-09-03 Trax Technology Solutions Pte Ltd. Method and a system for object recognition
US20160119540A1 (en) * 2014-10-23 2016-04-28 Xerox Corporation Model-based plane-like panorama and retail applications
US11587195B2 (en) 2015-05-20 2023-02-21 Digimarc Corporation Image processing methods and arrangements useful in automated store shelf inspections
US10552933B1 (en) 2015-05-20 2020-02-04 Digimarc Corporation Image processing methods and arrangements useful in automated store shelf inspections
US20170046868A1 (en) * 2015-08-14 2017-02-16 Samsung Electronics Co., Ltd. Method and apparatus for constructing three dimensional model of object
US10360718B2 (en) * 2015-08-14 2019-07-23 Samsung Electronics Co., Ltd. Method and apparatus for constructing three dimensional model of object
US10565548B2 (en) * 2016-03-29 2020-02-18 Bossa Nova Robotics Ip, Inc. Planogram assisted inventory system and method
US11030717B2 (en) * 2016-07-06 2021-06-08 Gopro, Inc. Apparatus and methods for multi-resolution image stitching
US11475538B2 (en) 2016-07-06 2022-10-18 Gopro, Inc. Apparatus and methods for multi-resolution image stitching
US10482574B2 (en) * 2016-07-06 2019-11-19 Gopro, Inc. Systems and methods for multi-resolution image stitching
US11463700B2 (en) * 2018-01-03 2022-10-04 Huawei Technologies Co., Ltd. Video picture processing method and apparatus
US11774842B2 (en) 2019-08-16 2023-10-03 Bossa Nova Robotics Ip, Inc. Systems and methods for image capture and shelf content detection
US11107238B2 (en) * 2019-12-13 2021-08-31 Zebra Technologies Corporation Method, system and apparatus for detecting item facings
WO2022187354A1 (en) * 2021-03-04 2022-09-09 Focal Systems, Inc. On-shelf image based barcode reader for inventory management system
US11842321B1 (en) * 2021-03-17 2023-12-12 Amazon Technologies, Inc. Image-based detection of planogram product spaces
WO2023125362A1 (en) * 2021-12-29 2023-07-06 影石创新科技股份有限公司 Image display method and apparatus, and electronic device
CN115314750A (en) * 2022-08-10 2022-11-08 润博全景文旅科技有限公司 Video playing method, device and equipment

Also Published As

Publication number Publication date
US20160309082A1 (en) 2016-10-20
US10122915B2 (en) 2018-11-06
WO2015104705A1 (en) 2015-07-16

Similar Documents

Publication Publication Date Title
US10122915B2 (en) Method and device for panoramic image processing
US10387996B2 (en) System and method for panoramic image processing
KR101608869B1 (en) Method for indicating a virtual ruler on separated image or medical image of object, medical image obtaining apparatus, and apparatus and method for displaying separated image or medical image with a virtual ruler
KR102217204B1 (en) Omnistereo capture for mobile devices
US20170213375A1 (en) Image processing device and image processing method for image correction, and non-transitory computer readable recording medium thereof
TWI460682B (en) Panorama processing
JP5573349B2 (en) Panorama development image photographing system and method
CN106133794B (en) Information processing method, information processing apparatus, and program
EP2620914A2 (en) Three-dimensional scan recovery
US20170140574A1 (en) Image processing device and image processing method
JP2010109452A (en) Vehicle surrounding monitoring device and vehicle surrounding monitoring method
JP4901212B2 (en) Medical image processing device
US20180213156A1 (en) Method for displaying on a screen at least one representation of an object, related computer program, electronic display device and apparatus
US11361455B2 (en) Systems and methods for facilitating the identifying of correspondences between images experiencing motion blur
US20240046430A1 (en) Corrective Lighting for Video Inpainting
KR101597915B1 (en) Image processing apparatus and image processing method
WO2016147778A1 (en) Image processing device, method, and program
US10216373B2 (en) Image processing apparatus for position adjustment between multiple frames included in a video
JP2006061274A (en) Program and endoscope system
US20220409324A1 (en) Systems and methods for telestration with spatial memory
WO2008073563A1 (en) Method and system for gaze estimation
US20130201178A1 (en) System and method providing a viewable three dimensional display cursor
KR20160040492A (en) Method for indicating a virtual ruler on separated image or medical image of object, medical image obtaining apparatus, and apparatus and method for displaying separated image or medical image with a virtual ruler
JP2009212711A (en) Image processor
JP6962242B2 (en) Information processing device, superimposition display program, superimposition display method

Legal Events

Date Code Title Description
AS Assignment

Owner name: TRAX TECHNOLOGY SOLUTIONS PTE LTD., SINGAPORE

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:MAAYAN, HILIT;COHEN, DANIEL SHIMON;GREENSHPAN, JACOB;SIGNING DATES FROM 20140214 TO 20140220;REEL/FRAME:032683/0459

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION

AS Assignment

Owner name: PACIFIC WESTERN BANK, NORTH CAROLINA

Free format text: SECURITY INTEREST;ASSIGNOR:TRAX TECHNOLOGY SOLUTIONS PTE LTD.;REEL/FRAME:049564/0715

Effective date: 20190510

AS Assignment

Owner name: TRAX TECHNOLOGY SOLUTIONS PTE LTD., SINGAPORE

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:PACIFIC WESTERN BANK;REEL/FRAME:052662/0568

Effective date: 20200513