SE542615C2

SE542615C2 - Systems, methods and computer programs for container space allocation monitoring

Info

Publication number: SE542615C2
Application number: SE1751462A
Authority: SE
Inventors: Barbara Terebieniec; Cristian Klein
Original assignee: Barbara Terebieniec; Cristian Klein
Priority date: 2017-11-28
Filing date: 2017-11-28
Publication date: 2020-06-16
Also published as: SE1751462A1; WO2019108117A1

Abstract

The present disclosure relates to a system (100) for container space allocation monitoring. The system (100) comprises at least one camera (10a-c) placed to face a respective region in front of an opening (112) of a container (110). The system further comprises a database (14) configured to store information relating to items stored in the container and their respective positions within the container space (120). The system also comprises control circuitry (12). The control circuitry is configured to detect a symbol of an item based on processing one or more images captured by the at least one camera. The control circuitry is further configured to determine a position of the detected symbol based on image processing of at least one image comprising the detected symbol. The control circuitry is also configured to determine a trajectory of the detected symbol based on a set of determined at least one positions of the detected symbol. The control circuitry is additionally configured to determine insertion or removal of the item and a position of the item relative to a position of the container space based on the determined trajectory, and to update the stored information in the database based on the determined insertion or removal of the item and the determined position of the item relative to the position of the container space.The present disclosure also relates to corresponding methods and computer programs.

Description

1 Systems, methods and computer programs for container space allocation monitoringTECHNICAL FIELD The present disclosure re|ates to efficient identification of item location with respect to acontainer space. ln particular, the present disclosure re|ates to systems, methods and computer programs for container space allocation monitoring.BACKGROUND ln many lab settings, a biological sample is to be repeatedly stored in and removed from afreezer, often with considerable time between insertion and removal of the biological sample.The biological sample is typically held within a vial. The vial is then placed in a box arranged tohold a plurality of vials. The box may in turn be arranged in one of several racks of the freezer.A typical freezer may contain on the order of five hundred boxes. ln many workplaces thelocation ofthe box in which a biological sample has been placed is tracked using pen and paperor an ad-hoc digital table to be filled in by the person responsible for the biological sample.

Finding the right box again after having inserted a biological sample can be time consuming.

One reason for this is that in many workplaces, there is naturally a high staff turn-around andinformation on where the leaving staff kept their biological samples is often lost. Anothersource for the trouble in recovering the right biological sample is that during freezer failures ormaintenance, the person(s) responsible for taking care of the failure or maintenance are oftendifferent from the people working with the biological samples. Hence, it is common that theperson(s) responsible for taking care of the failure or maintenance to inadvertently shuffle theracks holding the boxes with biological samples around, and sometimes even moving the boxesholding the biological sample vials around as well to facilitate their own work. Since thepersons working in lab environments are typically high skilled workers, with high matchingsalaries, the additional time spent on identifying the location of a stored sample adds up overtime and results in lost efficiency. A further potential cause for lost time is a need to identifywhat boxes are still in use and what boxes are obsolete, since this would require going through all the boxes in the freezer one by one.

The above scenario is described as relating to storage of items in freezers in a lab environment.

However, there are other scenarios in which similar difficulties occur. lt can for example relate 2to libraries where the items are books. lt may also be applicable to shopping malls or grocerystores wherein each item for sale may be monitored accordingly. Finally, it may concern post offices where items are parcels stored on shelves. ln each of the above scenarios, there is a need to decrease amount of time spent looking for items which have been stored away in a storage for storing a large quantity of items.SUMMARY One of the main problems faced when inserting and removing items in containers requiringmultiple users to keep track of the items in order to maintain a well-organized storage withinthe container is that the process of monitoring insertion and removal, as well as keeping trackof where every item is stored, may take up a lot of time. The present disclosure suggestssystems, methods and computer programs where symbols, e.g. barcodes, are placed on anitem, such as a box holding an organic sample, and the symbol is detected by one or morecameras. Based on trajectories and/or current content of the container, control circuitry of thesystem determines if the item is being inserted or removed. The container may comprisestorage components, such as racks, which may also be labelled with unique symbols, therebyenabling matching the symbol of the item with a symbol of a storage component in order to associate the item with a location within the container.

I\/|ore specifically, the present disclosure relates to a system for container space allocationmonitoring. The system comprises at least one camera placed to face a respective region infront of an opening of a container. The system further comprises a database configured tostore information relating to items stored in the container and their respective positions withinthe container space. The system also comprises control circuitry. The control circuitry isconfigured to detect a symbol of an item based on processing one or more images captured bythe at least one camera. The control circuitry is further configured to determine a position ofthe detected symbol based on image processing of at least one image comprising the detectedsymbol. The control circuitry is also configured to determine a trajectory of the detectedsymbol based on a set of determined at least one positions of the detected symbol. Thecontrol circuitry is additionally configured to determine insertion or removal of the item and aposition of the item relative to a position of the container space based on the determined trajectory. The control circuitry is yet further configured to update the stored information in 3 the database based on the determined insertion or removal of the item and the determinedposition of the item relative to the position of the container space. The system thereby usesthe trajectory of the determined symbol and the stored information in the database to derive acontextual relationship which is used to determine if the item carrying the symbol is beinginserted or removed from the container and a position of the item relative to a position of thecontainer space. The position of the item relative to a position of the container space could beas simple as ”outside” or ”inside” the container space, but could in some examples use asymbol of a storage component ofthe container space to determine which storage componentthe item has been inserted in or removed from. The position of the item's symbol, relative tothe container space and/or relative to a symbol of a storage component, may be recorded in the database to speed up retrieval of items stored in the container space.

According to some aspects, at least part ofthe control circuitry and one or more of the at leastone camera are comprised in a single logical unit such that detecting the symbol of an item,determining the position of the detected symbol, determining the trajectory of the detectedsymbol and determining insertion or removal of the item and a position of the item relative toa position of the container space are performed within the single logical unit. According tosome further aspects, the single logical unit comprises a single-board computer. By integratingthe camera(s) with the downstream processing performed by the control circuitry, encoding and decoding of image data can be omitted.

According to some aspects, the system further comprises a motion filter arranged todetermine changes between consecutive images captured by the at least one camera. Themotion filter thereby facilitates real-time identification of symbols without the need for substa ntial com putational resources.

According to some aspects, the system further comprises a video encoder and a video decoder.The video encoder is configured to encode image data in a compressed format. The videodecoder is configured to decode/decompress the compressed format image data. The use ofvideo encoders and video decoders enables effective ways of separating the componentsassociated with generating image data, such as cameras and motion filters (in addition to videoencoders), from components associated with interpretation of the generated image data. ln particular, a video encoder enables efficient transfer of image data between the image data 4generating components and the image data interpreting components. According to somefurther aspects, the at least one camera comprises the motion filter and/or the video encoder.The camera thereby not only captures images, but also performs local image processing in the Camera.

According to some aspects, the control circuitry is further configured to ensure correctchronological order of a plurality of separately processed images. This enables parallelprocessing of the image data associated with the images. lf the control circuitry is able toprocess image data relating to different images in parallel, the processing of one imagecorresponding to a second time later than a first time may be finished before another imagecorresponding to the first time, which would cause the processed image data to appear out of order; by ensuring correct chronological order, such problems can be avoided.

According to some aspects, the system further comprises a memory buffer configured toreceive the compressed format image data from the video encoder and store the compressedformat image data until a remaining processing pipeline is ready to process the storedcompressed format image data. The system thereby becomes more stable with respect todownstream processes, which often are more computationally expensive. The buffer canensure smooth real-time monitoring of the container. ln case the at least one cameracomprises a motion filter and/or a video encoder, the processed image data may betransmitted to the memory buffer directly from the at least one camera. According to someaspects, the at least one camera also comprises the memory buffer. ln such case, the at leastone camera transmits image data from the buffer to downstream processes, such as apotential video decoder, when a remaining processing pipeline is ready to process the stored compressed format image data.

According to some aspects, the image processing of the at least one image comprising thedetected symbol is configured to determine an estimated relative position of the detectedsymbol to another symbol and/or an environmental marker. ln addition to provide informationof where in space the item carrying the detected symbol is, i.e. its present location, theestimated relative position may be used to establish a contextual relationship between thecontainer and the item. lf the position of the container with respect to the other symbol or environmental marker is known, changes in the relative position ofthe detected symbol to the other symbol and/or the environmental marker can tell if the item is moving towards or awayfrom the container. lf, for instance, the database has no indication that the item is consideredstored within the container and a series of relative positions indicates a movement of the itemtowards the container, upon which the symbol of the item is lost from view of the at least onecamera, the situation may be interpreted as the item being delivered to the container, asindicated by the series of relative positions, and placed inside the container, as indicated by theat least one camera no longer being able to track the symbol of the item. The contextualrelationship is thus one of insertion, and the database can be updated accordingly. The relativeposition of the detected symbol to another symbol may further be used to determine arelative position of the item within the container space and/or within a storage component, typically a storage component comprising the other symbol.

According to some aspects, the image processing of the at least one image comprising thedetected symbol is configured to determine an estimated distance and/or orientation of thedetected symbol relative to the at least one camera. By determining the estimated distanceand/or orientation, positional accuracy of the item with respect to the container can beestablished or improved (if the position of the item is also determined in other ways). Theestimated distance and/or orientation may further facilitate establishment of a contextualrelationship which enables determining if the item is being inserted or removed from the container.

According to some aspects, the at least one camera is a single camera. A single cameratypically reduces cost with respect to solutions involving a plurality of cameras, as well as reducing space requirements and complexity of the system.

According to some aspects, the at least one camera comprises a plurality of cameras placed toface different regions in front of the opening of the container. The plurality of cameras therebyis more likely to detect a symbol which may be obscured from certain camera viewpoints atparticular times. The plurality of cameras further facilitates establishing a three-dimensionalinterpretation ofthe detected symbol as well as the space in which the item having the symbol mOVeS.

According to some aspects, the at least one camera comprises a stereo camera and/or a depth- sensing camera. The stereo and/or depth-sensing ability provides ways for single cameras to 6determine a position of the item with respect to the container based on the detected symbol.The size and orientation of the detected symbol can be used to deduce a distance andorientation of the item with respect to the camera, and with knowledge of the camera with respect to the container, a position of the item with respect to the container.

According to some aspects, the symbol of the item comprises one of a one-dimensionalbarcode, a two-dimensional barcode, an optical character recognition, OCR, number, andhuman-readable symbols, letters or digits that can be translated into machine code usingoptical character recognition. Barcodes and OCR numbers allow for large sets of different itemsand storage components to be assigned unique symbols. The barcodes and OCR numbers further enable the use of available methods for identifying the barcodes and OCR numbers.

According to some aspects, the control circuitry is further configured to detect a symbol of astorage component of the container, and to determine the insertion or removal of the itemand the position of the item relative to a position of a detected symbol of the storagecomponent of the container space. By assigning storage components, such as racks, theirrespective unique symbols, the symbol of the item can be correlated to the symbol of thestorage component in order to keep track ofwhere, i.e. for which storage component, the itemwas inserted or removed. By monitoring the relative positions it is possible to determine if andhow the item is moving towards or away from the storage component, which may beinterpreted as removal and insertion of the item from/into the storage component andpossibly also where within the storage component the item was inserted or removed. ln otherwords, the symbol of the storage component may be used in combination with the symbol ofthe item to determine to/from where the item is inserted/removed, as well as assisting in establishing the context which enables determining ifthe item is being inserted or removed.

The present disclosure further relates to a method for container space allocation monitoring.The method comprises detecting a symbol of an item based on processing one or more imagescaptured by at least one camera. The method further comprises determining a position of thedetected symbol based on image processing of at least one image comprising the detectedsymbol. The method also comprises determining a trajectory of the detected symbol based ona set of determined at least one positions of the detected symbol. The method additionally comprises determining insertion or removal of the item and a position of the item relative to a 7position of a container space based on the determined trajectory. The method yet furthercomprises updating information stored in a database based on the determined insertion orremoval of the item and the determined position of the item relative to the position of thecontainer space. The disclosed method carries out the steps which the disclosed system isconfigured to carry out, and thus has all the technical effects and advantages of the disclosed system, as described above and below.

The method also relates to a computer program for container space allocation monitoring, thecomputer program comprising computer program code which, when executed by a processor,causes the processor to carry out the method as described above and below. The computerprogram thus has all the technical effects and advantages of the disclosed method and system, as described above and below.

BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 illustrates a system for container space allocation monitoring; Figure 2 is a box diagram of a system for container space allocation monitoring;Figure 3 is a box diagram of a system for container space allocation monitoring; andFigure 4 illustrates a method for container space allocation monitoring.

DETAILED DESCRIPTION Figure 1 illustrates a system 100 for container space allocation monitoring. Figure 1 furtherillustrates a container 110 having a container space 120 and an opening 112 for external accessto the container space 120. The container 110 may also comprise a closing mechanism (notshown), such as a door, configured to provide access to and close off the opening 112. Figure 1also illustrates an item 130 having a first symbol 132 and a storage component 140 of thecontainer space, wherein the storage component 140 has a second symbol 142. Neither one ofthe container 110, item 130, storage component 140 nor any of the first and second symbols132, 142 are part of the claimed system 100, but are illustrated herein merely to facilitateunderstanding of different aspects of the system 100, since the system is intended to operatewith respect to at least some of the container, item, storage component and the first and second symbols. 8The basic idea of the system 100 is to act analogous to a traffic camera, and use the capturedimages to identify symbols and their movement in space and time in order to determine ifitems are being inserted into or removed from the storage space, and, to the extent possible,with respect to the available positions within the storage space. The information relating toinsertion and removal of items to and from the storage space is recorded at a database, which may also be used to assist when determining if an item is being inserted or removed.

The system 100 comprises at least one camera 10a-c placed to face a respective region in frontof an opening of a container. The at least one camera is thereby arranged to capture images ofitems being inserted or removed and is able to register their associated symbols. ln a preferredaspect, one or more camera is arranged at a position above the opening of the container,facing straight down. The one or more camera thereby captures a bird's-eye view of what goeson in front of the opening of the container at any given moment. The bird's-eye view isparticularly suitable for capturing symbols, since most items inserted and removed will have apreferred side facing upwards, where an associated symbol will preferably be placed. Thebird's-eye view further facilitates generating trajectories of symbols in a two-dimensionalprojection over the floor in which the one or more camera is recording. According to someaspects, the at least one camera 10a-c is a single camera. A single camera typically reducescost with respect to solutions involving a plurality of cameras, as well as reducing spacerequirements and complexity of the system. lf a bird's-eye view type of monitoring is desired, asingle camera per container is typically enough to keep track of the items being inserted andremoved from the respective containers. According to some aspects, the single camera isarranged to face a respective region in front of an opening of each container in a plurality ofcontainers. ln other words, the region monitored by the single camera covers all of the respective regions in front of the plurality of containers.

According to some aspects, the at least one camera comprises a plurality of cameras placed toface different regions in front of the opening of the container. The plurality of cameras therebyis more likely to detect a symbol which may be obscured from certain camera viewpoints atparticular times. The plurality of cameras further facilitates establishing a three-dimensionalinterpretation of the detected symbol as well as the space in which the item having the symbolmoves. ln the case of several spatially distributed containers, a plurality of cameras can be distributed such that each container has one or more camera facing a region in front of the 9opening of each container. The system thereby extends the monitoring from one container to a plurality of containers.

According to some aspects, the at least one camera comprises a stereo camera and/or a depth-sensing camera. The stereo and/or depth-sensing ability provides ways for single cameras todetermine a position of the item with respect to the container based on the detected symbol.The size and orientation of the detected symbol can be used to deduce a distance andorientation of the item with respect to the camera, and with knowledge of the camera withrespect to the container, a position of the item with respect to the container. The positionsmay be estimated using a perspective-n-point algorithm to determine a pose of the detectedsymbol with respect to the camera. According to some aspects, three-dimensional reconstruction from a stereo image may be used to determine positioning of detected symbols.

The system further comprises a database 14 configured to store information relating to itemsstored in the container and their respective positions within the container space. When storingthe information, each item may be assigned a unique ID associated with a symbol of the item.Information relating to the position of the item may be as simple as an indicator if the item iscurrently stored within the container space or not, e.g. a binary number or an informative l statement such as ”inside” or ”outside”, but may also indicate where within the containerspace. If the container has storage components, such as racks, and the storage componentscan be identified during insertion or removal of an item, e.g. by detecting a symbol of thestorage component in question, the stored information may comprise information configuredto identify the storage component as well, such as a unique ID for the storage component, andpossibly also where within the storage component. For each item, the stored informationwould then comprise a unique ID for the item, a unique ID for the storage component in whichthe item is stored or has been removed from and an indicator if the item is currently storedwithin the container space or not. The stored information may further comprise additional information considered useful by a user of the system, such as information relating to persons and/or projects associated with the item or things stored within the item.

The system also comprises control circuitry 12. The control circuitry 12 is configured to detect a symbol of an item based on processing one or more images captured by the at least one camera. The processing may comprise the use of image recognition software and/or applyingfilters or software modules for identifying predetermined features associated with the symbol.ln other words, the symbol may have known features for which special purpose filters and/orsoftware modules are used to identify. The symbol of the item may comprise one of a one-dimensional barcode, a two-dimensional barcode, an optical character recognition, OCR,number, and human-readable symbols, letters or digits that can be translated into machinecode using optical character recognition. Such barcodes and OCR numbers have knownfeatures which may be used for detection. Barcodes and OCR numbers further enablesassigning unique |Ds to each item to be stored within the container. Thus, according to someaspects, the processing of the one or more images captured by the at least one cameracomprises identifying at least one of a one-dimensional barcode, a two-dimensional barcode,an optical character recognition, OCR, number, and human-readable symbols, letters or digits that can be translated into machine code using optical character recognition.

According to some aspects, the system further comprises a motion filter 15 arranged todetermine changes between consecutive images captured by the at least one camera 10a-c.The motion filter 15 thereby facilitates real-time identification of symbols without the need forsubstantial computational resources. The motion filter reduces the amount of image data thathas to be processed by downstream processes. The downstream processes may comprise boththe detection ofthe symbol of the item, as discussed above, as well as further data processingsteps in the form of determining if insertion or removal of the item is taking place. While localprocessing could be performed at the at least one camera in some examples, e.g. a cameramay comprise a motion filter, it is typically desirable to transfer (possibly filtered) image datato a computational resource of the control circuitry particularly suitable for processing the downstream image data.

Therefore, according to some aspects, the system further comprises a video encoder 16 and avideo decoder 17. The video encoder 16 is configured to encode image data in a compressedformat. The video decoder 17 is configured to decode/decompress the compressed formatimage data. The use of the video encoder 16 and the video decoder 17 enables efficienttransfer of image data from one part in the system to another. ln particular, the video encoder16 and the video decoder 17 facilitates splitting up the control circuitry 12 into logically and/or physically separate units, since image data transfer between the separate units of the control 11circuitry 12 can be performed more efficiently. Encoding the image data may further be usedto provide context and/or an optimal format for downstream image processing. The contextmay facilitate interpretation of the image data. The optimal format may facilitate the use of optimized algorithms to identify features of the image data.

According to some aspects, the system 100 further comprises a memory buffer 18 configuredto receive the compressed format image data from the video encoder 16 and store thecompressed format image data until a remaining processing pipeline is ready to process thestored compressed format image data. The system 100 thereby becomes more stable withrespect to downstream processes, which often are more computationally expensive an maynot be immediately ready to receive the image data. The buffer 18 can ensure smooth real-time monitoring of the container 110. ln case the at least one camera 10a-c comprises amotion filter 15 and/or a video encoder 16, the processed image data may be transmitted tothe memory buffer 18 directly from the at least one camera. According to some aspects, the atleast one camera 10a-c also comprises the memory buffer 18. ln such case, the at least onecamera transmits image data from the buffer 18 to downstream processes, such as a potentialvideo decoder 17, when a remaining processing pipeline is ready to process the stored compressed format image data. lt may sometimes be desirable to split up the processing of the image data over severalseparate computational resources. Since different computational resources may complete theirrespective tasks out-of-sync with the order in which the images associated with the image datawas recorded, there may be a need to sort at least some of the separately processed imageswith respect to the chronological order in which they were captured. Therefore, according tosome aspects, the control circuitry 12 is further configured to ensure correct chronological order of a plurality of separately processed images.

The control circuitry 12 is further configured to determine a position of the detected symbolbased on image processing of at least one image comprising the detected symbol. As statedabove, the determined positions form the basis for keeping track of the insertion and removalof items in the container. The position of the symbol may be derived from relationships with items in the environment and/or based on the image ofthe detected symbol itself. 12Thus, according to some aspects, the image processing of the at least one image comprisingthe detected symbol is configured to determine an estimated relative position of the detectedsymbol 132 to another symbol 142 and/or an environmental marker 152. With the position ofthe environmental marker 152 and/or the other symbol 142 being known, a spatialrelationship between the detected symbol 132 and the other symbol 142 and/or theenvironmental marker 152 can be established. For instance, if one 10a of the at least onecameras 10a-c is arranged to obtain a bird's-eye view of a region in front of the opening of thecontainer 110, the images of said camera may be used to determine a distance and direction ina two-dimensional plane parallel to a floor on which the container 110 is arranged betweenthe detected symbol 132 and the other symbol 142 and/or the environmental marker 152.Stated differently, the estimated relative position of the detected symbol 132 to the othersymbol 142 may be used to determine a relative position of the item 130 within the containerspace 120 and/or within a storage component 140 of the container space, e.g. a storagecomponent 140 comprising the other symbol 142. According to some aspects, theenvironment marker is arranged to provide a reference in a global coordinate system in whichboth the item 130 comprising the detected symbol 132 and the container 110 are arranged.According to some aspects, the other symbol 142 is arranged to provide a reference in localcoordinate system with respect to the container 110. The other symbol 142 may comprise asymbol arranged on a storage component 140 of the container space, e.g. a rack for receiving the item 130.

According to some aspects, the image processing of the at least one image comprising thedetected symbol is configured to determine an estimated distance and/or orientation of thedetected symbol relative to the at least one camera 10a-c. lf the detected symbol is known topossess certain geometrical properties, such as being square- or rectangular-shaped withknown side lengths, and/or having distinctive visual features, such as black and white squaresor stripes, the geometrical properties and/or visual features may be used to determine anestimated distance and/or orientation of the detected symbol relative to the at least onecamera. Once the symbol has been detected, the image of the symbol can be matched to atranslation and/or rotation of a symbol from a reference plane with respect to the at least one Camera. 13One or more positions at known times, e.g. by associating each position with a time stamp, canthen be used to form a trajectory. The trajectory may be used to keep track of where the item130 is in relation to the container 110 as well as providing context as to whether an item thatat one time instant appears or disappears in the vicinity of the opening 112 of the container110 is being inserted or removed from the container space 120. Therefore, the control circuitry12 is also configured to determine a trajectory of the detected symbol 132 based on a set of determined at least one positions of the detected symbol.

The control circuitry 12 is additionally configured to determine insertion or removal of the itemand a position of the item relative to a position of the container space based on thedetermined trajectory. Movement of the detected symbol 132 in space and time provides acontext which facilitates interpretation whether the item 130 is being inserted into or removedfrom the storage space 120. For example, a symbol of an item is detected within the vicinity ofthe opening 112 ofthe storage space and the system has to decide if the item is being insertedor removed (or neither). The system may consult the information stored in the database 14 tosee if the item 130 associated with the detected symbol 132 was previously stored in thecontainer space 120 or not. lf the item was registered as stored, the system has to determine ifthe item is to be considered removed. A trajectory of the detected symbol 132 which leadsaway from the container, e.g. after the detected symbol has disappeared from view of the atleast one camera 10a-c, may indicate that the item is considered removed from the containerspace. A trajectory which makes a narrow U-turn in vicinity of the opening 112 ofthe containeror a symbol 142 of a storage component 140 of the container space, after which the detectedsymbol may disappear from view of the at least one camera 10a-c, may be interpreted as theitem being taken out temporarily and then returned, thus still being stored within thecontainer space. lf two storage components having unique symbols are detected during thetrajectory of the item, or rather the detected symbol 132 of the item 130, the trajectory maybe used to indicate the item being moved from one storage component to another. Likewise,the trajectory may be used to indicate insertion of an item. lf the trajectory of the detectedsymbol 132 approaches the storage space and then disappears, this may be used to interpret the item 130 having the detected symbol 132 as being inserted. lf one or more symbols of corresponding one or more storage components are detected at the same time, the trajectory may further be used to indicate in which storage component the 14item is stored. ln other words, according to some aspects, the control circuitry 12 is furtherconfigured to detect a symbol of a storage component of the container, and to determine theinsertion or removal of the item and the position of the item relative to a position of a detected symbol of the storage component of the container space.

The control circuitry 12 is yet further configured to update the stored information in thedatabase based on the determined insertion or removal of the item and the determined position of the item relative to the position of the container space.

According to some aspects, at least part of the control circuitry 12 and one or more of the atleast one camera 10a-c are comprised in a single logical unit (not shown) such that detectingthe symbol of an item, determining the position of the detected symbol, determining thetrajectory of the detected symbol and determining insertion or removal of the item and aposition of the item relative to a position of the container space are performed within thesingle logical unit. According to some further aspects, the single logical unit comprises a single-board computer. By integrating the camera(s) with the downstream processing performed bythe control circuitry, encoding and decoding of image data can be omitted. The single logicalunit may further comprise the part of the control circuitry 12 configured to update the storedinformation in the database based on the determined insertion or removal ofthe item and thedetermined position of the item relative to the position of the container space. According tosome aspects, the single logical unit also comprises the motion filter 15. According to some aspects, the single logical unit further comprises the memory buffer 18.ln some examples the database 34 is comprised in the single logical unit. lt may be desirable to connect several logical units of the single logical unit time describedabove, e.g. several single-board computer systems, to the same database. ln such cases it maybe desirable to avoid performing the same type of updates in the database. ln some examplesthe control circuitry of the respective logical unit is configured to only update the storedinformation in the database if the information the control circuitry wants to store at the database differs from the currently stored information.

Alternatively, logical unit could share a single, external portion of the control circuitryconfigured to update the stored information in the database based on the determined insertion or removal of the item and the determined position of the item relative to the position of the container space. According to some aspects, the single, external portion of thecontrol circuitry configured to update the stored information in the database is comprised in the database.

Figure 2 is a box diagram of a system 200 for container space allocation monitoring. Theillustrated system provides examples of how systems can be arranged to divide thecomputational tasks from the moment of image capture to updating a database betweencomputational resources of different capacity. The system is here illustrated having a lowperformance computational subsystem 202 and a high performance computational subsystem 204, as will be illustrated further below.

The system 200 comprises at least one camera 20 placed to face a respective region in front ofan opening of a container (not shown). For reasons of convenience, the at least one camera 20will be discussed in terms of a single camera, but it should be understood that the illustratedexamples may comprise more than one camera of any type discussed herein. The systemfurther comprises a database 24 configured to store information relating to items stored in the container and their respective positions within the container space.

The system also comprises control circuitry. The control circuitry handles the computationaltasks and will be illustrated as distributed modules for clarity. Though the modules areillustrated as separate entities, it is to be understood that the modules may be integrated intoone or more common logical unit(s). The modules can be implemented in any suitable combination of software and hardware.

The control circuitry is configured to detect a symbol of an item based on processing one ormore images captured by the at least one camera, as illustrated by a symbol detector anddecoder module 221. The control circuitry is further configured to determine a position of thedetected symbol based on image processing of at least one image comprising the detectedsymbol, as illustrated by a symbol positioner module 222. The control circuitry is alsoconfigured to determine a trajectory of the detected symbol based on a set of determined atleast one positions of the detected symbol as illustrated by a symbol tracker module 223. Thecontrol circuitry is yet further configured to determine insertion or removal of the item and aposition of the item relative to a position of the container space based on the determined trajectory, as illustrated by a gesture decoder module 224. The control circuitry is additionally 16configured to update the stored information in the database 24 based on the determinedinsertion or removal of the item and the determined position of the item relative to the position ofthe container space, as illustrated by a database writer module 225.

The system 200 preferably further comprises a motion filter 25 arranged to determine changes between consecutive images captured by the camera 20.

According to some aspects, the control circuitry is further configured to ensure correctchronological order of a plurality of separately processed images, as illustrated by a reorder module 226.

According to some preferred aspects, the system further comprises a video encoder 26 and avideo decoder 27, as illustrated by a video encoder module 26 and a video decoder module 27,respectively. The video encoder 26 is configured to encode image data in a compressed format.

The video decoder 27 is configured to decode/decompress the compressed format image data.

The system may also comprise a memory buffer 28, as illustrated by a buffer module 28,configured to receive the compressed format image data from the video encoder 26 and storethe compressed format image data until a remaining processing pipeline is ready to process the stored compressed format image data.

Possible aspects and capabilities of the at least one camera 20, the motion filer 25, the videoencoder 26, the memory buffer 28, the video decoder 27, the database 24 as well asimplementations ofthe control circuitry, illustrated at least in part here by the symbol detectorand decoder module 221, the reorder module 226, the symbol positioner module 222, thesymbol tracker module 223, the gesture decoder module 224 and the database writer module225, have been discussed in relation to Figure 1 above and applies in the illustrated example of Figure 2 as well. ln the illustrated example the camera 20 and any motion filter 25, video encoder 26 andmemory buffer 28 are arranged together in a low performance computational subsystem 202.lf present, any of the motion filter 25, video encoder 26 and memory buffer 28 may bearranged as part of or integrated with the camera 20. According to some aspects, the camera20, the motion filter 25, the video encoder 26 and the memory buffer 28 comprised in a single- boa rd com puter system. 17Likewise, the symbol detector and decoder module 221, the symbol positioner module, thesymbol tracker module 223, the gesture decoder module 224, the database writer module225, and, if present, the reorder module 226 and the video decoder 27 are arranged togetherin a high performance computational subsystem 204. The high performance computationalsubsystem 204 may be implemented in a cloud virtual machine. According to some aspects, the database 24 is also comprised in the high performance computational subsystem 204.

The low performance computational subsystem 202 and the high performance computational subsystem 204 are communicatively connected.

Figure 3 is a box diagram of a system 300 for container space allocation monitoring. Theillustrated system provides examples of how image processing and downstream aspects of thecontrol circuitry may be integrated with one or more camera in a single-board computersystem. ln other words, Figure 3 aims at illustrating how the low performance computationalsubsystem 202 and the high performance computational subsystem 204 of Figure 2 above may both be comprised in a single-board computer system 303, along with potential advantages.

The system 300 comprises at least one camera 30 placed to face a respective region in front ofan opening of a container (not shown). For reasons of convenience, the at least one camera 30will be discussed in terms of a single camera, but it should be understood that the illustrated examples may comprise more than one camera of any type discussed herein.

The system further comprises a database 34 configured to store information relating to items stored in the container and their respective positions within the container space.

The control circuitry is configured to detect a symbol of an item based on processing one ormore images captured by the at least one camera, as illustrated by a symbol detector anddecoder module 321. The control circuitry is further configured to determine a position of the detected symbol based on image processing of at least one image comprising the detected 18symbol, as illustrated by a symbol positioner module 322. The control circuitry is alsoconfigured to determine a trajectory of the detected symbol based on a set of determined atleast one positions of the detected symbol as illustrated by a symbol tracker module 323. Thecontrol circuitry is yet further configured to determine insertion or removal of the item and aposition of the item relative to a position of the container space based on the determinedtrajectory, as illustrated by a gesture decoder module 324. The control circuitry is additionallyconfigured to update the stored information in the database 34 based on the determinedinsertion or removal of the item and the determined position of the item relative to the position ofthe container space, as illustrated by a database writer module 325.

The system 300 preferably further comprises a motion filter 35 arranged to determine changes between consecutive images captured by the camera 30.

According to some aspects, the control circuitry is further configured to ensure correctchronological order of a plurality of separately processed images, as illustrated by a reorder module 326.

One of the advantages of the examples of Figure 3 with respect to the examples illustrated in relation to Figure 2 is that the video encoder and video decoders can be omitted.

The system may also comprise a memory buffer 38, as illustrated by a buffer module 38,configured to receive the compressed format image data from either directly from the camera30 or via the motion filter 35 (if present) and store the compressed format image data until a remaining processing pipeline is ready to process the stored compressed format image data.

Possible aspects and capabilities of the at least one camera 30, the motion filer 35, thememory buffer 38, the database 34 as well as implementations of the control circuitry,illustrated at least in part here by the symbol detector and decoder module 321, the reordermodule 326, the symbol positioner module 322, the symbol tracker module 323, the gesturedecoder module 324 and the database writer module 325, have been discussed in relation to Figure 1 above and applies in the illustrated example of Figure 3 as well.

The camera 30 and at least the part of the control circuitry comprising the symbol detector and decoder module 321, the reorder module 326, the symbol positioner module 322, the symbol 19tracker module 323, the gesture decoder module 324 and the database writer module 325 are comprised in a single-board computer system 303.

The database 34 is illustrated as a unit separate from the single-board computer system 303, but may in some examples be comprised in the single-board computer system 303 as well. lt may be desirable to connect several single-board computer systems 303 to the samedatabase 34. ln such cases it may be desirable to avoid performing the same type of updates inthe database 34. ln some examples the database writer module 325 is configured to onlyupdate the stored information in the database if the information the database writer module 325 wants to store at the database 34 differs from the currently stored information.

Alternatively, each single-board computer systems 303 could share a single, external databasewriter module 325 (not shown). According to some aspects, the single, external database writer module is comprised in the database 34 (not shown).

Figure 4 illustrates a method for container space allocation monitoring. The method comprisesdetecting S10 a symbol of an item based on processing one or more images captured by atleast one camera. The method further comprises determining S20 a position of the detectedsymbol based on image processing of at least one image comprising the detected symbol. Themethod also comprises determining S30 a trajectory of the detected symbol based on a set ofdetermined at least one positions of the detected symbol. The method additionally comprisesdetermining S40 insertion or removal of the item and a position of the item relative to aposition of a container space based on the determined trajectory. The method yet furthercomprises updating S50 information stored in a database based on the determined insertion orremoval of the item and the determined position of the item relative to the position of thecontainer space. The method carries out the steps for which the disclosed system for containerspace allocation monitoring as described above and below is configured, and consequently hasall the technical effects and advantages of the system for container space allocation monitoring.

The present disclosure also relates to a computer program for container space allocationmonitoring. The computer program comprises computer program code which, when executedby a processor, causes the processor to carry out the method for container space allocation monitoring as described above and below.

Claims

1. CLAlMS A system (100, 200, 300) for container space allocation monitoring, the system (100, 200, 300) comprising at least one camera (10a-c, 20, 30), the at least one camera being placed outside a container and facing a region in front of an opening of the container to capture activities outside the opening of the container, a database (14, 24, 34) configured to store information re|ating to items stored in the container and their respective positions within the container space (120), and control circuitry (12) configured to detect a first symbol (132) of an item (130) based on processing one or moreimages captured by the at least one camera , detect a second symbol (142) arranged on a rack (140) for receiving the item(130), determine a position of the detected first symbol (132) based on imageprocessing of at least one image comprising the detected first symbol (132),determine a trajectory of the detected first symbol (132) based on a set ofdetermined at least one positions of the detected first symbol (132),determine insertion or removal of the item (130) and a position of the itemrelative to a position of the container space based on the determinedtrajectory, wherein the position of the item is also determined relative to a position of the second symbol and update the stored information in the database based on the determined insertion or removal of the item and the determined position of the item relative to the position of the container space, wherein the control circuitry is configured to detect the first symbol (132) based on processing one or more images captured by the at least one camera, wherein the processing comprises the use of image recognition software and/or applying filters or software modules for identifying predetermined features associated with the first symbol, wherein the control circuitry is arranged to determine an estimated relative position ofthe detected first symbol (132) to the second symbol (142), and wherein the estimatedrelative position of the detected first symbol (132) to the second symbol (142) is usedto determine a relative position of the item within the container space and/or within the rack, wherein the database is configured to store information relating to items (130) storedin the container and their respective positions within the container space relative to the second symbol, and wherein the database is further configured to, when storing the information, assigning each item a unique ID associated with the first symbol (130). The system according to claim 1, wherein at least part ofthe control circuitry and oneor more of the at least one camera are comprised in a single logical unit such thatdetecting the first symbol, determining the position of the detected first symbol,determining the trajectory of the detected first symbol and determining insertion orremoval of the item and a position of the item relative to a position of the container space are performed within the single logical unit. The system according to claim 1 or 2, wherein the system further comprises a motionfilter (15, 25, 35) arranged to determine changes between consecutive images captured by the at least one camera (10a-c, 20, 30). The system according to any of the preceding claims, wherein the control circuitry(12) is further configured to ensure correct chronological order of a plurality of separately processed images. The system according to any of the preceding claims, wherein the system furthercomprises a video encoder (16, 26) and a video decoder (17, 27), wherein the video encoder (16, 26) is configured to encode image data in a compressed format, and 10. 11. 1

2. 3wherein the video decoder (17, 27) is configured to decode/decompress the compressed format image data. The system according to claim 5, further comprising a memory buffer (18, 28, 38)configured to receive the compressed format image data from the video encoder andstore the compressed format image data until a remaining processing pipeline is ready to process the stored compressed format image data. The system according to any of the preceding claims, wherein the image processingof the at least one image comprising the detected first symbol is configured todetermine an estimated relative position of the detected first symbol to an environmental marker. The system according to any of the preceding claims, wherein the image processingof the at least one image comprising the detected first symbol is configured todetermine an estimated distance and/or orientation of the detected first symbol relative to the at least one camera. The system according to any of the preceding claims, wherein the at least one camera (10a-c, 20, 30) is a single camera. The system according to any of claims 1-8, wherein the at least one camera (10a-c,20, 30) comprises a plurality of cameras placed to face different regions in front of the opening of the container. The system according to any of the preceding claims, wherein the at least one camera (10a-c, 20, 30) comprises a stereo camera and/or a depth-sensing camera. The system according to any of the preceding claims, wherein the first symbolcomprises one of a one-dimensional barcode, a two-dimensional barcode, an opticalcharacter recognition, OCR, number, and human-readable symbols, letters or digits that can be translated into machine code using optical character recognition. 1

3. A method for container space allocation monitoring, the method comprising detecting (S10) a first symbol of an item based on processing one or more imagescaptured by at least one camera, detecting a second symbol arranged on a rack (140) for receiving the item (130),determining (S20) a position of the detected first symbol based on imageprocessing of at least one image comprising the detected first symbol,determining (S30) a trajectory of the detected first symbol based on a set ofdetermined at least one positions of the detected first symbol, determining (S40) insertion or removal of the item and a position of the itemrelative to a position of a container space based on the determined trajectory,and updating (S50) information stored in a database based on the determinedinsertion or removal of the item and the determined position of the item relativeto the position of the container space, wherein the one or more images arecaptured from a position outside the container and captures a region in front ofan opening of the container to capture activities outside the opening of thecontainer, wherein the position of the item is also determined relative to a position of thesecond symbol, wherein the first symbol is detected based on processing one or more imagescaptured by the at least one camera, wherein the processing comprises the use ofimage recognition software and/or applying filters or software modules foridentifying predetermined features associated with the first symbol, wherein an estimated relative position of the detected first symbol (132) to thesecond symbol (142) is determined , and wherein the estimated relative positionof the detected first symbol (132) to the second symbol (142) is used todetermine a relative position of the item within the container space and/or within the rack, 5- wherein the database stores information relating to items stored in the containerand their respective positions within the container space relative to the second symbol, and 5 - wherein the database assigns, when storing the information, each item a unique ID associated with the first symbol. 1

4. A computer program for container space a||ocation monitoring, the computer programcomprising computer program code which, when executed by a processor, causes the 10 processor to carry out the method according to c|aim 13.