KR20230038960A - Large-scale category object detection and recognition method for inventory management of autonomous unmanned stores - Google Patents

Abstract

The present invention relates to a large-scale category object detection and recognition method for inventory management of an autonomous store. The large-scale category object detection and recognition method for inventory management of an autonomous store can overcome a weakness of an object detection deep learning model trained with a corresponding data set due to various data classes and features; use a model that can adapt in an environment with different data features through multiple branch tree structures using active semi-supervised learning for training a strong object detection deep learning model; be used for an online vision artificial intelligence (AI) deep learning platform technology, which can adaptively extend and update a deep learning advanced image recognition function, an AI POS, and an inventory-combined smart store; and commercialize behavior recognition technology even in service robot-related and smart home industry-related fields required to quickly prepare for chronological changes.

Description

Large-scale category object detection and recognition method for inventory management of autonomous unmanned stores}

The present invention relates to a method for detecting and recognizing large-scale category objects for inventory management of autonomous unmanned stores, and more particularly, to hierarchical meta learning for deep learning (MLDL) for large-scale category objects for inventory management of autonomous unmanned stores. It relates to a detection and recognition method using learning).

Recently, many image data sets have various data classes and features for extracting general features.

However, due to these various data classes and characteristics, the object detection deep learning model trained with the corresponding data set does not show good performance in a large-scale category object recognition environment.

Although great progress has been made in the field of deep learning object detection since Alexnet, object detection is still a challenging problem in heterogeneous and sparse data distributions. Many existing researchers have devised methods that can respond to large categories using various methods to obtain better accuracy in object detection problems.

In addition, we focused on sub-class transformation, where most of the sub-classes were built based only on similar information within the sub-class.

However, large-scale category detection and recognition is a very difficult problem because feature information for each object is slightly different even within the same class.

On the other hand, recently, with Amazon Go at the head, the demand for store AI image analysis solutions through the advancement of vision AI deep learning recognition technology has emerged, increasing work efficiency through store artificial intelligence inventory management, pickup management, and payment management, and COVID-19. Due to rising labor costs such as non-contact and 52-hour work due to COVID-19, the realization of artificial intelligence inventory and POS-based (semi-)unmanned stores that are operated with an unmanned or minimum number of people is accelerating.

Accordingly, in Korea, hybrid convenience stores, which are semi-unmanned stores, are currently operating at about 600 locations. However, the shelf inventory of convenience stores, which must recognize and manage over 800 categories, is still dependent on manual work.

And, overseas, Amazon Go, Amazon Grocery, and Amazon Fresh were introduced, but due to the problem of large-scale category recognition rate, they are staying at the barcode-centered smart cart stage.

As a result, there are many technical barriers to ultimately evolving to fully unmanned stores, such as fully autonomous vehicles.

Reference 1: Korean Patent Registration No. 10-0988754 Reference 2: Korean Patent Registration No. 10-2114808

Therefore, the present invention overcomes the disadvantage of performance degradation in a large-scale category object environment and robust The purpose of this study is to provide a method for detecting and recognizing large-scale categorical objects using hierarchical meta learning for deep learning (MLDL) to train an object detection deep learning model.

In addition, an object of the present invention is to provide a method for detecting and recognizing large-scale category objects for inventory management of an autonomous unmanned store that can adapt to an autonomous unmanned store environment having complex data characteristics like a large-scale category by using such a structure.

The present invention to solve such a technical problem;

A large-scale category object detection and recognition method for inventory management of an autonomous unmanned store, comprising: a first step of collecting multi-image image data scanned by N-cameras (N>= 1) by an inventory robot; A second step of collecting store meta information associated with the video image data; A third step of deriving a meta factor by performing meta-learning on the store meta information; And a fourth step of generating a hierarchical deep learning model based on the image data archive, the meta data archive, and the meta factor. .

In this case, the second step is characterized in that the store meta information is collected from one or more input devices.

And, the meta factor is the detection recognition of one or more of the characteristics of the collected image data, the reliability of the distributed image characteristics, the bin-based characteristics, the video instance-based characteristics, the image collection time, the location of the inventory robot, and the lighting state. Characterized in that it is a factor that affects performance.

In addition, the fourth step is characterized by constructing an initial model with a small amount of labeled data and gradually optimizing performance through an augmented ASSL learning process.

In addition, after the fourth step, a fifth step of inputting multiple images scanned by the N-camera from the inventory robot to the hierarchical deep learning detection recognizer to derive a detection recognition result; characterized in that it further comprises.

In this case, the fifth step is characterized in that the hierarchical deep learning detection recognizer receives feedback from the MLDL model generator and performs detection and recognition of the large-scale category object using the inventory robot and meta factor information of the store.

And, after the fifth step, a sixth step of updating the image data archive and the metadata archive by using the derived meta factor and the detection recognition result of the hierarchical deep learning detection recognizer; and a seventh step of updating a hierarchical deep learning model based on the updated image data archive, meta data archive, and meta factor.

In addition, the fifth step is characterized in that the multiple images scanned by the N-camera from the inventory robot are input to the hierarchical deep learning detection recognizer to continuously derive detection and recognition results.

In this case, the fifth step may include applying a time-window technique for advanced detection and recognition of multiple product objects on the shelf.

In addition, the time-window technique, prior to generating the final tracking path from the object detection result of each image frame, optimizes multi-object tracking segments (partial tracking paths are performed in batch units). Applying a time-window technique for extraction; And since there are many variables that affect object identification in the operating location of the inventory robot, NMS (Non-Maximum Suppression) is a type of algorithm, WNMS (Weighted It is characterized by including; applying a Non-Maximum Suppression).

According to the present invention, when detecting and recognizing large-scale category objects for inventory management of autonomous unmanned stores, it is possible to overcome the disadvantages of object detection deep learning models trained with the corresponding data set due to various data classes and characteristics, and robust object detection deep To train a learning model, a model that can adapt to environments with different data characteristics can be used through a multi-branch tree structure using active semi-supervised learning.

In addition, the present invention can be applied to smart stores that combine online vision AI deep learning platform technology, artificial intelligence POS, and inventory that can adaptively expand and update deep learning advanced image recognition functions.

In particular, the present invention can promote full-scale commercialization of smart store technology by securing practical vision AI technology with online deep learning technology, and in the case of offline deep learning systems, which are currently commonly used in terms of technology, in the field where learning is not performed In the case of supervised learning-based deep learning, which is currently mainstream, it overcomes the limitations in recognizing and analyzing real environments where change and diversity exist due to a rapid drop in detection ability. Although there was a setback in general-purpose use because it was possible to recognize, this also has an advantage that can be overcome.

In addition, the present invention can commercialize action recognition technology in service robots, smart home industries and related industries that need to quickly prepare for changes in the times to activate related industries, as well as to realize the results of the present invention The hierarchical online deep learning platform can be applied to service robots and consumer robots after it has been successfully commercialized for serving robots, providing a basis for activating the robot market.

1 is a diagram illustrating a hierarchical MLDL detection and recognition platform for inventory management of an autonomous unmanned store according to an embodiment of the present invention.
2 is a diagram for explaining an HFM construction method according to an embodiment of the present invention.
3 is a diagram for explaining a hierarchical MLDL detection and recognition process corresponding to a new class or an existing class having ambiguous characteristics according to an embodiment of the present invention.
4 is a diagram illustrating an image object detection method using an existing CNN and a CNN-based image object detection method using a time-window technique.
5 is a diagram for explaining an ASSL training method according to an embodiment of the present invention.
6 is a diagram for explaining an augmented learning method of a CNN algorithm according to an embodiment of the present invention.
7 is a diagram illustrating an autonomous unmanned store system H/W and F/W device for loading a hierarchical MLDL augmented vision AI solution according to an embodiment of the present invention.

Hereinafter, the characteristics of a method for detecting and recognizing large-scale category objects for inventory management of an autonomous unmanned store according to the present invention will be understood by an embodiment described in detail with reference to the accompanying drawings.

Prior to this, the terms or words used in this specification and claims should not be construed as being limited to the usual or dictionary meaning, and the inventor appropriately uses the concept of the term in order to explain his/her invention in the best way. It should be interpreted as a meaning and concept consistent with the technical idea of the present invention based on the principle that it can be defined.

Therefore, the embodiments described in this specification and the configurations shown in the drawings are only one of the most preferred embodiments of the present invention, and do not represent all the technical ideas of the present invention, so at the time of this application, they can be replaced. It should be understood that there may be many equivalents and variations.

First of all, the automatic inventory management of the shelf for the operation of the autonomous unmanned store according to the present invention is a core element of the autonomous store, and large-scale product category image detection and recognition technology is essential. In the case of a convenience store, which is a representative unmanned store target, it is necessary to manage an inventory of 700 to 800 items, and in the case of a grocery store (grocery store), up to about 3,000 items.

In particular, the low detection and recognition rate of product images on the shelf, where the product is atypical and the uncertainty is very high depending on the display characteristics, is a major obstacle to commercialization of autonomous unmanned stores.

That is, the existing object detection and recognition technology has a low recognition rate, and it is difficult to use it in a real environment, especially in the case of large-scale class objects.

On the other hand, various big data have been built based on the recent growth of the Internet, and computing power has grown significantly with the development of semiconductors and parallel processing technology. Complex neural network models have emerged and various deep learning technologies are being actively spread. As these advanced deep learning technologies are applied to object detection methods, the overall recognition rate has increased significantly. However, most of the technologies currently used for object detection are based on supervised learning algorithms that initially require enormous amounts of data, and have a fatal disadvantage in that their performance drops sharply when used in large-scale category (hundreds or more) object groups.

Accordingly, in the present invention, an ASSL (Active Semi-Supervised Learning) method capable of quickly recovering performance deteriorated due to changes in external factors such as the environment in a large-scale category object detection and recognition application and a Hierarchical Feature Modeling (HFM) method capable of responding to environmental changes Hierarchical MLDL advanced technology that combines technology and meter learning is applied.

In particular, the present invention proposes an inventory technology based on hierarchical meta learning for deep learning (MLDL) as a platform for automatically managing the inventory of autonomous unmanned stores.

The present invention combines a convolutional neural network (CNN) deep learning algorithm, a hierarchical feature modeling (HFM), and a soft class ensemble (SCE) algorithm, and applies a hierarchical ML technique to use context information. It is a deep learning method that applies MLDL technology to perform detection and recognition of large-scale category product objects in autonomous unmanned store shelf inventory. Existing convolutional neural networks are trained based on a flat one-dimensional category structure, so they do not exist in a previously learned environment or exist. In order to prevent the problem of a sharp drop in recognition performance when a new object that does not appear, modeling the data distribution of the object to be recognized based on HFM and applying a hierarchical CNN-based augmented vision AI technology are included.

The hierarchical meta-learning deep learning (MLDL) is a method that combines the data collection and processing process with the execution environment such as the time and place of the store, and generates an optimal vision AI model robust to large-scale categories of unmanned store product objects and environmental changes Constructing an initial model with a relatively small amount of labeled data, unlike conventional deep learning-based vision AI that relies on a large amount of labeled data in batches, and optimizing progressive performance through an augmented ASSL learning process; MLDL concept generation step to maintain the quality of the continuous vision AI model by reflecting changes in the operating environment of the inventory robot, such as time and place; performing hierarchical deep learning based on the generated concept; receiving feedback on an error of a result of detection and recognition; Conducting meta-learning based on the generated concept; receiving feedback on errors in the learned meta information; and upgrading the MLDL concept generator using the feedback information.

The hierarchical meta-learning deep learning (MLDL)-based large-scale category object detection and recognition platform integration is a large-scale category object detection and recognition platform that combines data collection, processing, and learning environments to create and verify optimal vision AI models that are robust to environmental changes. Unlike the existing deep learning-based vision AI, which relies on a large amount of high-cost labeled data in batches, relatively low initial setup costs are required through the augmented vision AI package, and augmented ASSL learning Through the process, it is possible to optimize performance through incremental or crowd sourcing, maintain the quality of continuous vision AI models by reflecting changes in the operating environment, and optimize vision AI-based inventory robots and systems.

Referring to FIG. 1, the hierarchical MLDL detection and recognition platform for guaranteeing recognition performance for shelf inventory management by converging meta-factor information rather than simple detection and recognition for large-scale category object detection and recognition for autonomous unmanned stores of the present invention , The MLDL generator (10) generates ML and DL models based on the data archive (11) and the meta factor archive (12), and the inventory robot converts multiple images scanned with N-cameras into a hierarchical deep learning detection recognizer (20 ) to derive a detection recognition result, and the derived search recognition result is transmitted to the MLDL model generator 10, store meta information is collected by one or more input devices, and meta-learning 30 is performed, The detection recognition result derived from the hierarchical deep learning detection recognizer 20 and the meta factor result derived through meta learning 30 are transferred to the MLDL model generator 10, and the meta factor and detection recognition result are transferred to the MLDL model generator 10. Based on the results, ML and DL models are created (or updated), and the hierarchical deep learning detection recognizer (20) receives feedback from the MLDL model generator (10) and detects and recognizes them using the inventory robot and meta factor information of the store. Do it. This method fuses meta factor information rather than simple hierarchical detection and recognition, so that recognition performance for shelf inventory management can be guaranteed.

Here, the meta factor directly or indirectly affects the performance of detection and recognition, such as collected image characteristics, reliability of distributed image characteristics, bin-based characteristics, image instance-based characteristics, image collection time, location of inventory robots, and lighting conditions. is a factor that gives

In the method for detecting and recognizing large-scale category objects for inventory management of autonomous unmanned stores of the present invention, the inventory robot uses N-cameras (N>= 1) A first step of collecting multi-image image data to be scanned; A second step of collecting store meta information associated with the video image data; A third step of deriving a meta factor by performing meta-learning on the store meta information; and a fourth step of generating a hierarchical deep learning model (ML, DL model) based on the image data archive, the meta data archive, and the meta factor.

More specifically, in the present invention, the large-scale category object detection and recognition method fuses meta factor information rather than simple detection and recognition to ensure recognition performance for shelf inventory management, in the MLDL model generator 10 data archive ( 11) and generating ML and DL models based on the meta factor archive 12; Deriving a detection recognition result by inputting multiple images scanned by an N-camera from an inventory robot to the hierarchical deep learning detection recognizer 20; transferring the derived search recognition result to the MLDL model generator 10; Performing meta-learning (30) using store meta-information collected by one or a plurality of input devices and inputting it to the hierarchical deep learning detection recognizer (20) to derive a detection recognition result; Transferring the meta factor result derived by performing the meta learning 30 to the MLDL model generator 10; generating (updating), by the MLDL model generator 10, ML and DL models based on meta factors and detection and recognition results; In the hierarchical deep learning detection/recognizer 20, receiving feedback from the MLDL model generator 10 and performing detection/recognition using the inventory robot and store meta-factor information.

The large-scale category object detection and recognition method for inventory management of autonomous unmanned stores of the present invention uses a relatively small amount of labeled data to create an initial model, unlike conventional deep learning-based vision AI that relies on a large amount of labeled data in batches. and optimize the incremental performance through the augmented ASSL learning process.

On the other hand, the present invention models the data distribution of the object to be recognized based on HFM to prevent a problem in which the recognition performance rapidly deteriorates when a new object that is not in the previously learned environment or does not exist appears and responds to environmental changes.

와 시멘틱 물체 공간

가 주어졌을 때 정의할 수 있는 HFM은 상단에 위치한 Super-카테고리 노드, 중간에 위치한 Augmented-카테고리 노드, 하단에 위치한 Sub-카테고리 노드로 구성의 최적화 하는 단계; Super-카테고리 노드

는 전체 학습 데이터

의 subset인

로 학습하여 다수의 Augmented-카테고리를 출력하는 알고리즘 단계; Augmented 카테고리

은 Super-카테고리 학습 데이터인

의 subset인

으로 학습하고, 다수의 sub-카테고리를 출력하는 알고리즘 단계;를 포함하며, 베이스라인 HFM은 각각 Super-, Augmented-, Sub- 물체 공간인

,

,

에서 각자의 사물 검출, 인식을 수행하는 구조이며, 서비스 시스템 데이터 분석을 수행한 후 최적화 기법을 포함한다.Referring to FIG. 2, the HFM construction method includes constructing an HFM by utilizing a hierarchical structure class obtained as a result of soft clustering in which similar categories are classified from a dataset result having category labels; Hierarchically expressing category information to construct an HFM having a hierarchical structure; Applying a hierarchical soft clustering method of class clusters; Modeling the HFM divided into H-level, which is the highest parent node, M-level, which is a real object node of the middle layer, and L-level, which is a detailed classification node; learning data

and semantic object space

HFM that can be defined when is given is a super-category node located at the top, an Augmented-category node located in the middle, and a sub-category node located at the bottom. Super-Category Node

is the entire training data

which is a subset of

Algorithm step for outputting a plurality of Augmented-categories by learning with ; Augmented category

is the super-category training data

which is a subset of

Including; an algorithm step for learning and outputting a plurality of sub-categories; wherein the baseline HFM is Super-, Augmented-, and Sub-object space, respectively.

,

,

It is a structure that performs each object detection and recognition in , and includes an optimization technique after performing service system data analysis.

And, referring to FIG. 3, the HFM method discriminates between a new class and an existing class having ambiguous characteristics by using a hierarchical CNN instead of using a flat CNN for the target object class based on the baseline HFM. doing; Applying a hierarchical CNN structure capable of reinforcement learning to the changed environment at each HFM node; An external open-set class learning method in which new classes that have not been seen before are added, and an internal open-set class learning method that compensates for existing classes but whose recognition rate is low due to a changed environment. Dividing into; In order to integrate the internal/external openset method, a hierarchical deep learning method is applied to the object object/recognition detector and applied to each openset class using an augmented vision AI learning method.

In addition, the present invention applies a time-window technique for advanced detection and recognition of multiple product objects on a shelf provided in an autonomous unmanned store. Referring to FIG. 4 , object detection shows a result of identifying a region of interest with respect to one image frame. In order to ensure detection performance for videos in which image frames are sequentially input for multi-object tracking, an optimized multi-object tracking segment (partial tracking path) prior to generating the final tracking path from the object detection result of each video frame ) in batch units) applying a time-window technique for extracting; Since there are many variables that affect object identification in the operating location of the inventory robot, WNMS (Weighted Non-Weighted Non-Maximum Suppression), a type of NMS (Non-Maximum Suppression) algorithm, is used as a method to ensure the accuracy of multi-object identification within the extracted tracking segment. -Applying Maximum Suppression); Applying the NMS algorithm as a post-processing of multi-object identification in an unstable environment, determining an optimal identification result among several identification candidates within an image frame; The WNMS algorithm clusters similar multi-object identification results, assigning weights based on the identification scores used at this time, and summing them; Since very sensitive identification results are derived depending on the clustering method, the clustering threshold according to environmental changes is optimally tuned, and the identification results of adjacent image frames in the image sequence are simultaneously referred and compared to increase the effectiveness of WNMS and secure the stability of multi-object identification. step; includes.

카테고리 공간을, 각각의 Super-카테고리

는

카테고리 공간을, 그리고 augmented 카테고리

은

카테고리 공간을 바탕으로 딥컨볼루션 신경망을 학습하는 단계; 딥컨볼루션 신경망은 이미지로부터 관심 있는 영역(Region of Interest, 이하 RoI)을 입력값으로 받으면 그 결과로 해당 RoI에 대한 특징점을 추출하고, 그 특징점은 검출기를 통해 인식 점수(CS; Confidence score)를 추출하는 단계; 검출기로부터 얻어낸 CS는 절대적인 값이 아닌 상대적인 값으로써 특징점을 학습한 데이터 공간이 각각

,

,

으로 다르기 때문에 직접적으로 CS값들을 비교하거나 합성할 수 없으므로 이를 해결하기 위해 다른 데이터 공간에서 얻어낸 CS값들을 모두 절대적인 기준으로 변형하여 합성할 수 있도록 CS 결과를 확률 공간으로 전환시키는 단계; 관심 영역(RoI)

이 주어졌을 때, super-카테고리

를 위해 학습시킨 linear Classifier

결과를 확률 공간으로 projection 시키는 확률 함수

를

로 정의하는 단계; 파라미터

와

는 logistic regression을 통해 결정하기 위한 알고리즘 개발확률 함수

를 이용한 최종적인 super-카테고리 검출을 위한 인식 점수

를

로 정의하는 단계; Augemented-카테고리

과 sub-카테고리

또한, 위와 같은 방법으로 각각

,

을 얻어내는 단계; 계층 구조 기반 검출 결과 앙상블(HCE)을 통해 관심 영역(RoI)

에 대해 종합 검출 점수(OCS; Overall Confidence Score)인

를 추출하는 단계; 이 함수에 접근하기 위한 알고리즘을 통해 타겟 시스템의 HFM을 최적화하는 단계;를 포함한다.In addition, the soft class ensemble (SCE) algorithm optimized for the target system, in order to obtain the final object detection result from the HFM structure, is an ensemble that can integrate the object detection results of each hierarchical structure. apply step; At the top node by HFM

category space, each Super-category

Is

Category space, and augmented categories

silver

Learning a deep convolutional neural network based on the category space; A deep convolutional neural network receives a region of interest (RoI) from an image as an input value, extracts a feature point for the corresponding RoI as a result, and the feature point receives a confidence score (CS) through a detector. extracting; The CS obtained from the detector is a relative value rather than an absolute value.

,

,

Since CS values cannot be directly compared or synthesized because they are different, in order to solve this problem, all CS values obtained from other data spaces are transformed into an absolute standard and converted into a probability space so that they can be synthesized; Region of interest (RoI)

is given, the super-category

The linear Classifier trained for

A probability function that projects the result into a probability space

cast

defining as; parameter

and

is an algorithm development probability function to determine through logistic regression

Recognition score for final super-category detection using

cast

defining as; Augmented-Categories

and sub-categories

In addition, in the same way as above, each

,

obtaining; Region of interest (RoI) via hierarchical structure-based detection result ensemble (HCE)

The Overall Confidence Score (OCS) for

Extracting; optimizing the HFM of the target system through an algorithm to access this function;

Finally, referring to FIG. 5, the present invention includes the step of applying the CNN learning technology using ASSL, the image data distribution modeling technology using HFM, and the classifier convergence technology using SCE, even in a real-time environment where various noises exist. It includes hierarchical deep learning-based MLDL technology that robustly recognizes multiple objects.

The hierarchical CNN algorithm method for large-scale category detection and recognition of the present invention includes applying an Active Semi-Supervised Learning (ASSL) algorithm that can be quickly learned and adapted for the hierarchical CNN algorithm; Based on this, applying to dynamic HFM to determine a hierarchical deep learning structure suitable for the use environment of the target service system; It includes a high-level visual recognition technology combined with an augmented ASSL learning method to recognize open-set objects occurring in the field of the target service system.

At this time, referring to FIG. 6, the augmented learning method that combines Forward Learning and Rollback Learning, breaking away from Forward Batch Learning, which is generally used to solve the uncertainty of the object class classification problem, uses the actual labels of unlabeled data sets in advance. estimating a new label by defining a loss measurement function because it is unknown to ; An algorithm step of selecting an appropriate boundary of the current model considering both the worst and best cases; an algorithm step of measuring an expected error, defining an objective function for optimizing learning efficiency, and selecting data that minimizes the objective function; In the ASSL algorithm, when calculating the objective function, all unlabeled data are considered and expanded; Since the retraining method of existing deep learning models examines all unlabeled data and all possible labels, applying a method of performing fast augmented learning in units of small bins to minimize many calculations; augmentation including; It is desirable to optimize by applying the ASSL technique to the target service system.

In order to manage the shelf of the autonomous unmanned store of the present invention, it is necessary to build an inventory robot and system H/W (Hardware) and F/W (Firmware). Referring to FIG. 7, in the present invention, the inventory robot 40, Build a hierarchical MLDL platform using a smart shelf 50 and a cloud server system.

Of course, the hierarchical MLDL-based large-scale category product object detection and recognition device for shelf inventory management using the N-camera inventory robot and system for autonomous unmanned store shelf management of the present invention is included.

In addition, artificial intelligence POS (Point of sales, point-of-sale information management) 60, secure speed gate 70 and smart shelf 50 are supplemented for autonomous unmanned store shelf management of the present invention, and cloud-based MLDL technology It is desirable to support testbed operation environment establishment and application tests.

The devices described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. Devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), a programmable logic unit (PLU), It may be implemented using one or more general purpose or special purpose computers, such as a microprocessor or any other processing device capable of executing and responding to instructions.

Such a processing device may run an operating system (OS) and one or more software applications running on the operating system.

A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include.

For example, the processing device includes a plurality of processors or a processor and a controller, or other processing configurations such as parallel processors are also possible.

In addition, the software may include a computer program, code, instructions, or a combination of one or more of these, and may independently or in combination configure a processing device to operate as desired ( collectively) to command processing units. The software and/or data may be any tangible machine, component, physical device, virtual equipment, computer storage medium, or It can be embodied in a device.

In addition, the software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer readable media.

In addition, the method according to the embodiment of the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program commands recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler.

As described above, although the embodiments have been described with limited examples and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

10: MLDL generator 11: Data archive
12: Metafactor Archive 20: Hierarchical Deep Learning Detection Recognizer
30: Meta-Learning 40: Inventory Robot
50: Smart shelf 60: AI POS
70: Secure Speed Gate

Claims (10)

In the large-scale category object detection and recognition method for inventory management of autonomous unmanned stores,
A first step in which the inventory robot collects multi-image image data scanned with N-cameras (N>= 1);
A second step of collecting store meta information associated with the video image data;
A third step of deriving a meta factor by performing meta-learning on the store meta information; and
A fourth step of generating a hierarchical deep learning model based on an image data archive, a meta data archive, and a meta factor; a large-scale category object detection and recognition method for inventory management of an autonomous unmanned store, comprising:
According to claim 1,
The method of detecting and recognizing large-scale category objects for inventory management of an autonomous unmanned store, characterized in that the second step is a step of collecting the store meta-information from one or more input devices.
According to claim 1,
The meta factor depends on the performance of detection and recognition of one or more of the characteristics of collected video image data, reliability of distributed video characteristics, bin-based characteristics, video instance-based characteristics, image collection time, inventory robot location, and lighting condition. A large-scale category object detection and recognition method for inventory management of an autonomous unmanned store, characterized in that it is an influencing factor.
According to claim 1,
The fourth step is a step of constructing an initial model with a small amount of labeled data and gradually optimizing performance through an augmented ASSL learning process.
According to claim 1,
After the fourth step, a fifth step of deriving a detection and recognition result by inputting multiple images scanned by the N-camera from the inventory robot to the hierarchical deep learning detection recognizer; A large-scale categorical object detection and recognition method for inventory management.
According to claim 5,
The fifth step is an autonomous unmanned store, characterized in that the hierarchical deep learning detection recognizer receives feedback from the MLDL model generator and performs detection and recognition of large-scale category objects using inventory robots and store meta-factor information. A method for detecting and recognizing large-scale categorical objects for inventory management.
According to claim 5,
After the fifth step, a sixth step of updating an image data archive and a metadata archive by using the derived meta factor and the detection and recognition result of the hierarchical deep learning detection recognizer; and
A seventh step of updating a hierarchical deep learning model based on the updated image data archive, meta data archive, and meta factor; large-scale category object detection and recognition method for inventory management of an autonomous unmanned store, characterized in that it further comprises .
According to claim 5,
The fifth step is a step of continuously deriving detection and recognition results by inputting multiple images scanned by N-cameras from the inventory robot to the hierarchical deep learning detection recognizer, a large-scale category for inventory management of autonomous unmanned stores, characterized in that Object detection recognition method.
According to claim 8,
The fifth step comprises the step of applying a time-window technique for advanced detection and recognition of multiple product objects on the shelf.
According to claim 9,
The time-window technique extracts an optimized multi-object tracking segment (partial tracking path performed in batch units) prior to generating the final tracking path from the object detection result of each image frame. Applying a time-window technique for; And since there are many variables that affect object identification in the operating location of the inventory robot, NMS (Non-Maximum Suppression) is a type of algorithm, WNMS (Weighted A method for detecting and recognizing large-scale category objects for inventory management of autonomous unmanned stores, comprising: applying non-maximum suppression.

Images