WO2021090753A1 - 処理装置、処理方法及びプログラム - Google Patents

処理装置、処理方法及びプログラム Download PDF

Info

Publication number
WO2021090753A1
WO2021090753A1 PCT/JP2020/040581 JP2020040581W WO2021090753A1 WO 2021090753 A1 WO2021090753 A1 WO 2021090753A1 JP 2020040581 W JP2020040581 W JP 2020040581W WO 2021090753 A1 WO2021090753 A1 WO 2021090753A1
Authority
WO
WIPO (PCT)
Prior art keywords
foreign matter
region
captured image
matter region
management target
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2020/040581
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
淳 内村
裕司 田原
莉奈 富田
康代 加増
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
NEC Corp
Original Assignee
NEC Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by NEC Corp filed Critical NEC Corp
Priority to JP2021554913A priority Critical patent/JP7476905B2/ja
Priority to US17/771,230 priority patent/US20220366695A1/en
Publication of WO2021090753A1 publication Critical patent/WO2021090753A1/ja
Anticipated expiration legal-status Critical
Priority to US18/232,763 priority patent/US20230386210A1/en
Priority to US18/232,760 priority patent/US20230386209A1/en
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/018—Certifying business or products
    • A—HUMAN NECESSITIES
    • A47—FURNITURE; DOMESTIC ARTICLES OR APPLIANCES; COFFEE MILLS; SPICE MILLS; SUCTION CLEANERS IN GENERAL
    • A47F—SPECIAL FURNITURE, FITTINGS, OR ACCESSORIES FOR SHOPS, STOREHOUSES, BARS, RESTAURANTS OR THE LIKE; PAYING COUNTERS
    • A47F5/00—Show stands, hangers, or shelves characterised by their constructional features
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/06—Buying, selling or leasing transactions
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0004—Industrial image inspection
    • G06T7/0008—Industrial image inspection checking presence/absence
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/60—Analysis of geometric attributes
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/90—Determination of colour characteristics
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/22—Image preprocessing by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
    • G—PHYSICS
    • G08—SIGNALLING
    • G08B—SIGNALLING SYSTEMS, e.g. PERSONAL CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
    • G08B13/00—Burglar, theft or intruder alarms
    • G08B13/18—Actuation by interference with heat, light, or radiation of shorter wavelength; Actuation by intruding sources of heat, light, or radiation of shorter wavelength
    • G08B13/189—Actuation by interference with heat, light, or radiation of shorter wavelength; Actuation by intruding sources of heat, light, or radiation of shorter wavelength using passive radiation detection systems
    • G08B13/194—Actuation by interference with heat, light, or radiation of shorter wavelength; Actuation by intruding sources of heat, light, or radiation of shorter wavelength using passive radiation detection systems using image scanning and comparing systems
    • G08B13/196—Actuation by interference with heat, light, or radiation of shorter wavelength; Actuation by intruding sources of heat, light, or radiation of shorter wavelength using passive radiation detection systems using image scanning and comparing systems using television cameras
    • G08B13/19639—Details of the system layout
    • G08B13/19641—Multiple cameras having overlapping views on a single scene
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30108—Industrial image inspection
    • G06T2207/30128—Food products
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30232—Surveillance
    • G—PHYSICS
    • G08—SIGNALLING
    • G08B—SIGNALLING SYSTEMS, e.g. PERSONAL CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
    • G08B13/00—Burglar, theft or intruder alarms
    • G08B13/18—Actuation by interference with heat, light, or radiation of shorter wavelength; Actuation by intruding sources of heat, light, or radiation of shorter wavelength
    • G08B13/189—Actuation by interference with heat, light, or radiation of shorter wavelength; Actuation by intruding sources of heat, light, or radiation of shorter wavelength using passive radiation detection systems
    • G08B13/194—Actuation by interference with heat, light, or radiation of shorter wavelength; Actuation by intruding sources of heat, light, or radiation of shorter wavelength using passive radiation detection systems using image scanning and comparing systems
    • G08B13/196—Actuation by interference with heat, light, or radiation of shorter wavelength; Actuation by intruding sources of heat, light, or radiation of shorter wavelength using passive radiation detection systems using image scanning and comparing systems using television cameras

Definitions

  • the present invention relates to a processing device, a processing method and a program.
  • Patent Document 1 stores the state of the shelves (reference state) after the store clerk organizes the products, and detects the change by comparing the state of the shelves after the customer acts on the shelves with the reference state. , Discloses a device that notifies that it is necessary to organize products on the shelves according to the detection result.
  • Foreign substances include items other than products placed on the product shelves, other products placed in the area where product A is displayed on the product shelves, floors in stores, tables, copy machines, counters, and parking lots in stores. Items that are not related to store management, etc.
  • An object of the present invention is to provide a technique for detecting a foreign substance existing in a managed object related to a store.
  • An acquisition method for acquiring captured images including management targets related to the store, A foreign matter region detecting means for detecting a foreign matter region which is a region where a foreign matter exists in the management target included in the captured image, and a foreign matter region detecting means.
  • a warning means for executing warning processing according to the size of the foreign matter region, and A processing device having the above is provided.
  • the computer Acquire the captured image including the management target related to the store, A foreign matter region, which is a region in which a foreign matter exists in the management target included in the captured image, is detected.
  • a processing method for executing warning processing is provided according to the size of the foreign matter region.
  • Computer Acquisition method for acquiring captured images including management targets related to stores, A foreign matter region detecting means for detecting a foreign matter region, which is a region in which a foreign matter exists in the management target included in the captured image.
  • a warning means that executes warning processing according to the size of the foreign matter region,
  • a program is provided to function as.
  • the present invention it is possible to detect a foreign substance existing in a management target related to a store.
  • FIG. 1 It is a figure which shows an example of the hardware composition of the processing apparatus of this embodiment. It is an example of the functional block diagram of the processing apparatus of this embodiment. It is a figure which shows typically an example of the information processed by the processing apparatus of this embodiment. It is a figure which shows typically an example of the information processed by the processing apparatus of this embodiment. It is a figure which shows typically an example of the information processed by the processing apparatus of this embodiment. It is a figure which shows typically an example of the information processed by the processing apparatus of this embodiment. It is a figure which shows typically an example of the information processed by the processing apparatus of this embodiment. It is a figure which shows typically an example of the information processed by the processing apparatus of this embodiment. It is a figure which shows typically an example of the information processed by the processing apparatus of this embodiment. It is a figure which shows typically an example of the information processed by the processing apparatus of this embodiment.
  • the processing device acquires a photographed image including a management target related to the store.
  • the management target is a target for which detection / removal of foreign matter is desired, and is not limited to, for example, a product display shelf, a floor, a table, a copier, a counter, a parking lot, and the like.
  • the processing device detects a foreign matter region, which is a region in which the foreign matter exists in the management target included in the captured image, and executes warning processing according to the size of the detected foreign matter region.
  • the processing device capable of detecting the foreign matter region in the management target included in the captured image, it is possible to automatically detect the foreign matter existing in the management target by image analysis. Then, since the processing device can perform warning processing according to the size of the detected foreign matter region, it is possible to warn against foreign matter of a negligible size that does not affect the store operation and noise of image data that is not foreign matter in the first place. False warnings based on can be avoided.
  • the functional unit included in the processing device of this embodiment is a CPU (Central Processing Unit) of an arbitrary computer, a memory, a program loaded in the memory, and a storage unit such as a hard disk for storing the program (from the stage of shipping the device in advance).
  • a storage unit such as a hard disk for storing the program (from the stage of shipping the device in advance).
  • it can also store programs downloaded from storage media such as CDs (Compact Discs) and servers on the Internet), and any combination of hardware and software centered on the network connection interface. Realized by. And, it is understood by those skilled in the art that there are various modifications of the realization method and the device.
  • FIG. 1 is a block diagram illustrating a hardware configuration of the processing device of the present embodiment.
  • the processing device includes a processor 1A, a memory 2A, an input / output interface 3A, a peripheral circuit 4A, and a bus 5A.
  • the peripheral circuit 4A includes various modules.
  • the peripheral circuit 4A does not have to be provided.
  • the processing device may be composed of one physically and / or logically integrated device, or may be composed of a plurality of physically and / or logically separated devices. When composed of a plurality of physically and / or logically separated devices, each of the plurality of devices can be provided with the above hardware configuration.
  • the bus 5A is a data transmission path for the processor 1A, the memory 2A, the peripheral circuits 4A, and the input / output interface 3A to send and receive data to and from each other.
  • the processor 1A is, for example, an arithmetic processing unit such as a CPU or a GPU (Graphics Processing Unit).
  • the memory 2A is, for example, a memory such as a RAM (RandomAccessMemory) or a ROM (ReadOnlyMemory).
  • the input / output interface 3A includes an interface for acquiring information from an input device, an external device, an external server, an external sensor, a camera, etc., an interface for outputting information to an output device, an external device, an external server, etc. ..
  • the input device is, for example, a keyboard, a mouse, a microphone, a touch panel, a physical button, a camera, or the like.
  • the output device is, for example, a display, a speaker, a printer, a mailer, or the like.
  • the processor 1A can issue commands to each module and perform calculations based on the calculation results thereof.
  • FIG. 2 shows an example of a functional block diagram of the processing device 10.
  • the processing device 10 includes an acquisition unit 11, a foreign matter region detection unit 12, and a warning unit 13.
  • the acquisition unit 11 acquires a photographed image including a management target related to the store.
  • the controlled object is an object for which the detection / removal of foreign matter is desired, and includes at least one of a product display shelf, a floor, a table, a copier, a counter, and a parking lot.
  • the management target may include other targets.
  • the acquisition unit 11 acquires a captured image generated by the camera that captures the management target.
  • the acquisition unit 11 may acquire a captured image after editing the captured image generated by the camera.
  • the editing process can be performed as needed according to the type of camera to be used, the orientation of the installed camera, etc. For example, a projective transformation, a process of expanding an image taken by a fisheye camera in a plane, and the like are exemplified. However, it is not limited to these.
  • the acquisition unit 11 may perform the editing.
  • an external device different from the processing device 10 may perform the editing, and the acquisition unit 11 may acquire the edited captured image.
  • the camera is fixed in place so as to shoot the managed object.
  • the orientation of the camera may also be fixed.
  • the camera may continuously capture moving images, or may capture still images at predetermined timings.
  • a plurality of cameras may be installed, and the acquisition unit 11 may acquire the captured image generated by each of the plurality of cameras, or one camera may be installed and the acquisition unit 11 acquires the captured image generated by the camera. You may.
  • a plurality of cameras are installed, and the acquisition unit 11 acquires captured images generated by each of the plurality of cameras.
  • FIG. 3 schematically shows an example of the photographed image P.
  • the management target is a product display shelf. The situation where the product 101 is displayed on the shelf board 100 is shown.
  • the foreign matter region detection unit 12 detects the foreign matter region in the management target included in the captured image.
  • the foreign matter region is a region where foreign matter is presumed to be present.
  • the foreign matter region detection unit 12 detects as a foreign matter region a region that is different from the designated color in the management target included in the captured image.
  • the foreign matter region detection unit 12 determines whether or not a permitted object exists in the region, and determines that the region different from the designated color in which the permitted object does not exist is a foreign matter region. May be detected as. Then, although the foreign matter region detection unit 12 is a region having a color different from the designated color, it is not necessary to detect the region where it is determined that the permitted object exists as the foreign matter region.
  • the designated color is determined for each management target.
  • the designated color is the color of the shelf board on which the product or thing is placed.
  • the designated color is the floor color.
  • the designated color is the color of the table on which the table objects are placed.
  • the designated color is the color of the upper surface of the copier on which an object may be placed.
  • the designated color is the color of the ground of the parking lot.
  • the processing device 10 may store information indicating an area in the captured image in which the management target exists and information indicating a designated color for each camera. Then, the foreign matter region detection unit 12 may specify a management target in the captured image generated by each camera based on the information, and specify a region having a color different from the designated color in the specified management target. Good.
  • the camera identification information that identifies each camera, the management target information that indicates the area in which the management target exists in the captured image, and the designated color of each management target are associated with each other.
  • the area where the management target exists is shown by specifying the quadrangular area by using the coordinates of the two-dimensional coordinate system defined for the captured image, but this is just an example.
  • the method is not limited to this method. As shown in the figure, there may be one management target in one captured image, or a plurality of management targets in one captured image. Which one will be used depends on how the camera is installed.
  • one color may be designated pinpointly, or it may be designated with a certain width.
  • a permit is an object that is permitted to exist in the management target.
  • the permit is a product.
  • the management target is a product display shelf, the permitted items may be set for each display area.
  • the permit is a product displayed in each display area. That is, the product A displayed in the display area A is a permitted item in the display area A, but is not a permitted item in the display area B.
  • the permitted items are deliveries that are temporarily placed on the floor.
  • the permitted items are goods, customer's luggage, and the like.
  • the permitted items are the customer's luggage, copy paper, and the like. If the management target is a parking lot, the permitted items are automobiles, motorcycles, and the like.
  • the processing device 10 may store information indicating a permitted object for each camera. Then, the foreign matter region detection unit 12 may grasp the permitted object in the management target included in the captured image generated by each camera based on the information.
  • the processing device 10 may store information indicating a permitted object for each camera. Then, the foreign matter region detection unit 12 may grasp the permitted object in the management target included in the captured image generated by each camera based on the information.
  • one management target is classified into a plurality of areas (plurality of display areas) like a product display shelf and a permit is specified for each area, as in the example of the camera identification information "C001" shown in the figure.
  • an estimation model for estimating an article type eg, rice ball, lunch box, automobile, motorcycle, customer's luggage, etc.
  • the foreign matter region detection unit 12 estimates the type of article existing in the region by inputting an image of the region having a color different from the designated color into the estimation model, and based on the estimation result, uses a color different from the designated color. It may be determined whether or not a permit exists in a certain area.
  • collation processing is performed between the image (template image) of the permitted object registered in the processing device 10 in advance for each display area and the image of the area having a color different from the designated color.
  • template matching etc.
  • the warning unit 13 executes warning processing according to the size of the foreign matter region detected by the foreign matter region detection unit 12. Specifically, the warning unit 13 executes a warning process when the size of the foreign matter region detected by the foreign matter region detecting unit 12 is equal to or larger than the reference value. In addition, the warning unit 13 determines whether or not the size is equal to or larger than the reference value for each foreign matter region that is a mass. That is, when a plurality of foreign matter regions separated from each other are detected, the warning unit 13 determines whether the size of each foreign matter region is equal to or larger than the reference value.
  • the reference value can be indicated by, for example, the number of pixels, but is not limited to this.
  • the reference value may be the same value for all captured images. However, for the following reasons, a reference value may be set for each camera that generated the captured image, and for each region in the captured image.
  • the size of foreign matter that needs to be removed may differ depending on the management target. For example, in the case of product display shelves, it is desirable to remove relatively small foreign substances in order to maintain a high level of cleanliness. On the other hand, in the case of parking lots, floors, etc., the level of cleanliness required is lower than that of product display shelves. For this reason, it may be permissible to leave relatively small foreign substances as they are, in balance with the burden on workers. Further, among the product display shelves, the required level of cleanliness may differ depending on the type of product to be displayed (eg, food, miscellaneous goods, books, etc.). In this way, the size of the foreign matter that needs to be removed may differ even within the same controlled object.
  • the type of product to be displayed eg, food, miscellaneous goods, books, etc.
  • the size in the captured image may differ depending on the orientation of the camera, the distance between the camera and the subject, and the like.
  • the processing device 10 may store information for which a reference value is set for each camera. Then, the warning unit 13 may determine a reference value based on the camera that generated the captured image including the detected foreign matter region, and determine whether the size of the detected foreign matter region is equal to or larger than the determined reference value. ..
  • the processing device 10 may store information in which a reference value is set for each position in the captured image. Then, the warning unit 13 may determine a reference value based on the position of the detected foreign matter region in the captured image, and may determine whether the size of the detected foreign matter region is equal to or larger than the determined reference value.
  • the warning process may be a process of notifying a predetermined user that a foreign substance has been detected by real-time processing in response to the detection by the foreign substance area detection unit 12.
  • the warning process accumulates information indicating a foreign matter area whose size is equal to or larger than the reference value, and is accumulated up to that point at a predetermined timing (eg, every hour, when a user has entered a browsing input, etc.).
  • It may be a process of notifying a predetermined user of the information (eg, transmitting a predetermined information to a predetermined terminal device).
  • the notification to the user may be the output of information via an output device such as a display, a projection device, or a speaker, the transmission of information via a mailer, or the like, or on an application or a web page.
  • Information may be displayed, a warning lamp may be lit, or the like, or the like.
  • the information output in the process of notifying the user may include a captured image in which a foreign matter region whose size is equal to or larger than the reference value is detected. Further, it may further include information for highlighting a foreign matter region whose size is equal to or larger than the reference value with a frame or the like. An example is shown in FIG. In the illustrated example, in the photographed image showing the product display shelf (managed object), the foreign matter region 103 whose detected size is equal to or larger than the reference value is highlighted by surrounding it with a frame 102.
  • the captured image generated before the captured image is generated by the camera that generated the captured image (eg, the frame image immediately before, the frame image several frames before, etc.). May be output together. This makes it easy to compare the state in which the foreign matter is present and the state in which the foreign matter is not present.
  • the information output in the process of notifying the user may include information indicating an instruction to the worker (eg, foreign matter removal, notification to a predetermined user, etc.).
  • the foreign matter region detection unit 12 performs a process of detecting a foreign matter region, which is a region in which a foreign matter exists in the management target included in the captured image (S11).
  • FIG. 10 shows an example of the flow of processing for detecting a foreign matter region in S11.
  • the foreign matter region detection unit 12 detects a region of the management target included in the captured image that is different from the designated color (S21). For example, the foreign body region detection unit 12 identifies the management target in the captured image and designates the management target based on the information shown in FIG. 4 and the information for identifying the camera that generated the acquired captured image. Identify the color. Then, the foreign matter region detection unit 12 detects a region having a color different from the designated color specified in the specified management target.
  • the foreign matter region detection unit 12 determines that there is no foreign matter region (S28).
  • the foreign matter region detection unit 12 classifies the detected region into a group of regions and designates one of them (Yes). S23). Then, the foreign matter region detection unit 12 determines whether or not the permitted object exists in the designated region (S24). For example, the foreign matter region detection unit 12 corresponds to the designated region based on the information shown in FIG. 5, the information for identifying the camera that generated the acquired captured image, the position of the designated region in the captured image, and the like. Identify the permit. Then, the foreign matter region detection unit 12 determines whether or not the permitted object exists in the designated region by using the method using the estimation model described above, template matching, or the like.
  • the foreign matter region detection unit 12 determines that the designated region is not a foreign matter region (S26). On the other hand, when it is determined that the permitted object does not exist (No in S24), the foreign matter region detection unit 12 determines that the designated region is a foreign matter region (S25).
  • the processing apparatus 10 ends the processing.
  • the warning unit 13 determines whether the size of the detected foreign matter region is equal to or larger than the reference value (S13). For example, the warning unit 13 determines a reference value based on the information shown in FIG. 6 or 7, information for identifying the camera that generated the acquired captured image, the position of the detected foreign matter region in the captured image, and the like. .. Then, the warning unit 13 determines whether the size of the detected foreign matter region is equal to or larger than the determined reference value.
  • the warning unit 13 executes a warning process. Since the details of the warning process are as described above, the description here will be omitted.
  • the processing apparatus 10 ends the processing.
  • the processing device 10 capable of detecting the foreign matter region in the management target included in the captured image
  • the foreign matter existing in the management target can be automatically detected by the image analysis.
  • the processing device 10 performs warning processing when the size of the detected foreign matter region is equal to or larger than the reference value, and does not perform warning processing when the size of the detected foreign matter region is less than the reference value. It is possible to avoid warnings about foreign substances that do not affect the operation and can be ignored, and false warnings based on noise of image data that is not foreign substances in the first place.
  • each management target or each predetermined area within the management target (example: display area of the product display shelf).
  • an appropriate reference value can be set according to, for example, the required cleanliness.
  • the reference value can be set for each camera or for each position in the captured image according to the orientation of the camera and the distance between the camera and the subject, the desired size is set regardless of the orientation of the camera and the distance between the camera and the subject. It is possible to detect more foreign substances with high accuracy.
  • the designated color can be specified and the region having a color different from the designated color can be detected as the foreign matter region, the computer burden of the process of detecting the foreign matter region can be relatively lightened.
  • the permitted object can be determined in advance and the area where the permitted object does not exist can be detected as a foreign substance area, the inconvenience of detecting an object whose existence in the management target does not matter as a foreign substance can be avoided.
  • the processing device 10 of the present embodiment differs from the first embodiment in the content of the process of detecting the foreign matter region by the foreign matter region detecting unit 12.
  • the foreign matter region detection unit 12 detects a region in which an object exists in the management target included in the captured image based on a well-known object detection technique. After that, the foreign matter region detection unit 12 determines whether or not the permitted object exists in the region where the object exists. Specifically, the foreign matter region detection unit 12 determines whether or not the detected object is a permitted object based on the characteristics of the appearance of the detected object and the permitted object. The determination is realized by the same method as "determination of whether or not a permitted object exists in a region having a color different from the designated color" described in the first embodiment. Then, the foreign matter region detection unit 12 detects the region (the region where the object exists) determined that the permitted object does not exist as the foreign matter region. On the other hand, the foreign matter region detection unit 12 does not detect the region (the region where the object exists) determined to have the permitted object as the foreign matter region.
  • FIG. 11 shows an example of the flow of processing for detecting a foreign matter region in S11.
  • the foreign matter region detection unit 12 performs a process of detecting an object in the management target included in the captured image based on an arbitrary object detection technique (S31). For example, the foreign matter region detection unit 12 identifies the management target in the captured image based on the information shown in FIG. 12 and the information that identifies the camera that generated the acquired captured image. Then, the foreign matter region detection unit 12 detects an object in the specified management target based on an arbitrary object detection technique.
  • S31 arbitrary object detection technique
  • the foreign matter region detection unit 12 determines that there is no foreign matter region (S38).
  • the foreign matter region detection unit 12 designates one of the detected objects (S33). Then, the foreign matter region detection unit 12 determines whether or not the permitted object exists in the region where the designated object exists (S34). For example, the foreign matter region detection unit 12 is designated based on the information shown in FIG. 5, the information for identifying the camera that generated the acquired captured image, the position in the captured image of the region where the designated object exists, and the like. Identify the permit corresponding to the object. Then, the foreign matter region detection unit 12 determines whether or not the permitted object exists in the region where the designated object exists by using the method using the above-mentioned estimation model, template matching, or the like.
  • the foreign matter region detection unit 12 determines that the region in which the designated object exists is not the foreign matter region (S36). On the other hand, when it is determined that the permitted object does not exist (No in S34), the foreign matter region detection unit 12 determines that the region in which the designated object exists is the foreign matter region (S35).
  • the operation and effect of the processing device 10 of the present embodiment will be described. According to the processing device 10 of the present embodiment, the same effects as those of the processing device 10 of the first embodiment are realized. In addition, since pre-registration of the designated color is not required, the processing load is reduced accordingly.
  • acquisition means “the own device goes to retrieve the data stored in another device or storage medium” based on the user input or the instruction of the program (active). Acquisition) ”, for example, requesting or inquiring about another device to receive the data, accessing another device or a storage medium to read the data, and the like. Further, “acquisition” means “inputting data output from another device to the own device (passive acquisition)” based on user input or program instruction, for example, distribution (or distribution (or). , Transmission, push notification, etc.) may be included. In addition, “acquisition” means to select and acquire from received data or information, and “edit data (text conversion, data sorting, partial data extraction, file format change, etc.)". It may include “to generate new data and acquire the new data”.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Business, Economics & Management (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Accounting & Taxation (AREA)
  • Finance (AREA)
  • Strategic Management (AREA)
  • General Business, Economics & Management (AREA)
  • Marketing (AREA)
  • Economics (AREA)
  • Development Economics (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Geometry (AREA)
  • Computing Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Quality & Reliability (AREA)
  • Databases & Information Systems (AREA)
  • Evolutionary Computation (AREA)
  • Signal Processing (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Image Analysis (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
  • Display Racks (AREA)
  • Closed-Circuit Television Systems (AREA)
  • Alarm Systems (AREA)
PCT/JP2020/040581 2019-11-05 2020-10-29 処理装置、処理方法及びプログラム Ceased WO2021090753A1 (ja)

Priority Applications (4)

Application Number Priority Date Filing Date Title
JP2021554913A JP7476905B2 (ja) 2019-11-05 2020-10-29 処理装置、処理方法及びプログラム
US17/771,230 US20220366695A1 (en) 2019-11-05 2020-10-29 Processing device, processing method, and non-transitory storage medium
US18/232,763 US20230386210A1 (en) 2019-11-05 2023-08-10 Processing device, processing method, and non-transitory storage medium
US18/232,760 US20230386209A1 (en) 2019-11-05 2023-08-10 Processing device, processing method, and non-transitory storage medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2019200590 2019-11-05
JP2019-200590 2019-11-05

Related Child Applications (3)

Application Number Title Priority Date Filing Date
US17/771,230 A-371-Of-International US20220366695A1 (en) 2019-11-05 2020-10-29 Processing device, processing method, and non-transitory storage medium
US18/232,763 Continuation US20230386210A1 (en) 2019-11-05 2023-08-10 Processing device, processing method, and non-transitory storage medium
US18/232,760 Continuation US20230386209A1 (en) 2019-11-05 2023-08-10 Processing device, processing method, and non-transitory storage medium

Publications (1)

Publication Number Publication Date
WO2021090753A1 true WO2021090753A1 (ja) 2021-05-14

Family

ID=75848247

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2020/040581 Ceased WO2021090753A1 (ja) 2019-11-05 2020-10-29 処理装置、処理方法及びプログラム

Country Status (3)

Country Link
US (3) US20220366695A1 (https=)
JP (1) JP7476905B2 (https=)
WO (1) WO2021090753A1 (https=)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2023096426A (ja) * 2021-12-27 2023-07-07 Ｉｈｉ運搬機械株式会社 工場内天井クレーンの安全通路検出装置

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114399733B (zh) * 2022-01-14 2025-09-26 京东方科技集团股份有限公司 图像检测方法、装置、设备和存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008005399A (ja) * 2006-06-26 2008-01-10 Matsushita Electric Ind Co Ltd 放置物検出装置及び放置物検出方法
JP2017182653A (ja) * 2016-03-31 2017-10-05 パナソニックＩｐマネジメント株式会社 商品モニタリング装置、商品モニタリングシステムおよび商品モニタリング方法
JP2018133042A (ja) * 2017-02-17 2018-08-23 セコム株式会社 放置物検出装置
JP2018151819A (ja) * 2017-03-13 2018-09-27 日本電気株式会社 管理装置、管理方法及びプログラム

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6151564A (en) * 1995-08-03 2000-11-21 Interval Research Corporation Coded object system and code recognition methods
US8577136B1 (en) * 2010-12-28 2013-11-05 Target Brands, Inc. Grid pixelation enhancement for in-stock analytics
EP3374947A4 (en) * 2015-11-09 2019-03-27 Simbe Robotics, Inc. PROCESS FOR TRACKING THE STORAGE STOCK IN A STORAGE
US10943363B2 (en) * 2016-07-21 2021-03-09 Nec Corporation Image processing apparatus, and image processing method
JP6960061B2 (ja) * 2018-01-10 2021-11-05 シムビ ロボティクス, インコーポレイテッドSimbe Robotics, Inc. 溢れおよび危険物を検出して対応する方法
US11398089B1 (en) * 2021-02-17 2022-07-26 Adobe Inc. Image processing techniques to quickly find a desired object among other objects from a captured video scene

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008005399A (ja) * 2006-06-26 2008-01-10 Matsushita Electric Ind Co Ltd 放置物検出装置及び放置物検出方法
JP2017182653A (ja) * 2016-03-31 2017-10-05 パナソニックＩｐマネジメント株式会社 商品モニタリング装置、商品モニタリングシステムおよび商品モニタリング方法
JP2018133042A (ja) * 2017-02-17 2018-08-23 セコム株式会社 放置物検出装置
JP2018151819A (ja) * 2017-03-13 2018-09-27 日本電気株式会社 管理装置、管理方法及びプログラム

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2023096426A (ja) * 2021-12-27 2023-07-07 Ｉｈｉ運搬機械株式会社 工場内天井クレーンの安全通路検出装置

Also Published As

Publication number Publication date
US20230386209A1 (en) 2023-11-30
US20220366695A1 (en) 2022-11-17
JPWO2021090753A1 (https=) 2021-05-14
JP7476905B2 (ja) 2024-05-01
US20230386210A1 (en) 2023-11-30

Similar Documents

Publication Publication Date Title
US20140003655A1 (en) Method, apparatus and system for providing image data to represent inventory
JP2018041261A (ja) 情報処理装置及びプログラム
JP6147676B2 (ja) 情報処理装置、店舗システム及びプログラム
US20230386210A1 (en) Processing device, processing method, and non-transitory storage medium
WO2020179480A1 (ja) 物品推定装置、物品推定方法、及びプログラム
TW201742007A (zh) 貨架取物偵測系統及方法
JP7537753B2 (ja) 管理装置及び管理方法
JP6558020B2 (ja) 検品処理装置、検品処理方法及びプログラム
JP7405144B2 (ja) 処理装置、処理方法及びプログラム
JP7235119B2 (ja) 処理装置、会計装置、処理方法及びプログラム
JP7435021B2 (ja) 情報処理装置、情報処理方法、およびプログラム
JP7619427B2 (ja) 処理装置、処理方法及びプログラム
JP7070654B2 (ja) 登録装置、登録方法及びプログラム
JP7613370B2 (ja) 処理装置、処理方法及びプログラム
US12205384B2 (en) Processing apparatus, processing method, and non-transitory storage medium
JPWO2019181033A1 (ja) 登録システム、登録方法及びプログラム
JP5529982B2 (ja) 情報処理装置
JP7544269B2 (ja) 欠品検出装置、欠品検出方法、及びプログラム
US20250200595A1 (en) Store rental support device, store rental support method, and recording medium
JP7428241B2 (ja) 処理装置、処理方法及びプログラム
JP7380689B2 (ja) 処理装置、処理方法及びプログラム
JP7574419B2 (ja) 事故要因管理システム、方法及びプログラム
JP7380869B2 (ja) 処理装置、前処理装置、処理方法、及び前処理方法
JPWO2019186975A1 (ja) 会計装置、会計システム、商品識別方法、およびプログラム
US20200342518A1 (en) Item recognition and presention within images

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20883882

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021554913

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20883882

Country of ref document: EP

Kind code of ref document: A1