CN117218762B - Intelligent container interaction control method, device and system based on machine vision - Google Patents

Intelligent container interaction control method, device and system based on machine vision Download PDF

Info

Publication number
CN117218762B
CN117218762B CN202311483613.0A CN202311483613A CN117218762B CN 117218762 B CN117218762 B CN 117218762B CN 202311483613 A CN202311483613 A CN 202311483613A CN 117218762 B CN117218762 B CN 117218762B
Authority
CN
China
Prior art keywords
image
intelligent container
self
hardware
checking
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202311483613.0A
Other languages
Chinese (zh)
Other versions
CN117218762A (en
Inventor
邱俊波
黄超群
陈俏锋
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Yopoint Smart Retail Technology Ltd
Original Assignee
Yopoint Smart Retail Technology Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Yopoint Smart Retail Technology Ltd filed Critical Yopoint Smart Retail Technology Ltd
Priority to CN202311483613.0A priority Critical patent/CN117218762B/en
Priority to CN202410339580.0A priority patent/CN118247884A/en
Publication of CN117218762A publication Critical patent/CN117218762A/en
Application granted granted Critical
Publication of CN117218762B publication Critical patent/CN117218762B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G07—CHECKING-DEVICES
    • G07F—COIN-FREED OR LIKE APPARATUS
    • G07F9/00—Details other than those peculiar to special kinds or types of apparatus
    • G07F9/002—Vending machines being part of a centrally controlled network of vending machines
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/10—Image acquisition
    • G06V10/12—Details of acquisition arrangements; Constructional details thereof
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G—PHYSICS
    • G07—CHECKING-DEVICES
    • G07C—TIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE
    • G07C9/00—Individual registration on entry or exit
    • G07C9/00174—Electronically operated locks; Circuits therefor; Nonmechanical keys therefor, e.g. passive or active electrical keys or other data carriers without mechanical keys
    • G07C9/00896—Electronically operated locks; Circuits therefor; Nonmechanical keys therefor, e.g. passive or active electrical keys or other data carriers without mechanical keys specially adapted for particular uses
    • G—PHYSICS
    • G07—CHECKING-DEVICES
    • G07F—COIN-FREED OR LIKE APPARATUS
    • G07F11/00—Coin-freed apparatus for dispensing, or the like, discrete articles
    • G07F11/004—Restocking arrangements therefor
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
    • Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
    • Y02P90/02—Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]

Landscapes

  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
  • Control Of Vending Devices And Auxiliary Devices For Vending Devices (AREA)

Abstract

The invention discloses an intelligent container interaction control method, device and system based on machine vision, and relates to the technical field of Internet of things. The method comprises the steps of controlling the intelligent container to carry out hardware self-inspection after receiving a starting command from a server, controlling an image acquisition device of the intelligent container to acquire a first continuous image, and controlling an electronic lock of the intelligent container to execute unlocking operation after the first continuous image is acquired successfully; and controlling the image acquisition device to start to acquire a second continuous image, transmitting the acquired second continuous image to the server, and driving the server to transmit order information to the intelligent terminal. The method ensures the normal operation and stability of the equipment through hardware self-checking, accurately identifies the commodity category and quantity and generates order information through real-time image acquisition and image analysis by sending to the server, so that the whole transaction process is more convenient and efficient, and good and convenient purchasing experience is provided for users.

Description

Intelligent container interaction control method, device and system based on machine vision
Technical Field
The invention relates to the technical field of Internet of things, in particular to an intelligent container interaction control method, device and system based on machine vision.
Background
The intelligent container is a self-service sales system, and provides goods and services in a more efficient and convenient mode through the Internet of things, machine vision technology and the like. Intelligent containers are often equipped with cameras, sensors, display screens, and internet-connected capabilities that enable automatic monitoring of cargo inventory, real-time data transmission, interaction with users, and payment processing. The intelligent container is available all-weather, is not limited by the opening time of a store, a user can purchase goods at any time without waiting, queuing time is reduced, in addition, the intelligent container can automatically monitor inventory and supplement goods, expiration and waste of goods are reduced, and retailers can better know market demands and user preferences through analyzing sales data so as to make a more intelligent goods-loading decision.
Intelligent containers bring numerous convenience to users, but there are often problems in the use process: hardware components of the intelligent container, such as a camera and an electronic lock, may malfunction, so that interaction is interrupted or wrong, the type or the number of commodities cannot be accurately identified due to image acquisition defects of the camera, interaction process is long or purchase is failed due to large data transmission delay, and the user experience is seriously affected by the problems.
Disclosure of Invention
In view of the above, the embodiments of the present invention provide a machine vision-based intelligent container interaction control method, apparatus and system, which are used for solving the problem of machine vision-based intelligent container interaction control in the prior art.
In a first aspect, an embodiment of the present invention provides a machine vision-based intelligent container interaction control method, which is applied to an intelligent container end, and the method includes:
responding to a starting command from a server, controlling a hardware device of the intelligent container to perform hardware self-checking and obtaining a hardware self-checking result, wherein the hardware self-checking result comprises normal or abnormal;
when the hardware self-checking result is normal, controlling an image acquisition device of the intelligent container to acquire a first continuous image according to preset time;
when the first continuous image acquisition is successful, controlling the electronic lock of the intelligent container to execute unlocking operation;
controlling the image acquisition device to start acquiring a second continuous image until a locking signal from the electronic lock is received;
and sending the acquired second continuous images to the server and driving the server to send order information to the intelligent terminal.
Preferably, the controlling the hardware device of the intelligent container to perform the hardware self-test and obtain the hardware self-test result in response to the start command from the server includes:
responding to the starting command, and controlling a main control system of the intelligent container to perform system initialization;
detecting and verifying the working state of a hardware device in the intelligent container, wherein the hardware device comprises at least one of an image acquisition device, an electronic lock and a network communication module;
and when the system is successfully initialized and the working states of the hardware devices are normal, the hardware self-checking result is normal, otherwise, the hardware self-checking result is abnormal.
Preferably, in response to a start command from the server, controlling the intelligent container to perform hardware self-inspection and obtain a hardware self-inspection result further includes:
transmitting process information generated in a hardware self-checking process to the server in real time, wherein the process information is input into a preset hardware self-checking prediction learning model of the server and generates hardware self-checking result prediction information;
acquiring the hardware self-checking result prediction information and determining necessary self-checking items and unnecessary self-checking items according to the hardware self-checking result prediction information;
and executing the hardware self-checking operation of the intelligent container according to the necessary self-checking item.
Preferably, the controlling the image capturing device to start capturing the second continuous image until receiving the lock signal from the electronic lock includes:
extracting frame images including intelligent container cabinet door information in the second continuous images, and marking the frame images as cabinet door images;
generating cabinet door state information in real time according to the cabinet door image, wherein the cabinet door state information at least comprises one of cabinet door opening and cabinet door closing;
acquiring the cabinet door state information when the locking signal is received;
and if the cabinet door state information is that the cabinet door is closed, controlling the image acquisition device to stop acquisition of the second continuous images.
Preferably, the image acquisition device at least comprises a first image acquisition device and a second image acquisition device, wherein the acquisition areas of the first image acquisition device and the second image acquisition device are not completely overlapped, and when the hardware self-checking result is normal, the image acquisition device for controlling the intelligent container to acquire the first continuous image according to the preset time comprises:
controlling the first image acquisition device and the second image acquisition device to acquire a first sub-continuous image and a second sub-continuous image according to preset time at the same time;
And stitching the first sub-continuous image and the second sub-continuous image to obtain the first continuous image.
Preferably, the first image acquisition device and the second image acquisition device are controlled to acquire a third sub-continuous image and a fourth sub-continuous image simultaneously, the second continuous image includes the third sub-continuous image and the fourth sub-continuous image, and the sending the acquired second continuous image to the server and driving the server to send order information to the intelligent terminal includes:
acquiring a first key frame image recorded with preset key information in the third sub-continuous image, and performing data segmentation processing on the first key frame image to acquire a first preset key image;
acquiring a second key frame image recorded with preset key information in the fourth sub-continuous image, and performing data segmentation processing on the second key frame image to acquire a second preset key image;
respectively acquiring the data volume of the first preset key image and the data volume of the second preset key image, and recording the data volume as a third data volume and a fourth data volume;
determining a third transmission channel corresponding to the first preset key image and a fourth transmission channel corresponding to the second preset key image according to the third data amount and the fourth data amount respectively;
And sending the first preset key image and the second preset key image to the server through the third transmission channel and the fourth transmission channel respectively, driving the server to process and analyze data according to the first preset key image and the second preset key image, and then sending order information to the intelligent terminal.
In a second aspect, an embodiment of the present invention provides a machine vision-based intelligent container interaction control method, which is applied to a server, and the method includes:
receiving a starting request from an intelligent terminal, and sending a starting command to an intelligent container, wherein the starting command drives the intelligent container to perform hardware self-inspection and acquire a first continuous image and a second continuous image, and the intelligent terminal comprises any one of a smart phone and an intelligent container man-machine interaction device;
receiving a second continuous image from the intelligent container, wherein the second continuous image is an image continuously acquired by the image acquisition device of the intelligent container from unlocking to locking of an electronic lock;
performing image recognition analysis processing on the second continuous images and generating order information;
And sending the order information to the intelligent terminal.
In a third aspect, an embodiment of the present invention provides an intelligent container interaction control device based on machine vision, applied to an intelligent container end, where the device includes:
the self-checking module is used for responding to a starting command from the server, controlling the hardware device of the intelligent container to carry out hardware self-checking and obtaining a hardware self-checking result, wherein the hardware self-checking result comprises normal or abnormal;
the first acquisition module is used for controlling the image acquisition device of the intelligent container to acquire a first continuous image according to preset time when the hardware self-checking result is normal;
the unlocking module is used for controlling the electronic lock of the intelligent container to execute unlocking operation when the first continuous image acquisition is successful;
the second acquisition module is used for controlling the image acquisition device to start to acquire a second continuous image until a locking signal from the electronic lock is received; the order acquisition module is used for sending the acquired second continuous images to the server and driving the server to send order information to the intelligent terminal.
In a fourth aspect, an embodiment of the present invention provides an intelligent container interaction control device based on machine vision, which is applied to a server, and the device includes:
The starting command sending module is used for receiving a starting request from the intelligent terminal and sending a starting command to the intelligent container, wherein the starting command drives the intelligent container to perform hardware self-inspection and acquire a first continuous image and a second continuous image, and the intelligent terminal comprises any one of a smart phone and an intelligent container man-machine interaction device;
the second image receiving module is used for receiving a second continuous image from the intelligent container, wherein the second continuous image is an image continuously acquired from the unlocking start to the locking start of the electronic lock by the image acquisition device of the intelligent container;
the order generation module is used for carrying out image recognition analysis processing on the second continuous images and generating order information;
and the order sending module is used for sending the order information to the intelligent terminal.
In a fifth aspect, an embodiment of the present invention provides an intelligent container system, the system comprising an intelligent container and a server, wherein the intelligent container comprises the machine vision based intelligent container interaction control device according to the third aspect, and the server comprises the machine vision based intelligent container interaction control device according to the fourth aspect.
In summary, the beneficial effects of the invention are as follows:
according to the intelligent container interaction control method, device and system based on machine vision, provided by the embodiment of the invention, the intelligent container is controlled to carry out hardware self-inspection and a hardware self-inspection result is obtained by responding to a starting command from a server; when the hardware self-checking result is normal, controlling an image acquisition device of the intelligent container to acquire a first continuous image according to preset time; when the first continuous image acquisition is successful, controlling the electronic lock of the intelligent container to execute unlocking operation; controlling the image acquisition device to start acquiring a second continuous image until a locking signal from the electronic lock is received; and sending the acquired second continuous images to the server and driving the server to send order information to the intelligent terminal, so that the intelligent container interactive control and commodity transaction process are realized. The method ensures the normal operation and stability of the equipment through hardware self-checking, accurately identifies the commodity category and quantity and generates order information through real-time image acquisition, monitoring and image analysis to the server, so that the whole transaction process is more convenient and efficient, and good and convenient purchasing experience is provided for users.
Drawings
In order to more clearly illustrate the technical solution of the embodiments of the present invention, the drawings required to be used in the embodiments of the present invention will be briefly described, and it is within the scope of the present invention to obtain other drawings according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of an intelligent container system according to a first embodiment of the present invention.
FIG. 2 is a schematic diagram of the structure of an intelligent container according to the first embodiment of the present invention.
FIG. 3 is a flow chart of a machine vision-based intelligent container interaction control method according to an embodiment of the invention.
Fig. 4 is a schematic diagram of a specific flow of step S1 in fig. 3.
Fig. 5 is a schematic diagram of a specific flow of step S1 in fig. 3.
Fig. 6 is a schematic diagram of a specific flow of step S3 in fig. 3.
Fig. 7 is a schematic diagram of a specific flow of step S4 in fig. 3.
Fig. 8 is a schematic diagram of a specific flow of step S5 in fig. 3.
FIG. 9 is a flow chart of a machine vision-based intelligent container interaction control method according to a second embodiment of the invention.
FIG. 10 is a schematic diagram of a machine vision-based intelligent container interaction control device according to a third embodiment of the present invention.
FIG. 11 is a schematic diagram of a machine vision-based intelligent container interaction control device according to a fourth embodiment of the present invention.
FIG. 12 is a schematic diagram of a system for controlling the operation of an intelligent container in accordance with an embodiment of the present invention.
Detailed Description
Features and exemplary embodiments of various aspects of the present invention will be described in detail below, and in order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in further detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely configured to illustrate the invention and are not configured to limit the invention. It will be apparent to one skilled in the art that the present invention may be practiced without some of these specific details. The following description of the embodiments is merely intended to provide a better understanding of the invention by showing examples of the invention.
It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.
Example 1
The embodiment of the invention provides an intelligent container interactive control method based on machine vision, which is suitable for an intelligent container end in an intelligent container system. As shown in FIG. 1, the intelligent container system comprises an intelligent container 1 and a server 2, wherein the intelligent container 1 and the server 2 are connected in a wired or wireless way, in practical application, after receiving a purchase request from a user, the intelligent container opens a cabinet door so as to allow the user to select goods, in the process, the intelligent container collects the process of selecting goods by the user through an image collection device such as an MIPI camera, an IPC camera or a USB camera arranged on the top, and sends collected images or videos to the server, the server analyzes and processes the image data to determine the type and quantity of the goods purchased by the user, then sends order information to a user intelligent mobile phone or a man-machine interaction device such as a display screen of the intelligent container, and the user pays corresponding amount to complete the transaction after determining the order information.
In one embodiment, as shown in fig. 1, a master control device 10 of the intelligent container is provided at the upper right corner of the intelligent container, wherein the master control device 10 is provided with two image acquisition devices: the first image acquisition device 11 and the second image acquisition device 12, wherein the first image acquisition device 11 is arranged in the main control device through a sliding groove (not shown), and the first image acquisition device slides in the sliding groove so as to adjust the distance between the first image acquisition device and the second image acquisition device 12. Preferably, the collection range of the first image collection device 11 is inside and outside the intelligent container cabinet, and the collection periphery of the second image collection device 12 is outside the intelligent container cabinet. The two image acquisition devices acquire images of the inside and the outside of the intelligent container cabinet body and send the images to the server to analyze user behaviors and operations, and the user takes the types, the quantity and the like of commodities, so that order information is determined. Preferably, the angle between the viewing angle direction of the first image acquisition device 11 and the plane of the intelligent container door in the vertical direction ranges from 55 ° to 65 °, and the angle between the viewing angle direction of the second image acquisition device 12 and the plane of the intelligent container door in the horizontal direction ranges from 55 ° to 65 °. During installation, the second image acquisition device 12 is fixedly installed at the upper right corner of the intelligent container door, the position of the first image acquisition device is adjusted according to the position, the acquisition angle and the acquisition range of the second image acquisition device 12, the acquisition range of the first image acquisition device 11 and the acquisition range of the second image acquisition device can reach the best effect by continuously adjusting the distance between the first image acquisition device 11 and the second image acquisition device 12 and the visual angles of the first image acquisition device 11 and the second image acquisition device, the best effect can be determined according to the actual application condition, the acquisition range of the first image acquisition device is required to be covered to a first preset area, the acquisition range of the second image acquisition device is required to be covered to a second preset area, and the overlapping part of the first preset area and the second preset area is more than or equal to a certain size, so that the acquisition range can reach the best effect.
Referring to fig. 3, the machine vision-based intelligent container interaction control method specifically includes the following steps:
s1, responding to a starting command from a server, controlling a hardware device of the intelligent container to carry out hardware self-checking and obtaining a hardware self-checking result, wherein the hardware self-checking result comprises normal or abnormal;
s2, when the hardware self-checking result is normal, controlling an image acquisition device of the intelligent container to acquire a first continuous image according to preset time;
s3, when the first continuous image acquisition is successful, controlling the electronic lock of the intelligent container to execute unlocking operation;
s4, controlling the image acquisition device to start to acquire a second continuous image until a locking signal from the electronic lock is received;
and S5, sending the acquired second continuous images to the server and driving the server to send order information to the intelligent terminal.
Specifically, when a user has a demand of purchasing goods in the intelligent container, the user can scan a two-dimensional code provided by the intelligent container through a smart phone or click a purchasing key in a man-machine interaction device (such as a display touch screen) of the intelligent container to send a starting request to a server, the server receives the starting request from the intelligent terminal and sends a starting command to the intelligent container, after receiving the starting command, the intelligent container main control system is used for ensuring that the intelligent container hardware can normally and stably operate in the whole transaction process, firstly, the hardware self-checking of the hardware device can be carried out, the hardware self-checking is finished to obtain the result of the hardware self-checking, namely, normal or abnormal, if the result of the hardware self-checking is abnormal, abnormal information is pushed to the intelligent terminal to inform the user that the intelligent container cannot be used and pushed to an operator of the intelligent container to inform the operator of timely maintenance.
When the hardware self-checking result is normal, the image acquisition device is controlled to acquire a first continuous image of preset time, wherein the preset time can be determined according to actual conditions, and is preferably 2s-3s. When the first continuous image is successfully collected, the intelligent container sends an unlocking instruction to the electronic lock and controls the electronic lock to unlock, a user can open the cabinet door and select goods, meanwhile, the image collecting device is controlled to continuously collect images of the process of selecting goods by the user to obtain a second continuous image, after the user finishes selecting and closes the cabinet door to trigger the electronic lock to send a locking signal, the collected second continuous image is sent to the server, the server analyzes and processes the collected second continuous image according to the collected second continuous image, the type and quantity of goods selected by the user are identified, the order amount is determined according to the type and quantity of goods, order information is pushed to the intelligent terminal to inform the user, and the user finishes payment according to the order information to finish commodity transaction. After the transaction is completed, the intelligent container is preferably controlled to be in a standby state to save power.
Referring to FIG. 4, in one embodiment, the controlling the intelligent container to perform a hardware self-test and obtain a hardware self-test result in response to a start command from a server, wherein the hardware self-test result includes normal or abnormal states including:
S11, responding to the starting command, and controlling a main control system of the intelligent container to initialize a system;
s12, detecting and verifying the working state of a hardware device in the intelligent container, wherein the hardware device comprises any one or more of an image acquisition device, an electronic lock and a network communication module;
s13, when the system is successfully initialized and the working states of the hardware devices are normal, the hardware self-checking result is normal, otherwise, the hardware self-checking result is abnormal.
Specifically, after receiving the start command, the intelligent container first executes basic system initialization, executes operations such as memory allocation and resource allocation, and after the system initialization is successful, starts to detect and verify the working state of the image acquisition device, the working state of the electronic lock, the working state of the network communication module, the working state of the temperature and humidity sensor in the container body, and the like, if the working states of the hardware devices such as the image acquisition device, the electronic lock, the network communication module, and the like are all normal, the hardware self-checking result is normal, otherwise, the hardware self-checking result is abnormal.
In one embodiment, referring to FIG. 5, in response to a start command from the server, controlling the intelligent container to perform a hardware self-test and obtain a hardware self-test result further comprises:
S14, transmitting process information generated in a hardware self-checking process to the server in real time, wherein the process information is input into a preset hardware self-checking prediction learning model of the server and generates hardware self-checking result prediction information;
s15, acquiring the hardware self-checking result prediction information and determining necessary self-checking items and unnecessary self-checking items according to the hardware self-checking result prediction information;
s16, executing the hardware self-checking operation of the intelligent container according to the necessary self-checking item.
In order to improve the self-checking efficiency of hardware, process information and result information generated in the self-checking process of hardware are sent to a server in real time, a machine learning model, namely a preset hardware self-checking prediction learning model, is preset in the server, the preset hardware self-checking prediction learning model can predict a hardware self-checking result under the current condition according to the process information generated in the self-checking process of hardware obtained in real time after training and learning according to historical data, and returns hardware self-checking result prediction information, an intelligent container determines necessary self-checking items and unnecessary self-checking items according to the hardware self-checking result prediction information, and exemplarily, the connection state of an image acquisition device is known to be abnormal according to the hardware self-checking result prediction information returned by the server, so that the detection and verification of the connection state of the image acquisition device are required to be executed; and obtaining the state stability of the hygrothermograph from the multiple hardware self-test results, wherein the detection of the hygrothermograph can be used as an unnecessary self-test item. The machine learning model is added in the hardware self-checking process to predict the self-checking result of the intelligent container under the current hardware condition, unnecessary self-checking steps can be skipped, the self-checking time is shortened, the use of system resources is searched, the efficiency is improved, and the hardware device which possibly has problems can be also helped to be predicted so as to inform operators to take preventive maintenance measures, and the user experience is improved.
In this embodiment, as shown in fig. 2, the image capturing device at least includes a first image capturing device and a second image capturing device, where the capturing areas of the first image capturing device and the second image capturing device do not completely overlap, so that the first continuous image includes a first sub-continuous image and a second sub-continuous image captured by the first image capturing device, please refer to fig. 6, and when the hardware self-test result is normal, the image capturing device of the intelligent container is controlled to capture the first continuous image according to a preset time and includes:
s21, controlling the first image acquisition device and the second image acquisition device to acquire a first sub-continuous image and a second sub-continuous image according to preset time at the same time;
s22, stitching the first sub-continuous image and the second sub-continuous image to obtain the first continuous image.
Specifically, when the hardware self-checking result is normal, the first image acquisition device and the second image acquisition device are respectively controlled to simultaneously acquire continuous images within preset time, and the continuous images are respectively recorded as a first sub-continuous image and a second sub-continuous image. And stitching the first sub-continuous image and the second sub-continuous image to obtain a first continuous image. Further, analysis processing can be performed to determine whether the first sub-continuous image and the second sub-continuous image meet the preset requirement, and exemplary resolution, frame rate, image acquisition range, and the like meet the preset requirement, and the acquired image is accurate only when the acquired image meets the preset requirement, and order information obtained by analysis processing according to the images is accurate. If the image does not meet the preset requirement, the image acquisition device of the intelligent container can be informed to correct the acquisition parameters and acquire the image meeting the preset requirement again.
When the first continuous image is successfully collected, the intelligent container controls the electronic lock to unlock, so that a user can open the cabinet door and pick up the commodity, and simultaneously, the image collecting device is controlled to continuously collect the image of the process of picking up the commodity by the user to obtain a second continuous image until the user finishes picking up the commodity and closes the cabinet door to trigger the electronic lock to send a locking signal, in order to ensure the integrity and accuracy of the second continuous image collecting process, please refer to fig. 7, the steps of controlling the image collecting device to start collecting the second continuous image until the locking signal from the electronic lock is received include:
s31, extracting a frame image including intelligent container door information in the second continuous image, and recording the frame image as a door image;
s32, generating cabinet door state information in real time according to the cabinet door image, wherein the cabinet door state information at least comprises one of cabinet door opening and cabinet door closing;
s33, acquiring the cabinet door state information when the locking signal is received;
and S34, if the cabinet door state information is that the cabinet door is closed, controlling the image acquisition device to stop acquisition of the second continuous images.
Specifically, information such as opening of the cabinet door by the user after unlocking, opening state of the cabinet door, selection of commodities, types and quantity of the selected commodities, closing of the cabinet door after finishing selection and the like is recorded in the second continuous image, the electronic lock is correspondingly locked and sends locking information to the main control system when the cabinet door is closed, and at the moment, the main control system considers that the user has finished commodity extraction and finishes current acquisition of the second continuous image. In order to avoid the situation that the electronic lock is in an abnormal state in the using process, locking information is sent out to finish the acquisition of the second continuous image in advance under the condition that the cabinet door is not closed, and the acquisition of the information is not completed, so that order accuracy is affected.
In the practical application process, a user can unintentionally or intentionally shield the commodity from the image which is acquired by the image acquisition device and cannot be acquired by the image acquisition device in the process of opening the cabinet door to take out the commodity, so that commodity information cannot be acquired. To avoid that the image capturing device cannot directly capture images of the commodity in the commodity taking process, in one embodiment, controlling the image capturing device to capture the second continuous image includes:
controlling the image acquisition device to acquire a motion track from opening the door to taking out the commodity;
determining a commodity area where the commodity taken out by the user is located according to the motion trail;
and comparing the frame image of the commodity area before the user takes out the commodity with the frame image of the commodity area after the user takes out the commodity, and determining the information of the commodity taken out by the user.
In this embodiment, the collection range of the image collection device is adjusted so that the image collection device can completely collect the whole image of the user and the image of each commodity area in the cabinet. Because multiple commodities with different prices are often placed in the intelligent container, preferably, when placing the commodities, similar commodities with the same or similar prices are placed in the same area, for example, commodities with the same or similar prices and lower prices, such as beverages, are placed on a first layer of shelves in the intelligent container, the first layer of shelves is the first commodity area, the beverages with higher prices are placed on a second layer of shelves (the second commodity area), the breads with the same or similar prices and lower prices are placed on a third layer of shelves (the third commodity area), the breads with higher prices are placed on a fourth layer of shelves (the fourth commodity area), and the like. When the first continuous image is acquired after the hardware self-checking, the acquired commodity placement information and the information of commodity areas where all commodities are located can be recorded. When a user opens an intelligent container door and starts to control an image acquisition device to acquire a second continuous image, the second continuous image records the motion trail of the user from the opening door, stretching hands into the container body, stopping the hands in a commodity area, moving the hands out of the container body and the like, if the user unintentionally or intentionally shields the commodity in the commodity taking process, and the image acquisition device cannot acquire commodity image information, at the moment, the commodity area where the commodity taken out of the image acquisition device is located is determined according to the motion trail of the user, meanwhile, the frame image of the commodity area before the user takes out the commodity and the frame image of the commodity area after the user takes out the commodity in the area are extracted, and the information such as the quantity and the price of the commodity taken out by the user is determined by comparing the two types of frame images.
In addition, the following situations may occur in practical applications: the acquisition frame rate of the image acquisition device is set to be low, and the user takes out goods too fast, so that the image acquisition image is blurred, and the confirmation of the goods information is influenced. In one embodiment, controlling the image acquisition device to acquire the second continuous image further comprises:
s301, extracting a plurality of frame images initially acquired by an image acquisition device, namely, an initial frame image;
s302, acquiring the definition grade of an initial frame image and judging whether the definition grade meets the preset image analysis requirement;
s303, if not, adjusting the acquisition frame rate of the image acquisition device according to the definition grade and the preset image analysis requirement;
s304, re-acquiring a plurality of previous frame images acquired by the image acquisition device after the frame rate is adjusted and recording the previous frame images as adjusted frame images;
s305, acquiring the definition grade of the adjusted frame image and judging whether the definition grade of the adjusted frame image meets the preset image analysis requirement; steps S303 to S305 are repeated until the sharpness level meets the preset image analysis requirement.
In order to ensure that the definition of the image acquired by the image acquisition device meets the requirement of subsequent image analysis, in this embodiment, a plurality of definition levels may be set before the image is acquired, and a preset image analysis requirement required to be met when the required commodity information is required to be acquired from the acquired image and a definition level corresponding to the preset image analysis requirement are determined. When image acquisition is started, firstly, images of a plurality of previous frames are extracted, whether the definition of the images meets the preset image analysis requirement is determined, and if the definition of the images does not meet the preset image analysis requirement, the acquisition frame rate of the image acquisition device is required to be adjusted to enable the images to meet the preset image analysis requirement. In other embodiments, the image capturing device may further adjust parameters affecting the image sharpness, such as exposure, light source, etc. while adjusting the capturing frame rate of the image capturing device, so that the image capturing device after parameter adjustment may clearly capture an image even if the user acts too fast, etc. so as to facilitate subsequent acquisition of merchandise information.
In practical applications, the image acquisition device may also be unintentionally or maliciously blocked. In one embodiment, the image acquisition device comprises at least two image acquisition devices. When one of the image acquisition devices is detected to be blocked, the image acquisition range of the other image acquisition device is automatically adjusted so that the acquired image can meet the requirement of acquiring commodity information through subsequent image analysis. In addition, the acquisition frame rate of the non-occluded image acquisition device can be improved, and parameters such as light source brightness, exposure, photosensitivity, automatic focusing and the like can be adjusted, so that images meeting the subsequent image analysis requirements can be acquired even if one image acquisition device is used. If both the image acquisition devices are detected to be shielded, the intelligent container can be controlled to give an alarm and start door locking operation, and the information that the image acquisition devices are shielded is reported to an operator through a server to inform the operator of timely eliminating shielding faults.
In one embodiment, an edge calculation module is disposed at the intelligent container end, and after controlling the image acquisition device to start acquiring the second continuous image until receiving the locking signal from the electronic lock, the method further comprises:
Judging whether the connection between the intelligent container and the server is normal or not;
if not, the second continuous images are sent to the edge calculation module to be subjected to image analysis processing, and order information is generated.
Under the condition that the communication network is normal, the acquired image is sent to a server side for image analysis processing, and then an order is generated and pushed to an intelligent terminal. When a communication network has larger delay or is abnormal, an edge calculation module is arranged at the intelligent container end to ensure normal operation of commodity transaction, and when the communication delay or the abnormality occurs, the acquired image can be directly transmitted to the edge calculation module, and an order is generated after commodity information is identified according to the acquired image, so that the situation that the order cannot be generated in time to complete the transaction when the communication delay or the abnormality occurs in connection with a server is avoided.
In one embodiment, controlling the image acquisition device to acquire the second successive images further comprises:
the acquired second continuous images are sent to an edge calculation module in real time to analyze and process the second continuous images;
when the edge calculation module detects that the user has taken out the commodity and does not take other commodities in a preset interval time according to the second continuous image, or detects that the user has left the intelligent container, the intelligent container is controlled to send a settlement instruction to a server so as to drive the server to generate order information;
And controlling the intelligent container door to execute closing operation.
If the user forgets to close the cabinet door after taking out the commodity or the cabinet door is not closed due to some reasons, in order not to influence commodity price settlement and avoid energy consumption, at the moment, whether the user takes out the commodity or not can be analyzed by utilizing the edge calculation module according to the image acquired by the image acquisition device, and no other operation for taking the commodity is performed within a preset interval time or the situation that the user leaves the intelligent container is detected, on the one hand, the intelligent container is controlled to timely send a settlement instruction to the server to inform the server to timely settle accounts and generate an order, and on the other hand, the cabinet door of the intelligent container is controlled to automatically execute closing operation, so that energy consumption is avoided.
In one embodiment, as shown in fig. 2, the image capturing device includes at least a first image capturing device and a second image capturing device, where the capturing areas of the first image capturing device and the second image capturing device do not completely overlap, the second continuous image includes a third sub-continuous image captured by the first image capturing device and a fourth sub-continuous image captured by the second image capturing device, referring to fig. 8, and the sending the captured second continuous image to the server and driving the server to send order information to the intelligent terminal includes:
S41, acquiring a first key frame image recorded with preset key information in the third sub-continuous image, and performing data segmentation processing on the first key frame image to acquire a first preset key image;
s42, acquiring a second key frame image recorded with preset key information in the fourth sub-continuous image, and performing data segmentation processing on the second key frame image to acquire a second preset key image;
s43, respectively acquiring the data volume of the first preset key image and the data volume of the second preset key image, and recording the data volume as a third data volume and a fourth data volume;
s44, determining a third transmission channel corresponding to the first preset key image and a fourth transmission channel corresponding to the second preset key image according to the third data amount and the fourth data amount respectively;
s45, sending the first preset key image and the second preset key image to the server through the third transmission channel and the fourth transmission channel respectively, driving the server to process and analyze data according to the first preset key image and the second preset key image, and then sending order information to the intelligent terminal.
Specifically, in order to improve the efficiency of data transmission and image processing, in the embodiment of the invention, before image data is transmitted to a server, intelligent data segmentation processing is performed on the image data, only key frame images and image information of key areas in the key frame images are extracted, the acquisition range of a first image acquisition device comprises the inside and the outside of an intelligent container, the acquisition range of a second image acquisition device is the outside of the intelligent container, the second continuous image comprises integral information of a user standing outside the intelligent container, intelligent container door information, integral information in the intelligent container, all commodity information, commodity selecting process information of the user and commodity extracting information of the user, because the determination of order information is mainly determined by selecting commodity process information of the user and extracting commodity information (namely preset key information) of the user, then images recording the information are recorded as key frame images, frame images recording the preset key information in a third continuous image are recorded as first key frame images, the frame images recording the preset key information in the fourth continuous image are recorded as second key frame images, further extracting the whole information of the user standing in the key frame images from the areas, and further carrying out accurate capture of the commodity information in the first key frame images and the second key frame images, because the order information is accurately obtained in the order is accurately in the shopping order frame information of the user, and the order is accurately taken in the human body is accurately, and the order is accurately in the order-shaped by taking the first and the face information is taken in the condition of the condition, removing images of a body area or a face area of a user only obtains area images which are used for determining and judging information effective for generating orders, such as the types and the quantity of commodities obtained by the user, and the area images are respectively recorded as a first preset key image and a second preset key image, and obtaining third data volume and fourth data volume corresponding to the first preset key image and the second preset key image, wherein the third data volume and the fourth data volume are reduced relative to the data volume of the third sub-continuous image and the fourth sub-continuous image, so that data transmission pressure can be reduced, and processing efficiency can be improved. Preferably, the first preset key image and the second preset key image are dynamically allocated with a suitable third transmission channel and fourth transmission channel according to the size of the third data amount and the fourth data amount, for example, when the third data amount is larger, the bandwidth of the third transmission channel is adjusted to be larger. After receiving the first preset key image and the second preset key image, the server analyzes and processes the first preset key image and the second preset key image to obtain information such as commodity types and data selected by the user, generates corresponding orders according to the information, and sends the orders to the smart mobile phone of the user or the man-machine interaction device of the intelligent container. The user pays the goods money according to the order information so as to complete the transaction.
In one embodiment, when the image acquisition device is abnormal due to occlusion or unreasonable parameter setting, or communication between the intelligent container and the server is abnormal, the edge calculation module arranged on the intelligent container can be used for automatic abnormality processing, and the intelligent container interaction control method based on machine vision comprises the following steps:
responding to a starting command from a server, controlling a hardware device of the intelligent container to perform hardware self-checking and obtaining a hardware self-checking result, wherein the hardware self-checking result comprises normal or abnormal;
when the hardware self-checking result is normal, controlling an image acquisition device of the intelligent container to acquire a first continuous image according to preset time;
when the first continuous image acquisition is successful, controlling the electronic lock of the intelligent container to execute unlocking operation;
when the first continuous image acquisition fails, acquiring an acquisition failure reason and sending the acquisition failure reason to the edge calculation module, wherein the edge calculation module formulates a corresponding image acquisition device processing strategy according to the acquisition failure reason;
controlling the intelligent container to execute unlocking operation after adjusting the image acquisition device according to the image acquisition device processing strategy;
Controlling the image acquisition device to start acquiring a second continuous image and transmitting the second continuous image to the edge calculation module in real time, wherein the edge calculation module is used for analyzing and processing the received second continuous image;
when the intelligent container receives a locking signal from the electronic lock within a preset maximum picking time, judging whether the connection between the intelligent container and the server is normal, if so, sending the acquired second continuous images to the server and driving the server to send order information to the intelligent terminal; if not, controlling the edge calculation module to generate order information according to the received second continuous images and sending the order information to the intelligent terminal;
when the intelligent container does not receive the locking signal within the preset maximum goods taking time, controlling the edge calculation module to judge whether a user finishes taking goods according to the second continuous image, if so, judging whether the connection between the intelligent container and the server is normal, and if so, controlling the intelligent container to send a settlement instruction to the server to drive the server to generate order information; and if the connection is abnormal, controlling the edge calculation module to generate order information according to the received second continuous images and sending the order information to the intelligent terminal.
Specifically, after the intelligent container is started in response to a starting command from a server and performs hardware self-inspection, the image acquisition device acquires a first continuous image according to preset time; if the image acquisition device does not have abnormal conditions, the first continuous image acquisition is successful, and the electronic lock of the intelligent container is controlled to execute unlocking operation; if the image acquisition device is improper in parameter setting or is blocked by part, the image acquisition device can not normally acquire images, and considers that the first continuous image acquisition fails, acquires acquisition failure reasons, and sends the acquisition failure reasons to the edge calculation module, wherein the edge calculation module formulates a corresponding image acquisition device processing strategy according to the acquisition failure reasons: for example, if the image acquisition device is maliciously shielded, starting the hidden other image acquisition device to acquire the image, or sending out a voice alarm to inform the user that the image acquisition device is not shielded; and if the parameter setting of the image acquisition device is improper, adjusting the parameter of the image acquisition device. The method comprises the steps that after an image acquisition device is adjusted to enable the image acquisition device to normally acquire images, unlocking operation is carried out, a user takes goods after a cabinet door is opened, the image acquisition device acquires the process of taking goods by the user, in order to ensure smooth transaction, the image acquisition device acquires second continuous images and simultaneously sends the images to an edge calculation module for real-time analysis of the images, when the intelligent container receives a locking signal from the electronic lock when the intelligent container is at a preset maximum goods taking time (the preset maximum goods taking time can be determined according to actual conditions, for example, the user can be informed that the transaction is generally completed within 10 minutes according to historical data analysis, the preset maximum goods taking time can be set to be 10 minutes), the user is informed that the connection of the intelligent container and the server is normal after the user takes goods and the cabinet door is closed, and if the connection is normal, the acquired second continuous images are sent to the server and the server is driven to send order information to the intelligent terminal; if the connection is abnormal due to network delay or malicious shielding of signals, the edge calculation module is controlled to generate order information according to the received second continuous images and send the order information to the intelligent terminal;
When the intelligent container does not receive the locking signal within the preset maximum goods taking time, the situation that the user possibly forgets to close the door or maliciously does not close the door is indicated, the edge calculation module is controlled to judge whether the user finishes taking goods according to the second continuous images, whether the user finishes taking goods or not is judged here, the edge calculation module can analyze and detect that the user takes goods and does not take other goods or the user leaves the intelligent container according to the second continuous images, if one of the conditions is that the user finishes taking goods, the user considers that the user finishes taking goods, at the moment, whether the intelligent container is normally connected with the server is judged, and if the intelligent container is normally connected, the intelligent container is controlled to send a settlement instruction to the server to drive the server to generate order information; and if the connection is abnormal, controlling the edge calculation module to generate order information according to the received second continuous images and sending the order information to the intelligent terminal. In this embodiment, when the image acquisition device is abnormal and/or communication connection with the server is abnormal during commodity transaction, the edge calculation module arranged at the intelligent container end is used for timely processing abnormal conditions, so that commodity transaction is smoothly performed, and user experience is improved.
Example two
The embodiment of the invention provides an intelligent container interaction control method based on machine vision, which is applied to a server, wherein the server is connected with an intelligent container in a wired or wireless mode, receives image data transmitted by the intelligent container and analyzes and processes the image data to assist the intelligent container to finish commodity identification and transaction. Referring to FIG. 9, the intelligent container interaction control method based on machine vision comprises the following steps:
s10, receiving a starting request from an intelligent terminal, and sending a starting command to an intelligent container, wherein the starting command drives the intelligent container to perform hardware self-inspection and acquire a first continuous image and a second continuous image, and the intelligent terminal comprises any one of a smart phone and an intelligent container man-machine interaction device;
s20, receiving a second continuous image from the intelligent container, wherein the second continuous image is an image continuously acquired from the unlocking start to the locking start of the electronic lock by the image acquisition device of the intelligent container;
s30, performing image recognition analysis processing on the second continuous images and generating order information;
and S40, sending the order information to the intelligent terminal.
Specifically, when a user has a demand of purchasing goods in the intelligent container, a two-dimensional code provided by the intelligent container can be scanned through a smart phone or a purchase key in a man-machine interaction device (such as a display touch screen) of the intelligent container is clicked to send a starting request to a server, the server receives the starting request from the intelligent terminal and sends a starting instruction to the intelligent container, after the intelligent container receives the starting command to start and completes self-checking of hardware, the intelligent container is controlled to acquire a first continuous image of preset time, after the first continuous image is successfully acquired, the intelligent container controls an electronic lock to unlock, the user can open a cabinet door and select goods, meanwhile, the image acquisition device is controlled to continuously acquire images of the process of selecting goods by the user to obtain a second continuous image, the acquired second continuous image is sent to the server after the user finishes selecting and closes the cabinet door to trigger the electronic lock to send a locking signal, the server analyzes and processes the acquired second continuous image, the type and quantity of goods selected by the user are identified, the amount of goods is determined according to the type and quantity of the order information, the user is pushed to the intelligent terminal to pay the user, and the goods transaction is completed according to the order information.
In summary, according to the machine vision-based intelligent container interaction control method provided by the embodiment of the invention, the intelligent container is controlled to perform hardware self-inspection and obtain a hardware self-inspection result by responding to the starting command from the server; when the hardware self-checking result is normal, controlling an image acquisition device of the intelligent container to acquire a first continuous image according to preset time; when the first continuous image acquisition is successful, controlling the electronic lock of the intelligent container to execute unlocking operation; controlling the image acquisition device to start acquiring a second continuous image until a locking signal from the electronic lock is received; and sending the acquired second continuous images to the server and driving the server to send order information to the intelligent terminal, so that the intelligent container interactive control and commodity transaction process are realized. The method ensures the normal operation and stability of the equipment through hardware self-checking, accurately identifies the commodity category and quantity and generates order information through real-time image acquisition, monitoring and image analysis to the server, so that the whole transaction process is more convenient and efficient, and good and convenient purchasing experience is provided for users.
Example III
Referring to fig. 10, an embodiment of the present invention provides an intelligent container interaction control device 200 based on machine vision, which is applied to an intelligent container end, the device 200 includes:
the self-checking module 201 is configured to respond to a start command from a server, control a hardware device of the intelligent container to perform hardware self-checking and obtain a hardware self-checking result, where the hardware self-checking result includes normal or abnormal;
the first acquisition module 202 is configured to control the image acquisition device of the intelligent container to acquire a first continuous image according to a preset time when the hardware self-checking result is normal;
the unlocking module 203 is configured to control the electronic lock of the intelligent container to perform an unlocking operation when the first continuous image acquisition is successful;
a second acquisition module 204, configured to control the image acquisition device to start acquiring a second continuous image until a locking signal from the electronic lock is received;
the order acquisition module 205 is applied to send the acquired second continuous images to the server and drive the server to send order information to the intelligent terminal.
Example IV
Referring to fig. 11, an embodiment of the present invention provides an intelligent container interaction control device 400 based on machine vision, and an application server, where the device 400 includes:
A start command sending module 401, configured to receive a start request from an intelligent terminal, and send a start command to an intelligent container, where the start command is to drive the intelligent container to perform hardware self-inspection, collect a first continuous image and a second continuous image, and the intelligent terminal includes any one of a smart phone and an intelligent container man-machine interaction device;
a second image receiving module 402, configured to receive a second continuous image from the intelligent container, where the second continuous image is an image continuously acquired by the image acquisition device of the intelligent container from unlocking to locking of an electronic lock;
the order generation module 403 is configured to perform image recognition analysis processing on the second continuous image and generate order information;
and the order sending module 404 is configured to send the order information to the intelligent terminal.
In summary, according to the intelligent container interaction control device based on machine vision provided by the embodiment of the invention, the intelligent container is controlled to perform hardware self-inspection and obtain a hardware self-inspection result by responding to the starting command from the server; when the hardware self-checking result is normal, controlling an image acquisition device of the intelligent container to acquire a first continuous image according to preset time; when the first continuous image acquisition is successful, controlling the electronic lock of the intelligent container to execute unlocking operation; controlling the image acquisition device to start acquiring a second continuous image until a locking signal from the electronic lock is received; and sending the acquired second continuous images to the server and driving the server to send order information to the intelligent terminal, so that the intelligent container interactive control and commodity transaction process are realized. The method ensures the normal operation and stability of the equipment through hardware self-checking, accurately identifies the commodity category and quantity and generates order information through real-time image acquisition, monitoring and image analysis to the server, so that the whole transaction process is more convenient and efficient, and good and convenient purchasing experience is provided for users.
Example five
In the above embodiment of the invention, an intelligent container system is provided, which comprises an intelligent container and a server, wherein the intelligent container and the server are connected in a wired or wireless manner,
the intelligent container comprises:
the machine vision-based intelligent container interaction control apparatus 200 applied to an intelligent container terminal according to the third embodiment comprises:
the self-checking module 201 is configured to control a hardware device of the intelligent container to perform hardware self-checking and obtain a hardware self-checking result in response to a start command from a server, where the hardware self-checking result includes normal or abnormal;
the first acquisition module 202 is configured to control the image acquisition device of the intelligent container to acquire a first continuous image according to a preset time when the hardware self-checking result is normal;
the unlocking module 203 is configured to control the electronic lock of the intelligent container to perform an unlocking operation when the first continuous image acquisition is successful;
a second acquisition module 204, configured to control the image acquisition device to start acquiring a second continuous image until a locking signal from the electronic lock is received;
the order acquisition module 205 is applied to send the acquired second continuous images to the server and drive the server to send order information to the intelligent terminal.
The server includes:
the intelligent container interaction control device 400 based on machine vision applied to a server side according to the fourth embodiment comprises:
a start command sending module 401, configured to receive a start request from an intelligent terminal, and send a start command to an intelligent container, where the start command is to drive the intelligent container to perform hardware self-inspection, collect a first continuous image and a second continuous image, and the intelligent terminal includes any one of a smart phone and an intelligent container man-machine interaction device;
a second image receiving module 402, configured to receive a second continuous image from the intelligent container, where the second continuous image is an image continuously acquired by the image acquisition device of the intelligent container from unlocking to locking of an electronic lock;
the order generation module 403 is configured to perform image recognition analysis processing on the second continuous image and generate order information;
and the order sending module 404 is configured to send the order information to the intelligent terminal. And sending the order information to the intelligent terminal.
Example six
In addition, the intelligent container interaction control method based on machine vision can be realized by an intelligent container main control system. FIG. 12 shows a hardware architecture diagram of an intelligent container master control system provided by an embodiment of the invention.
The intelligent container master control system may comprise a processor 301 and a memory 302 storing computer program instructions.
In particular, the processor 301 may include a Central Processing Unit (CPU), or an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), or may be configured as one or more integrated circuits that implement embodiments of the present invention.
Memory 302 may include mass storage for data or instructions. By way of example, and not limitation, memory 302 may comprise a Hard Disk Drive (HDD), floppy Disk Drive, flash memory, optical Disk, magneto-optical Disk, magnetic tape, or universal serial bus (Universal Serial Bus, USB) Drive, or a combination of two or more of the foregoing. Memory 302 may include removable or non-removable (or fixed) media, where appropriate. Memory 302 may be internal or external to the data processing apparatus, where appropriate. In a particular embodiment, the memory 302 is a non-volatile solid-state memory. In particular embodiments, memory 302 includes Read Only Memory (ROM). The ROM may be mask programmed ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically Erasable PROM (EEPROM), electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these, where appropriate.
The processor 301 reads and executes the computer program instructions stored in the memory 302 to implement any of the machine vision based intelligent container interaction control methods of the above embodiments.
In one example, the intelligent container master control system can also include a communication interface 303 and a bus 310. As shown in fig. 12, the processor 301, the memory 302, and the communication interface 303 are connected to each other by a bus 310 and perform communication with each other.
The communication interface 303 is mainly used to implement communication between each module, device, unit and/or apparatus in the embodiment of the present invention.
The bus 310 includes hardware, software, or both, that couples the components of the intelligent container master system to each other. By way of example, and not limitation, bus 310 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an infiniband interconnect, a Low Pin Count (LPC) bus, a memory bus, a micro channel architecture (MCa) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a video electronics standards association local (VLB) bus, or other suitable bus, or a combination of two or more of the above. Bus 310 may include one or more buses, where appropriate. Although embodiments of the invention have been described and illustrated with respect to a particular bus, the invention contemplates any suitable bus or interconnect.
It should be understood that the invention is not limited to the particular arrangements and instrumentality described above and shown in the drawings. For the sake of brevity, a detailed description of known methods is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method processes of the present invention are not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between steps, after appreciating the spirit of the present invention.
The functional blocks shown in the above-described structural block diagrams may be implemented in hardware, software, firmware, or a combination thereof. When implemented in hardware, it may be, for example, an electronic circuit, an Application Specific Integrated Circuit (ASIC), suitable firmware, a plug-in, a function card, or the like. When implemented in software, the elements of the invention are the programs or code segments used to perform the required tasks. The program or code segments may be stored in a machine readable medium or transmitted over transmission media or communication links by a data signal carried in a carrier wave. A "machine-readable medium" may include any medium that can store or transfer information. Examples of machine-readable media include electronic circuitry, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio Frequency (RF) links, and the like. The code segments may be downloaded via computer networks such as the internet, intranets, etc.
It should also be noted that the exemplary embodiments mentioned in this disclosure describe some methods or systems based on a series of steps or devices. However, the present invention is not limited to the order of the above-described steps, that is, the steps may be performed in the order mentioned in the embodiments, or may be performed in a different order from the order in the embodiments, or several steps may be performed simultaneously.
In the foregoing, only the specific embodiments of the present invention are described, and it will be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the systems, modules and units described above may refer to the corresponding processes in the foregoing method embodiments, which are not repeated herein. It should be understood that the scope of the present invention is not limited thereto, and any equivalent modifications or substitutions can be easily made by those skilled in the art within the technical scope of the present invention, and they should be included in the scope of the present invention.

Claims (8)

1. The intelligent container interaction control method based on machine vision is characterized by being applied to an intelligent container end and comprising the following steps of:
responding to a starting command from a server, controlling a hardware device of the intelligent container to carry out hardware self-checking and obtaining a hardware self-checking result, wherein the hardware self-checking result comprises normal or abnormal and comprises the following steps: responding to the starting command, and controlling a main control system of the intelligent container to perform system initialization; detecting and verifying the working state of a hardware device in the intelligent container, wherein the hardware device comprises at least one of an image acquisition device, an electronic lock and a network communication module; when the system is successfully initialized and the working states of the hardware devices are normal, the hardware self-checking result is normal, otherwise, the hardware self-checking result is abnormal; further comprises: transmitting process information generated in a hardware self-checking process to the server in real time, wherein the process information is input into a preset hardware self-checking prediction learning model of the server and generates hardware self-checking result prediction information; acquiring the hardware self-checking result prediction information and determining necessary self-checking items and unnecessary self-checking items according to the hardware self-checking result prediction information; executing the hardware self-checking operation of the intelligent container according to the necessary self-checking item;
When the hardware self-checking result is normal, controlling an image acquisition device of the intelligent container to acquire a first continuous image according to preset time;
when the first continuous image acquisition is successful, controlling the electronic lock of the intelligent container to execute unlocking operation;
controlling the image acquisition device to start acquiring a second continuous image until a locking signal from the electronic lock is received;
and sending the acquired second continuous images to the server and driving the server to send order information to the intelligent terminal.
2. The machine vision based intelligent container interaction control method of claim 1, wherein the controlling the image acquisition device to begin acquiring a second continuous image until a lockout signal from the electronic lock is received comprises:
extracting frame images including intelligent container cabinet door information in the second continuous images, and marking the frame images as cabinet door images;
generating cabinet door state information in real time according to the cabinet door image, wherein the cabinet door state information at least comprises one of cabinet door opening and cabinet door closing;
acquiring the cabinet door state information when the locking signal is received;
and if the cabinet door state information is that the cabinet door is closed, controlling the image acquisition device to stop acquisition of the second continuous images.
3. The machine vision-based intelligent container interaction control method according to claim 1, wherein the image acquisition device at least comprises a first image acquisition device and a second image acquisition device, wherein acquisition areas of the first image acquisition device and the second image acquisition device do not completely overlap, and when the hardware self-inspection result is normal, controlling the image acquisition device of the intelligent container to acquire a first continuous image according to a preset time comprises:
controlling the first image acquisition device and the second image acquisition device to acquire a first sub-continuous image and a second sub-continuous image according to preset time at the same time;
and stitching the first sub-continuous image and the second sub-continuous image to obtain the first continuous image.
4. The machine vision based intelligent container interaction control method of claim 3, wherein controlling the first image acquisition device to acquire a third sub-continuous image and controlling the second image acquisition device to acquire a fourth sub-continuous image, the second continuous image including the third sub-continuous image and the fourth sub-continuous image, the transmitting the acquired second continuous image to the server and driving the server to transmit order information to an intelligent terminal includes:
Acquiring a first key frame image recorded with preset key information in the third sub-continuous image, and performing data segmentation processing on the first key frame image to acquire a first preset key image;
acquiring a second key frame image recorded with preset key information in the fourth sub-continuous image, and performing data segmentation processing on the second key frame image to acquire a second preset key image;
respectively acquiring the data volume of the first preset key image and the data volume of the second preset key image, and recording the data volume as a third data volume and a fourth data volume;
determining a third transmission channel corresponding to the first preset key image and a fourth transmission channel corresponding to the second preset key image according to the third data amount and the fourth data amount respectively;
and sending the first preset key image and the second preset key image to the server through the third transmission channel and the fourth transmission channel respectively, driving the server to process and analyze data according to the first preset key image and the second preset key image, and then sending order information to the intelligent terminal.
5. An intelligent container interaction control method based on machine vision, which is applied to a server, is characterized by comprising the following steps:
Receiving a starting request from an intelligent terminal, and sending a starting command to an intelligent container, wherein the starting command drives the intelligent container to perform hardware self-inspection, acquire a first continuous image and a second continuous image, the intelligent terminal comprises any one of a smart phone and an intelligent container man-machine interaction device, and the process of performing hardware self-inspection by the intelligent container comprises the following steps: responding to the starting command, and controlling a main control system of the intelligent container to perform system initialization; detecting and verifying the working state of a hardware device in the intelligent container, wherein the hardware device comprises at least one of an image acquisition device, an electronic lock and a network communication module; when the system is successfully initialized and the working states of the hardware devices are normal, the hardware self-checking result is normal, otherwise, the hardware self-checking result is abnormal; further comprises: transmitting process information generated in a hardware self-checking process to the server in real time, wherein the process information is input into a preset hardware self-checking prediction learning model of the server and generates hardware self-checking result prediction information; acquiring the hardware self-checking result prediction information and determining necessary self-checking items and unnecessary self-checking items according to the hardware self-checking result prediction information; executing the hardware self-checking operation of the intelligent container according to the necessary self-checking item;
Receiving a second continuous image from the intelligent container, wherein the second continuous image is an image continuously acquired by the image acquisition device of the intelligent container from unlocking to locking of an electronic lock;
performing image recognition analysis processing on the second continuous images and generating order information;
and sending the order information to the intelligent terminal.
6. An intelligent counter interaction control device based on machine vision is applied to intelligent counter end, characterized in that, the device includes:
the self-checking module is used for responding to a starting command from the server, controlling the hardware device of the intelligent container to carry out hardware self-checking and obtaining a hardware self-checking result, wherein the hardware self-checking result comprises normal or abnormal and comprises the following steps: responding to the starting command, and controlling a main control system of the intelligent container to perform system initialization; detecting and verifying the working state of a hardware device in the intelligent container, wherein the hardware device comprises at least one of an image acquisition device, an electronic lock and a network communication module; when the system is successfully initialized and the working states of the hardware devices are normal, the hardware self-checking result is normal, otherwise, the hardware self-checking result is abnormal; further comprises: transmitting process information generated in a hardware self-checking process to the server in real time, wherein the process information is input into a preset hardware self-checking prediction learning model of the server and generates hardware self-checking result prediction information; acquiring the hardware self-checking result prediction information and determining necessary self-checking items and unnecessary self-checking items according to the hardware self-checking result prediction information; executing the hardware self-checking operation of the intelligent container according to the necessary self-checking item;
The first acquisition module is used for controlling the image acquisition device of the intelligent container to acquire a first continuous image according to preset time when the hardware self-checking result is normal;
the unlocking module is used for controlling the electronic lock of the intelligent container to execute unlocking operation when the first continuous image acquisition is successful;
the second acquisition module is used for controlling the image acquisition device to start to acquire a second continuous image until a locking signal from the electronic lock is received;
the order acquisition module is used for sending the acquired second continuous images to the server and driving the server to send order information to the intelligent terminal.
7. An intelligent counter interaction control device based on machine vision, which is applied to a server side, is characterized in that the device comprises:
the starting command sending module is used for receiving a starting request from the intelligent terminal and sending a starting command to the intelligent container, wherein the starting command drives the intelligent container to perform hardware self-inspection, collect a first continuous image and a second continuous image, the intelligent terminal comprises any one of a smart mobile phone and an intelligent container man-machine interaction device, and the process of performing hardware self-inspection by the intelligent container comprises the following steps: responding to the starting command, and controlling a main control system of the intelligent container to perform system initialization; detecting and verifying the working state of a hardware device in the intelligent container, wherein the hardware device comprises at least one of an image acquisition device, an electronic lock and a network communication module; when the system is successfully initialized and the working states of the hardware devices are normal, the hardware self-checking result is normal, otherwise, the hardware self-checking result is abnormal; further comprises: transmitting process information generated in a hardware self-checking process to the server in real time, wherein the process information is input into a preset hardware self-checking prediction learning model of the server and generates hardware self-checking result prediction information; acquiring the hardware self-checking result prediction information and determining necessary self-checking items and unnecessary self-checking items according to the hardware self-checking result prediction information; executing the hardware self-checking operation of the intelligent container according to the necessary self-checking item;
The second image receiving module is used for receiving a second continuous image from the intelligent container, wherein the second continuous image is an image continuously acquired from the unlocking start to the locking start of the electronic lock by the image acquisition device of the intelligent container;
the order generation module is used for carrying out image recognition analysis processing on the second continuous images and generating order information;
and the order sending module is used for sending the order information to the intelligent terminal.
8. An intelligent container system, characterized in that the system comprises an intelligent container and a server, wherein the intelligent container and the server are connected in a wired or wireless way,
the intelligent container comprises:
the intelligent machine vision-based container interaction control device of claim 6, comprising:
the self-checking module is used for responding to a starting command from the server, controlling the hardware device of the intelligent container to carry out hardware self-checking and obtaining a hardware self-checking result, wherein the hardware self-checking result comprises normal or abnormal and comprises the following steps: responding to the starting command, and controlling a main control system of the intelligent container to perform system initialization; detecting and verifying the working state of a hardware device in the intelligent container, wherein the hardware device comprises at least one of an image acquisition device, an electronic lock and a network communication module; when the system is successfully initialized and the working states of the hardware devices are normal, the hardware self-checking result is normal, otherwise, the hardware self-checking result is abnormal; further comprises: transmitting process information generated in a hardware self-checking process to the server in real time, wherein the process information is input into a preset hardware self-checking prediction learning model of the server and generates hardware self-checking result prediction information; acquiring the hardware self-checking result prediction information and determining necessary self-checking items and unnecessary self-checking items according to the hardware self-checking result prediction information; executing the hardware self-checking operation of the intelligent container according to the necessary self-checking item;
The first acquisition module is used for controlling an image acquisition device of the intelligent container to acquire a first continuous image according to preset time when the hardware self-checking result is normal;
the unlocking module is used for controlling the electronic lock of the intelligent container to execute unlocking operation when the first continuous image acquisition is successful;
the second acquisition module is used for controlling the image acquisition device to start to acquire a second continuous image until a locking signal from the electronic lock is received;
the order acquisition module is used for sending the acquired second continuous images to the server and driving the server to send order information to the intelligent terminal;
the server includes:
the intelligent machine vision-based container interaction control device of claim 7, comprising:
the starting command sending module is used for receiving a starting request from the intelligent terminal and sending a starting command to the intelligent container, wherein the starting command drives the intelligent container to perform hardware self-inspection and acquire a first continuous image and a second continuous image, and the intelligent terminal comprises any one of a smart phone and an intelligent container man-machine interaction device;
the second image receiving module is used for receiving a second continuous image from the intelligent container, wherein the second continuous image is an image continuously acquired from the unlocking start to the locking start of the electronic lock by the image acquisition device of the intelligent container;
The order generation module is used for carrying out image recognition analysis processing on the second continuous images and generating order information;
and the order sending module is used for sending the order information to the intelligent terminal.
CN202311483613.0A 2023-11-09 2023-11-09 Intelligent container interaction control method, device and system based on machine vision Active CN117218762B (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN202311483613.0A CN117218762B (en) 2023-11-09 2023-11-09 Intelligent container interaction control method, device and system based on machine vision
CN202410339580.0A CN118247884A (en) 2023-11-09 2023-11-09 Intelligent container, and interactive control method and system of intelligent container

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311483613.0A CN117218762B (en) 2023-11-09 2023-11-09 Intelligent container interaction control method, device and system based on machine vision

Related Child Applications (1)

Application Number Title Priority Date Filing Date
CN202410339580.0A Division CN118247884A (en) 2023-11-09 2023-11-09 Intelligent container, and interactive control method and system of intelligent container

Publications (2)

Publication Number Publication Date
CN117218762A CN117218762A (en) 2023-12-12
CN117218762B true CN117218762B (en) 2024-02-09

Family

ID=89039365

Family Applications (2)

Application Number Title Priority Date Filing Date
CN202311483613.0A Active CN117218762B (en) 2023-11-09 2023-11-09 Intelligent container interaction control method, device and system based on machine vision
CN202410339580.0A Pending CN118247884A (en) 2023-11-09 2023-11-09 Intelligent container, and interactive control method and system of intelligent container

Family Applications After (1)

Application Number Title Priority Date Filing Date
CN202410339580.0A Pending CN118247884A (en) 2023-11-09 2023-11-09 Intelligent container, and interactive control method and system of intelligent container

Country Status (1)

Country Link
CN (2) CN117218762B (en)

Citations (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN200986723Y (en) * 2005-11-15 2007-12-05 北京兆维科技股份有限公司 Photo-taking card vendor
CN102547364A (en) * 2010-12-30 2012-07-04 乐金电子(中国)研究开发中心有限公司 Digital television after-sale service system and digital television receiver
CN202632412U (en) * 2012-05-24 2012-12-26 浙江贝克曼智能商务有限公司 Self-service vending equipment based on personal computer (PC)
CN107895430A (en) * 2017-12-06 2018-04-10 湖南农博众商电子商务有限责任公司 A kind of new retail terminal system of intelligence and its vending method
CN108460908A (en) * 2018-03-02 2018-08-28 北京京东尚科信息技术有限公司 Automatic vending method and system and automatic vending device and automatic vending machine
CN108831013A (en) * 2018-05-22 2018-11-16 深圳云行智能科技有限公司 Self-service method, apparatus, system and computer equipment
CN108921048A (en) * 2018-06-14 2018-11-30 深圳码隆科技有限公司 A kind of shopping settlement method, device and user terminal
CN109783295A (en) * 2017-11-10 2019-05-21 中国电子科技集团公司第四十一研究所 Test macro self-test method for producing software and device based on configuration
CN110555939A (en) * 2018-05-30 2019-12-10 展讯通信（上海）有限公司 Automatic vending machine
CN113053024A (en) * 2021-03-17 2021-06-29 广东便捷神科技股份有限公司 Control method of unmanned retail terminal
CN215932724U (en) * 2021-05-17 2022-03-01 厦门盈趣汽车电子有限公司 Display screen self-checking system and unmanned selling device
CN115081714A (en) * 2022-06-24 2022-09-20 天津凯发电气股份有限公司 One-key switching station self-checking optimization method for urban rail transit
CN116483460A (en) * 2023-02-15 2023-07-25 北京五八信息技术有限公司 Application starting method and device, electronic equipment and storage medium

Patent Citations (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN200986723Y (en) * 2005-11-15 2007-12-05 北京兆维科技股份有限公司 Photo-taking card vendor
CN102547364A (en) * 2010-12-30 2012-07-04 乐金电子(中国)研究开发中心有限公司 Digital television after-sale service system and digital television receiver
CN202632412U (en) * 2012-05-24 2012-12-26 浙江贝克曼智能商务有限公司 Self-service vending equipment based on personal computer (PC)
CN109783295A (en) * 2017-11-10 2019-05-21 中国电子科技集团公司第四十一研究所 Test macro self-test method for producing software and device based on configuration
CN107895430A (en) * 2017-12-06 2018-04-10 湖南农博众商电子商务有限责任公司 A kind of new retail terminal system of intelligence and its vending method
CN108460908A (en) * 2018-03-02 2018-08-28 北京京东尚科信息技术有限公司 Automatic vending method and system and automatic vending device and automatic vending machine
CN108831013A (en) * 2018-05-22 2018-11-16 深圳云行智能科技有限公司 Self-service method, apparatus, system and computer equipment
CN110555939A (en) * 2018-05-30 2019-12-10 展讯通信（上海）有限公司 Automatic vending machine
CN108921048A (en) * 2018-06-14 2018-11-30 深圳码隆科技有限公司 A kind of shopping settlement method, device and user terminal
CN113053024A (en) * 2021-03-17 2021-06-29 广东便捷神科技股份有限公司 Control method of unmanned retail terminal
CN215932724U (en) * 2021-05-17 2022-03-01 厦门盈趣汽车电子有限公司 Display screen self-checking system and unmanned selling device
CN115081714A (en) * 2022-06-24 2022-09-20 天津凯发电气股份有限公司 One-key switching station self-checking optimization method for urban rail transit
CN116483460A (en) * 2023-02-15 2023-07-25 北京五八信息技术有限公司 Application starting method and device, electronic equipment and storage medium

Also Published As

Publication number Publication date
CN117218762A (en) 2023-12-12
CN118247884A (en) 2024-06-25

Similar Documents

Publication Publication Date Title
CN110009836A (en) The system and method for deep learning based on EO-1 hyperion photography technology
CN206757798U (en) Automatic vending machine
EP3767598A1 (en) Vending device and method
CN106781014A (en) Automatic vending machine and its operation method
CN107146325A (en) Automatic vending machine and its control method, control device and system
CN201255897Y (en) Human flow monitoring device for bus
JP5789170B2 (en) Parking lot management system
WO2020107951A1 (en) Image-based product checkout method and apparatus, medium, and electronic device
CN113114895A (en) Control method of container, container and readable storage medium
EP3901841B1 (en) Settlement method, apparatus, and system
CN109887161A (en) A kind of self-service cabinet theft preventing method, system and the self-service cabinet with anti-theft function
CN110136326A (en) A kind of sales counter control method and device
US20230005348A1 (en) Fraud detection system and method
CN111222870A (en) Settlement method, device and system
CN111178860A (en) Settlement method, device, equipment and storage medium for unmanned convenience store
WO2021195523A1 (en) Systems and methods for detecting a mis-scan of an item for purchase
CN113763136B (en) Intelligent order generation method for video segmentation processing based on weight change of commodity area
CN117218762B (en) Intelligent container interaction control method, device and system based on machine vision
CN112991379A (en) Unmanned vending method and system based on dynamic vision
RU2714901C1 (en) Method and system for pattern recognition from a video stream
CN113723384B (en) Intelligent order generation method based on fusion after multi-view image acquisition and intelligent vending machine
CN110443946A (en) Vending machine, the recognition methods of type of goods and device
CN115170999A (en) Intelligent order generation method for carrying out image analysis based on commodity weight combination
CN113723383A (en) Order generation method for synchronously identifying commodities in same region at different visual angles and intelligent vending machine
CN113727029B (en) Intelligent order generation method for combining collected images at multiple visual angles and intelligent vending machine

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant