WO2019233098A1 - 一种商品购买判定方法、装置和用户终端 - Google Patents

一种商品购买判定方法、装置和用户终端 Download PDF

Info

Publication number
WO2019233098A1
WO2019233098A1 PCT/CN2019/070034 CN2019070034W WO2019233098A1 WO 2019233098 A1 WO2019233098 A1 WO 2019233098A1 CN 2019070034 W CN2019070034 W CN 2019070034W WO 2019233098 A1 WO2019233098 A1 WO 2019233098A1
Authority
WO
WIPO (PCT)
Prior art keywords
shopping
user
gesture
product
image
Prior art date
Application number
PCT/CN2019/070034
Other languages
English (en)
French (fr)
Inventor
黄鼎隆
斯科特•马修•罗伯特
马咪娜
王海涵
Original Assignee
深圳码隆科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳码隆科技有限公司 filed Critical 深圳码隆科技有限公司
Publication of WO2019233098A1 publication Critical patent/WO2019233098A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/06—Buying, selling or leasing transactions
    • G06Q30/0601—Electronic shopping [e-shopping]
    • G06Q30/0641—Shopping interfaces
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/107—Static hand or arm
    • G06V40/113—Recognition of static hand signs
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language

Definitions

  • the present application relates to the field of image recognition technology, and more particularly, to a method, a device, and a user terminal for determining a product purchase.
  • Material circulation is the most basic element of human society.
  • the retail industry or an open related industry that sells directly to consumers hereinafter referred to as the retail industry or retail enterprises
  • the retail industry or retail enterprises as an important means of current material circulation needs to employ a large number of cashiers to realize the purchase and sale of goods.
  • the shopping items are counted one by one. After the statistics are completed, the payment is completed with the consumer.
  • the automatic vending container solves the shortcomings of manual statistics and manual settlement in the retail industry. It is a consumer-oriented unmanned sales and automatic settlement sales method. Through the selection of the user, the types and total value of the products selected by the user can be counted for the user to settle and complete the payment process. However, the current vending container can only perform automatic settlement based on the user's selection, and cannot accurately identify the user's shopping behavior and obtained in an open retail shopping environment, thereby achieving settlement.
  • the present application provides a method, a device and a user terminal for determining the purchase of a product to solve the shortcomings of the prior art.
  • this application provides a method for determining the purchase of a product, including:
  • the shopping start instruction based on the timestamp, image acquisition of the user's shopping gesture is started, and the image acquisition of the user's shopping gesture is stopped when receiving the shopping termination instruction returned by the infrared signal triggered by the user's leaving gesture To obtain a continuous shooting image with a time stamp between the time when the shopping start instruction and the shopping termination instruction are received;
  • image recognition is performed on the continuous shooting images with timestamps to determine the user's product shopping data, and the products are settled according to the product shopping data;
  • the product shopping data includes the products retrieved by the user Variety, take-out time, and quantity of merchandise corresponding to the merchandise variety.
  • the "based on neural network learning to perform image recognition on the time-stamped continuous shooting image to determine user's product shopping data” includes:
  • image recognition is performed on the gesture image of each frame to determine the commodity shopping data of the user.
  • the image recognition of the gesture image of each frame based on neural network learning to determine the user's product shopping data includes:
  • the "based on the neural network learning to perform gesture feature localization on gestures in the gesture image of each frame to obtain target gesture feature trajectory data” includes:
  • the minimum screenshot of the gesture image of each frame is synthesized into a characteristic motion trajectory according to the sequence of timestamps, and target gesture characteristic trajectory data is generated based on the characteristic motion trajectories and corresponding timestamps.
  • the "recognizing the target gesture characteristic trajectory data to determine the product variety taken out by the user in the target gesture characteristic trajectory data” includes:
  • the key frames of each of the items are identified to determine a commodity variety taken out by the user in the target gesture characteristic trajectory data.
  • the "recognizing the key frame of each of said items to determine the product variety taken out by said user in said target gesture feature trajectory data" includes:
  • a key frame recognition result corresponding to the key frame of the item is calculated, and the target gesture feature trajectory data extracted by the user is determined according to the key frame recognition result.
  • Commodity variety
  • the present application also provides a product purchase determination device, including: a receiving module, an acquisition module, and an identification module;
  • the receiving module is configured to receive a shopping start instruction returned by an infrared signal triggered by a user's shopping gesture
  • the acquisition module is configured to start image acquisition of the user's shopping gesture based on the shopping start instruction based on the time stamp, and stop receiving the shopping termination instruction returned by the infrared signal triggered by the user's leaving gesture. Said image acquisition of a user's shopping gesture, to obtain a continuous shooting image with a time stamp between the time when the shopping start instruction and the shopping termination instruction are received;
  • the recognition module is configured to perform image recognition on the continuous-shot image with time stamps based on the learning of the neural network to determine the user's product shopping data, and settle the product according to the product shopping data; the product
  • the shopping data includes the types of goods taken out by the user, the time of taking out, and the number of goods corresponding to the types of goods.
  • the present application further provides a user terminal including a memory and a processor, where the memory is configured to store a commodity purchase determination program, and the processor runs the commodity purchase determination program to enable the user terminal
  • the product purchase determination method described above is executed.
  • the present application also provides a computer-readable storage medium, where the computer-readable storage medium stores a commodity purchase determination program, and the commodity purchase determination program is implemented as described above when executed by a processor.
  • Product purchase judgment method
  • the present invention provides a method, a device and a user terminal for judging a commodity purchase.
  • the method provided in this application triggers an infrared signal through a shopping gesture to obtain a shopping start instruction, and returns a shopping termination instruction when the infrared signal triggered by the user leaves the gesture, and the time between receiving the shopping startup instruction and the shopping termination instruction.
  • the segment performs image collection of the user's shopping gestures and recognizes the continuous shooting image to finally determine the user's purchase behavior and the variety, quantity, and shopping time of the purchased items, so as to perform settlement.
  • image recognition technology is used to judge the user's shopping behavior, and the intelligent identification of the variety and quantity of the purchased product is realized.
  • the user's purchase behavior of the product and the identification and judgment of the purchased product reduce labor costs, greatly shorten the settlement time, high settlement efficiency, simple shopping process, and improved user experience.
  • FIG. 1 is a schematic structural diagram of a hardware operating environment involved in a solution of an embodiment of a method for judging a product purchase of this application;
  • FIG. 2 is a schematic flowchart of a first embodiment of a method for determining a purchase of a product in this application;
  • FIG. 3 is a schematic flowchart of a second embodiment of a method for judging a product purchase in this application
  • FIG. 4 is a schematic flowchart of a third embodiment of a method for determining a purchase of a product in this application;
  • FIG. 5 is a schematic flowchart of a fourth embodiment of a method for determining a purchase of a commodity in this application
  • FIG. 6 is a schematic flowchart of a fifth embodiment of a method for determining a purchase of a commodity in this application;
  • FIG. 7 is a schematic flowchart of a sixth embodiment of a method for determining a purchase of a commodity in this application.
  • FIG. 8 is a schematic diagram of functional modules of a product purchase determination device of the present application.
  • first and second are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as “first” and “second” may explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "a plurality” is two or more, unless it is specifically and specifically defined otherwise.
  • the terms “installation,” “connected,” “connected,” and “fixed” should be understood broadly unless otherwise specified and limited, for example, they may be fixed connections or removable connections Or integrated; it can be mechanical or electrical; it can be directly connected or indirectly connected through an intermediate medium; it can be the internal connection of two elements or the interaction between two elements.
  • installation should be understood broadly unless otherwise specified and limited, for example, they may be fixed connections or removable connections Or integrated; it can be mechanical or electrical; it can be directly connected or indirectly connected through an intermediate medium; it can be the internal connection of two elements or the interaction between two elements.
  • FIG. 1 is a schematic structural diagram of a hardware operating environment of a terminal involved in a solution according to an embodiment of the present application.
  • the terminal may be a PC provided in an automatic container machine, or may be a mobile terminal device such as a smart phone, a tablet computer, an e-book reader, an MP3 player, an MP4 player, and a portable computer.
  • a mobile terminal device such as a smart phone, a tablet computer, an e-book reader, an MP3 player, an MP4 player, and a portable computer.
  • it may be a computer hardware device provided in the automatic container machine itself.
  • the terminal may include a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002.
  • the communication bus 1002 is used to implement connection and communication between these components.
  • the user interface 1003 may include a display screen, an input unit such as a keyboard, a remote controller, and the optional user interface 1003 may further include a standard wired interface and a wireless interface.
  • the network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface).
  • the memory 1005 may be a high-speed RAM memory or a stable memory, such as a magnetic disk memory.
  • the memory 1005 may optionally be a storage device independent of the foregoing processor 1001.
  • the terminal includes an image acquisition device, which may specifically be a camera, a camera, and the like.
  • the terminal also includes an infrared sensing device to judge a user's shopping behavior.
  • the terminal may further include an RF (Radio Frequency) circuit, a sensor, an audio circuit, a WiFi module, and the like.
  • the mobile terminal may be configured with other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, and the like, which will not be repeated here.
  • the terminal shown in FIG. 1 does not constitute a limitation on the terminal, and may include more or fewer components than those shown in the figure, or combine some components, or arrange different components.
  • the memory 1005 as a computer-readable storage medium may include an operating system, a data interface control program, a network connection program, and a commodity purchase determination program.
  • the present invention provides a method, a device and a user terminal for judging a commodity purchase.
  • the method realizes the judgment of the user's shopping behavior and the intelligent identification of the variety and quantity of the purchased product through image recognition technology in an open shopping environment, thereby realizing the purchase behavior of the user's product.
  • the identification and judgment of the purchased products reduces labor costs, greatly shortens the settlement time, high settlement efficiency, simple shopping process and improved user experience.
  • a first embodiment of the present application provides a method for determining a purchase of a product, including:
  • Step S10000 receiving a shopping start instruction returned by an infrared signal triggered by a user's shopping gesture
  • the method for judging the purchase of a product provided in this embodiment can be applied to an open shopping environment, such as a shopping mall, a supermarket, or other places, that is, unattended, and users can take goods freely.
  • it may be an open-type vending machine. After the user starts shopping, the door of the vending machine is opened, and the user directly takes the internal goods, and the vending machine uses the image acquisition device to make shopping for the user. Identification and statistics of behaviors and purchased goods, to achieve the final settlement of the purchased goods of users.
  • the user's shopping gesture is the action of the user to take the target product.
  • This action can include two processes, that is, the process of selecting the grab or place process and the process of retrieving or handing out the goods. Through these two processes, For the pickup of the target product.
  • the user can move to the target product empty-handed, grab the goods, and retrieve the target product.
  • the above process may also be directed to a process in which a user holds a commodity to a target location, places it at the target location, and exits empty-handed.
  • the infrared signal is the infrared signal sent by the infrared sensing device in this embodiment.
  • the infrared signal is arranged so that the user can trigger the corresponding infrared sensor location when a shopping gesture occurs.
  • the infrared signal is sent and a shopping start command is returned.
  • the shopping start instruction can start a targeted instruction according to the corresponding position of the corresponding infrared sensing device, that is, setting different infrared sensing devices in different areas, and for users to grab products in different areas,
  • the infrared signals of different areas are triggered to return the shopping start instruction corresponding to the corresponding area.
  • the infrared signal of the infrared sensing device on the lower level is triggered, and a shopping corresponding to the lower level space is returned correspondingly. Start instruction.
  • Step S20000 According to the shopping start instruction, based on the timestamp, image acquisition of the user's shopping gesture is started, and when the shopping termination instruction returned by the infrared signal triggered by the user ’s leaving gesture is received, the user stops the shopping gesture. Image acquisition to obtain a continuous shooting image with a time stamp between the time when the shopping start instruction and the shopping termination instruction are received;
  • the image acquisition device may be a device with an image capturing function such as a video camera or a camera.
  • the image acquisition can be to set different image acquisition devices in different shopping areas, and start image acquisition devices in different areas according to the shopping start instructions corresponding to the different areas, so as to facilitate the collection and identification of corresponding shopping behaviors.
  • the infrared signal of the infrared sensing device on the lower level is triggered, and a shopping corresponding to the lower space is returned accordingly.
  • Start the instruction At this time, a plurality of cameras at different angles set at corresponding positions in the lower space begin to work, and start to collect images of the user's shopping behavior.
  • the timestamp is a time label corresponding to the image captured by the image acquisition device.
  • Each frame of the captured image corresponds to a timestamp.
  • the captured continuous-shot images with timestamps form a user gesture. Motion track.
  • the continuous shooting image refers to the continuous tracking of user gestures during the user's shopping process, and continuous shooting of the user's shopping gesture images, to determine the variety of products picked up and dropped by the user during the entire process, that is, obtained through continuous shooting
  • the image identifies the variety of the product purchased by the user, and performs settlement for the removed product.
  • continuous shooting images can be image video data, that is, video images.
  • it may also be a dynamic image of a user ’s shopping gesture that can record the entire shopping process.
  • Step S30000 based on the learning of the neural network, image recognition is performed on the continuous-shot image with time stamp to determine the user's product shopping data, and the product is settled according to the product shopping data; the product shopping data includes the user The commodity type to be taken out, the time to take out, and the number of commodities corresponding to the commodity type.
  • neural network learning which is artificial neural network (Artificial Neural Network, ANN)
  • ANN Artificial Neural Network
  • a neural network is a computing model that consists of a large number of nodes (or neurons) connected to each other. Each node represents a specific output function, called an activation function.
  • the connection between each two nodes represents a weighted value for the signal passing through the connection, called the weight, which is equivalent to the memory of an artificial neural network.
  • the output of the network varies depending on how the network is connected, the weight value and the excitation function.
  • the network itself is usually an approximation of an algorithm or function in nature, or it may be an expression of a logical strategy.
  • the system can obtain the ability to recognize the shopping gestures in the image and the products corresponding to the shopping gestures.
  • the method provided by this embodiment triggers an infrared signal through a shopping gesture to obtain a shopping start instruction, and returns a shopping termination instruction when the user leaves the infrared signal triggered by the gesture, and the time period between receiving the shopping startup instruction and the shopping termination instruction Image collection of the user's shopping gesture is performed, and the continuous shooting image is recognized to finally determine the user's purchase behavior and the variety, quantity, and shopping time of the purchased item, so as to perform settlement.
  • image recognition technology is used to judge the user's shopping behavior, and the intelligent identification of the variety and quantity of the purchased product is realized.
  • the user's purchase behavior of the product and the identification and judgment of the purchased product reduce labor costs, greatly shorten the settlement time, high settlement efficiency, simple shopping process, and improved user experience.
  • a second embodiment of the present application provides a method for determining a purchase of a product.
  • the step S30000 “based on the learning of a neural network, the continuous shooting with time stamp Image recognition to determine user's product shopping data "includes:
  • Step S31000 converting the continuous shooting images with timestamps into gesture images of consecutive frames according to the timestamp order
  • the continuous shooting image may be moving image video data, and the continuous shooting image is converted into several frames of gesture images.
  • the foregoing frames may be set according to specific image recognition capabilities, for example, 10 frames.
  • the number of frames of the gesture image converted by the continuous shooting image can be set according to the speed of the user's shopping gesture. If the speed of the user's shopping gesture is high, the number of frames is large; if the speed is slow, the number of frames is small. In addition, it may also be a fixed value. For example, in this embodiment, several frames of image conversion for each shopping gesture of the user may be set to 10-15 frames.
  • continuous shooting is required for each shopping gesture of the user.
  • a large number of pictures of shopping gestures need to be taken and saved, occupying a certain amount of storage space and system resources.
  • the pre-processing of the time-lapsed images with time stamps can be specifically performed as follows:
  • the method may further include:
  • Unified resolution adjustment is performed on the gesture image after the image is intercepted to obtain the gesture image that is compatible with image recognition and has a uniform size.
  • image optimization processing is performed on the acquired gesture image.
  • image brightness, contrast, and color saturation processing are performed, that is, to ensure image quality while ensuring image quality.
  • the gesture image is intercepted to remove unnecessary parts except the area occupied by the shopping gesture and the product, and only the smallest image of the area occupied by the shopping gesture and the product in the continuous shooting image in each frame is intercepted.
  • the resolution of the continuous shooting image is adjusted so that the gesture image is adapted to the minimum resolution of image recognition, and all the captured images are unified in size, thereby achieving the effect of unification and reducing the memory or capacity occupied by the image.
  • Step S32000 based on the neural network learning, image recognition is performed on the gesture image of each frame to determine the commodity shopping data of the user.
  • image recognition is performed on the gesture image of each frame to determine the user's product shopping data, to determine what information was taken out during the user's shopping process, how much quantity was taken out, and so on.
  • a third embodiment of the present application provides a method for determining a purchase of a product.
  • the step S32000 “based on the neural network learning, performs the gesture image of each frame Image recognition to determine the product shopping data for the user "includes:
  • Step S32100 based on the neural network learning, performing gesture feature positioning on the gesture in the gesture image of each frame to obtain target gesture feature trajectory data;
  • the gesture feature is the feature information of the user ’s shopping gesture in the image, including the image feature of the user ’s hand and the image feature of the product grasped by the user ’s hand, where the image feature of the hand may be the gesture feature , Palm position characteristics, finger shape characteristics, etc., the image characteristics of the product may include the shape, size and other information of the product.
  • the gesture image contains gesture feature trajectory data.
  • the gesture features contained in the image can be located to determine whether there is a user's gesture in the gesture image and whether the user's hand is grasped. commodity.
  • Step S32200 identifying the target gesture characteristic trajectory data to determine a commodity variety taken out by the user in the target gesture characteristic trajectory data
  • step S32300 the product types retrieved by the user in the target gesture feature trajectory data are counted to generate product shopping data.
  • the gesture features in the gesture image of each frame are located to find the gesture features in the gesture image, thereby obtaining the target gesture feature trajectory data.
  • the target gesture characteristic trajectory data is identified, and the product variety corresponding to the product taken out by the user in the target gesture characteristic trajectory data is obtained through image recognition, and statistics are generated to generate product shopping data.
  • a fourth embodiment of the present application provides a method for determining a purchase of a product.
  • the step S32100 “based on neural network learning, for each gesture image in each frame, Gesture gesture positioning to obtain target gesture feature trajectory data "includes:
  • Step S32110 based on the neural network learning, perform gesture feature positioning on the gesture in the gesture image of each frame, determine a feature area frame of the gesture feature, and intercept a minimum including the gesture feature according to the feature area frame.
  • step S32120 the minimum screenshot of the gesture image of each frame is synthesized into a characteristic motion trajectory according to the sequence of timestamps, and target gesture characteristic trajectory data is generated based on the characteristic motion trajectory and a corresponding time stamp.
  • the feature area frame is a frame of the feature area of the origin or the center of the area that includes the user's gesture when performing image recognition, and a screenshot is performed according to the feature area frame.
  • the feature area frame is the smallest screenshot including the gesture feature.
  • the shape of the characteristic area frame may be rectangular, square, or any other shape.
  • the minimum screenshot of the gesture image of each frame when the user is shopping is synthesized into a characteristic motion trajectory, and the target gesture characteristic trajectory data with a time stamp is generated according to the characteristic motion trajectory and a time stamp.
  • the minimum screenshot containing gesture features is obtained, and the minimum screenshots corresponding to all frames are synthesized into the feature motion trajectory, which greatly reduces the system resources occupied by the image during the image retransmission recognition process.
  • the minimum screenshot contains There is necessary gesture feature information that needs to be identified, and irrelevant image information is deleted, the capacity of the storage space occupied by the image is reduced, and the efficiency of image recognition can be improved to a certain extent.
  • a fifth embodiment of the present application provides a method for determining a purchase of a product.
  • the step S32200 "recognizes the target gesture feature trajectory data to determine the target gesture feature trajectory data.
  • the commodity varieties taken out by the user appearing in the target gesture feature trajectory data include:
  • Step S32210 extracting a feature image of a user's initial shopping state in the target gesture feature trajectory data, and using the feature image as an initial feature template;
  • the initial state of the user's shopping is the state when the user's gesture is empty-handed and the product is grabbed. There is no product in the user's gesture and the product is not in contact with the product.
  • the initial feature template of the user's shopping is used as the initial feature template of the reference feature to further determine whether there is a product in the user's hand.
  • step S32220 the minimum screenshot of each frame in the target gesture feature trajectory data is compared with the initial feature template, and a key frame of an item containing the commodity in the minimum screenshot of each frame is determined;
  • step S32230 the key frames of each of the items are identified, so as to determine a commodity variety taken out by the user appearing in the target gesture feature trajectory data.
  • the minimum screenshots of all the frames are compared with the initial feature template to find the key frames of the articles containing the product, that is, to find the picture frames in which the product is grasped by the user.
  • the initial feature template corresponding to the initial state is compared to find the key frame of the item that the user has grasped. This step is to determine whether the user has obtained the product, so as to further identify the acquired product. .
  • the user's own gesture image can be compared. Similarity in skin color, gestures, and actions improves the accuracy of the user's shopping behavior judgment.
  • a fifth embodiment of the present application provides a method for determining a purchase of a product. Based on the fifth embodiment shown in FIG. 6 described above, the step S32230 “recognizes each key frame of the item to determine the The commodity variety taken out by the user appearing in the target gesture characteristic trajectory data "includes:
  • Step S32231 converting the key frame of the item into a grayscale image, and R, G, and B three-color channel images
  • grayscale digital image is an image with only one sample color per pixel. This type of image is usually displayed as a grayscale from the darkest black to the brightest white, although in theory this sample can be of different shades of any color, or even different colors at different brightnesses.
  • Gray images are different from black and white images. In the field of computer graphics, black and white images have only two colors: black and white. Gray images have many levels of color depth between black and white.
  • the RGB color mode is a color standard in the industry. It is obtained by changing the three color channels of red (R), green (G), and blue (B) and superimposing them on each other. RGB is the color representing the three channels of red, green and blue.
  • the item key frames are converted into grayscale images, R-channel images, G-channel images, and B-channel images.
  • Step S32232 based on the preset product feature database, matching the preset product feature images in the preset product feature database with the grayscale image, the R, G, and B three-color channel images to obtain corresponding ones. Recognition result
  • the preset product feature database is a database that stores preset product feature images of multiple angles and features of all products on sale, which may include product images of different angles, images of different parts of the product, and may also include Product information such as shape, color, size, etc.
  • the grayscale image, the R channel image, the G channel image, and the B channel image are respectively matched with a preset product feature image in a preset product feature database to obtain a corresponding recognition result.
  • Step S32233 calculating a keyframe recognition result corresponding to the keyframe of the item according to a preset weight occupied by each recognition result, and determining the occurrence of the target gesture feature trajectory data according to the keyframe recognition result. Variety of products taken out by the user.
  • the preset weight may be a calculation weight for the importance of the recognition result obtained by matching and comparing the grayscale image, the R channel image, the G channel image, and the B channel image, respectively.
  • the gray image weight is 40%, and the R channel image, G channel image, and B channel image are all 20%.
  • a preset weight for each key frame is obtained through calculation, and a key frame identification result corresponding to the key frame of the item is calculated.
  • the key frame recognition result may also be similarity, that is, comparing with the preset product feature image in the preset product feature database, and obtaining the similarity, and calculating the preset weight to obtain The keyframe identification result corresponding to the item keyframe.
  • the present application also provides a product purchase determination device, including: a receiving module 10, a collection module 20, and an identification module 30;
  • the receiving module 10 is configured to receive a shopping start instruction returned by an infrared signal triggered by a user's shopping gesture
  • the acquisition module 20 is configured to start image collection of the user's shopping gesture based on the shopping start instruction based on the timestamp, and stop receiving the shopping termination instruction returned by the infrared signal triggered by the user's leaving gesture Acquiring images of the user ’s shopping gestures to obtain a time-stamped continuous shooting image between the time when the shopping start instruction and the shopping termination instruction are received;
  • the recognition module 30 is configured to perform image recognition on the continuous-shot image with time stamps based on the learning of the neural network to determine the user's product shopping data, and settle the product according to the product shopping data;
  • the product shopping data includes the types of products taken out by the user, the time of taking out, and the number of products corresponding to the types of the products.
  • the present application also provides a user terminal, which includes a memory and a processor, where the memory is used to store a commodity purchase determination program, and the processor runs the commodity purchase determination program to cause the user terminal to execute as described above.
  • Product purchase judgment method is used to determine the price of a commodity purchase determination program.
  • the present application also provides a computer-readable storage medium, where the computer-readable storage medium stores a commodity purchase determination program, and when the commodity purchase determination program is executed by a processor, the commodity purchase determination method described above is implemented.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Business, Economics & Management (AREA)
  • Human Computer Interaction (AREA)
  • Multimedia (AREA)
  • Finance (AREA)
  • Accounting & Taxation (AREA)
  • Social Psychology (AREA)
  • Psychiatry (AREA)
  • General Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Health & Medical Sciences (AREA)
  • Development Economics (AREA)
  • Economics (AREA)
  • Marketing (AREA)
  • Strategic Management (AREA)
  • General Business, Economics & Management (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
  • User Interface Of Digital Computer (AREA)
  • Image Analysis (AREA)

Abstract

一种商品购买判定方法、装置和用户终端，其中所述方法包括：接收由用户购物手势触发红外信号所返回的购物启动指令；根据购物启动指令开始进行图像采集，并在接收到购物终止指令时停止对用户购物手势的图像采集，得到带有时间戳的连拍图像；对带有时间戳的连拍图像进行图像识别，以确定用户的商品购物数据，并根据商品购物数据对商品进行结算。上述方法实现了在开放式的购物环境中，通过图像识别技术，对于用户的购物行为的判断，以及对于所购物品的品种和数量的智能识别，实现了对于用户的商品购买的行为和所购商品的识别和判断，降低了人力成本，大大缩短了结算时间，结算效率高，购物过程简单，提高了用户体验。

Description

一种商品购买判定方法、装置和用户终端
本申请是以申请号为201810582232.0、申请日为2018年6月7日的中国专利申请为基础，并主张其优先权，该申请的全部内容在此作为整体引入本申请中。
技术领域
本申请涉及图像识别技术领域，更具体地说，涉及一种商品购买判定方法、装置和用户终端。
背景技术
物质流通是人类社会的最基本要素。零售行业或直接面向消费者销售的开放式的相关行业(以下简称为零售行业或零售企业)作为当前物质流通的重要手段，需要雇佣大量的收银人员来实现商品的买卖工作，收银人员对消费者所购物品一件一件进行统计，统计完成后，与消费者共同完成支付。
而自动售货柜，则解决了零售行业的人工统计和人工结算的缺陷，是一种面向消费者的无人售货、自动结算的售卖方式。可通过用户的选择，统计出用户所选择的商品的品种种类和总价值，以供用户进行结算，完成支付过程。但目前的自动售货柜只能进行根据用户的选择，自动出货自动结算，并不能在开放的零售购物环境中实现对于用户的购物的行为和所取得的进行准确的识别，从而实现结算。
总之，目前现有的开放式的购物环境中，只能通过人工对于消费者所购得的商品进行逐件的检查、统计和总价计算，耗费大量人力成本，结算时间长效率低，购物过程繁琐、麻烦，用户体验差。
申请内容
有鉴于此，本申请提供一种商品购买判定方法、装置和用户终端以解决现有技术的不足。
为解决上述问题，本申请提供一种商品购买判定方法，包括：
接收由用户购物手势触发红外信号所返回的购物启动指令；
根据所述购物启动指令，基于时间戳，对所述用户购物手势开始进行图像采集，并在接收到用户离开手势触发的红外信号所返回的购物终止指令时停止对所述用户购物手势的图像采集，得到在接收到所述购物启动指令和所述购物终止指令的时间之间的带有时间戳的连拍图像；
基于神经网络学习，对所述带有时间戳的连拍图像进行图像识别，以确定用户的商品购物数据，并根据所述商品购物数据对商品进行结算；所述商品购物数据包括用户取出的商品品种、取出时间和与所述商品品种对应的商品数量。
优选地，所述“基于神经网络学习，对所述带有时间戳的连拍图像进行图像识别，以确定用户的商品购物数据”包括：
将所述带有时间戳的连拍图像按照所述时间戳顺序转换为连续的若干帧的手势图像；
基于神经网络学习，对每一帧的所述手势图像进行图像识别，以确定所述用户的所述商品购物数据。
优选地，所述“基于神经网络学习，对每一帧的所述手势图像进行图像识别，以确定所述用户的所述商品购物数据”，包括：
基于神经网络学习，对每一帧的所述手势图像中的手势进行手势特征定位，得到目标手势特征轨迹数据；
对所述目标手势特征轨迹数据进行识别，以确定所述目标手势特征轨迹数据中所述用户取出的商品品种；
对所述目标手势特征轨迹数据中所述用户取出的商品品种进行统计，生成商品购物数据。
优选地，所述“基于神经网络学习，对每一帧的所述手势图像中的手势进行手势特征定位，得到目标手势特征轨迹数据”包括：
基于神经网络学习，对每一帧的所述手势图像中的手势进行手势特征定位，确定所述手势特征的特征区域框，并根据所述特征区域框截取包括所述手势特征的最小截图；
将每一帧所述手势图像的所述最小截图依据时间戳的顺序合成为特征运动轨迹，并基于所述特征运动轨迹和与其对应的时间戳生成目标手势特征轨迹数据。
优选地，所述“对所述目标手势特征轨迹数据进行识别，以确定所述目标手势特征轨迹数据中所述用户取出的商品品种”包括：
提取所述目标手势特征轨迹数据中的用户购物初始状态的特征图像，并将所述特征图像作为初始特征模板；
将所述目标手势特征轨迹数据中的每一帧所述最小截图与所述初始特征模板进行比对，确定每一帧所述最小截图中的包含有所述商品的物品关键帧；
对每个所述物品关键帧进行识别，以确定所述目标手势特征轨迹数据中所述用户取出的商品品种。
优选地，所述“对每个所述物品关键帧进行识别，以确定所述目标手势特征轨迹数据中所述用户取出的商品品种”，包括：
将所述物品关键帧转换为灰度图像，以及R、G、B三色通道图像；
基于预设商品特征库，将所述预设商品特征库中的预设商品特征图像分别与所述灰度图像、所述R、G、B三色通道图像进行匹配，获得对应的 识别结果；
根据每个识别结果的所占预设权重，计算得到所述物品关键帧对应的关键帧识别结果，并根据所述关键帧识别结果确定所述目标手势特征轨迹数据中出现的所述用户取出的商品品种。
此外，为解决上述问题，本申请还提供一种商品购买判定装置，包括：接收模块、采集模块和识别模块；
所述接收模块，用于接收由用户购物手势触发红外信号所返回的购物启动指令；
所述采集模块，用于根据所述购物启动指令，基于时间戳，对所述用户购物手势开始进行图像采集，并在接收到用户离开手势触发的红外信号所返回的购物终止指令时停止对所述用户购物手势的图像采集，得到在接收到所述购物启动指令和所述购物终止指令的时间之间的带有时间戳的连拍图像；
所述识别模块，用于基于神经网络学习，对所述带有时间戳的连拍图像进行图像识别，以确定用户的商品购物数据，并根据所述商品购物数据对商品进行结算；所述商品购物数据包括用户取出的商品品种、取出时间和与所述商品品种对应的商品数量。
此外，为解决上述问题，本申请还提供一种用户终端，包括存储器以及处理器，所述存储器用于存储商品购买判定程序，所述处理器运行所述商品购买判定程序以使所述用户终端执行如上述所述商品购买判定方法。
此外，为解决上述问题，本申请还提供一种计算机可读存储介质，所述计算机可读存储介质上存储有商品购买判定程序，所述商品购买判定程序被处理器执行时实现如上述所述商品购买判定方法。
本申请提供的一种商品购买判定方法、装置和用户终端。其中，本申 请所提供的方法通过购物手势触发红外信号，进而获得购物启动指令，并且，在用户离开手势触发的红外信号返回购物终止指令，在接收到购物启动指令和购物终止指令之间的时间段进行对用户购物手势的图像采集，并对该连拍图像进行识别，以最终确定用户的购买行为和所购买的物品的品种、数量，购物时间等信息，从而进行结算。通过本申请所提供的商品购买判定方法，实现了在开放式的购物环境中，通过图像识别技术，对于用户的购物行为的判断，以及对于所购物品的品种和数量的智能识别，实现了对于用户的商品购买的行为和所购商品的识别和判断，降低了人力成本，大大缩短了结算时间，结算效率高，购物过程简单，提高了用户体验。
附图说明
图1为本申请商品购买判定方法实施例方案涉及的硬件运行环境的结构示意图；
图2为本申请商品购买判定方法第一实施例的流程示意图；
图3为本申请商品购买判定方法第二实施例的流程示意图；
图4为本申请商品购买判定方法第三实施例的流程示意图；
图5为本申请商品购买判定方法第四实施例的流程示意图；
图6为本申请商品购买判定方法第五实施例的流程示意图；
图7为本申请商品购买判定方法第六实施例的流程示意图；
图8为本申请商品购买判定装置的功能模块示意图。
本申请目的的实现、功能特点及优点将结合实施例，参照附图做进一步说明。
具体实施方式
下面详细描述本申请的实施例，其中自始至终相同或类似的标号表示相同或类似的元件或具有相同或类似功能的元件。
此外，术语“第一”、“第二”仅用于描述目的，而不能理解为指示或暗示相对重要性或者隐含指明所指示的技术特征的数量。由此，限定有“第一”、“第二”的特征可以明示或者隐含地包括一个或者更多个该特征。在本申请的描述中，“多个”的含义是两个或两个以上，除非另有明确具体的限定。
在本申请中，除非另有明确的规定和限定，术语“安装”、“相连”、“连接”、“固定”等术语应做广义理解，例如，可以是固定连接，也可以是可拆卸连接，或成一体；可以是机械连接，也可以是电连接；可以是直接相连，也可以通过中间媒介间接相连，可以是两个元件内部的连通或两个元件的相互作用关系。对于本领域的普通技术人员而言，可以根据具体情况理解上述术语在本申请中的具体含义。
应当理解，此处所描述的具体实施例仅仅用以解释本申请，并不用于限定本申请。
如图1所示，图1是本申请实施例方案涉及的终端的硬件运行环境的结构示意图。
本申请实施例终端可以是的设于自动货柜机中的PC，也可以是智能手机、平板电脑、电子书阅读器、MP3播放器、MP4播放器、便携计算机等可移动式终端设备。此外，也可以为自动货柜机本身所带有的计算机硬件装置。
如图1所示，该终端可以包括：处理器1001，例如CPU，网络接口1004，用户接口1003，存储器1005，通信总线1002。其中，通信总线1002用于实现这些组件之间的连接通信。用户接口1003可以包括显示屏、输入单元比如键盘、遥控器，可选用户接口1003还可以包括标准的有线接口、无线接口。网络接口1004可选的可以包括标准的有线接口、无线接口(如WI-FI 接口)。存储器1005可以是高速RAM存储器，也可以是稳定的存储器，例如磁盘存储器。存储器1005可选的还可以是独立于前述处理器1001的存储装置。
此外，终端包括图像采集设备，具体可以为摄像头，相机等。
此外，终端还包括红外传感设备，用以对用户的购物行为进行判断。
可选地，终端还可以包括RF(Radio Frequency，射频)电路，传感器、音频电路、WiFi模块等等。此外，移动终端还可配置陀螺仪、气压计、湿度计、温度计、红外线传感器等其他传感器，在此不再赘述。
本领域技术人员可以理解，图1中示出的终端并不构成对终端的限定，可以包括比图示更多或更少的部件，或者组合某些部件，或者不同的部件布置。
如图1所示，作为一种计算机可读存储介质的存储器1005中可以包括操作系统、数据接口控制程序、网络连接程序以及商品购买判定程序。
本申请提供的一种商品购买判定方法、装置和用户终端。其中，所述方法实现了在开放式的购物环境中，通过图像识别技术，对于用户的购物行为的判断，以及对于所购物品的品种和数量的智能识别，实现了对于用户的商品购买的行为和所购商品的识别和判断，降低了人力成本，大大缩短了结算时间，结算效率高，购物过程简单，提高了用户体验。
实施例1：
参照图2，本申请第一实施例提供一种商品购买判定方法，包括：
步骤S10000，接收由用户购物手势触发红外信号所返回的购物启动指令；
上述，本实施例所提供的商品购买判定方法，可以适用于开放式的购 物环境，例如商场、超市等场所，即无人值守，用户可进行自由拿取货物。在本实施例中，可以为开放式的自动售货机，用户开始购物后，自动售货机的柜门打开，用户进行对内部的货物直接拿取，而自动售货机通过图像采集设备对用户的购物行为、所购物品进行识别和统计，实现对于用户的所购商品的最终结算。
上述，用户购物手势，为用户对目标商品的拿取的动作，该动作可以包括两个过程，即可以为选择抓取或放置过程和货品取回或递出过程，通过这两个过程，实现对于目标商品的拿取。上述过程，可以为用户空手向目标商品移动，抓取货物，并且将目标商品取回。此外，上述过程也可以针对于用户手持商品向目标位置递出，放置于目标位置，并且空手退出的过程。
上述，红外信号，为本实施例中的红外传感设备所发出的红外线信号，通过在特定位置设置红外传感设备，布置红外线信号，以便于用户在出现购物手势时，触发相应的红外传感器所发出的红外线信号，并返回一购物启动指令。
上述，购物启动指令，可以依据相应的红外传感设备所对应的位置，启动有针对性的指令，即在不同的区域设置不同的红外传感设备，对于用户在不同区域进行抓取商品时，触发的不同区域的红外信号，返回相应区域所对应的购物启动指令。例如，开放式的自动售货机存在有上中下3层购物空间，用户在去拿取下层的商品时，触发下层的红外传感设备的红外信号，对应的返回一个与下层空间相对应的购物启动指令。
步骤S20000，根据所述购物启动指令，基于时间戳，对所述用户购物手势开始进行图像采集，并在接收到用户离开手势触发的红外信号所返回的购物终止指令时停止对所述用户购物手势的图像采集，得到在接收到所 述购物启动指令和所述购物终止指令的时间之间的带有时间戳的连拍图像；
上述，图像采集的设备，可以为摄像机、相机等具有图像拍摄功能的装置。
图像采集可以为在不同的购物区域设置不同的图像采集设备，并且根据不同区域对应的购物启动指令，开启不同区域的图像采集设备，以便于对相应的购物行为进行采集和识别。例如，开放式的自动售货机存在有上中下3层购物空间，用户在去拿取下层的商品时，触发下层的红外传感设备的红外信号，对应的返回一个与下层空间相对应的购物启动指令，此时，设于下层空间的相应位置的多个不同角度的摄像头开始工作，开始对于用户的购物行为进行图像采集。
上述，时间戳，为图像采集设备所采集图像对应的时间标签，所采集图像的每一帧对应一个时间戳，按照时间戳的顺序，所采集到的带有时间戳的连拍图像形成用户手势的运动轨迹。
上述，连拍图像，即为在用户购物的过程中，持续不断的追踪用户手势，连续拍摄用户购物的手势图像，判断整个过程中用户拿起、放下的商品品种，即通过所连续拍摄得到的图像对用户所购商品的品种进行识别，并对所取出的商品进行结算。
上述，连拍图像，可以为图像视频数据，即为视频图像。此外，也可以为可记录购物完整过程的用户购物手势的动态图像。
步骤S30000，基于神经网络学习，对所述带有时间戳的连拍图像进行图像识别，以确定用户的商品购物数据，并根据所述商品购物数据对商品进行结算；所述商品购物数据包括用户取出的商品品种、取出时间和与所述商品品种对应的商品数量。
上述，需要说明的是，神经网络学习，即为人工神经网络(Artificial Neural Network，即ANN)，是20世纪80年代以来人工智能领域兴起的研究热点。它从信息处理角度对人脑神经元网络进行抽象，建立某种简单模型，按不同的连接方式组成不同的网络。在工程与学术界也常直接简称为神经网络或类神经网络。神经网络是一种运算模型，由大量的节点(或称神经元)之间相互联接构成。每个节点代表一种特定的输出函数，称为激励函数(activation function)。每两个节点间的连接都代表一个对于通过该连接信号的加权值，称之为权重，这相当于人工神经网络的记忆。网络的输出则依网络的连接方式，权重值和激励函数的不同而不同。而网络自身通常都是对自然界某种算法或者函数的逼近，也可能是对一种逻辑策略的表达。
通过神经网络学习，使系统获得对图像中的购物手势以及与购物手势对应的商品的识别的能力。通过神经网络学习，识别用户的带有时间戳的连拍图像进行图像识别，从而得到用户的购物手势所取得的商品的品种、数量和时间，以便于最终对用户所抓取的商品进行最终结算。
本实施例所提供的方法通过购物手势触发红外信号，进而获得购物启动指令，并且，在用户离开手势触发的红外信号返回购物终止指令，在接收到购物启动指令和购物终止指令之间的时间段进行对用户购物手势的图像采集，并对该连拍图像进行识别，以最终确定用户的购买行为和所购买的物品的品种、数量，购物时间等信息，从而进行结算。通过本申请所提供的商品购买判定方法，实现了在开放式的购物环境中，通过图像识别技术，对于用户的购物行为的判断，以及对于所购物品的品种和数量的智能识别，实现了对于用户的商品购买的行为和所购商品的识别和判断，降低了人力成本，大大缩短了结算时间，结算效率高，购物过程简单，提高了 用户体验。
实施例2：
参照图3，本申请第二实施例提供一种商品购买判定方法，基于上述图2所示的第一实施例，所述步骤S30000“基于神经网络学习，对所述带有时间戳的连拍图像进行图像识别，以确定用户的商品购物数据”包括：
步骤S31000，将所述带有时间戳的连拍图像按照所述时间戳顺序转换为连续的若干帧的手势图像；
上述，在本实施例中连拍图像可以为动态图像视频数据，将该连拍图像转换为若干帧的手势图像，上述若干帧，可以依据具体图像识别能力进行设置，例如10帧。
所连拍图像转换的手势图像的帧数，可以根据用户购物手势的速度进行设置，用户购物手势速度快，则帧数多；速度慢，则帧数少。此外，也可以为固定数值，例如，在本实施例中，可以设置对用户每次的购物手势的图像转换的若干帧为10-15帧。
在某种实施方式中，由于对于用户的每个购物手势均需要进行连续拍摄，在用户的购物过程中，需要拍摄并保存有大量的购物手势的图片，在一定程度占用存储空间、系统资源，甚至在与云端进行数据交互时会占用大量网络资源，影响网络带宽速度，严重的会导致一定程度上的拖慢整个网络的情况。所以，在本实施中，可以在得到在接收到所述购物启动指令和所述购物终止指令的时间之间的带有时间戳的连拍图像后，并且在进行图片识别前，对所获取到的带有时间戳的连拍图像进行预处理，具体可以为如下步骤：
在所述步骤S31000之后，还可以包括：
对每一帧的所述手势图像进行亮度、对比度、色彩饱和度的优化处理；
对亮度、对比度、色彩饱和度的优化处理后的所述手势图像进行图像截取，以得到包含有用户购物手势和商品的所述手势图像；
对图像截取后的所述手势图像进行分辨率统一化调整，以得到与图像识别相适应且统一大小的所述手势图像。
上述，在进行图像处理的步骤中，对所采集到的所述手势图像进行图像优化处理，首先进行图像亮度、对比度、色彩饱和度处理，即为在保证图像质量的情况下，以达到对图像的所占内存或容量进行相应降低的效果。
上述，对于手势图像进行截取，去除掉其中的除了购物手势及商品所占区域以外的多余部分，只截取每一帧连拍图像中的包含有购物手势及商品的所占区域的最小图像。
上述，调整连拍图像的分辨率，以便于手势图像适应于图像识别的最小分辨率，且将所有截取后的图像进行大小统一，从而达到统一化并且降低图像所占内存或容量的效果。
步骤S32000，基于神经网络学习，对每一帧的所述手势图像进行图像识别，以确定所述用户的所述商品购物数据。
上述，对于其中的每一帧的手势图像进行图像识别，从而确定用户的商品购物数据，以确定用户购物过程中取出了什么，取出了多少数量等等信息。通过对于连拍图像的单帧手势图像的识别，从而可以更加准确地对于用户的购物手势进行定位并进行进一步的判断。
实施例3：
参照图4，本申请第三实施例提供一种商品购买判定方法，基于上述图3所示的第二实施例，所述步骤S32000“基于神经网络学习，对每一帧的所 述手势图像进行图像识别，以确定所述用户的所述商品购物数据”，包括：
步骤S32100，基于神经网络学习，对每一帧的所述手势图像中的手势进行手势特征定位，得到目标手势特征轨迹数据；
上述，手势特征即为在图像中，包含有用户的购物手势的特征信息，包括用户的手的图像特征和用户手抓取的商品的图像特征，其中，手的图像特征可以为抓取手势特征，手心位置特征，手指形状特征等等，商品的图像特征可以包括商品的外形、大小等等信息。
上述，手势图像中包含有手势特征轨迹数据，通过神经网络学习，可对图像中的所包含的手势特征进行定位，从而判断出手势图像中是否有用户的手势存在并且，用户的手中是否抓取商品。
步骤S32200，对所述目标手势特征轨迹数据进行识别，以确定所述目标手势特征轨迹数据中所述用户取出的商品品种；
步骤S32300，对所述目标手势特征轨迹数据中所述用户取出的商品品种进行统计，生成商品购物数据。
上述，在进行图像识别时，对于每一帧的手势图像中的手势特征进行定位，从而找到手势图像中的手势特征，从而得到目标手势特征轨迹数据。并且，对目标手势特征轨迹数据进行识别，通过图像识别，得到目标手势特征轨迹数据中的用户所取出的商品对应的商品品种，并进行统计以生成商品购物数据。
实施例4：
参照图5，本申请第四实施例提供一种商品购买判定方法，基于上述图4所示的第三实施例，所述步骤S32100“基于神经网络学习，对每一帧的所述手势图像中的手势进行手势特征定位，得到目标手势特征轨迹数据” 包括：
步骤S32110，基于神经网络学习，对每一帧的所述手势图像中的手势进行手势特征定位，确定所述手势特征的特征区域框，并根据所述特征区域框截取包括所述手势特征的最小截图；
步骤S32120，将每一帧所述手势图像的所述最小截图依据时间戳的顺序合成为特征运动轨迹，并基于所述特征运动轨迹和与其对应的时间戳生成目标手势特征轨迹数据。
上述，特征区域框为进行图像识别时，所定位出的手势特征为原点或区域中心的且包括用户的手势的特征区域的框，并进行根据特征区域框所进行的截图。其中，特征区域框，即为包括所述手势特征的最小截图。
上述特征区域框的形状可以为长方形、正方形或其他任意形状。
上述，将用户进行购物时的每一帧所述手势图像的所述最小截图合成为特征运动轨迹，并且，根据该特征运动轨迹和时间戳，生成带有时间戳的目标手势特征轨迹数据。通过进行图像截取，得到了包含有手势特征的最小截图，从而将所有帧对应的最小截图合成为特征运动轨迹，大大减少了图像再传输识别过程中的图像所占用的系统资源，最小截图中包含有所需要进行识别的必要的手势特征信息，并且删除了不相关的图像信息，减少了图像所占用存储空间的容量，可在一定程度上提高图像识别的效率。
实施例5：
参照图6，本申请第五实施例提供一种商品购买判定方法，基于上述图5所示的第四实施例，所述步骤S32200“对所述目标手势特征轨迹数据进行识别，以确定所述目标手势特征轨迹数据中出现的所述用户取出的商品品种”包括：
步骤S32210，提取所述目标手势特征轨迹数据中的用户购物初始状态的特征图像，并将所述特征图像作为初始特征模板；
上述，用户购物初始状态，即为用户的手势为空手，进行对商品抓取时的状态，其中，用户的手势中并没有商品，且并没有接触商品。
将用户购物初始状态作为基准特征的初始特征模板，进行进一步对用户手中是否有商品进行判断。
步骤S32220，将所述目标手势特征轨迹数据中的每一帧所述最小截图与所述初始特征模板进行比对，确定每一帧所述最小截图中的包含有所述商品的物品关键帧；
步骤S32230，对每个所述物品关键帧进行识别，以确定所述目标手势特征轨迹数据中出现的所述用户取出的商品品种。
上述，在本实施例中，通过将所有帧的最小截图与初始特征模板进行比对，以找出其中包含有商品的物品关键帧，即为找出其中用户的手中抓有商品的图片帧。通过与初始状态对应的初始特征模板进行比对，从而找出用户手中抓有商品的物品关键帧，该步骤为对于用户是否拿取到商品进行判断，以便于进一步对所拿取的商品进行识别。此外，通过与初始状态对应的初始特征模板进行比对，实现了对用户的自己的手势图像进行比对，肤色、手势、动作的相似，提高了对于用户购物行为判断的准确性。
实施例6：
参照图7，本申请第五实施例提供一种商品购买判定方法，基于上述图6所示的第五实施例，所述步骤S32230“对每个所述物品关键帧进行识别，以确定所述目标手势特征轨迹数据中出现的所述用户取出的商品品种”，包括：
步骤S32231，将所述物品关键帧转换为灰度图像，以及R、G、B三色通道图像；
上述，需要说明的是，灰度数字图像是每个像素只有一个采样颜色的图像。这类图像通常显示为从最暗黑色到最亮的白色的灰度，尽管理论上这个采样可以任何颜色的不同深浅，甚至可以是不同亮度上的不同颜色。灰度图像与黑白图像不同，在计算机图像领域中黑白图像只有黑白两种颜色，灰度图像在黑色与白色之间还有许多级的颜色深度。
上述，RGB色彩模式是工业界的一种颜色标准，是通过对红(R)、绿(G)、蓝(B)三个颜色通道的变化以及它们相互之间的叠加来得到各式各样的颜色的，RGB即是代表红、绿、蓝三个通道的颜色。
将物品关键帧转换为灰度图像、R通道图像、G通道图像、B通道图像。
步骤S32232，基于预设商品特征库，将所述预设商品特征库中的预设商品特征图像分别与所述灰度图像、所述R、G、B三色通道图像进行匹配，获得对应的识别结果；
上述，预设商品特征库，即为保存有所有在售商品的多角度和特征的预设商品特征图像的数据库，其中，可以包括不同角度的商品图像、商品不同部位的图像，以及还可以包括商品的形状、色彩、大小尺寸信息等等数据信息。
上述，将灰度图像、R通道图像、G通道图像、B通道图像分别与预设商品特征库中的预设商品特征图像进行匹配，以获得对应的识别结果。
步骤S32233，根据每个识别结果的所占预设权重，计算得到所述物品关键帧对应的关键帧识别结果，并根据所述关键帧识别结果确定所述目标手势特征轨迹数据中出现的所述用户取出的商品品种。
上述，预设权重，可以为对于灰度图像、R通道图像、G通道图像、B 通道图像进行分别匹配和比对，所获得的识别结果的所占重要性的计算权重。例如，灰度图像权重为40％，R通道图像、G通道图像、B通道图像均为20％。
通过计算得到每一关键帧的预设权重，计算得到物品关键帧对应的关键帧识别结果。
上述，关键帧识别结果，也可以为相似性，即与所述预设商品特征库中的预设商品特征图像进行分别比对，所得到的相似性，并通过预设权重进行计算，从而得到物品关键帧对应的关键帧识别结果。
此外，本申请还提供一种商品购买判定装置，包括：接收模块10、采集模块20和识别模块30；
所述接收模块10，用于接收由用户购物手势触发红外信号所返回的购物启动指令；
所述采集模块20，用于根据所述购物启动指令，基于时间戳，对所述用户购物手势开始进行图像采集，并在接收到用户离开手势触发的红外信号所返回的购物终止指令时停止对所述用户购物手势的图像采集，得到在接收到所述购物启动指令和所述购物终止指令的时间之间的带有时间戳的连拍图像；
所述识别模块30，用于基于神经网络学习，对所述带有时间戳的连拍图像进行图像识别，以确定用户的商品购物数据，并根据所述商品购物数据对商品进行结算；所述商品购物数据包括用户取出的商品品种、取出时间和与所述商品品种对应的商品数量。
此外，本申请还提供一种用户终端，包括存储器以及处理器，所述存 储器用于存储商品购买判定程序，所述处理器运行所述商品购买判定程序以使所述用户终端执行如上述所述商品购买判定方法。
此外，本申请还提供一种计算机可读存储介质，所述计算机可读存储介质上存储有商品购买判定程序，所述商品购买判定程序被处理器执行时实现如上述所述商品购买判定方法。
上述本申请实施例序号仅仅为了描述，不代表实施例的优劣。
通过以上的实施方式的描述，本领域的技术人员可以清楚地了解到上述实施例方法可借助软件加必需的通用硬件平台的方式来实现，当然也可以通过硬件，但很多情况下前者是更佳的实施方式。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在如上所述的一个存储介质(如ROM/RAM、磁碟、光盘)中，包括若干指令用以使得一台终端设备(可以是手机，计算机，服务器，或者网络设备等)执行本申请各个实施例所述的方法。
以上仅为本申请的优选实施例，并非因此限制本申请的专利范围，凡是利用本申请说明书及附图内容所作的等效结构或等效流程变换，或直接或间接运用在其他相关的技术领域，均同理包括在本申请的专利保护范围内。

Claims (9)

  1. 一种商品购买判定方法，其特征在于，包括：
    接收由用户购物手势触发红外信号所返回的购物启动指令；
    根据所述购物启动指令，基于时间戳，对所述用户购物手势开始进行图像采集，并在接收到用户离开手势触发的红外信号所返回的购物终止指令时停止对所述用户购物手势的图像采集，得到在接收到所述购物启动指令和所述购物终止指令的时间之间的带有时间戳的连拍图像；
    基于神经网络学习，对所述带有时间戳的连拍图像进行图像识别，以确定用户的商品购物数据，并根据所述商品购物数据对商品进行结算；所述商品购物数据包括用户取出的商品品种、取出时间和与所述商品品种对应的商品数量。
  2. 如权利要求1所述商品购买判定方法，其特征在于，所述“基于神经网络学习，对所述带有时间戳的连拍图像进行图像识别，以确定用户的商品购物数据”包括：
    将所述带有时间戳的连拍图像按照所述时间戳顺序转换为连续的若干帧的手势图像；
    基于神经网络学习，对每一帧的所述手势图像进行图像识别，以确定所述用户的所述商品购物数据。
  3. 如权利要求2所述商品购买判定方法，其特征在于，所述“基于神经网络学习，对每一帧的所述手势图像进行图像识别，以确定所述用户的所述商品购物数据”，包括：
    基于神经网络学习，对每一帧的所述手势图像中的手势进行手势特征定位，得到目标手势特征轨迹数据；
    对所述目标手势特征轨迹数据进行识别，以确定所述目标手势特征轨 迹数据中所述用户取出的商品品种；
    对所述目标手势特征轨迹数据中所述用户取出的商品品种进行统计，生成商品购物数据。
  4. 如权利要求3所述商品购买判定方法，其特征在于，所述“基于神经网络学习，对每一帧的所述手势图像中的手势进行手势特征定位，得到目标手势特征轨迹数据”包括：
    基于神经网络学习，对每一帧的所述手势图像中的手势进行手势特征定位，确定所述手势特征的特征区域框，并根据所述特征区域框截取包括所述手势特征的最小截图；
    将每一帧所述手势图像的所述最小截图依据时间戳的顺序合成为特征运动轨迹，并基于所述特征运动轨迹和与其对应的时间戳生成目标手势特征轨迹数据。
  5. 如权利要求4所述商品购买判定方法，其特征在于，所述“对所述目标手势特征轨迹数据进行识别，以确定所述目标手势特征轨迹数据中所述用户取出的商品品种”包括：
    提取所述目标手势特征轨迹数据中的用户购物初始状态的特征图像，并将所述特征图像作为初始特征模板；
    将所述目标手势特征轨迹数据中的每一帧所述最小截图与所述初始特征模板进行比对，确定每一帧所述最小截图中的包含有所述商品的物品关键帧；
    对每个所述物品关键帧进行识别，以确定所述目标手势特征轨迹数据中所述用户取出的商品品种。
  6. 如权利要求5所述商品购买判定方法，其特征在于，所述“对每个所述物品关键帧进行识别，以确定所述目标手势特征轨迹数据中所述用户取 出的商品品种”，包括：
    将所述物品关键帧转换为灰度图像，以及R、G、B三色通道图像；
    基于预设商品特征库，将所述预设商品特征库中的预设商品特征图像分别与所述灰度图像、所述R、G、B三色通道图像进行匹配，获得对应的识别结果；
    根据每个识别结果的所占预设权重，计算得到所述物品关键帧对应的关键帧识别结果，并根据所述关键帧识别结果确定所述目标手势特征轨迹数据中出现的所述用户取出的商品品种。
  7. 一种商品购买判定装置，其特征在于，包括：接收模块、采集模块和识别模块；
    所述接收模块，用于接收由用户购物手势触发红外信号所返回的购物启动指令；
    所述采集模块，用于根据所述购物启动指令，基于时间戳，对所述用户购物手势开始进行图像采集，并在接收到用户离开手势触发的红外信号所返回的购物终止指令时停止对所述用户购物手势的图像采集，得到在接收到所述购物启动指令和所述购物终止指令的时间之间的带有时间戳的连拍图像；
    所述识别模块，用于基于神经网络学习，对所述带有时间戳的连拍图像进行图像识别，以确定用户的商品购物数据，并根据所述商品购物数据对商品进行结算；所述商品购物数据包括用户取出的商品品种、取出时间和与所述商品品种对应的商品数量。
  8. 一种用户终端，其特征在于，包括存储器以及处理器，所述存储器用于存储商品购买判定程序，所述处理器运行所述商品购买判定程序以使所述用户终端执行如权利要求1-6中任一项所述商品购买判定方法。
  9. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质上存储有商品购买判定程序，所述商品购买判定程序被处理器执行时实现如权利要求1-6中任一项所述商品购买判定方法。
PCT/CN2019/070034 2018-06-07 2019-01-02 一种商品购买判定方法、装置和用户终端 WO2019233098A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810582232.0 2018-06-07
CN201810582232.0A CN108921645B (zh) 2018-06-07 2018-06-07 一种商品购买判定方法、装置和用户终端

Publications (1)

Publication Number Publication Date
WO2019233098A1 true WO2019233098A1 (zh) 2019-12-12

Family

ID=64419066

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/070034 WO2019233098A1 (zh) 2018-06-07 2019-01-02 一种商品购买判定方法、装置和用户终端

Country Status (2)

Country Link
CN (1) CN108921645B (zh)
WO (1) WO2019233098A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111540447A (zh) * 2020-04-20 2020-08-14 深圳市华智信息科技有限公司 智能库存数据更新方法、电子装置及计算机可读存储介质
CN112735030A (zh) * 2020-12-28 2021-04-30 深兰人工智能(深圳)有限公司 售货柜的视觉识别方法、装置、电子设备和可读存储介质

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108921645B (zh) * 2018-06-07 2021-07-13 深圳码隆科技有限公司 一种商品购买判定方法、装置和用户终端
CN111382650B (zh) * 2018-12-28 2023-05-02 阿里巴巴集团控股有限公司 商品购物处理系统、方法、装置及电子设备
CN111415461B (zh) 2019-01-08 2021-09-28 虹软科技股份有限公司 物品识别方法及系统、电子设备
CN109979130A (zh) * 2019-03-29 2019-07-05 厦门益东智能科技有限公司 一种商品自动识别和结算售货柜、方法以及系统
CN110276305B (zh) * 2019-06-25 2021-06-15 广州众聚智能科技有限公司 一种动态商品识别方法
CN110781752A (zh) * 2019-09-27 2020-02-11 上海数烨数据科技有限公司 一种动态视觉智能柜多品类场景下的物体识别方法
CN112016398A (zh) * 2020-07-29 2020-12-01 华为技术有限公司 手持物体识别方法及装置
CN112306604B (zh) * 2020-08-21 2022-09-23 海信视像科技股份有限公司 一种传输文件的进度显示方法及显示设备
CN112149577A (zh) * 2020-09-24 2020-12-29 信雅达系统工程股份有限公司 基于神经网络图像识别的智能结算系统
CN111931740B (zh) * 2020-09-29 2021-01-26 创新奇智(南京)科技有限公司 商品销量识别方法及装置、电子设备、存储介质
CN113194252B (zh) * 2021-04-28 2023-05-09 深圳市乐科智控科技有限公司 一种动态识别智能柜的数据流量消耗优化方法及其系统

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150323997A1 (en) * 2014-05-06 2015-11-12 Symbol Technologies, Inc. Apparatus and method for performing a variable data capture process
CN107507339A (zh) * 2017-08-11 2017-12-22 深兰科技（上海）有限公司 智能冰柜
CN107871111A (zh) * 2016-09-28 2018-04-03 苏宁云商集团股份有限公司 一种行为分析方法及系统
CN107944859A (zh) * 2017-11-15 2018-04-20 深圳码隆科技有限公司 一种基于图像识别技术的无人售货方法和设备
CN108921645A (zh) * 2018-06-07 2018-11-30 深圳码隆科技有限公司 一种商品购买判定方法、装置和用户终端

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2014195903A1 (en) * 2013-06-05 2014-12-11 Smartli Ltd Methods and devices for smart shopping
CN106952402B (zh) * 2017-03-22 2019-09-06 帮团成都电子商务有限责任公司 一种数据处理方法及装置
CN107134053B (zh) * 2017-04-19 2019-08-06 石道松 智能售货门店
CN107610379A (zh) * 2017-09-11 2018-01-19 江苏弘冠智能科技有限公司 一种购物识别方法及购物车识别装置

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150323997A1 (en) * 2014-05-06 2015-11-12 Symbol Technologies, Inc. Apparatus and method for performing a variable data capture process
CN107871111A (zh) * 2016-09-28 2018-04-03 苏宁云商集团股份有限公司 一种行为分析方法及系统
CN107507339A (zh) * 2017-08-11 2017-12-22 深兰科技（上海）有限公司 智能冰柜
CN107944859A (zh) * 2017-11-15 2018-04-20 深圳码隆科技有限公司 一种基于图像识别技术的无人售货方法和设备
CN108921645A (zh) * 2018-06-07 2018-11-30 深圳码隆科技有限公司 一种商品购买判定方法、装置和用户终端

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111540447A (zh) * 2020-04-20 2020-08-14 深圳市华智信息科技有限公司 智能库存数据更新方法、电子装置及计算机可读存储介质
CN111540447B (zh) * 2020-04-20 2023-10-03 深圳市华智信息科技有限公司 智能库存数据更新方法、电子装置及计算机可读存储介质
CN112735030A (zh) * 2020-12-28 2021-04-30 深兰人工智能(深圳)有限公司 售货柜的视觉识别方法、装置、电子设备和可读存储介质

Also Published As

Publication number Publication date
CN108921645B (zh) 2021-07-13
CN108921645A (zh) 2018-11-30

Similar Documents

Publication Publication Date Title
WO2019233098A1 (zh) 一种商品购买判定方法、装置和用户终端
WO2019237729A1 (zh) 一种购物结算方法、装置和用户终端
CN108922026B (zh) 一种自动售货机的补货管理方法、装置和用户终端
WO2020010827A1 (zh) 一种基于开放式自动售货机的购物结算方法和装置
US11335092B2 (en) Item identification method, system and electronic device
TWI470563B (zh) 偵測影像中臉部屬性之方法、執行影像分析處理之處理系統及電腦可讀取儲存媒體
WO2019057168A1 (zh) 一种货物订单处理方法、装置、服务器、购物终端及系统
CN109241883B (zh) 一种基于自动售货机的退货控制方法及其装置
CN109979130A (zh) 一种商品自动识别和结算售货柜、方法以及系统
Vazquez-Fernandez et al. Built-in face recognition for smart photo sharing in mobile devices
JP2012243313A (ja) 画像処理方法および画像処理装置
CN103617420A (zh) 基于图像特征匹配的商品快速识别方法及系统
EP3006551B1 (en) Image processing device, image processing method, program, and storage medium
CN111222389B (zh) 商超货架上商品的分析方法和系统
CN110348293B (zh) 一种商品识别方法及系统
CN107808126A (zh) 车辆检索方法及装置
CN112306442A (zh) 跨设备的内容投屏方法、装置、设备及存储介质
Greco et al. Digital signage by real-time gender recognition from face images
CN110858277A (zh) 一种获得姿态分类模型的方法以及装置
CN110689389A (zh) 基于计算机视觉的购物清单自动维护方法及装置、存储介质、终端
CN110765825A (zh) 物品摆放状态获取方法和系统
WO2024077971A1 (zh) 活体检测方法及装置
CN114764897A (zh) 行为识别方法、装置、终端设备以及存储介质
Kaushik et al. Revamping Supermarkets With AI and RSSi
JP6575628B1 (ja) 情報処理装置、情報処理システム、制御方法、及びプログラム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19815713

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19815713

Country of ref document: EP

Kind code of ref document: A1