CN118015663A

CN118015663A - Staff identification method, device and equipment

Info

Publication number: CN118015663A
Application number: CN202410420828.6A
Authority: CN
Inventors: 林志伟; 丁益; 任益波
Original assignee: Zhejiang Shenxiang Intelligent Technology Co ltd
Current assignee: Zhejiang Shenxiang Intelligent Technology Co ltd
Priority date: 2024-04-09
Filing date: 2024-04-09
Publication date: 2024-05-10
Anticipated expiration: 2044-04-09
Also published as: CN118015663B

Abstract

The embodiment of the application provides a staff identification method, a device and equipment, wherein the method comprises the following steps: the electronic equipment determines historical sampling data corresponding to the user according to the video stream acquired by the shooting equipment; the historical sampling data comprises user tracks, user behavior events and user characteristics; calculating and analyzing the historical sampling data to determine target employee data corresponding to the target employee; acquiring sampling data to be identified of a user to be identified; and under the condition that the sampling data to be identified is matched with the target employee data, determining that the user to be identified is the target employee. In this way, the electronic equipment determines the target staff based on the track, the behavior event and the characteristics of the user to obtain the target staff data of the target staff, and then determines the target staff through the comparison and matching of the sampling data to be identified and the target staff data, so that the accurate identification of the staff is realized, and meanwhile, the target staff data can be automatically acquired and updated based on the historical sampling data, so that the cost of data updating and maintenance is reduced.

Description

Staff identification method, device and equipment

Technical Field

The present application relates to the field of computer technologies, and in particular, to a method, an apparatus, and a device for identifying employees.

Background

In an on-line retail scenario, merchants have ubiquitous demand for passenger flow statistics to adjust sales strategies based on passenger flow analysis. Staff is often excluded from the passenger flow statistics process, so that staff identification is required in the process.

In the related art, staff identification is usually realized based on staff clothing detection, and the staff identification mode is single, the identification accuracy is not high, staff clothing in different areas and different stores may not be uniform, so that the cost of data updating and maintenance is high.

Disclosure of Invention

The application provides a staff identification method, device and equipment, which are used for improving the accuracy of staff identification and reducing the cost of data updating and maintenance.

In a first aspect, an embodiment of the present application provides an employee identifying method, including:

according to the video stream acquired by the shooting equipment, determining historical sampling data corresponding to a user; the historical sampling data comprises user tracks, user behavior events and user characteristics;

Calculating and analyzing the historical sampling data to determine target employee data corresponding to target employees;

acquiring sampling data to be identified of a user to be identified;

and under the condition that the sampling data to be identified is matched with the target employee data, determining that the user to be identified is a target employee.

In one possible implementation manner, the determining historical sampling data corresponding to each user according to the video stream acquired by the shooting device includes:

Determining foot point information of the user in each image frame of a video stream; the foot point information comprises foot point coordinates and time information;

And determining the user track corresponding to the same user according to the foot point information.

In one possible implementation, the user behavior event comprises a user region event; according to the video stream collected by the shooting equipment, the historical sampling data corresponding to each user is determined, and the method comprises the following steps:

determining actual position information of the user according to the foot point information of the user;

determining a moving path corresponding to the user based on the actual position information;

and generating a user area event in the case that the moving path enters or leaves the target area.

In one possible implementation, the user behavior event includes a user in-out event; according to the video stream collected by the shooting equipment, the historical sampling data corresponding to each user is determined, and the method comprises the following steps:

determining a target line segment corresponding to a target access in the video stream;

determining the position relationship between the foot point information of the user and the target line segment according to the user track;

And under the condition that the position relation meets a first preset condition, determining that the user has an overline behavior, and generating the user access event.

In one possible implementation manner, the determining the position relationship between the foot point information of the user and the target line segment includes:

determining the linear distance from the foot point information to the target line segment, and determining a target cross product between the foot point information and the target line segment;

and determining the position relationship between the foot point information and the target line segment according to the linear distance and the target cross product.

based on a preset human body detection model, determining alternative user characteristics corresponding to a user and quality scores corresponding to the alternative user characteristics;

Tracking and detecting the same user according to the alternative user characteristics;

after tracking is finished, determining the alternative user characteristics of which the quality scores meet a preset score condition as user characteristics corresponding to the user; the user features include human body features and human head features.

In a possible implementation manner, the calculating and analyzing the historical sampling data to determine target employee data includes:

Filtering and screening each user according to the historical sampling data to obtain alternative employee data;

And determining target employee data corresponding to the target employee according to the occurrence frequency of the candidate employee data corresponding to the preset history period.

In a possible implementation manner, the filtering and screening the users according to the historical sampling data to obtain alternative employee data includes:

Determining the number of tracks and the residence time corresponding to each user according to the historical sampling data;

Screening and filtering the user according to the track number and the residence time to obtain first alternative employee data corresponding to a first alternative employee;

And merging the first alternative staff according to the first alternative staff characteristics corresponding to the first alternative staff to obtain alternative staff and alternative staff data corresponding to the alternative staff.

In a possible implementation manner, the determining, according to the occurrence frequency of the candidate employee data corresponding to the preset history period, target employee data corresponding to the target employee includes:

acquiring user characteristics of the alternative staff corresponding to the alternative staff in a preset history period;

Merging the user characteristics of the alternative staff according to the similarity among the user characteristics of the alternative staff, and determining the sampling time corresponding to each user characteristic of the alternative staff;

determining the occurrence frequency of the alternative staff in a preset history period according to the sampling time;

And determining that each candidate staff is a target staff when the occurrence frequency meets a second preset condition, and determining candidate staff data corresponding to the candidate staff as the target staff data.

In a possible implementation manner, the sampling data to be identified includes a first user characteristic, a first user behavior event and a first user track corresponding to the user to be identified; and under the condition that the sampling data to be identified is matched with the target employee data, determining that the user to be identified is a target employee comprises the following steps:

If the first user characteristics are matched with the target employee user characteristics in the target employee data, determining that the user to be identified is a target employee;

Determining the times of entering and exiting of a target area and the residence time of the target area corresponding to the user to be identified according to the first user behavior event;

if the number of times of entering and exiting of the target area and the residence time of the target area meet a third preset condition, determining that the user to be identified is a target employee;

Determining the number of the on-site tracks and the on-site residence time corresponding to the user to be identified according to the first user track and the first user behavior event;

and if the number of the tracks in the field and the residence time in the field meet a fourth preset condition, determining the user to be identified as a target employee.

In a second aspect, an embodiment of the present application provides an employee identifying apparatus, including:

The first determining module is used for determining historical sampling data corresponding to a user according to a video stream acquired by the shooting equipment; the historical sampling data comprises user tracks, user behavior events and user characteristics;

the second determining module is used for carrying out calculation analysis on the historical sampling data and determining target employee data corresponding to the target employee;

The acquisition module is used for acquiring sampling data to be identified of the user to be identified;

And the third determining module is used for determining that the user to be identified is a target employee under the condition that the sampling data to be identified is matched with the target employee data.

In a possible implementation manner, the first determining module is specifically configured to:

In one possible implementation, the user behavior event includes a user in-out event; the first determining module is specifically configured to:

In a possible implementation manner, the second determining module is specifically configured to:

In a possible implementation manner, the sampling data to be identified includes a first user characteristic, a first user behavior event and a first user track corresponding to the user to be identified; the third determining module is specifically configured to:

In a third aspect, an embodiment of the present application provides an electronic device, including: a memory and a processor;

The memory stores computer-executable instructions;

The processor executing computer-executable instructions stored in the memory causing the processor to perform the employee identification method of any of the first aspects.

In a fourth aspect, embodiments of the present application provide a computer-readable storage medium having stored therein computer-executable instructions for implementing the employee identification method of any of the first aspects when the computer-executable instructions are executed by a processor.

In a fifth aspect, an embodiment of the present application provides a computer program product comprising a computer program which, when executed by a processor, implements the employee identification method of any of the first aspects.

In the embodiment of the application, the electronic equipment determines the historical sampling data corresponding to the user according to the video stream acquired by the shooting equipment; the historical sampling data comprises user tracks, user behavior events and user characteristics; calculating and analyzing the historical sampling data to determine target employee data corresponding to the target employee; acquiring sampling data to be identified of a user to be identified; and under the condition that the sampling data to be identified is matched with the target employee data, determining that the user to be identified is the target employee. In this way, the electronic equipment determines the target staff based on the track, the behavior event and the characteristics of the user to obtain the target staff data of the target staff, and then determines the target staff through the comparison and matching of the sample data to be identified and the target staff data after obtaining the sample data to be identified of the user to be identified, so that the accurate identification of the staff is realized, and meanwhile, the target staff data can be automatically acquired and updated based on the historical sample data, so that the cost of data updating and maintenance is reduced.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute a limitation on the application. In the drawings:

Fig. 1 is a schematic diagram of an application scenario provided in an exemplary embodiment of the present application;

Fig. 2 is a schematic flow chart of an employee identification method according to an exemplary embodiment of the present application;

FIG. 3 is a schematic diagram of a determining flow of historical sampled data according to an exemplary embodiment of the present application;

FIG. 4 is a schematic diagram of a pinhole imaging model provided in an exemplary embodiment of the present application;

FIG. 5 is a schematic diagram of a conversion relationship between a plurality of coordinate systems according to an exemplary embodiment of the present application;

FIG. 6 is a schematic diagram of pinhole imaging provided by an exemplary embodiment of the present application;

FIG. 7 is a schematic diagram of a pinhole imaging coordinate system position modification provided by an exemplary embodiment of the present application;

FIG. 8 is a schematic diagram of a coordinate transformation provided by an exemplary embodiment of the present application;

FIG. 9 is a schematic diagram of an in-line store scene coordinate transformation according to an exemplary embodiment of the present application;

FIG. 10 is a schematic diagram of a target area within an offline store scene provided by an exemplary embodiment of the present application;

FIG. 11 is a schematic diagram of a target segment according to an exemplary embodiment of the present application;

FIG. 12 is a schematic diagram illustrating a determination of a user entry and exit event according to an exemplary embodiment of the present application;

FIG. 13 is a schematic diagram of a logic for determining historical sample data according to an exemplary embodiment of the present application;

FIG. 14 is a flowchart of another employee identification method according to an exemplary embodiment of the present application;

FIG. 15 is a schematic diagram of determination logic for alternative employee data provided by an exemplary embodiment of the present application;

FIG. 16 is a schematic diagram of determining logic for target employee data according to an exemplary embodiment of the present application;

FIG. 17 is a schematic diagram of a judgment logic for employee identification according to an exemplary embodiment of the present application;

FIG. 18 is a schematic diagram of a system architecture for employee identification provided by an exemplary embodiment of the present application;

fig. 19 is a schematic structural diagram of an employee identification apparatus according to an exemplary embodiment of the present application;

fig. 20 is a schematic structural diagram of an electronic device according to an exemplary embodiment of the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the technical solutions of the present application will be clearly and completely described below with reference to specific embodiments of the present application and corresponding drawings. It will be apparent that the described embodiments are only some, but not all, embodiments of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application. The user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, presented data, etc.) related to the present application are information and data authorized by the user or fully authorized by each party, and the collection, use and processing of related data is required to comply with the relevant laws and regulations and standards of the relevant country and region, and is provided with corresponding operation entries for the user to select authorization or rejection.

In an on-line retail scene, in order to realize various analyses based on passenger flows, such as full-field passenger flow analysis, shop passenger flow analysis, floor passenger flow analysis, employee reception conversion rate, customer portrait analysis, entrance and exit analysis, business state passenger flow analysis and the like, accurate passenger flow data are required to be acquired by a merchant. However, since a certain number of internal personnel, namely staff, exist in the business process, the staff can enter and exit for many times, which has a certain negative effect on passenger flow analysis. Therefore, in the passenger flow statistical analysis process, the manufacturer is more often required to identify and reject staff, so that an accurate data basis can be provided for online operation and online digital management more accurately.

In the related art, staff identification is usually realized based on detection analysis of staff clothes, but due to the fact that staff wearing is not uniform, tooling is changed seasonally and the like in an actual scene, color negative interference is strong, accuracy of staff identification is low, and due to the fact that staff clothes are not uniform in different areas and different stores, a large amount of clothes data needs to be stored in the background, and accordingly cost of data updating and maintenance is high. In addition, staff identification in the related art can also be realized by adopting modes such as face recognition and the like, and the accuracy of staff identification is not high due to people flow shielding.

In order to solve the problems, the application provides an employee identification method, device and equipment, wherein electronic equipment acquires user tracks, user behavior events and user characteristics, automatically determines target employees, forms a target employee database of the target employees, subsequently acquires sampling data to be identified of users to be identified, and identifies the target employees based on whether the sampling data to be identified and the target employee data are matched. Therefore, the electronic equipment can perform staff identification based on multiple dimensions, has stronger adaptability, improves the accuracy of staff identification, can dynamically update target staff data based on historical sampling data, and reduces the cost of data update maintenance.

Fig. 1 is a schematic diagram of an application scenario provided in an exemplary embodiment of the present application. As shown in fig. 1, in the related art, when staff identification is performed by using a manufacturer excess, staff in a field is generally determined based on detection and identification of clothing of staff, and the accuracy of the staff identification is not high, and the cost of data update and maintenance is high.

In the embodiment of the application, the electronic equipment analyzes and processes the user track, the user behavior event and the user characteristic based on the historical sampling data to obtain target employee data corresponding to the target employee; and then the electronic equipment acquires the sampling data to be identified of the user to be identified, determines whether the employee to be identified is a target employee according to the matching condition of the sampling data to be identified and the target employee data, realizes accurate identification of the employee, can realize automatic dynamic update of the target employee data, and reduces the cost of data maintenance.

The technical scheme shown in the application is described in detail by specific examples. It should be noted that the following embodiments may exist alone or in combination with each other, and for the same or similar content, the description will not be repeated in different embodiments.

Fig. 2 is a flowchart of an employee identification method according to an exemplary embodiment of the present application. Referring to fig. 2, the employee identification method may include:

s201, determining historical sampling data corresponding to a user according to a video stream acquired by shooting equipment; the historical sample data includes user trajectories, user behavior events, and user characteristics.

The execution subject of the embodiment of the application can be electronic equipment or an employee identification device arranged in the electronic equipment. The employee identification apparatus may be implemented in software or a combination of software and hardware. For ease of understanding, hereinafter, an execution body will be described as an example of an electronic device. The electronic device may specifically refer to a server, a cloud end, a mobile terminal, or the like, which is not limited in the embodiment of the present application.

In the embodiment of the application, the employee identification specifically can be to identify and confirm the identity of the employee through a specific authentication means. The shooting device may be used for video shooting and image acquisition, and may specifically refer to a camera or the like, where the type of the camera may be a network video camera (Internet Protocol Camera, IPC) and capable of sending video data to a server based on a communication network. The video stream may refer to video data collected by a photographing apparatus, and the video stream may include a plurality of image frames therein. A user may refer to a pedestrian included in a video stream, which may include customers and employees. The historical sampling data may refer to various sampling data corresponding to the user, and may specifically include user tracks, user behavior events, user features, and the like. The user track may refer to a moving path formed by user foot points. User behavior events may characterize a particular behavior event of a user, such as entering and exiting a feature region, etc. The user behavior event may specifically include a user area event, a user in-out event (or a user line crossing event), and the like, where the user area event refers to a user area event in which a user enters or leaves a target area, and the target area may be a specific area such as a foreground, an employee rest area, and the like in an off-line retail scene; a user entry and exit event may refer to an event in which a user enters or exits from a store entrance.

The user characteristics may refer to characteristic information corresponding to the user, and specifically may include human body characteristics, human head characteristics, and the like. The body characteristics may include, among other things, the user's body shape, clothing, accessories (e.g., backpacks, jewelry, etc.), and body posture, etc. The head features may include facial features as well as hair features (color, length, density, and texture), head contour features (shape and size of head), and ear features (shape and size), as well as other marking features of head. The facial features may include, among other things, the shape and location of the facial features (eyes, nose, mouth, ears), facial contours (including chin, cheek shapes), skin tone and skin texture, facial expressions, facial markers (e.g., moles, scars), skin texture, and age features (e.g., wrinkles), among others. Of course, the user features may also include other types, which are not limited in this regard by embodiments of the present application.

In this step, the electronic device may acquire a video stream acquired by the capturing device, and perform detection analysis on the video stream to determine historical sampling data corresponding to the user. Specifically, the electronic device may determine user features corresponding to each user based on a preset human body detection algorithm, and then track and detect the same user based on a tracking algorithm to generate a user track; in the tracking process, when a user triggers a behavior event, a corresponding user behavior event may be generated. Therefore, the electronic equipment can determine the historical sampling data corresponding to the user through the detection processing of the video stream, the accuracy and the comprehensiveness of the historical sampling data of different users are ensured, and the accuracy of the subsequent employee identification can be improved.

S202, calculating and analyzing the historical sampling data to determine target employee data corresponding to the target employee.

In the embodiment of the application, the target staff can refer to the user identified as the staff. The target employee data may refer to sampling data corresponding to the target employee, for example, user features of the target employee, and the target employee data may be stored in a target employee database, so as to facilitate subsequent data comparison and matching.

In this step, after the electronic device obtains the historical sampling data of the user, the electronic device may perform calculation processing on the historical sampling data to determine the target employee and the target employee data. Specifically, the electronic device may perform screening and filtering based on the number of tracks, the residence time, and the like, and filter out users belonging to the customer, so as to determine alternative employees; and merging user characteristics of the candidate staff, determining the target staff based on the occurrence frequency in a preset history period (such as in a week), and realizing accurate judgment of the target staff. Of course, the electronic device may determine the target employee data based on other manners, which are not limited by embodiments of the application.

S203, acquiring sampling data to be identified of the user to be identified.

S204, under the condition that the sampling data to be identified is matched with the target employee data, determining that the user to be identified is the target employee.

In the embodiment of the application, the user to be identified can refer to the user needing staff identification in the video stream. The sample data to be identified may refer to sample data corresponding to a user to be identified, and specifically may include a first user feature, a first user behavior event, and a first user track corresponding to the user to be identified. In this step, after determining the target employee data, the electronic device may collect the sample data to be identified of the user to be identified, and then compare and match the sample data to be identified with the target employee data, where the electronic device may determine that the employee to be identified is the target employee. Specifically, the electronic device may compare and match the first user feature of the user to be identified with the target employee user feature in the target employee data, and determine whether the user to be identified is a target employee according to the matching result; in addition, the electronic device can also determine the number of times of entering and exiting of the target staff in the target area or the store and the residence time based on the target staff data, set the judging conditions based on the number of times of entering and exiting and the residence time, and then perform matching judgment on the number of times of entering and exiting of the user to be identified in the target area or the store and the residence time, so as to determine whether the user to be identified is the target staff.

On the basis of the above embodiment, fig. 3 is a schematic diagram of a determining flow of historical sampled data according to an exemplary embodiment of the present application. Referring to fig. 3, the determining of the historical sampling data may specifically include:

S301, based on a preset human body detection model, determining alternative user characteristics corresponding to a user and quality scores corresponding to the alternative user characteristics.

In the embodiment of the application, the preset human body detection model may refer to a preset human body detection algorithm model, which is used for performing feature detection and identification on the image frames of the video stream. The candidate user features may be user features detected by the electronic device based on a preset human body detection model, and the candidate user features may be in the form of feature vectors and represent abstract features of the image. The preset human body detection model can obtain a group of characteristic vectors representing the image structure and content by representing the image matrix as a vector form and then applying the techniques such as characteristic value decomposition or singular value decomposition. The quality score may refer to a quality score corresponding to a user feature, and generally the clearer the user image portion in the video flowsheet frame, the higher the quality score of the alternative user feature. The sharpness of the user image portion of a video stream image frame is affected by light shading, occlusion, etc.

In this step, after the electronic device obtains the video stream collected by the photographing device, the electronic device may analyze the image frames in the video stream frame by frame, determine, based on a preset human detection model, candidate user features of each user in each image frame, and determine a quality score corresponding to the candidate user features, and subsequently may screen the user features based on the candidate user features.

S302, tracking and detecting the same user according to the alternative user characteristics.

S303, after tracking is finished, determining the alternative user characteristics with the quality scores meeting the preset score conditions as user characteristics corresponding to the user; the user features include human body features and human head features.

In the embodiment of the present application, the preset score condition may refer to a preset quality score judgment condition, and specifically may be that the quality score is highest or that the quality score is greater than a preset score threshold value, etc. Specifically, after the electronic device determines the candidate user characteristics and the quality scores of the users in the image frames, the electronic device can perform characteristic comparison judgment based on a tracking algorithm, track and detect the same user, and after tracking is finished (for example, the users cannot be detected in a camera, etc.), the electronic device can screen out the candidate user characteristics with the highest quality score from the candidate user characteristics of the same user in different image frames as the user characteristics, or use one or more candidate user characteristics with the quality scores larger than a preset score threshold as the user characteristics.

In the embodiment of the application, the electronic equipment detects and identifies the image frames of the video stream based on the preset human body detection algorithm, determines the alternative user characteristics and the quality scores of the users, then tracks and detects the same user based on the tracking algorithm such as the characteristic detection and the like, and determines the alternative user characteristics with the quality scores meeting the preset score conditions as the user characteristics corresponding to the users after the tracking is finished, so that the accuracy and the rationality of the user characteristic determination can be improved, the electronic equipment stores the user characteristics based on the form of the characteristic vectors, the image information such as the faces, the fingerprints and the like of the users are not required to be stored, the data leakage risk can be reduced, and the data safety is ensured.

S304, determining foot point information of a user in each image frame of the video stream; the foot point information comprises foot point coordinates and time information; and determining the user track corresponding to the same user according to the foot point information.

In the embodiment of the application, the foot point information can refer to footprint point information or step point information of projection mapping of a user on the ground. The user trajectory may be a path formed by a series of foot points connected. The foot point information may include foot point coordinates, which may refer to pixel coordinates of a user foot point in an image frame, and time information; the time information may refer to a time corresponding to the foot point.

In this step, after the electronic device obtains the video stream, it may detect the foot point coordinates and time information of foot points of each user in the video stream frame, generate the foot point information of the user, then track and detect the same user, and generate the user track corresponding to the user based on the foot point information of the user. Therefore, the electronic equipment can accurately detect the user track by determining the foot point information and further tracking and detecting the foot point information to obtain the user track.

S305, determining actual position information of the user according to the foot point information of the user; and determining a moving path corresponding to the user based on the actual position information.

In the embodiment of the application, the user behavior event can comprise a user area event and a user access event. The actual position information may refer to a spatial position coordinate corresponding to the foot point information of the user in the actual scene of the online store, and may specifically be represented by world coordinates, longitude and latitude, and the like. The movement path may refer to a movement route of the user in an off-line store actual scene.

Specifically, when determining the user area event, the electronic device first ensures that coordinates between the target area and the foot point information are unified, that is, both may perform position identification and determination under the pixel coordinate system, or may perform position identification and determination under the world coordinate system. Both of these methods require a conversion process for the coordinates. The following focuses on the implementation of coordinate conversion in images and actual scenes:

The coordinate transformation involves a camera calibration process, which is to find out quantitative relation of an object by finding out a conversion mathematical relation between the image and the real world, so as to realize the purpose of measuring actual data from the image, and involves parameters such as internal parameters (parameters related to the characteristics of the camera, such as focal length, pixel size, distortion coefficient of the camera, and the like) and external parameters (parameters in a world coordinate system, such as position, rotation direction, and the like) of the camera. In general, coordinate transformation is specifically a homography matrix that needs to be solved from the world coordinate system to the pixel coordinates. The foot point information determined in the image frame adopts pixel coordinates, but in an actual scene, an associated Computer aided design (Computer AIDED DESIGN, CAD) map, namely a world coordinate system in the actual scene, is needed, and a homography matrix between the two needs to be solved at the moment so as to ensure accurate conversion between the two.

Homography matrices can be used to describe the positional mapping of objects between world coordinates and pixel coordinates. The coordinates corresponding to the world coordinate system are real-field coordinates, which refer to the coordinates in the real world, and different interpretation exists in different calibration methods. In an indirect calibration method, a plurality of fixed buildings can be selected as original points, a plurality of fixed buildings are selected as original points in a real field, physical distances (units are millimeters) between the selected original points and the selected original points are measured by using scales in the real field, the horizontal direction is generally taken as the positive direction of an X axis, the vertical direction is taken as the positive direction of a Y axis, and the obtained points are real field coordinates. In the direct calibration method, in some fields with low precision requirements and simpler field topology results, the homography matrix can be calculated by directly using the pixel coordinates and the map coordinates, the real-field coordinates at the moment are coordinates in the map, and the real-field coordinates at the moment can be converted into millimeters because the scale of map coordinates is constant (for example, 1:10000). The origin of coordinates refers to the origin of the reference coordinate system used as the measurement of the real field coordinates in the indirect calibration method. When calibrating a camera, an origin point needs to be selected first. The virtual field coordinates generally refer to pixel coordinates, i.e. coordinates of a certain point in the image, the origin is generally at the upper left corner of the image, the horizontal right direction is the positive X-axis direction, and the vertical downward direction is the positive Y-axis direction.

The camera imaging model may be a simple bore imaging model. In the small bore imaging model, the camera can be abstracted into its simplest form: a small hole and an imaging plane, the small hole being located between the imaging plane and the real three-dimensional scene, any light from the real world reaching the imaging plane only through the small hole. Therefore, there is a correspondence between the imaging plane and the real three-dimensional scene as seen through the aperture, i.e. there is a transformation between the two-dimensional image point in the image and the three-dimensional point of the real three-dimensional world. By finding this transformation relationship, three-dimensional information of the scene can be restored by using the two-dimensional point information in the image. Illustratively, FIG. 4 is a schematic diagram of a pinhole imaging model provided in an exemplary embodiment of the present application. As shown in fig. 4, the imaging plane is placed in front of the aperture and the imaged image is also upright.

In the aperture imaging model, the aperture imaging mode can lead to low imaging brightness of an object only by transmitting light through an aperture part, so that a lens is introduced into a camera, the brightness is adjusted, but due to the manufacturing process of the lens, imaging generates various distortion, and a distortion coefficient (belonging to an internal reference of the camera) can be expressed by mathematics. As described before, one of the purposes of camera calibration is to establish a correspondence of an object from a three-dimensional world to coordinate points on an imaging plane, and the following are specific definitions of several coordinates. The world coordinate system (world coordinate system) is a user-defined three-dimensional world coordinate system, which is introduced to describe the position of the object in the real world, and may be in meters (m). The camera coordinate system (camera coordinate system) may refer to a coordinate system established on the camera, defined for describing the object position from the camera's perspective, as a middle loop communicating the world coordinate system and the image/pixel coordinate system, in m. The image coordinate system (image coordinate system) is introduced for describing the projection transmission relation of the object from the camera coordinate system to the image coordinate system in the imaging process, so that the coordinates in the pixel coordinate system can be conveniently obtained, and the unit is m. The pixel coordinate system (pixel coordinate system) is introduced to describe the coordinates of the image point on the digital image (photo) after the object is imaged, and is the coordinate system where the information actually read from the camera is located, and the unit is number of pixels.

Fig. 5 is a schematic diagram illustrating a conversion relationship between a plurality of coordinate systems according to an exemplary embodiment of the present application. Wherein, the world coordinate system is X _w、Y_w、Z_w, and the origin is O _w. The camera coordinate system is X _c、Y_c、Z_c, and the origin is O _c. The image coordinate system is x and y, and the origin is o. The pixel coordinate system is u, v. The Z _c axis of the camera coordinate system coincides with the optical axis, and is perpendicular to the plane of the image coordinate system and passes through the origin of the image coordinate system, and the distance between the camera coordinate system and the image coordinate system is the focal length f (i.e., the origin of the image coordinate system coincides with the focal point). The pixel coordinate system plane u-v coincides with the image coordinate system plane x-y, but the origin of the pixel coordinate system is located in the upper left corner of the figure, so that reading and writing from the first address of the stored information can be realized. Specifically, based on fig. 5, the following is a conversion process between coordinate systems:

first, from the world coordinate system to the camera coordinate system. The process of converting an object from the world coordinate system to the camera coordinate system can be obtained by rotation and translation, and the transformation matrix corresponding to the conversion process can be represented by a homogeneous coordinate matrix formed by combining a rotation matrix and a translation vector, for example, the following formula (1):

in the above formula (1), R is a rotation matrix, and t is a translation vector, and since it is assumed that the plane in which the object point is located in the world coordinate system passes through the origin of the world coordinate system and is perpendicular to the axis Z _w (i.e. the plane of the chessboard coincides with the plane X _w-Y_w, which can facilitate subsequent calculation), Z _w is 0 and can be directly converted into the form of formula (1). The transformation matrix (including R, t, 0 ₃ ^T, 1) is an extrinsic matrix, which is related to external parameters of the camera only, and changes with the position of the object.

And secondly, from a camera coordinate system to an image coordinate system. The conversion process converts three-dimensional coordinates into two-dimensional coordinates, namely a projection perspective process, wherein the projection perspective process is to project an object onto a projection surface by using a central projection method, so that a single-sided projection chart which is relatively close to a visual effect is obtained, namely an imaging mode which enables a human eye to see the near-far size of a scene. In the case of pinhole imaging, the imaging effect is the same as that of lens imaging except that imaging brightness is low, but the optical path is simpler. Fig. 6 is a schematic diagram of pinhole imaging provided in an exemplary embodiment of the present application. As shown in fig. 6, the pinhole plane (camera coordinate system) is between the image plane (image coordinate system) and the object point plane (checkerboard plane), and the resultant image is an inverted real image.

In order to make the description more convenient mathematically, the positions of the camera coordinate system and the image coordinate system can be exchanged, and fig. 6 is changed to the arrangement mode of fig. 7, and it should be noted that the process has no practical physical meaning, and is only convenient for performing the calculation processing. Becomes the arrangement shown in the following figure. Fig. 7 is a schematic diagram of a pinhole imaging coordinate system position modification according to an exemplary embodiment of the present application. In fig. 7, assuming that there is a point M in the camera coordinate system, the coordinates of the imaging point P in the ideal image coordinate system (without distortion) are (can be derived from the principle of similar triangles):

the above formula is converted into homogeneous coordinate representation form, and the following formula (2) can be obtained, namely, a transformation matrix from a camera coordinate system to an image coordinate system:

Thirdly, from an actual image coordinate system to a pixel coordinate system. Since the origin of the defined pixel coordinate system is not coincident with the origin of the image coordinate system, assuming that the coordinates of the origin of the image coordinate system under the pixel coordinate system are (u ₀,v₀), the dimensions of each pixel point in the x-axis and y-axis directions of the image coordinate system are: d _x、d_y, and the coordinates of the image point in the actual image coordinate system are (x _c,y_c), then the coordinates of the image point in the pixel coordinate system are obtained as follows:

The following formula (3) can be obtained by converting the above formula into the homogeneous coordinate representation:

The (x _p, y_p) in the formula (2) is the same as the (x _c, y_c) in the formula (3), and is the coordinates in the image coordinate system. Multiplying the conversion matrix of the formula (2) and the formula (3) to obtain an internal reference matrix M, wherein each value in the internal reference matrix M is only related to the internal parameters of the camera and does not change along with the position change of an object, and the internal reference matrix M is specifically represented by the following formula (4):

based on the foregoing, fig. 8 is a schematic diagram of coordinate transformation according to an exemplary embodiment of the present application. The transformation matrix between world coordinate system to camera coordinate system, camera coordinate system to image coordinate system, image coordinate system to pixel coordinate system is shown in fig. 8.

On the basis of fig. 8, the coordinate mapping relationship of the final pixel coordinate system and the world coordinate system based on the derivation process of the above formula can be represented by the following formula (5):

In formula (5), u, v denote coordinates in the pixel coordinate system, s denote scale factors, f _x、f_y、u₀、v₀, γ (two coordinate axis deflection parameters due to manufacturing errors are usually small) denote 5 camera references, R, t denote camera references, and X _w、Y_w、Z_w (assuming that the calibration checkerboard is in the plane of Z _w =0 in the world coordinate system) denote coordinates in the world coordinate system.

Based on the above, a homography (Homography) transform can be used to describe the positional mapping of objects between the world coordinate system and the pixel coordinate system, with the transform matrix to which the homography transform corresponds being referred to as the homography matrix. In connection with the above formula (5), the homography matrix may be defined as the following formula (6):

Wherein M is an internal reference matrix, wherein since γ is smaller, it can be approximated as equation (4). Based on the formula (6), it can be known that the homography matrix simultaneously contains camera internal parameters and external parameters, and based on the homography matrix, conversion of image coordinates and a world coordinate system can be realized.

In addition, when in-field calibration, a direct calibration method, a Zhang's calibration method and the like can be adopted, wherein the direct calibration method is that in some fields with relatively simple in-field topological result and required precision, the homography matrix H can be calculated by directly using pixel coordinates and CAD map coordinates. The Zhang's calibration method refers to calculating the homography matrix H from the image plane to the world plane based on four sets of calibration points.

Based on the above, after determining the homography matrix between the world coordinate system and the pixel coordinate system, the electronic device performs coordinate conversion on the foot point coordinates in the foot point information of the user, determines the world coordinates of the user in the actual scene, and obtains the actual position information of the user. The specific determination process may be that the electronic device determines four calibration points based on a CAD map of an actual scene, so as to realize a conversion relationship between pixel coordinates and longitude and latitude coordinates, and specifically may be that the pixel coordinates are converted into mercator coordinates first, then the longitude and latitude coordinates are converted, and so on. Therefore, based on the foot point information of the user, the electronic equipment performs coordinate transformation to obtain the actual position information of the user in the field, and then the actual moving path of the user can be obtained, so that whether the user triggers the user area event or not can be determined based on the spatial relationship between the moving path and the target area.

S306, generating a user area event when the moving path enters or leaves the target area.

In the embodiment of the application, when the moving path of the user in the actual scene leaves or enters the target area from the target area, the electronic equipment can generate the user area event corresponding to the user. The specific position of the target area may be preset by the user, and may correspond to an employee rest area or foreground in the offline store scene, and the like. In this way, the electronic device obtains the actual position information of the user in the actual scene by performing coordinate transformation on the foot point information of the user, further obtains the moving path of the user, and generates the user area event based on the spatial relationship between the moving path and the target area, wherein the user area event can comprise the user area entry and exit type (such as entering the target area and leaving the target area), the user area entry and exit time and the like, so that the accurate and rapid detection of the user area event is realized.

Illustratively, fig. 9 is a schematic diagram of an in-line coordinate transformation for an off-line store scene according to an exemplary embodiment of the present application. As shown in fig. 9, for the check points in the image, the electronic device may determine a homography matrix between the pixel coordinates and the world coordinates based on 4 preset calibration points, and further obtain the world coordinates of the check points based on the pixel coordinates of the check points, so as to implement coordinate conversion between the pixel coordinates and the world coordinates.

Illustratively, fig. 10 is a schematic diagram of a target area in an offline store scene according to an exemplary embodiment of the present application. As shown in fig. 10, the electronic device specifically may be an employee rest area, a foreground, or a non-customer access area, etc. by calibrating the target area 1001 in the actual scene. After acquiring the foot point information of the user, the electronic device obtains the actual position information of the user through coordinate conversion of the foot point information, further obtains a moving path in the user field, and then judges the control relation between the moving path and the target area, when the moving path of the user enters the target area 1001 or leaves the target area 1001, the electronic device can generate a user area event, so that accurate judgment of the user entering or leaving the target area is realized, and an accurate data basis is provided for the subsequent employee identification.

S307, determining a target line segment corresponding to the target access in the video stream.

S308, determining the position relationship between the foot point information of the user and the target line segment according to the user track.

In the embodiment of the present application, the target entrance may be an entrance of an offline store. The target line segment may refer to a reference line segment corresponding to an entrance in an off-line store. The positional relationship may refer to a spatial relationship between the foot point information and the target line segment, such as whether the user foot point is to the left or right of the target line segment, and the like. In this step, the electronic device may define a target line segment at the target entrance and exit, and generate a user entrance and exit event when the user trajectory enters and exits, and then determine a residence time (a time interval between an approach time and an departure time) of the user in the presence based on the user entrance and exit event, so as to determine a time characteristic of the user.

In one possible implementation, step S308 may be specifically implemented by:

Determining the linear distance from the foot point information to the target line segment, and determining a target cross product between the foot point information and the target line segment; and determining the position relationship between the foot point information and the target line segment according to the linear distance and the target cross product.

In the embodiment of the present application, the straight line distance may refer to a straight line distance between a coordinate of a foot point in foot point information of a user and a target line segment, and may be calculated by a distance from the point to the straight line, specifically may be a ratio of a cross product of three points (two end points of the target line segment and the foot point) to a euclidean distance of the target line segment, that is, a projection distance of the point on the straight line is divided by a unit length (a lineated euclidean distance). The target cross product may refer to a vector cross product between the foot point information and the target line segment, which may be used to determine on which side of the target line segment the foot point is.

Specifically, the vector (or vector) cross product can be used to determine whether the point is on a side of a straight line, and thus can be used to determine whether the point is within a triangle, whether two rectangles overlap, and so on. The modulus of the cross product of the vectors represents the area of the parallelogram enclosed by the two vectors. Assuming the vector p= (x 1, y 1), q= (x 2, y 2), the vector cross product is defined as the signed area of one parallelogram, i.e.: p×q=x1×y2—x2×y1, with the result being a pseudo vector. At this time, there are properties p×q= - (q×p) and p× (-Q) = - (p×q). Based on the vector cross product, the electronic device can determine the clockwise-anticlockwise relationship of the two vectors with respect to each other by its sign, i.e. if p×q >0, then P is in the clockwise direction of Q; if P X Q <0, then P is in the counterclockwise direction of Q; if p×q=0, P and Q are collinear, but may be co-directional or reverse. The cross product direction is perpendicular to both vectors that make up the cross product, so the cross product vector is the normal vector of the plane that the two vectors make up. If the vector cross product is a zero vector, then the two vectors are in parallel relationship. Therefore, in the embodiment of the present application, the electronic device may define that the target cross product >0 indicates that the foot point is on the left side of the target line segment, the target cross product <0 indicates that the foot point is on the right side of the target line segment, and the target cross product=0 indicates that the foot point is on the target line segment (i.e., the point falls inside the line segment and the foot of the point is within the scribe line).

Illustratively, fig. 11 is a schematic diagram of a target line segment according to an exemplary embodiment of the present application. As shown in fig. 11, the electronic device may calibrate the target line segment 1101 based on the configuration information of the user, and may subsequently determine whether the user crosses the target line segment 1101 based on the foot point information of the user or the user trajectory composed of the foot point information, so as to determine whether to generate the user entrance/exit event.

Illustratively, fig. 12 is a schematic diagram of a user event entry and exit determination according to an exemplary embodiment of the present application. As shown in fig. 12, the electronic device is calibrated with a target line segment 1201 at a target entrance of a down-line store, and a user track composed of user foot point information is 1202. The electronic device may determine, according to each foot point in the user track 1202, a linear distance between the foot point and the target line segment 1201, and determine a target cross product of the foot point and the target line segment 1201, and then determine, based on the linear distance and the target cross product, a positional relationship between the foot point information of the user and the target line segment 1201, and further determine, based on a change in the positional relationship in the user track 1202, whether a scribing behavior exists for the user, that is, whether a user access event is generated.

Illustratively, table 1 provides a schematic illustration of a process for computing user entry and exit events according to an exemplary embodiment of the present application. The method comprises the following steps:

TABLE 1

As shown in table 1, the user trajectory foot point information is continuously approaching the target line segment, the electronic device may record the point before the line crossing (beforePoint), and when the point-to-line distance exceeds the virtual-to-real line offset, the corresponding value is continuously updated. The current point position is momentPoint; goodMoment-Point represents a "good" Point that is within a virtual-to-real line offset, which is the distance between the dashed line and the solid line (i.e., target line segment 1201) in fig. 12, and the foot Point is plumbed within the scribe line, which may specifically be 20, 30, etc. (in centimeters, etc.). PENDINGSTATE is a preparation state, that is, the foot point information reaches the condition check before the line crossing, and the user track where the foot point information is located can be continuously judged later.

S309, under the condition that the position relation meets a first preset condition, determining that the user has an overline behavior, and generating a user access event.

In the embodiment of the present application, the first preset condition may be that a foot point after the line crossing exists, and the duration of the foot point after the line crossing exceeds a preset time threshold. The preset time threshold is a threshold of the user loitering time, that is, in some cases, the user may loitering at the entrance of the online store and does not actually enter the store, and the preset time threshold may specifically be 1 second, 2 seconds, etc. Of course, the first preset condition may be other types of conditions, and specifically may be flexibly set based on actual requirements, which is not limited in the embodiment of the present application.

In this step, after determining the positional relationship between the foot point information and the target line segment in the user track, the electronic device may determine whether the positional relationship meets a first preset condition, that is, whether there is a foot point after the line crossing, and the duration of the foot point after the line crossing exceeds a preset time threshold, if the first preset condition is met, the electronic device may determine that the user to which the user track belongs has a line crossing behavior, and may generate a user access event correspondingly, where the user access event may specifically include a user line crossing type (for example, from within to outside of the field, from outside of the field to within of the field), a user line crossing time, and so on.

Illustratively, fig. 13 is a schematic diagram of a logic for determining historical sampled data according to an exemplary embodiment of the present application. As shown in fig. 13, the electronic apparatus acquires a video stream acquired by a photographing apparatus (camera or the like), and analyzes the video stream frame by frame. Firstly, the electronic equipment extracts alternative user characteristics of users in image frames based on a preset human body detection algorithm, and determines quality scores corresponding to the alternative user characteristics of each user; then continuously tracking the same pedestrian based on the alternative user characteristics, a tracking algorithm and the like, and determining the foot point information of the user to obtain a user track corresponding to the user; and in the tracking process, the electronic device can determine whether the user has target area in-out behaviors and overline behaviors based on the user track, and correspondingly can determine whether to generate user behavior events (user area events and user in-out events). After the tracking is finished, the electronic device may select, as the user feature, the candidate user feature with the highest quality score based on the quality score of the candidate user feature corresponding to the user. Therefore, the electronic equipment can comprehensively and accurately acquire the historical sampling data based on feature detection, quality score calculation, user tracking and event judgment, and the accuracy of identifying target staff is improved.

It should be understood that, in various embodiments of the present application, the sequence number of each process does not mean that the execution sequence of each process should be determined by its functions and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

On the basis of the above embodiment, fig. 14 is a schematic flow chart of another employee identification method according to an exemplary embodiment of the present application. As shown in fig. 14, the employee identification method includes:

S1401, determining historical sampling data corresponding to a user according to a video stream acquired by shooting equipment; the historical sample data includes user trajectories, user behavior events, and user characteristics.

S1402, filtering and screening each user according to the historical sampling data to obtain alternative employee data.

In the embodiment of the application, the candidate staff data can refer to sampling data which is obtained through preliminary screening and filtering and is possibly staff. After the electronic device acquires the historical sampling data of a plurality of users, the historical sampling data can be merged, counted and analyzed to identify target staff and target staff data. The merging process may refer to a merging process based on pedestrian Re-identification (ReID). When the target employee data is determined, the electronic device may perform preliminary filtering to obtain candidate employees that may be employees, obtain candidate employee data corresponding to the candidate employees, and perform secondary detection and identification on the candidate employee data to identify the target employees and obtain the target employee data.

In one possible implementation, step 1402 may be specifically implemented by the following steps (1) to (3):

(1) And determining the track number and the stay time corresponding to each user according to the historical sampling data.

(2) And screening and filtering the user according to the track number and the residence time to obtain first alternative employee data corresponding to the first alternative employee.

In the embodiment of the application, the track number can refer to the number of the user tracks in the online store scene. The stay time may refer to the time that a user stays within an online store scene, and may be determined based on user entry and exit events. The first alternative employee may refer to intermediate filtering results obtained after the customer is initially filtered.

Specifically, the electronic device may determine the number of tracks and the residence time of each user according to the historical sampling data, and then may perform screening based on the number of tracks and the residence time, where the specific screening condition may be that the number of tracks is greater than a first threshold and the residence time is greater than a second threshold, or that the number of tracks is greater than a third threshold and the residence time is greater than a fourth threshold, where the first threshold is less than the third threshold and the second threshold is greater than the fourth threshold. In general, when an employee is active in a store, it is a case that the employee stays in the store for a long time, such as a cashier, etc., where the user trajectory of the employee is continuous, the number of trajectories is small but the stay time is long; another situation is that the employee needs to go to the store frequently to reach customers, such as shopping guides, etc., and the user track of the employee is discontinuous because the employee enters and exits the store frequently, and the number of tracks of the employee is large but the stay time is not long. The specific value of the threshold may be flexibly set based on the actual requirement, and of course, other filtering conditions may also be adopted, for example, the number of tracks in N consecutive days is greater than the fifth threshold, the residence time is greater than the sixth threshold, etc., where N is 5 days, 7 days, etc., which is not limited in the embodiment of the present application. Therefore, the electronic equipment can filter the sampling data of most customers based on the track number and the stay time to obtain the first alternative staff and the first alternative staff data, and the subsequent calculated amount is reduced.

(3) And merging the first alternative staff according to the first alternative staff characteristics corresponding to the first alternative staff to obtain alternative staff and alternative staff data corresponding to the alternative staff.

In the embodiment of the application, after the first alternative employee data is obtained, the electronic device can merge the first alternative employee based on the user features in the first alternative employee data, namely the first alternative employee features, specifically can filter the first alternative employee features according to the quality score, filter the first alternative employee features with lower quality score, and simultaneously merge two first alternative employee features with higher similarity into one set based on similarity (SIMILARITY SCORE) calculation, so that the number of subsequent user feature comparison is reduced. Finally, the electronic device can select the user characteristics of one candidate staff from each set to store, and obtain the candidate staff and candidate staff data, wherein the candidate staff data can comprise the user characteristics of the candidate staff and the characteristic sampling time. In this way, the electronic equipment performs preliminary filtering and screening on the historical sampling data to obtain the candidate staff and the candidate staff data corresponding to the candidate staff, so that the efficiency and accuracy of target staff identification can be improved.

Illustratively, fig. 15 is a schematic diagram of determining logic of alternative employee data according to an exemplary embodiment of the present application. As shown in fig. 15, the electronic device determines, according to the historical sampling data, a residence time and a track number of the user in the presence, and filters sampling data of most customers based on the residence time and the track number to obtain a first candidate employee and first candidate employee data. For first alternative employee characteristics in the first alternative employee data, first filtering out first alternative employee characteristics with low quality scores based on quality scores, then comparing the remaining first alternative employee characteristics in pairs, combining two first alternative employee characteristics with higher similarity (for example, greater than 0.9) into one set, and finally selecting one first alternative employee characteristic from each set by the electronic equipment to put in storage to obtain alternative employees and alternative employee data.

S1403, determining target employee data corresponding to the target employee according to the occurrence frequency of the candidate employee data corresponding to the preset history period.

In the embodiment of the present application, the preset history period may refer to a preset time period, and specifically may be 5 days, 7 days, and the like. The frequency of occurrence may refer to the number of days that the alternative employee data continuously appears. After determining the candidate employee data, the electronic device may conduct accurate identification of the target employee based on whether the candidate employee data appears for a plurality of consecutive days.

In one possible embodiment, step S1403 may be specifically implemented by the following steps (4) to (7):

(4) And acquiring the user characteristics of the alternative staff corresponding to the alternative staff in the preset history period.

(5) And merging the user characteristics of the alternative staff according to the similarity among the user characteristics of the alternative staff, and determining the sampling time corresponding to each user characteristic of the alternative staff.

In the embodiment of the application, the electronic equipment can acquire the user characteristics of the alternative staff corresponding to the alternative staff in the preset history period, and particularly can be the head characteristics and the like. And then, the electronic equipment can perform pairwise similarity calculation on the user features of the alternative staff, combine the user features of the alternative staff with higher similarity into a set, and determine the sampling time of the user features of each alternative staff in the set.

(6) And determining the occurrence frequency of the alternative staff in a preset history period according to the sampling time.

(7) And determining the candidate staff as target staff for each candidate staff under the condition that the occurrence frequency meets a second preset condition, and determining the candidate staff data corresponding to the candidate staff as target staff data.

In the embodiment of the present application, the second preset condition may mean that the candidate staff appears in the store for 5 consecutive days (or 7 days, etc.). According to each sampling time in the user characteristic set of the candidate staff of the same candidate staff, the electronic device can determine the occurrence frequency of the candidate staff in the preset history period, if the occurrence frequency meets a second preset condition, for example, the candidate staff appears in a store for 5 consecutive days, the electronic device can determine that the candidate staff is a target staff, and can determine that the candidate staff data corresponding to the candidate staff is target staff data. In this way, the electronic equipment accurately identifies the target staff based on the occurrence frequency of the candidate staff in the preset history period, and the accuracy of identifying the target staff can be improved.

Illustratively, fig. 16 is a schematic diagram of determining logic of target employee data according to an exemplary embodiment of the present application. As shown in fig. 16, the electronic device may perform task scheduling calculations at daily timings, for example, may obtain candidate employee user characteristics for all candidate employees within a preset history period, e.g., 7 days, at 9a morning of each day; then the electronic device can calculate based on the similarity, and combine the user characteristics of the alternative staff with the similarity higher than a certain threshold (for example, 0.8, 0.9, etc., representing the same person) to form an alternative staff user characteristic set corresponding to the alternative staff, and can determine the sampling time of each characteristic in the set; and then, the electronic device can determine the appearance frequency of the candidate staff based on the sampling time, and based on whether the appearance frequency meets a second preset condition, such as whether the appearance frequency appears for 5 consecutive days, if so, the electronic device can determine the candidate staff as a target staff, and can use the candidate staff data of the candidate staff as target staff data to form a feature database corresponding to the target staff.

S1404, acquiring sampling data to be identified of a user to be identified.

In the embodiment of the application, the sampling data to be identified comprises a first user characteristic, a first user behavior event and a first user track corresponding to the user to be identified. After determining the target employee data of the target employee, the electronic device may collect sample data to be identified of the user to be identified, and subsequently may determine whether the user to be identified is the target employee based on a matching condition between the sample data to be identified and the target employee data.

S1405, if the first user characteristics are matched with the target employee user characteristics in the target employee data, determining that the user to be identified is a target employee.

In the embodiment of the application, the electronic equipment can firstly identify and match the user characteristics. Specifically, if the similarity between the first user feature of the user to be identified and a certain target employee user feature in the target employee data is higher than a preset first similarity threshold, for example, 0.8, 0.9, etc., the electronic device may directly determine that the user feature matches, where the user to be identified is a target employee.

In another possible implementation manner, if the similarity between the first user feature of the user to be identified and a certain target employee user feature in the target employee data is higher than a preset second similarity threshold but lower than the preset first similarity threshold, the preset second similarity threshold is 0.5, 0.6, or the like, that is, the user feature of the user to be identified may be matched with the target employee user feature at this time, the electronic device may sample the first user track to obtain a number a of first user features, where the number a may be 10 or 20, or the like, and then the electronic device may calculate a number ratio of the a number of first user features, where the similarity between the first user feature and the target employee user feature is greater than the preset second similarity threshold, and if the number ratio is greater than the preset ratio threshold, the electronic device may determine that the user feature of the user to be identified is matched with the target employee user feature, and the user to be identified is the target employee. In an exemplary embodiment, among the a first user features of the first user track, there are B first user features having a similarity with the target employee user feature greater than a preset second similarity threshold, where the electronic device may determine whether the B/a is greater than a preset ratio threshold, and if so, the electronic device may determine that the user to be identified is the target employee.

In the embodiment of the application, the electronic equipment performs employee identification based on the user characteristics, wherein the user characteristics can comprise human body characteristics, head characteristics and the like, the biological unique characteristics such as faces and the like are not used, the identification accuracy is higher, the user characteristics have long-term performance, the information such as faces, fingerprints and the like are not required to be stored, and the data security risk is reduced.

S1406, determining the number of times of entering and exiting of a target area and the residence time of the target area corresponding to a user to be identified according to a first user behavior event; and if the number of times of entering and exiting the target area and the residence time of the target area meet a third preset condition, determining the user to be identified as a target employee.

In the embodiment of the application, the number of times of entering and exiting the target area may refer to the number of times that the user to be identified enters and exits the target area in the store. The target area stay time may refer to the number of stays of the user to be identified in the store target area. The third preset condition may mean that the number of times the target area is entered and exited is greater than a preset number of times threshold, and the target area residence time is greater than a first preset duration, etc. The preset time threshold and the first preset time period may be determined based on the average number of times of entering and exiting of the target employee in the target area in the target employee data, the average residence time period, and the like, or may be determined based on other manners, which is not limited in the embodiment of the present application.

In this step, the electronic device may also determine whether the user to be identified is a target employee according to the number of times the user to be identified enters and exits in the target area and the residence time. The electronic device may determine a third preset condition according to the target employee data, for example, determine the third preset condition based on an average number of times of entering and exiting of the target employee in the target area, an average residence time, and the like, then the electronic device may determine the number of times of entering and exiting of the target area and a residence time of the target area corresponding to the user to be identified based on the user area event in the first user behavior event of the user to be identified, and if the number of times of entering and exiting of the target area and the residence time of the target area meet the third preset condition, the electronic device may determine that the user to be identified is the target employee. Therefore, besides the user characteristic matching, the electronic equipment can perform employee identification based on the in-and-out times and the residence time of the target area in the field, and the accuracy and the comprehensiveness of employee identification can be improved.

S1407, determining the number of tracks and the on-site residence time corresponding to the user to be identified according to the first user track and the first user behavior event; and if the track number and the in-field residence time meet the fourth preset condition, determining the user to be identified as a target employee.

In the embodiment of the application, the track number corresponding to the user to be identified can refer to the total track number of the user to be identified in the presence. The on-site dwell time may refer to the total time that the user to be identified remains in the site. The fourth preset condition may be that the number of tracks is greater than a preset number threshold and the in-field residence time is greater than a second preset duration, where the preset number threshold and the second preset duration may be determined based on the average number of tracks of the target employee and the in-field average residence time, and may be determined based on other manners, which is not limited in this embodiment of the present application.

In this step, the electronic device may also determine whether the employee to be identified is a target employee based on the number of tracks of the user to be identified in the store and the in-store residence time. The electronic device may determine the fourth preset condition from the target employee data, e.g., may determine the fourth preset condition based on an average number of trajectories of the target employee in the field and an average residence time in the field. Then, the electronic device may determine, based on the user area event in the first user behavior event, the number of in-field tracks and the in-field residence time corresponding to the user to be identified, and if the number of in-field tracks and the in-field residence time meet a fourth preset condition, the electronic device may determine that the user to be identified is a target employee. In this way, besides the user characteristic matching and the target area space and time characteristic matching, the electronic equipment can perform staff identification according to the number of tracks in the field of the user to be identified and the stay time in the field, so that the accuracy and the comprehensiveness of staff identification can be further improved.

It should be noted that when the electronic device performs employee identification, three matching modes of user feature matching, target area space-time feature matching and field space-time feature matching may have a precedence relationship, that is, priority is from high to low, or three types of matching may be performed simultaneously, specifically, may be determined based on the processing capability of the electronic device, and the embodiment of the present application is not limited to this.

Fig. 17 is a schematic diagram illustrating a judgment logic of employee identification according to an exemplary embodiment of the present application. As shown in fig. 17, after the electronic device performs REID merging on the collected user to be identified to obtain the sample data to be identified, the electronic device may first match the first user feature of the user to be identified with the target employee user feature, and if the similarity between the first user feature and the target employee user feature is greater than a preset first similarity threshold, or if the number ratio of the similarity between the first user feature and the target employee user feature is greater than a preset second similarity threshold is greater than a preset ratio threshold, the electronic device may determine that the first user feature of the user to be identified matches the target employee user feature, where the user to be identified is a target employee.

If the first user characteristic of the user to be identified is not matched with the user characteristic of the target employee, the electronic device can further determine the number of times of entering and exiting the target area and the target area residence time corresponding to the user to be identified according to the user area event in the first user behavior event, and if the number of times of entering and exiting the target area and the target area residence time meet a third preset condition, the electronic device can determine that the user to be identified is the target employee.

If the third preset condition is not met, the electronic equipment can further determine the number of the on-site tracks and the on-site residence time corresponding to the user to be identified according to the user access event in the first user behavior event, and if the number of the on-site tracks and the on-site residence time meet the fourth preset condition, the electronic equipment can determine that the user to be identified is a target employee; if not, the electronic device may determine that the employee to be identified is not a target employee. Of course, in some scenarios, the electronic device may also perform auxiliary matching recognition in combination with human body features such as a tool of the user to be recognized, so as to further improve accuracy of employee recognition, which is not limited by the embodiment of the present application.

Based on any one of the above embodiments, fig. 18 is a schematic diagram of a system architecture for employee identification according to an exemplary embodiment of the present application. As shown in fig. 18, the system architecture of employee identification may include a camera IPC, an edge server, a network management cluster, a Message Queue (MQ) middleware cluster, a cloud micro-service cluster, a database, an REID server, an offline data computing platform, and the like. Wherein, 0.0 represents that the shooting equipment (namely, the camera IPC) is connected to the platform of the Internet of things through the triplet login registration, so as to construct a communication channel. Wherein, the triplet may refer to a unique identifier of a Device in the internet of things, and generally includes three elements, namely a Device ID (Device ID), a product ID (Product ID), and a Device key (DEVICE SECRET). The device ID is a unique identifier of the device, the product ID is a unique identifier of a product to which the device belongs, and the device key is a key for encrypted communication between the device and the server. Through the device triplets, the internet of things platform can accurately identify and manage each device and ensure the safety and reliability of communication between the devices.

0.1 Indicates that the cloud micro service application cluster can be configured via message queue telemetry transport (Message Queuing Telemetry Transport, MQTT) or hypertext transfer protocol (Hypertext Transfer Protocol, HTTP) channels, such as for on-site calibration information, on-site digitized information (map coordinate information), etc. And 0.2, the edge computing server can report data through the MQTT or HTTP channel to process the server, such as MQTT reporting sampling data, HTTP uploading pictures, and the like. And 0.3, the gateway can forward to the cloud micro-service cluster for processing after receiving the request.

And 1.1, carrying out calculation processing by the edge server from the camera pull stream, calling an algorithm model to identify and adding analysis and calculation logic, and obtaining historical sampling data of a user. 1.2 shows that the edge server reports various sampling data including historical sampling data, sampling data to be identified and the like through the MQTT channel. And 1.3, performing service asynchronous buffer processing through the MQ middleware, and forwarding to cloud micro-service cluster processing. 1.4, 1.5 and 1.6 represent that the cloud micro service cluster performs data processing and asynchronously stores the sampled data for REID merging. 1.7 and 1.8 represent REID pulling sampling data, REID merging and data persistence and library falling. 1.9, 1.10 represent offline data computing clusters making some data analysis and synchronizing the results to the database.

In the related art, a part of the business process or a store recognizes employees based on clothing worn by the employees and clothing features of the clothing. In the employee identification mode, as a plurality of chain branches may exist in different areas of a brand, the dressing requirements of each store are not uniform and are changed in four seasons, a model is required to be retrained based on garment identification, samples and data are required to be collected for each training, time is consumed, the cost is high, part of the garment is completely black/pure and has no obvious attribute, the garment cannot be distinguished from customer garments, and the accuracy of employee identification is low.

In addition, in the related art, part of the business process or the store is to perform image recognition based on Gaussian probability distribution of the personnel image data, and determine the personnel image data. The employee identification mode is sensitive to abnormal values, the abnormal data interference is large, and the accuracy is low; and the hypothetical data of gaussian distributions are symmetrical, unimodal, and continuously distributed, while many data in the real world do not meet these assumptions. If the distribution shape of the data does not match the Gaussian distribution, distortion of the modeling and estimation results of the image data may be caused; in addition, the image recognition needs to be pre-learning processing, and in an actual scene, the scene that staff wears non-uniformly and the like is existed, so that color interference is easy to be caused, and the stability of staff recognition is not high. In addition, other staff identification modes based on face identification and fingerprint identification need to be stored, privacy leakage risks exist, update and maintenance costs are high, and identification accuracy is low.

In the embodiment of the application, the system architecture for staff identification fully utilizes the edge computing capability and the engineering capability of the server. And when the edge calculation is utilized, the relation between a large number of foot points and overlines is calculated (used for judging that people enter a store and exit the store), the calculation quantity on the cloud and the server pressure are reduced, the personnel track REID is integrated (which tracks are judged to be the same person and are also called REID in the industry) by utilizing the data processing capacity of the server, the rapid calculation of the data is realized, and the adaptability is higher. In space analysis, the embodiment of the application determines the track and the relative space position of the user in the field according to the foot point information (foot point coordinates and the like), and identifies whether the personnel are in target areas such as an employee rest area, a foreground and the like for a long time or not, so as to perform space feature auxiliary identification; from time analysis, the electronic equipment makes staff identification auxiliary judgment from time characteristics according to the residence time of the user in the presence and the residence time of the user in the target area. On the aspect of algorithm characteristics, the electronic equipment performs matching identification according to relatively stable user characteristics such as head characteristics, and the like, so that the database data has long-term property and higher identification accuracy. On the basis of maintaining the bottom library cost, the engineering end capability is relied on, the features of the bottom library in the field are automatically built at the server end, manual maintenance is not needed, dynamic update can be automatically realized, and the method has higher accuracy and effectiveness.

In the embodiment of the application, the electronic equipment does not depend on staff tools for identification, and the user characteristics such as head characteristics and the like are reliably identified, so that negative effects such as color interference and the like are avoided, and the applicability is strong; and the track behavior characteristics of the staff, including time characteristics and space characteristics, are fully utilized, and the accuracy of staff identification is improved. In addition, the electronic equipment does not use the biological unique characteristics such as human face and the like for identification, so that the data security risk is reduced. The staff identification process of the embodiment of the application has the advantages of simple realization, less calculation amount, flexible realization, no need of training mass feature models, full utilization of the calculation capability of the edge end and the data processing advantage of the server end, support of mass data processing on the architecture and wider adaptability.

Fig. 19 is a schematic structural diagram of an employee identifying apparatus according to an exemplary embodiment of the present application, please refer to fig. 19, the employee identifying apparatus 190 includes:

A first determining module 1901, configured to determine historical sampling data corresponding to a user according to a video stream acquired by a capturing device; the historical sampling data comprises user tracks, user behavior events and user characteristics;

the second determining module 1902 is configured to perform calculation analysis on the historical sampling data, and determine target employee data corresponding to a target employee;

An acquisition module 1903, configured to acquire sampling data to be identified of a user to be identified;

a third determining module 1904 is configured to determine that the user to be identified is a target employee if the sample data to be identified matches the target employee data.

In one possible implementation, the first determining module 1901 is specifically configured to:

determining foot point information of a user in each image frame of the video stream; the foot point information comprises foot point coordinates and time information;

determining actual position information of a user according to foot point information of the user;

in the case where the movement path enters or leaves the target area, a user area event is generated.

In one possible implementation, the user behavior event includes a user entry and exit event; the first determining module 1901 is specifically configured to:

And under the condition that the position relation meets a first preset condition, determining that the user has an overline behavior, and generating a user access event.

After tracking is finished, determining the alternative user characteristics with the quality scores meeting the preset score conditions as user characteristics corresponding to the user; the user features include human body features and human head features.

In one possible implementation, the second determining module 1902 is specifically configured to:

screening and filtering the users according to the track number and the residence time to obtain first alternative employee data corresponding to the first alternative employees;

acquiring user characteristics of alternative staff corresponding to the alternative staff in a preset history period;

And determining that each candidate staff is a target staff when the occurrence frequency meets a second preset condition, and determining candidate staff data corresponding to the candidate staff as target staff data.

In one possible implementation manner, the sampling data to be identified includes a first user characteristic, a first user behavior event and a first user track corresponding to the user to be identified; the third determining module 1903 is specifically configured to:

Determining the number of times of entering and exiting of a target area corresponding to a user to be identified and the residence time of the target area according to a first user behavior event;

determining the number of the tracks in the field and the residence time in the field corresponding to the user to be identified according to the first user track and the first user behavior event;

And if the number of tracks in the field and the residence time in the field meet a fourth preset condition, determining the user to be identified as a target employee.

The employee identifying device 190 provided in the embodiment of the present application may execute the technical scheme shown in the above method embodiment, and its implementation principle and beneficial effects are similar, and will not be described herein again.

Fig. 20 is a schematic structural diagram of an electronic device according to an exemplary embodiment of the present application, referring to fig. 20, the electronic device 200 may include a processor 2001 and a memory 2002. The processor 2001 and the memory 2002 are connected to each other by a bus 2003, for example.

Memory 2002 stores computer-executable instructions;

Processor 2001 executes computer-executable instructions stored in memory 2002, causing processor 2001 to perform the employee identification method as described in the method embodiments above.

Accordingly, an embodiment of the present application provides a computer-readable storage medium having stored therein computer-executable instructions for implementing the employee identification method of the above-described method embodiment when the computer-executable instructions are executed by a processor.

Accordingly, embodiments of the present application may also provide a computer program product, including a computer program, which, when executed by a processor, may implement the employee identification method shown in the foregoing method embodiments.

It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In one typical configuration, a computing device includes one or more processors, input/output interfaces, network interfaces, and memory.

The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.

Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.

The foregoing is merely exemplary of the present application and is not intended to limit the present application. Various modifications and variations of the present application will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. which come within the spirit and principles of the application are to be included in the scope of the claims of the present application.

Claims

1. A method for identifying employees, comprising:

acquiring sampling data to be identified of a user to be identified;

2. The method according to claim 1, wherein determining historical sampling data corresponding to each user according to the video stream collected by the photographing device comprises:

3. The method of claim 1, wherein the user behavior event comprises a user region event; according to the video stream collected by the shooting equipment, the historical sampling data corresponding to each user is determined, and the method comprises the following steps:

4. The method of claim 1, wherein the user behavior event comprises a user entry and exit event; according to the video stream collected by the shooting equipment, the historical sampling data corresponding to each user is determined, and the method comprises the following steps:

5. The method of claim 4, wherein determining the positional relationship between the user's foot point information and the target line segment comprises:

6. The method according to claim 1, wherein determining historical sampling data corresponding to each user according to the video stream collected by the photographing device comprises:

7. The method of claim 1, wherein said computationally analyzing the historical sampled data to determine target employee data comprises:

8. The method of claim 7, wherein said filtering the individual users based on the historical sample data to obtain alternative employee data comprises:

9. A method as defined in claim 7, wherein the determining the target employee data corresponding to the target employee according to the frequency of occurrence of the candidate employee data corresponding to the preset history period includes:

10. The method according to any one of claims 1 to 9, wherein the sample data to be identified comprises a first user feature, a first user behavior event, and a first user trajectory corresponding to the user to be identified; and under the condition that the sampling data to be identified is matched with the target employee data, determining that the user to be identified is a target employee comprises the following steps:

11. An employee identification apparatus, comprising:

12. An electronic device, comprising: a memory and a processor;

The memory stores computer-executable instructions;

The processor executing computer-executable instructions stored in the memory causes the processor to perform a method of identifying an employee as claimed in any one of claims 1 to 10.

13. A computer readable storage medium having stored therein computer executable instructions for implementing the employee identification method of any of claims 1 to 10 when the computer executable instructions are executed by a processor.

14. A computer program product comprising a computer program which, when executed by a computer, implements a method of identifying staff as claimed in any one of claims 1 to 10.