CN115116130A

CN115116130A - Call action recognition method, device, equipment and storage medium

Info

Publication number: CN115116130A
Application number: CN202210661996.5A
Authority: CN
Inventors: 姜楠; 聂磊
Original assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Current assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Priority date: 2022-06-13
Filing date: 2022-06-13
Publication date: 2022-09-27

Abstract

The present disclosure provides a call action recognition method, device, equipment and storage medium, which relate to the technical field of artificial intelligence, and in particular to the technical fields of image processing, deep learning, computer vision, and the like. The specific implementation scheme is as follows: carrying out region detection on an image to be detected to obtain a target image region where a person is located; matching the target image area with motion trail information to obtain a target motion trail matched with the target image area, wherein the motion trail information is as follows: information describing the motion trail of the person is determined according to the region of the same person in the image; carrying out call action recognition on the target image area to obtain a recognition result; under the condition that the recognition result represents and recognizes the call action, updating the number of times of recognizing the call action corresponding to the target motion track; and determining whether the call action really exists according to the updated times. By applying the scheme, the accuracy of the call action recognition can be improved.

Description

Call action recognition method, device, equipment and storage medium

Technical Field

The present disclosure relates to the field of artificial intelligence technology, and more particularly, to the field of image processing, deep learning, and computer vision.

Background

In a work scene such as a biochemical enterprise, if a person uses a mobile phone to communicate, a fire may occur, and normal operation of electrons in the work scene may be affected. Therefore, it is necessary to identify the call action of the person in the work scene so as to find the call action of the person in the work scene in time and prevent the call action.

Disclosure of Invention

The disclosure provides a call action recognition method, a call action recognition device, equipment and a storage medium.

In one aspect of the present disclosure, a method for recognizing a call action is provided, including:

carrying out region detection on an image to be detected to obtain a target image region where a person is located;

matching the target image area with motion trail information to obtain a target motion trail matched with the target image area, wherein the motion trail information is as follows: information describing the motion trail of the person is determined according to the region of the same person in the image;

carrying out call action recognition on the target image area to obtain a recognition result;

under the condition that the recognition result represents and recognizes the call action, updating the number of times of recognizing the call action corresponding to the target motion track;

and determining whether the call action really exists according to the updated times.

In another aspect of the present disclosure, a call action recognition apparatus is provided, including:

the image area obtaining module is used for carrying out area detection on the image to be detected to obtain a target image area where a person is located;

a motion track obtaining module, configured to match the target image area with information of a motion track, to obtain a target motion track matched with the target image area, where the information of the motion track is: information describing the motion trail of the person is determined according to the region of the same person in the image;

the recognition result obtaining module is used for carrying out call action recognition on the target image area to obtain a recognition result;

the action frequency updating module is used for updating the frequency of the identified call action corresponding to the target motion track under the condition that the call action is identified by the identification result representation;

and the call action determining module is used for determining whether the call action really exists according to the updated times.

In still another aspect of the present disclosure, there is provided an electronic device including:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the call action recognition method.

In still another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the above-mentioned call motion recognition method.

In yet another aspect of the present disclosure, a computer program product is provided, which includes a computer program, and the computer program realizes the above-mentioned call action recognition method when being executed by a processor.

As can be seen from the above, in the solution provided in the embodiment of the present disclosure, after the target image area where the person is located is detected, the target motion trajectory of the person in the target image area is determined by matching the motion trajectories, so that when a call action is identified in the target image area, counting based on the identification result can be performed according to different persons, and then, the counting result is the count of the call actions detected for the same person in the multi-frame image, where the larger the count is, the larger the number of times of the call actions detected for the person is, the higher the probability that the person is actually in a call is, and therefore, the counting result can more accurately reflect whether the person in the target image area is in a call. Therefore, the scheme provided by the embodiment of the disclosure can successfully detect whether the person really has the call action, and can improve the accuracy of detecting the call action.

It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present disclosure, nor do they limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.

Drawings

The drawings are included to provide a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:

FIG. 1 is a schematic diagram of a scenario provided by an embodiment of the present disclosure;

fig. 2 is a schematic flow chart of a first call action recognition method according to an embodiment of the present disclosure;

fig. 3 is a schematic flow chart of a first motion trajectory matching method provided by the embodiment of the present disclosure;

fig. 4 is a flowchart illustrating a second call action recognition method according to an embodiment of the disclosure;

fig. 5 is a schematic flow chart of a second motion trajectory matching method provided by the embodiment of the present disclosure;

fig. 6 is a schematic structural diagram of a call action recognition device according to an embodiment of the present disclosure;

fig. 7 is a block diagram of an electronic device for implementing a call action recognition method provided by an embodiment of the present disclosure.

Detailed Description

Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which various details of the embodiments of the disclosure are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

An application scenario of the embodiment of the present disclosure is explained below.

Referring to fig. 1, a shooting place is a work place. In the scheme provided by the implementation of the disclosure, image acquisition equipment such as a camera can be arranged around the workplace in advance, and the workplace is shot to obtain images. The image acquisition device can continuously take images at a plurality of moments in time within a period of time, and fig. 1 shows the images at 16:16 moments in time after taking.

The scheme provided by the embodiment of the disclosure can be used for carrying out region detection on all the shot images to obtain the image region where the person is located in the image, and refer to the image region where the square frame is located in fig. 1. By performing call recognition on the image area, whether a person in the image area is calling can be determined.

The execution main body of the embodiment of the present disclosure may be an electronic device in communication connection with the image capturing device, and the electronic device may be capable of receiving an image captured by the image capturing device and performing call action recognition based on the received image, for example, the electronic device may be a server, a desktop computer, or the like.

The following describes the call action recognition method according to the embodiment of the present disclosure in detail by using a specific embodiment.

In one embodiment of the present disclosure, referring to fig. 2, a flowchart of a first call action recognition method is provided, which includes the following steps S201 to S205.

Step S201: and carrying out region detection on the image to be detected to obtain a target image region where the person is located.

The target image area is determined by the position of the person in the image to be detected. The target image region may include a partial region of the person, for example, a head-shoulder region of the person, or may include the entire region of the person.

Two implementations of region detection to obtain the target image region are described below.

In one implementation, the image to be detected may be input into a preset detection model to obtain a target image region including a person. The detection model may be a target detection model obtained by training in advance by using a sample image of an area where the tagged person is located, and specifically, the target detection model may be a target detection model based on yolov5, a target detection model based on yolo3D, or the like.

In another implementation mode, the image characteristics of the image to be detected can be extracted, and then the person in the image to be recognized is detected according to the extracted image characteristics, so that the target image area is determined. For example, after the image features are extracted, a connected component in the image may be determined according to the image features, and then an area containing the connected component may be used as a target image area containing a person.

Step S202: and matching the target image area with the motion track information to obtain a target motion track matched with the target image area.

The motion trail of the person represents the motion state of the person and can be determined according to a position sequence formed by the positions of the persons in the multi-frame images.

Wherein, the motion trail information is: and determining information describing the motion state of the person according to the regions of the same person in different images. Each motion track corresponds to the motion state of the same person, and different motion tracks correspond to the motion states of different persons.

Because image acquisition equipment is constantly gathering the image, above-mentioned electronic equipment carries out the regional discernment in personage place to each image that image acquisition equipment gathered, and then along with the image is constantly gathered, can form different movement track based on the personage place region that discerns, and each movement track can be along with the constantly gathering and constantly updating of image, above-mentioned movement track is: the motion trail formed according to the previous image obtained by shooting prior to the current image to be detected is the motion trail obtained when the region of the image to be detected is detected. Namely, the motion trail mentioned in each step in the embodiments of the present disclosure is the obtained motion trail.

The target motion trajectory refers to a motion trajectory matched with the target image area. The matching relationship between the motion trail and the target image area can be determined by the position relationship between the target image area and the motion trail. For example, if the target image area is located on the motion trajectory a or the target image area is close to the motion trajectory a, the target image area is considered to match the motion trajectory a, and the motion trajectory a is a target motion trajectory matching the target image area.

When the position relationship between the target image area and the motion trail is judged, the motion trail can be adopted to predict the area position where the person appears in the image to be detected, and the area position is compared with the area position of the target image area to determine whether the target image area is matched with the motion trail. The specific implementation manner can be seen in the following embodiment shown in fig. 3, and will not be described in detail here.

In another case, when the position relationship between the target image area and the motion track is determined, a distance threshold may be preset, and a pixel point may be selected in the target image area, and if the distance between the pixel point and the motion track is smaller than the preset distance threshold, it is determined that the target image area is close to the motion track, so as to determine that the target image area is matched with the motion track.

Step S203: and carrying out call action recognition on the target image area to obtain a recognition result.

In an embodiment of the present disclosure, the target image area may be input into a preset classification model to obtain an action type of a person in the target image area, and the call action recognition is implemented by determining whether the obtained action type belongs to a call action type. Specifically, the classification model may be a network model obtained by training in advance using a sample image labeled with an action type, and the network model may be a Resnet50 model, a MobileNet model, or the like. In this case, the accuracy of obtaining the action type result can be improved through iterative training of the network model.

In another embodiment of the present disclosure, the human body key point positions of the person, including the hand position, the mouth position, and the like, may be acquired from the target image area. According to the relation between the positions of the key points, whether the person makes a conversation gesture can be determined, and therefore whether the person is in conversation can be judged.

The identification result obtained after the call identification comprises the following steps: and characterizing the characterization information identifying the call action or not identifying the call action. In addition, the recognition result may include: probability representing the accuracy of the representation information, rectangular region position representing the position of the target object region, and the like.

Step S204: and under the condition that the conversation action is identified by the identification result representation, updating the number of times of identifying the conversation action corresponding to the target motion track.

In one embodiment of the present disclosure, the number of times the call action is recognized is updated, i.e., the value of the number of times is increased. For example, the initial value of the number of recognized call actions corresponding to the target motion trajectory may be set to 0, and for a plurality of frames of images to be detected, the number of times of the target motion trajectory matching the target image area where the call action is recognized may be increased by 1 every time there is one frame of image to be detected in which the call action is recognized as the recognition result obtained by performing call action recognition according to the foregoing steps.

In another embodiment of the present disclosure, when the above times are updated, the image quality of the image to be detected may also be considered, and the specific implementation manner is as follows:

under the condition that the call action is identified by the identification result representation, carrying out image quality detection on the image to be detected to obtain a quality representation value for describing the quality of the image to be detected; and if the quality characterization value is larger than a preset characterization threshold value, updating the number of times of recognizing the call action corresponding to the target motion track.

The recognition result represents and recognizes the call action, namely, the recognition result contains representation information representing and recognizing the call action.

The quality characteristic value can be represented by the definition degree of the image to be detected, and the definition degree of the image to be detected can be determined based on whether the image to be detected has the conditions of blurring, backlight and the like. If the obtained definition degree is higher, the corresponding quality characterization value is larger, otherwise, the quality characterization value is smaller.

Or, the image to be detected may be input into a preset quality classification model to obtain a quality characterization value output by the quality classification model. The quality classification model can be a network model which is trained by adopting sample images with labeled quality types in advance, and the network model can be a Resnet50 model, a MobileNet model and the like. When the sample image is labeled, the sample image can be labeled into a high-quality type and a low-quality type, wherein the high-quality type is represented by a quality representation value with a larger numerical value, and the low-quality type is represented by a quality representation value with a smaller numerical value, so that the trained quality classification model can output the quality representation value serving as a quality classification result when performing quality classification on the image to be detected.

The characterization threshold is a preset value. And the quality characterization value is greater than the characterization threshold value, which indicates that the image quality of the image to be detected is higher, and in this case, the number of times of recognizing the call action corresponding to the target motion track is updated.

As can be seen from the above, in the scheme provided in this embodiment, when the number of times of the call action is updated and identified, counting is performed based on the identification result of the to-be-detected image with a quality characteristic value greater than the characteristic threshold, that is, with a higher image quality. The image information used for identifying the image to be detected with higher image quality during the action identification is more accurate, correspondingly, the accuracy of the identification result and the counting result can be improved, and the times of false alarm caused by the call action at the false detection position in the image to be detected are reduced.

Moreover, the image to be detected is screened through image quality, so that the scheme provided by the embodiment of the disclosure can be suitable for scenes in which low-quality images are easy to obtain, such as scenes with violent change of light, scenes blocked or not clearly seen by a mobile phone, and the like, and has better universality.

Step S205: and determining whether the call action really exists according to the updated times.

The more times of the call action are identified, namely the more images to be detected with the call action as the identification result, the more image information of the images to be detected can be understood to determine that the call action exists, the more the reference image information is sufficient, and the more accurate the call action is judged to exist.

Specifically, whether the number of times of the identified call action in the first number of frames of images which are continuously collected recently reaches a preset number of times can be judged according to the updated number of times; if so, judging that the conversation action really exists; otherwise, judging that the call action does not really exist.

The first number of frames of images may be a plurality of frames of images to be detected, which are continuously acquired and captured by the image acquisition device for a plurality of times within a recent short period of time, for example, within 1 minute and within 30 seconds. For each image to be detected, the number of times of recognizing the call action corresponding to the target motion trajectory can be updated according to the foregoing steps S201 to S204. In this case, the number of times of recognizing the call action, which is obtained by accumulating each motion trajectory, may be obtained for comparison with a preset number of times, and if the preset number of times is reached, it is determined that the call action actually exists, otherwise, the call action does not exist.

The preset number of times may be determined according to the number of the first number of frame images, for example, the preset number of times may be 80% of the number, that is, when 5 frames of the first number of frame images are provided, at least 4 frames of the first number of frame images recognize a call, and it is determined that there is a call for a person in the image to be detected.

In this case, the images to be detected with a higher proportion in the first number of frame images are judged to have a call action, and because the first number of frame images are continuously acquired images, the time interval of image acquisition is shorter, and the action of the person in a short time is relatively stable, so that when the call action is identified, if the identification results of a sufficient number indicate that the person has a call in the latest time, the probability that the person has the call action can be considered to be higher, and the accuracy of the call action identification is further improved.

A specific implementation of matching the target image area and the motion trajectory in step S202 is described below.

In an embodiment of the present disclosure, referring to fig. 3, in the foregoing step S202, the target image area may be respectively matched with the information of each motion trajectory according to the following steps S301 to S304.

Step S301: and predicting the motion state parameters of the person corresponding to the motion trail according to the state prediction parameters corresponding to the motion trail.

The motion trajectory may be obtained from a previous image captured prior to the current image to be detected. For example, a preset image acquisition device a may be used to continuously shoot a work site to obtain a group of images to be detected with shooting times in sequence, where the current image to be detected is the nth image of the group of images to be detected arranged in sequence according to the shooting time, and then the first N-1 images are the previous images.

The state prediction parameters may be used to predict a motion state at a later time from a motion state at a previous time on the motion trajectory. Specifically, the state prediction parameter may be obtained according to a kalman filter coefficient, a corresponding time update function and a measurement update function in the kalman filter are configured, and the parameter of the motion state at the previous time is used as an input value and input to the kalman filter to obtain the predicted parameter of the motion state. The time updating function is used for obtaining a predicted value representing the motion state at the later moment according to the parameter of the motion state at the previous moment, and the measurement updating function is used for correcting the obtained predicted value to obtain the parameter of the motion state.

The parameter of the motion state is an attribute parameter indicating the motion state of the person, and may include a velocity, an acceleration, and the like of the motion of the person.

Step S302: and determining a candidate image area of the person corresponding to the motion track in the image to be detected according to the motion state parameters.

After the motion state parameters are obtained, the positions of the people in the image to be detected, which move to, can be obtained according to the positions of the people in the motion trail at the moment before the corresponding moment of the image to be detected and the motion process represented by the motion state parameters; in this case, the image region including the position in the image to be detected is taken as a candidate image region, that is, a region in which a person may appear in the image to be detected.

Step S303: and extracting the regional characteristics of the target image region.

In an embodiment of the present disclosure, a feature vector corresponding to a feature describing the image may be obtained as a region feature according to a color, a texture, and a shape of the image in the target image region.

Step S304: and if the region characteristics are matched with the character characteristics of the character corresponding to the motion track and the position of the target image region is matched with the position of the determined candidate image region, determining the motion track as the target motion track matched with the target image region.

The character features of the character corresponding to the motion trajectory may be determined according to the first character features extracted from the region where the character is in the image used when the motion trajectory is formed.

The character features can be obtained by a classification network model adopted by a reID (Re-Identification) technology, the network model adopts the ID (Identification) of the character as a label, and the network model is trained to extract the character features of the corresponding character through images. The obtained character features have an association relation with the ID of the character corresponding to the motion trail. In this case, the ID of the person corresponding to the movement trajectory may be determined by the similarity between the first person feature and the person feature extracted by the classification network model, and the person feature corresponding to the ID may be obtained based on the ID.

In this case, the matching of the motion trajectory may be performed in the case where there is a person who disappears in a part of the image to be detected, for example, a part of the person may walk in capturing the image to be detected, thus walking out of the capturing range, or the like. In this case, the character feature can still be confirmed by reID in the above process and used for matching with the region feature.

In one embodiment of the present disclosure, whether the region feature matches the character feature may be determined based on a relationship between a first feature vector representing the region feature and a second feature vector representing the character feature. Specifically, the relationship between the first feature vector and the second feature vector may be represented by a distance or a cosine similarity between the first feature vector and the second feature vector, which is not limited in the embodiments of the present disclosure.

As can be seen from the above, in the scheme provided by the embodiment of the present disclosure, when the target image region is matched with the motion trajectory, not only the region characteristics are matched with the person characteristics of the person corresponding to the motion trajectory, but also the matching relationship between the positions of the target image region and the candidate image region is referred to, and whether the target image region is matched with the motion trajectory is determined by using various information, so that the reference information is sufficient, and the obtained matching result is more accurate.

In one implementation, in step S304, if the determination of the target motion trajectory fails, that is, the position of the target image region does not match the position of the candidate image region determined by any motion trajectory, or the region feature does not match the person feature of the person corresponding to any motion trajectory, the target motion trajectory may be determined from the motion trajectories according to the intersection ratio between the target image region and the candidate image region corresponding to each motion trajectory.

The intersection ratio is used to indicate the degree of coincidence between the target image region and the candidate image region, and the higher the degree of coincidence, the higher the probability that the target image region matches the candidate image region, i.e., the higher the probability that the target image region matches the motion trajectory of the prediction candidate image region. Specifically, a ratio threshold may be preset, and if the intersection ratio between the target image area and the candidate image corresponding to the motion trajectory X reaches the ratio threshold, it is determined that the target image area is matched with the motion trajectory X. In this case, even if the determination of the trajectory fails, the target region and the motion trajectory can be matched more accurately by the intersection and comparison between the target image region and the candidate image region.

In step S304, after the target motion trajectory is obtained, the state prediction parameters corresponding to the target motion trajectory may be updated according to the target image area.

Different state prediction parameters correspond to different updating modes. For example, when the state prediction parameter is a parameter obtained according to a kalman filter coefficient, the predicted value and the measured value may be fused to obtain a fusion value, and an updated kalman filter coefficient may be obtained according to the fusion value, so as to obtain an updated state prediction parameter.

The predicted value is: a parameter indicative of a predicted location of the candidate image region; the measured values are: parameter indicating position of target image area

Therefore, the state prediction parameters are updated, so that the updated state prediction parameters can accord with the actual motion state of the person corresponding to the target image area in the image to be detected, the accuracy of the state prediction parameters is improved, and when the state prediction parameters are used for carrying out prediction again, the accuracy of the prediction result is correspondingly improved.

In an embodiment of the present disclosure, matching the target image area with the information of the motion trajectory to obtain a target motion trajectory matched with the target image area includes:

determining the available motion trail which is successfully matched with the image area for the continuous latest preset number of times from the motion trail; and matching the target image area with the information of the available motion trail to obtain a target motion trail matched with the target image area.

In this embodiment, when the motion trajectory is used for matching, a motion trajectory that is successfully matched with an image region based on a preset number of times in a continuously and recently obtained image to be detected is used, that is, the available motion trajectory is used. In a long time, a person may completely leave an area photographed by the image capturing device, for example, a worker may leave a work place with the image capturing device during work, and an effective motion trajectory cannot be obtained according to a photographed image, so that an image to be detected continuously photographed in a recent time period is required to be an acquisition source of the motion trajectory. On the contrary, if the motion track is matched with the image area successfully for the latest preset number of times, the person corresponding to the motion track is always in the shot area, sufficient image information can be obtained when the person is subjected to call action recognition, and the recognition result is accurate.

The matching is performed in the same manner as the previous step in the embodiment of fig. 3, except that the names of the motion trajectory and the motion trajectory may be conceptually replaced, and are not described in detail here. In this case, the motion trajectory is successfully matched with a considerable number of image regions, which indicates that the obtained motion trajectory conforms to the motion state of the person described by a considerable number of image information, and improves the accuracy of the obtained motion trajectory.

The overall flow of the call recognition method according to the embodiment of the present disclosure is described below with reference to fig. 4 and 5.

In fig. 4, the head and shoulder area of the human body is a target image area obtained by detecting an image to be detected; the Resnet head and shoulder classification shows that the Resnet model identifies the call action of the target image area to obtain an identification result; the Resnet quality model is used for judging whether the image to be detected is a high-quality image; the accumulated identification result indicates that the number of times of call operation identified in the identification result of the high-quality image to be detected is updated. And voting to judge whether the human body plays a mobile phone or not, namely determining whether the character really has a communication action or not according to the updated times.

And the Deep Sort human head and shoulder tracking is used for matching the target image area with the obtained motion trail and confirming the specific person to which the identification result belongs. The embodiment of fig. 5 is a specific matching method.

Referring to fig. 5, the kalman filter prediction is used to obtain motion state parameters of a person corresponding to the motion trajectory and predict the candidate image region, and the specific implementation manner is referred to the foregoing steps S301 to S302, and is not described in detail here.

Confirming that the motion trail corresponding to the Kalman filtering is an available motion trail; if the motion trajectory is not available, the prediction is not performed using the motion trajectory.

The available motion trajectory is used for performing cascade matching with a target image region obtained by head and shoulder detection, and the specific implementation manner refers to the steps in the foregoing fig. 3 embodiment, which are not described in detail here.

And tracking on matching indicates that a target motion track matched with the target image area exists, and in this case, Kalman filtering updating is carried out, namely state prediction parameters corresponding to the target motion track are updated according to the target image area.

Tracking on a mismatch and detection on a mismatch indicate failure to determine the target motion trajectory. And tracking on unmatched state, wherein the target image area is not matched with any motion trail. The detection on the unmatched state indicates that there is a motion track unmatched with any target image area, for example, if 3 target image areas corresponding to the person object are detected in the image a to be detected, but 4 motion tracks are obtained, at least one motion track is not matched with any target image area. In this case and in the case of a non-available motion trajectory of the motion trajectory, matching is performed in the manner of cross-over ratio in the foregoing embodiment, so as to obtain a target motion trajectory, that is, an IOU Match (cross-over Union Match) in the graph.

Corresponding to the call action recognition method, the embodiment of the disclosure also provides a call action recognition device.

In an embodiment of the present disclosure, referring to fig. 6, a schematic structural diagram of a call action recognition device is provided, where the device includes:

an image area obtaining module 601, configured to perform area detection on an image to be detected to obtain a target image area where a person is located;

a motion trajectory obtaining module 602, configured to match the target image area with information of a motion trajectory to obtain a target motion trajectory matched with the target image area, where the information of the motion trajectory is: information describing the motion trail of the person is determined according to the region of the same person in the image;

a recognition result obtaining module 603, configured to perform call action recognition on the target image area to obtain a recognition result;

an action number updating module 604, configured to update the number of times of recognizing the call action corresponding to the target motion trajectory when the recognition result represents that the call action is recognized;

and a call action determining module 605, configured to determine whether a call action actually exists according to the updated number.

In an embodiment of the present disclosure, the action number updating module 604 is specifically configured to, when the recognition result represents that a call action is recognized, perform image quality detection on the image to be detected to obtain a quality representation value for describing quality of the image to be detected; and if the quality characterization value is larger than a preset characterization threshold value, updating the number of times of recognizing the call action corresponding to the target motion track.

As can be seen from the above, in the scheme provided in this embodiment, when the number of times of the call action is updated and identified, counting is performed based on the identification result of the to-be-detected image with a quality characteristic value greater than the characteristic threshold, that is, with a higher image quality. The image information used for identification of the image to be detected with higher image quality during action identification is more accurate, correspondingly, the accuracy of the identification result and the counting result can be improved, and the times of false alarm caused by the call action at the false detection position in the image to be detected are reduced.

Moreover, the image to be detected is screened based on the image quality, so that the scheme provided by the embodiment of the disclosure can be suitable for scenes in which low-quality images are easy to obtain, such as scenes with violent change of light, scenes in which a mobile phone is shielded or cannot be seen clearly, and the like, and has better universality.

In an embodiment of the present disclosure, the motion trajectory obtaining module 602 is specifically configured to match the target image area with information of each motion trajectory respectively according to the following manners: predicting the motion state parameters of the figure corresponding to the motion trail according to the state prediction parameters corresponding to the motion trail; determining a candidate image area of a person corresponding to the motion track appearing in the image to be detected according to the motion state parameter; extracting the regional characteristics of the target image region; and if the region characteristics are matched with the character characteristics of the character corresponding to the motion track and the position of the target image region is matched with the position of the determined candidate image region, determining the motion track as the target motion track matched with the target image region.

In an embodiment of the present disclosure, in a case that determining the target motion trajectory fails, the apparatus further includes:

and the motion track determining module is used for determining the target motion track from the motion tracks according to the intersection ratio between the target image area and the candidate image area corresponding to each motion track.

In this case, even if the determination of the trajectory fails, the target region and the motion trajectory can be matched more accurately by the degree of coincidence between the target image region and the candidate image region.

In an embodiment of the present disclosure, after obtaining the target motion trajectory, the apparatus further includes:

and the parameter updating module is used for updating the state prediction parameters corresponding to the target motion trail according to the target image area.

Therefore, the state prediction parameters are updated, so that the updated state prediction parameters can accord with the actual motion state of the corresponding person in the target image area in the image to be detected, the accuracy of the state prediction parameters is improved, and when the state prediction parameters are used for carrying out prediction again, the accuracy of the prediction result is correspondingly improved.

In an embodiment of the present disclosure, the motion trajectory obtaining module 602 is specifically configured to determine, from the motion trajectories, an available motion trajectory successfully matched with the image region for the consecutive nearest preset number of times; and matching the target image area with the information of the available motion trail to obtain a target motion trail matched with the target image area.

In this case, the motion trajectory is successfully matched with a considerable number of image regions, which indicates that the obtained motion trajectory conforms to the motion state of the person described by a considerable number of image information, and improves the accuracy of the obtained motion trajectory.

In an embodiment of the present disclosure, the call action determining module 605 is specifically configured to determine whether the number of times of call actions identified in the first number of frames of images that are continuously collected recently has a preset number of times according to the updated number of times; if yes, judging that the call action really exists; otherwise, judging that the call action does not really exist.

Under the condition, the image to be detected reaching a high proportion in the first number of frame images judges that a call action exists, the first number of frame images are continuously acquired images, the time interval of image acquisition is short, the action of the person in a short time is relatively stable, when the person is identified, if the identification results of enough number all indicate that the person has a call in the latest time, the probability that the person has the call action is high based on the consistency of the identification result pair, and the accuracy of call action identification is further improved.

In the technical scheme of the disclosure, the processes of collecting, storing, using, processing, transmitting, providing, disclosing and the like of the related person image information all conform to the regulations of related laws and regulations and do not violate the good custom of the public order.

The present disclosure also provides an electronic device, a readable storage medium, and a computer program product according to embodiments of the present disclosure.

In one embodiment of the present disclosure, there is provided an electronic device including:

at least one processor; and

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the call action recognition method of any of the preceding embodiments.

In an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute a call motion recognition method according to any one of the preceding embodiments is provided.

In an embodiment of the present disclosure, a computer program product is provided, which includes a computer program, and when being executed by a processor, the computer program implements the call action recognition method according to any one of the foregoing embodiments.

FIG. 7 illustrates a schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the disclosure described and/or claimed herein.

As shown in fig. 7, the device 700 comprises a computing unit 701 which may perform various suitable actions and processes according to a computer program stored in a Read Only Memory (ROM)702 or a computer program loaded from a storage unit 708 into a Random Access Memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other by a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.

Various components in the device 700 are connected to the I/O interface 705, including: an input unit 706 such as a keyboard, a mouse, or the like; an output unit 707 such as various types of displays, speakers, and the like; a storage unit 708 such as a magnetic disk, optical disk, or the like; and a communication unit 709 such as a network card, modem, wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.

Computing unit 701 may be a variety of general purpose and/or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various dedicated Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, and so forth. The calculation unit 701 executes the respective methods and processes described above, such as a call action recognition method. For example, in some embodiments, the call action recognition method may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of a computer program may be loaded onto and/or installed onto device 700 via ROM 702 and/or communications unit 709. When loaded into RAM 703 and executed by the computing unit 701, may perform one or more steps of the above described call action recognition method. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the call action recognition method in any other suitable manner (e.g., by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), system on a chip (SOCs), Complex Programmable Logic Devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server of a distributed system, or a server with a combined blockchain.

It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present disclosure may be executed in parallel or sequentially or in different orders, and are not limited herein as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved.

The above detailed description should not be construed as limiting the scope of the disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A call action recognition method comprises the following steps:

matching the target image area with motion trail information to obtain a target motion trail matched with the target image area, wherein the motion trail information is as follows: determining information describing the motion trail of the same person according to the region of the same person in the image;

under the condition that the conversation action is identified by the identification result representation, updating the number of times of identifying the conversation action corresponding to the target motion track;

2. The method according to claim 1, wherein the updating the number of times of the recognized call action corresponding to the target motion trajectory when the recognition result represents that the call action is recognized comprises:

under the condition that the recognition result represents and recognizes the call action, carrying out image quality detection on the image to be detected to obtain a quality representation value for describing the quality of the image to be detected;

and if the quality characterization value is larger than a preset characterization threshold value, updating the number of times of recognizing the call action corresponding to the target motion track.

3. The method according to claim 1 or 2, wherein the matching the target image area with motion trajectory information to obtain a target motion trajectory matched with the target image area comprises:

respectively matching the target image area with each piece of motion trail information according to the following modes:

predicting the motion state parameters of the figure corresponding to the motion trail according to the state prediction parameters corresponding to the motion trail;

determining a candidate image area of a person corresponding to the motion track appearing in the image to be detected according to the motion state parameter;

extracting the regional characteristics of the target image region;

and if the region characteristics are matched with the character characteristics of the character corresponding to the motion track and the position of the target image region is matched with the position of the determined candidate image region, determining the motion track as the target motion track matched with the target image region.

4. The method of claim 3, in the event that determining the target motion trajectory fails, further comprising:

and determining the target motion track from the motion tracks according to the intersection ratio between the target image area and the candidate image area corresponding to each motion track.

5. The method of claim 3, further comprising, after obtaining the target motion trajectory:

and updating the state prediction parameters corresponding to the target motion trail according to the target image area.

6. The method according to claim 1 or 2, wherein the matching the target image area with the information of the motion trajectory to obtain the target motion trajectory matched with the target image area comprises:

determining the available motion trail which is successfully matched with the image area for the continuous latest preset number of times from the motion trail;

and matching the target image area with the information of the available motion trail to obtain a target motion trail matched with the target image area.

7. The method of claim 1 or 2, wherein the determining whether a call action actually exists according to the updated number of times comprises:

judging whether the number of times of the call action identified in the first number of frames of images which are continuously collected recently is preset according to the updated number of times;

if so, judging that the conversation action really exists;

otherwise, judging that the call action does not really exist.

8. A call action recognition device comprising:

9. The apparatus of claim 8, wherein,

the action frequency updating module is specifically used for carrying out image quality detection on the image to be detected under the condition that the recognition result represents and recognizes the call action, so as to obtain a quality representation value for describing the quality of the image to be detected; and if the quality characterization value is larger than a preset characterization threshold value, updating the number of times of recognizing the call action corresponding to the target motion track.

10. The apparatus of claim 8 or 9,

the motion trail obtaining module is specifically configured to match the target image area with information of each motion trail respectively according to the following modes: predicting the motion state parameters of the figure corresponding to the motion trail according to the state prediction parameters corresponding to the motion trail; determining a candidate image area of a person corresponding to the motion track appearing in the image to be detected according to the motion state parameter; extracting the regional characteristics of the target image region; and if the region characteristics are matched with the character characteristics of the character corresponding to the motion track and the position of the target image region is matched with the position of the determined candidate image region, determining the motion track as the target motion track matched with the target image region.

11. The apparatus of claim 10, in case of failure to determine the target motion trajectory, further comprising:

and the motion track determining module is used for determining the target motion track from the motion tracks according to the intersection ratio between the target image area and the candidate image areas corresponding to the motion tracks.

12. The apparatus of claim 10, after obtaining the target motion trajectory, further comprising:

13. The apparatus of claim 8 or 9,

the motion track obtaining module is specifically used for determining an available motion track which is successfully matched with the image area for the most recent preset number of times from the motion tracks; and matching the target image area with the information of the available motion trail to obtain a target motion trail matched with the target image area.

14. The apparatus of claim 8 or 9,

the call action determining module is specifically used for judging whether the number of times of call actions identified in the first number of frames of images which are continuously collected recently is preset according to the updated number of times; if so, judging that the conversation action really exists; otherwise, judging that the call action does not really exist.

15. An electronic device, comprising:

at least one processor; and

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

16. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any one of claims 1-7.

17. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-7.