CN112132118B - Character relation recognition method and device, electronic equipment and computer storage medium - Google Patents
Character relation recognition method and device, electronic equipment and computer storage medium Download PDFInfo
- Publication number
- CN112132118B CN112132118B CN202011317975.9A CN202011317975A CN112132118B CN 112132118 B CN112132118 B CN 112132118B CN 202011317975 A CN202011317975 A CN 202011317975A CN 112132118 B CN112132118 B CN 112132118B
- Authority
- CN
- China
- Prior art keywords
- character
- information
- determining
- role
- person
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
Abstract
The embodiment of the application provides a person relation identification method, a person relation identification device, electronic equipment and a computer storage medium, which are applied to the technical field of image processing, and the person relation identification method comprises the following steps: carrying out scene recognition on an image to be recognized containing at least two characters to obtain a scene recognition result; respectively extracting the characteristics of at least two persons in the image to be recognized, and determining the personal information of the at least two persons based on the extracted characteristics of the persons; determining character distribution information corresponding to the character information according to the character information and the scene recognition result; wherein the role distribution information is used for indicating the probability that each character belongs to a plurality of roles; and determining the character relationship among the characters according to the character distribution information. The method and the device can improve the accuracy of character relation recognition.
Description
Technical Field
The embodiment of the application relates to the technical field of image processing, in particular to a person relation identification method, a person relation identification device, an electronic device and a computer storage medium.
Background
In daily work and life, the relationship between people is also called social relationship intricacy and complexity, and comprises the following steps: brothers, sisters, parents and children, teachers and students, colleagues, leadership and the like. At present, social relationship identification has wide application in various scenes such as character relationship determination, social event understanding and the like.
One way of social relationship recognition is based on image recognition to recognize social relationships from images and to better understand the behavior, emotion, etc. of people. In the related art, the social relationship between persons can be predicted by performing face recognition and analysis on images including the persons through a pre-trained model. However, the accuracy of the social relationship predicted by the method is low.
Disclosure of Invention
In view of the above, embodiments of the present application provide a human relationship identification scheme, so as to overcome at least some of the problems in the prior art.
According to a first aspect of the embodiments of the present application, there is provided a person relationship identification method, including:
carrying out scene recognition on an image to be recognized containing at least two characters to obtain a scene recognition result;
respectively extracting the characteristics of at least two persons in the image to be recognized, and determining the person information of the at least two persons based on the extracted person characteristics;
determining role distribution information corresponding to the character information according to the character information and the scene identification result; wherein the character distribution information is used for indicating the probability that each character belongs to a plurality of characters;
and determining the character relationship among the characters according to the character distribution information.
According to a second aspect of the embodiments of the present application, there is provided a human relationship recognition apparatus including:
the scene recognition module is used for carrying out scene recognition on the image to be recognized containing at least two characters to obtain a scene recognition result;
the figure information determining module is used for respectively extracting the features of at least two figures in the image to be recognized and determining the figure information of the at least two figures based on the extracted figure features;
the role distribution information determining module is used for determining role distribution information corresponding to the character information according to the character information and the scene recognition result; wherein the character distribution information is used for indicating the probability that each character belongs to a plurality of characters;
and the person relationship determining module is used for determining the person relationship among the persons according to the role distribution information.
According to a third aspect of embodiments of the present application, there is provided an electronic apparatus, including: a processor; and a memory configured to store computer executable instructions that, when executed, cause the processor to implement the method of the first aspect described above.
According to a fourth aspect of embodiments herein, there is provided a computer storage medium storing computer-executable instructions that, when executed, implement the method of the first aspect described above.
In the embodiment of the application, because the roles of the characters are usually different in different scenes, the character relationships between the characters are usually different. Therefore, the scene where the person is located is identified from the image, and the characteristics of the person are extracted, so that the person information in multiple dimensions, such as gender, age, posture and the like, can be obtained. In this way, the personal relationship between the persons is determined based on the character distribution information corresponding to the personal information of the plurality of dimensions, and the accuracy of the determined personal relationship can be improved.
Drawings
Some specific embodiments of the present application will be described in detail hereinafter by way of illustration and not limitation with reference to the accompanying drawings. The same reference numbers in the drawings identify the same or similar elements or components. Those skilled in the art will appreciate that the drawings are not necessarily drawn to scale. In the drawings:
FIG. 1 is a flowchart of a method for identifying relationships between persons according to an embodiment of the present application;
FIG. 2 is a flowchart illustrating a method for identifying relationships between persons according to an embodiment of the present application;
FIG. 3 is a diagram of an image to be recognized according to an embodiment of the present application;
FIG. 4 is a schematic diagram of a person relationship identification method according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of a human relationship identification apparatus according to an embodiment of the present application;
fig. 6 is a schematic structural diagram of an electronic device in an embodiment of the present application.
Detailed Description
The following further describes specific implementation of the embodiments of the present invention with reference to the drawings.
Referring to fig. 1, fig. 1 is a flowchart of a method for identifying a person relationship in an embodiment of the present application, which may include the following steps:
step S110, carrying out scene recognition on the image to be recognized containing at least two characters to obtain a scene recognition result.
In the embodiment of the application, the image to be recognized may be any image including at least two persons, so that the relationship between the at least two persons in the image to be recognized may be recognized. Because the character relations are different under different scenes, the image to be recognized can be subjected to scene recognition to obtain a scene recognition result. Scene recognition may be used to identify the physical location where the person is located, and the scene recognition result may be, for example, a classroom, a playground, an office, and the like. Therefore, more accurate character relations can be obtained according to the scene recognition result and by combining character characteristics.
Step S120, respectively extracting characteristics of at least two persons in the image to be recognized, and determining the person information of the at least two persons based on the extracted characteristics of the persons.
Specifically, for each person in the image, the person may typically include a variety of person features, such as facial features, behavioral features, pose features, and the like. By analyzing the character characteristics of each character, various kinds of character information of the character can be obtained. For example, the gender, age, face orientation and posture (e.g., standing, sitting, walking, etc.) of the person, etc. may be obtained.
Step S130, determining character distribution information corresponding to the personal information according to the personal information and the scene recognition result.
In the embodiment of the application, the character can be determined to possibly belong to a plurality of roles according to the scene recognition result, and then the character information is further combined, so that the character distribution information of the character belonging to the plurality of roles can be predicted, wherein the character distribution information is used for indicating the probability of each character belonging to the plurality of roles. For example, in a certain scenario, a person may belong to role a, role B or role C, and the probability that the person belongs to role a, role B or role C is usually different according to different persona information of the same person. That is, the character distribution information corresponding to different pieces of personal information is usually different.
The plurality of characters corresponding to each character may be the same as or different from the plurality of characters corresponding to other characters. For example, the image to be recognized may include a first character and a second character, where the first character corresponds to three characters, namely a character a, a character B, and a character C, the second character corresponds to three characters, namely a character a, a character B, and a character C, and the second character also corresponds to four characters, namely a character B, a character C, a character D, and a character E. Accordingly, the character distribution information of the person one and the character division information of the person two may be different.
In an implementation manner of the application, a preset scene relation matrix corresponding to character information can be determined according to the character information and a scene recognition result; wherein, presetting the scene relation matrix comprises: and the role distribution information respectively corresponds to a plurality of different preset personal information. The preset personal information may be personal information preset according to an actual scene, and the preset personal information may be a certain information or a range. For example, the preset personal information may be a plurality of different age groups when the personal information is age, and the preset personal information may be standing posture, sitting posture, or the like when the personal information is posture.
After the preset scene relationship matrix is obtained, the role distribution information corresponding to the character information in the preset scene relationship matrix can be determined. Specifically, preset personal information corresponding to the personal information may be determined, and the role distribution information corresponding to the preset personal information may be used as the role distribution information corresponding to the personal information.
For example, the preset personal information in the preset scene relationship matrix is a plurality of different age groups, the plurality of roles are respectively role a, role B, role C and role D, in the first age group, the probabilities of belonging to role a, role B, role C and role D are corresponding, in the second age group, the probabilities of belonging to role a, role B, role C and role D are also corresponding, and so on. If the personal information of the person is age and belongs to age group one, the probability that the person belongs to role a, role B, role C and role D, that is, the role distribution information corresponding to the personal information can be obtained.
Step S140, determining the relationship between the persons according to the character distribution information.
As is apparent from the above description, different pieces of personal information each correspond to character distribution information, and the character distribution information indicates the probability that each person belongs to a plurality of characters. Therefore, for each person, the character distribution information corresponding to the plurality of pieces of personal information of the person is analyzed to identify the character of the person. Then, the human relationship among the human characters is determined according to the roles of the human characters.
According to the character relation identification method, due to the fact that the characters are different in roles generally and the character relations between the characters are different generally in different scenes. Therefore, the scene where the person is located is identified from the image, and the characteristics of the person are extracted, so that the person information in multiple dimensions, such as gender, age, posture and the like, can be obtained. In this way, the personal relationship is specified based on the character distribution information corresponding to the personal information of a plurality of dimensions, and the accuracy of the specified personal relationship can be improved.
Referring to fig. 2, fig. 2 is a flowchart of a method for identifying a relationship between persons in the embodiment of the present application, which may include the following steps:
step S210, carrying out scene recognition on the image to be recognized containing at least two characters to obtain a scene recognition result.
This step is the same as step S110 in the embodiment of fig. 1, and specific reference may be made to the description in step S110, which is not described herein again.
Step S220, performing object recognition on the image to be recognized, and obtaining information of an object in the image to be recognized.
In the embodiment of the application, the object in the image to be recognized can be recognized, and the information of the object in the image to be recognized can be obtained through perception and recognition of the object and the environment of the three-dimensional world. Alternatively, the object in the image to be recognized may be recognized, and the category of the object may be determined. For example, for an image to be recognized in which the scene is a classroom, objects such as desks, seats, windows, blackboards, and lecture tables can be recognized by object detection techniques. For the image to be identified with the scene of the basketball court, a basketball stand and the like can be identified.
Additionally, a distance intersection ratio between objects may also be determined, and/or a distance intersection ratio between a person and an object may be determined. In target detection, the intersection ratio is an algorithm for calculating the mutual overlapping ratio of different images, and refers to the ratio of the intersection and union of two rectangles (or other shapes), and the value of the intersection ratio is between [0 and 1 ]. When the intersection ratio is equal to 0, the two rectangular frames are not intersected; when the intersection ratio equals 1, the prediction box coincides with the truth box. The distance intersection ratio takes the overlapping degree, the distance and the scale between the two frames into consideration, and the distance of the graph is considered on the basis of perfecting the calculation function of the image overlapping degree. The larger the distance intersection ratio between two objects, the closer the two objects are, the more overlapping areas. Similarly, the larger the intersection ratio of the distance between the person and the object, the closer the person and the object are, the more the overlapping area is.
In step S230, the scene recognition result is corrected based on the object information, so as to obtain a corrected scene recognition result.
It should be noted that after the information of the object is determined, it can be determined whether the information of the object matches the scene recognition result, and if so, it indicates that the scene recognition result is accurate; if the scene identification result is not matched with the scene identification result, the scene identification result may be corrected, so as to obtain a corrected scene identification result.
Specifically, on the basis of step S220, the scene recognition result may be corrected based on the category of the objects and the distance intersection ratio between the objects. Alternatively, the scene recognition result is corrected based on the category of the object and the distance intersection ratio between the person and the object. Or, the scene recognition result is corrected based on the category of the objects, the distance intersection ratio between the objects, and the distance intersection ratio between the person and the object. Thus, a more accurate scene recognition result can be obtained. It should be noted that, the scene recognition result may be directly modified, or the scene recognition result may be further supplemented, and the like.
For example, if the scene recognition result of the image to be recognized obtained in step S210 is a library, and the class of the object in the image to be recognized includes a cash register, the scene recognition result may be considered as an error in the library, and the scene recognition result may be corrected to be a bookstore.
Step S240, respectively performing feature extraction on at least two people in the image to be recognized, and determining the person information of the at least two people based on the extracted person features.
In general, a person in an image may contain various features, such as attribute features and behavior features, and the attribute features include: gender and age, etc., and behavioral characteristics including posture characteristics, etc. Therefore, the facial features of at least two persons in the image to be recognized can be extracted respectively, and the attribute information of the persons can be confirmed based on the extracted facial features. For example, the attribute information of the person may include: gender, age, and face orientation, etc. Or, the human body posture features of at least two people in the image to be recognized can be extracted respectively, and the posture information of the people can be confirmed based on the extracted human body posture features. The pose information may include: standing, sitting, etc. Of course, the attribute information and the posture information of the person may be acquired at the same time, and the attribute information and the posture information may be used as the person information.
In still another implementation manner of the present application, when the intersection ratio of the distances between the person and the object is greater than a preset threshold, the intersection ratio of the distances between the person and the object may be further used as the person information of the person. The preset threshold may be a value set according to actual conditions, and for example, may be 0.5, 0.6, and the like, which is not limited herein. For example, when the scene recognition result is a classroom, the recognized objects include a desk, a seat, and the like, and if the distance intersection ratio of the person to the desk and the seat is greater than a preset threshold, the distance intersection ratio may also be used as the person information of the person.
Step S250, determining character distribution information corresponding to the personal information according to the personal information and the corrected scene recognition result.
It should be noted that the method for determining the role distribution information corresponding to the personal information in this step is similar to the processing procedure in step S130, and is not described herein again.
Step S260, a role value corresponding to a plurality of roles of each person is obtained according to the role distribution information.
As described above, each person may include a plurality of pieces of personal information each corresponding to the character distribution information, and therefore, the character distribution information corresponding to the plurality of pieces of personal information may be analyzed to obtain character values corresponding to the plurality of characters. The role value of a character corresponding to a certain role reflects the probability that the character belongs to the role, and the larger the role value is, the higher the probability that the character belongs to the role is.
In one implementation manner of the present application, the role distribution information corresponding to each role may be subjected to weighting processing, so as to obtain role values of each person corresponding to a plurality of roles. Wherein, the role distribution information corresponding to each role comprises the probability that the persona belongs to the role in different dimensions (namely, in the dimension of each persona information).
The description will be given taking the personal information as age and sitting posture as an example. The information about the distribution of the role corresponding to the age can be seen in the second column of table 1, the information about the distribution of the role corresponding to the sitting posture can be seen in the third column of table 1, the information about the distribution of the role corresponding to the role a can be seen in the second row of table 1, the information about the distribution of the role corresponding to the role B can be seen in the third row of table 1, and the information about the distribution of the role corresponding to the role C can be seen in the fourth row of table 1. Then, the character value of the character corresponding to character a may be determined based on 10% and 5%, for example, 10% and 5% may be weighted to obtain the character value. Likewise, the character value of the character corresponding to character B may be determined from 90% and 5%, and the character value of the character corresponding to character C may be determined from 0 and 90%.
It should be noted that, if the person is identified in history, the relevant data of the person may be stored locally, and the relevant data includes: the character corresponds to the coefficient value of each character. Then, in determining the angular color value, after the weighting process is performed, the resultant value may be multiplied by the coefficient value, and the result after the multiplication may be taken as the angular color value. If there is no data associated with the persona locally, default coefficient values may be set, e.g., the coefficient value for each persona may be set to 0.5 or other values, etc.
Can be expressed by the formula:
；
wherein the content of the first and second substances,
a character value indicating that a character corresponds to character m, I indicates the total amount of personal information,
indicating the probability that the person corresponding to the ith personal belonging information belongs to the character m,
and I is an integer from 1 to I.
In step S270, the character relationship between the characters is determined based on the respective corresponding character values.
After determining the character values, the characters of the respective characters may be determined based on the respective corresponding character values of the respective characters. For example, the character corresponding to the maximum hue value may be the character of the character. Alternatively, the persona of the persona may also be determined by the following algorithm:
if the number of the role values corresponding to any two characters is M and N respectively, and M and N are integers larger than 1, one of M angular color values and one of N angular color values can be combined and input into a pre-constructed objective function to obtain M multiplied by N function values; that is, M angular color values and N angular color values are input to the objective function in a combined manner, resulting in M × N function values. And using the roles corresponding to the two angular color values corresponding to the maximum function value as the roles of the two characters. Wherein the objective function is used to represent the degree of the human relationship between two humans.
In the embodiment of the present application, if the probability coefficient values of the two roles are locally stored, a value multiplied by the probability coefficient may be used as the function value of the objective function, and if the probability coefficient values of the two roles are not locally stored, a default coefficient value (for example, may be 1 or the like) may be multiplied, and the obtained value may be used as the final function value.
Thus, the objective function can be expressed as:
。
wherein the content of the first and second substances,
a probability coefficient representing a character m and a character n,
a character value indicating that the character corresponds to character m,
indicating that another persona corresponds to the persona value for persona n.
For any two characters, according to the objective function, two roles, namely the roles of the two characters, when the objective function takes the maximum value can be obtained according to the corresponding angle values of the two characters.
Then, the character relationship between the characters can be determined based on the characters of the characters and a preset character relationship matrix. The preset role relationship matrix is a role relationship matrix established in advance according to the roles and comprises a mapping relationship between a role pair (namely two roles) and the roles. Therefore, according to the two obtained roles, the corresponding character relation can be obtained by searching in the preset role relation matrix.
In one implementation manner of the application, after determining the character relationship between the characters, the scene recognition result and the related data of the characters can be stored locally, so that when the same scene as the scene recognition result is recognized in other images and the characters are recognized, the character relationship between the characters and other characters in other images is determined according to the related data; wherein the relevant data comprises: the role of the person, the role value of the person, and the person relationship. Of course, the correlation data may also comprise a plurality of coefficient values as described above.
By storing the related data, the same scene and the same person can be directly obtained from the locally stored data when the person relationship is identified next time, so that the repeated calculation of the data is avoided, and the efficiency of the person relationship identification can be improved.
In the embodiment of the application, in order to improve the accuracy of the character relationship identification, a human-computer interaction interface can be provided for a user to correct the determined role and character relationship. That is, the user may manually determine the roles of the two characters, and may directly correct the roles when it is determined that there is an error in the obtained characters. Further, the user may judge the personal relationship between two persons, and may correct the personal relationship when it is determined that there is an error in the obtained personal relationship.
Specifically, the user can modify the role of the character through the man-machine interaction interfaceAnd generating a role modification instruction. And responding to a role modification instruction input by a user, modifying the role of the character, and updating the modified role to the local. Similarly, the user may modify the personal relationship to generate a personal relationship modification instruction. And responding to a character relation modification instruction input by the user, modifying the character relation, and updating the modified character relation to the local. In this way, it is ensured that the locally stored data are all correct. When the character or the relationship of the character is corrected, the correlation coefficient value (for example,
and
etc.) to be modified.
According to the character relation identification method, the scene where the character is located and the character information (such as sex, age, posture and the like) are identified from the image, the scene identification result is combined with the character information, the character distribution information corresponding to each character information can be accurately determined, and namely the probability that the character belongs to a plurality of characters in a plurality of different dimensions can be accurately determined. Moreover, the role values of the characters corresponding to the characters are determined by performing weighting processing on the character distribution information corresponding to the character information of the plurality of dimensions, so that the accuracy of character distribution can be further improved. Furthermore, the extreme value processing is carried out on the objective function according to the role values of all the people to determine the people relation, so that the accuracy of the determined people relation can be improved. In addition, the accuracy of the human relationship recognition can be improved by the secondary correction method. Wherein, the first correction is the correction of a single role, and the second correction is the correction of the relationship between the roles.
The following describes a person relationship recognition method according to an embodiment of the present application, with reference to the image to be recognized shown in fig. 3 and a schematic diagram of the person relationship recognition method shown in fig. 4.
As shown in fig. 4, scenes in an image may be recognized, and for each person, the character of the person may be determined based on the scene, attribute information and posture information of the person, the category of the object, the distance cross-over ratio between the objects, the distance cross-over ratio between the person and the object, relevant data of the person (i.e., historically stored data), and the like. The human relationship between any two humans is then determined. The method comprises the following specific steps:
for the image to be recognized shown in fig. 3, the background of the image may be recognized as a classroom, and the image may be in class, at a family meeting, etc. Here, the person 310 and the person 320 in the image are taken as an example for explanation.
By recognizing the facial features, behavior features and the like of the person 310 and the person 320 through the deep learning neural network, it can be determined that the age of the person 310 is 11 years, the posture information is a sitting posture, the age of the person 320 is 26 years, and the posture information is a standing posture. Also, a desk, a seat, a blackboard, and the like can be recognized. The distance intersection ratio of the person 310 and the desk and the seat is obtained through calculation and is larger than a preset threshold value, the distance intersection can be compared with the person information of the person 310, the distance intersection ratio of the person 320 and the blackboard is obtained through calculation and is larger than the preset threshold value, and the intersection can be compared with the person information of the person 320. From the above information, table 2 can be obtained.
According to the classroom background and the character information, the related preset scene relation matrix can be searched, and the preset scene relation matrix comprises the following tables 3-5, wherein the table 3 is a preset scene relation matrix related to age, the table 4 is a preset scene relation matrix related to posture, and the table 5 is a preset scene relation matrix related to distance intersection and comparison. Of course, more character information can be obtained, and the corresponding preset scene relation matrix is searched for, so as to obtain the corresponding role distribution information.
The character distribution information corresponding to the personal information is determined based on the personal information of the person 310, and is shown in table 6, which includes probabilities that the person 310 belongs to the respective characters in a plurality of different dimensions.
Assuming that no relevant data for persona 310 is stored locally,
can be 0.5, the character 310 can be calculated for each character by the following method.
Since there are 5 characters, the character 310 has a corresponding character value of 5.
Similarly, the character distribution information corresponding to the personal information is determined from the personal information of the person 320, as shown in table 7.
according to the objective function:
and m and n corresponding to the maximum value of the calculated function value are calculated, assuming that there is no probability coefficient for storing the character m and the character n in the local,
default coefficient values may be used. Of course, if stored locally
Then locally stored coefficient values may be used. By calculation, it is found that the function value of the objective function is the largest in the case where m is a student and n is a teacher, and therefore, it can be determined that the role of the character 310 is a student and the role of the character 320 is a teacher.
In the embodiment of the present application, the preset role relationship matrix can be referred to in table 8.
Further, assuming that both the genders of the person 310 and the person 320 are recognized as female, it can be determined that the person 310 is a girl student and the person 320 is a girl teacher, and the relationship between the persons can be obtained as a teacher-student relationship from table 8.
An embodiment of the present application provides a person relationship recognition apparatus, as shown in fig. 5, the person relationship recognition apparatus 500 includes:
a scene recognition module 510, configured to perform scene recognition on an image to be recognized that includes at least two characters, so as to obtain a scene recognition result;
a personal information determining module 520, configured to perform feature extraction on at least two persons in the image to be recognized, and determine personal information of the at least two persons based on the extracted personal features;
a character distribution information determining module 530, configured to determine character distribution information corresponding to the personal information according to the personal information and the scene recognition result; wherein the role distribution information is used for indicating the probability that each character belongs to a plurality of roles;
and a personal relationship determining module 540, configured to determine a personal relationship between the persons according to the role distribution information.
In one implementation manner of the present application, the human relationship recognition apparatus 500 further includes:
the object identification module is used for carrying out object identification on the image to be identified to obtain the information of the object in the image to be identified;
the scene recognition result correction module is used for correcting the scene recognition result based on the information of the object to obtain a corrected scene recognition result;
the role distribution information determining module 530 is specifically configured to determine role distribution information corresponding to the personal information according to the personal information and the corrected scene identification result.
In one implementation manner of the present application, the object identification module includes:
and the object type determining unit is used for carrying out object identification on the image to be identified and determining the type of the object in the image to be identified.
In an implementation manner of the present application, the object identification module further includes:
and the distance intersection ratio determining unit is used for determining the distance intersection ratio between the objects and/or determining the distance intersection ratio between the person and the objects.
In one implementation manner of the present application, the personal information determining module 520 is further configured to determine the distance intersection ratio between the person and the object as the personal information of the person when the distance intersection ratio between the person and the object is greater than a preset threshold.
In an implementation manner of the present application, the person information determining module 520 is specifically configured to extract face features of at least two persons in an image to be recognized respectively, and determine attribute information of the persons based on the extracted face features; and/or
And respectively extracting human body posture characteristics of at least two people in the image to be recognized, and confirming the posture information of the people on the basis of the extracted human body posture characteristics.
In an implementation manner of the present application, the role distribution information determining module 530 is specifically configured to determine a preset scene relationship matrix corresponding to the personal information according to the personal information and a scene recognition result; wherein, presetting the scene relation matrix comprises: the role distribution information corresponding to a plurality of different preset persona information respectively; and determining character distribution information corresponding to the character information in a preset scene relation matrix.
In one implementation manner of the present application, the human relationship determining module 540 includes:
the role value determining unit is used for obtaining role values of all the characters corresponding to a plurality of roles according to the role distribution information;
and the character relation determining unit is used for determining the character relation among the characters based on the corresponding angle color values of the characters.
In an implementation manner of the present application, the role value determining unit is specifically configured to perform weighting processing on role distribution information corresponding to each role to obtain role values of each persona corresponding to a plurality of roles.
In an implementation manner of the present application, the character relationship determining unit is specifically configured to determine a role of each character based on a corresponding role value of each character; and determining the character relationship among the characters based on the characters of the characters and a preset character relationship matrix.
In an implementation manner of the present application, the character role determination unit is specifically configured to determine whether the number of role values corresponding to any two characters is M and N, where M and N are integers greater than 1;
combining one of the M angular color values and one of the N angular color values and inputting the combination into a pre-constructed objective function to obtain M multiplied by N function values; the objective function is used for representing the degree of the character relationship between two characters; and using the roles corresponding to the two angular color values corresponding to the maximum function value as the roles of the two characters.
In one implementation manner of the present application, the human relationship recognition apparatus 500 further includes:
the data storage module is used for storing the scene recognition result and the related data of the person to the local, so that when the scene which is the same as the scene recognition result is recognized in other images and the person is recognized, the person relationship between the person and other persons in other images is determined according to the related data;
wherein the relevant data comprises: the role of the person, the role value of the person, and the person relationship.
In one implementation manner of the present application, the human relationship recognition apparatus 500 further includes:
the role modification module is used for responding to a role modification instruction input by a user, modifying the role of the character and updating the modified role to the local;
and the character relation modification module is used for responding to a character relation modification instruction input by a user, modifying the character relation and updating the modified character relation to the local.
The specific details of each module or unit in the above-mentioned person relationship identification apparatus have been described in detail in the corresponding person relationship identification method, and therefore are not described herein again.
It should be noted that although in the above detailed description several modules or units of the device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit, according to embodiments of the application. Conversely, the features and functions of one module or unit described above may be further divided into embodiments by a plurality of modules or units.
Based on the foregoing method embodiment, an embodiment of the present application further provides an electronic device, configured to execute the method described in any of the foregoing embodiments, as shown in fig. 6, where the electronic device includes:
a processor 602, a communication interface 604, a memory 606, and a communication bus 608.
The processor 602, the communication interface 604, and the memory 606 communicate with each other via a communication bus 608.
A communication interface 604 for communicating with other terminal devices or servers.
The processor 602 is configured to execute the program 610, and may specifically perform the relevant steps in the above-described person relationship identification method embodiment.
In particular, program 610 may include program code comprising computer operating instructions.
The processor 602 may be a central processing unit CPU or an application Specific Integrated circuit asic or one or more Integrated circuits configured to implement embodiments of the present invention. The terminal device comprises one or more processors, which can be the same type of processor, such as one or more CPUs; or may be different types of processors such as one or more CPUs and one or more ASICs.
And a memory 606 for storing a program 610. Memory 606 may comprise high-speed RAM memory, and may also include non-volatile memory (non-volatile memory), such as at least one disk memory.
The electronic device of the embodiments of the present application exists in various forms, including but not limited to:
(1) a mobile communication device: such devices are characterized by mobile communications capabilities and are primarily targeted at providing voice, data communications. Such terminals include: smart phones (e.g., iphones), multimedia phones, functional phones, and low-end phones, among others.
(2) Ultra mobile personal computer device: the equipment belongs to the category of personal computers, has calculation and processing functions and generally has the characteristic of mobile internet access. Such terminals include: PDA, MID, and UMPC devices, etc., such as ipads.
(3) A portable entertainment device: such devices can display and play multimedia content. This type of device comprises: audio, video players (e.g., ipods), handheld game consoles, electronic books, and smart toys and portable car navigation devices.
(4) And other electronic equipment with data interaction function.
The embodiment of the present application further provides a computer-readable storage medium, where computer-executable instructions are stored, and when executed, the computer-executable instructions implement the character relationship identification method described in any embodiment of the present application.
Thus, particular embodiments of the present subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may be advantageous.
The method illustrated in the above embodiments may be implemented by a computer chip or an entity, or by a product with certain functions. One typical implementation device is a computer. In particular, the computer may be, for example, a personal computer, a laptop computer, a cellular telephone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, apparatus, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The application may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular transactions or implement particular abstract data types. The application may also be practiced in distributed computing environments where transactions are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the apparatus, electronic device, and computer-readable storage embodiment, since they are substantially similar to the method embodiment, the description is relatively simple, and reference may be made to some descriptions of the method embodiment for relevant points.
The above description is only an example of the present application and is not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims (15)
1. A person relationship recognition method is characterized by comprising the following steps:
carrying out scene recognition on an image to be recognized containing at least two characters to obtain a scene recognition result;
respectively extracting the characteristics of at least two persons in the image to be recognized, and determining the person information of the at least two persons based on the extracted person characteristics;
determining role distribution information corresponding to the character information according to the character information and the scene identification result; wherein the character distribution information is used for indicating the probability that each character belongs to a plurality of characters;
determining the character relationship among all characters according to the character distribution information;
wherein, the determining the role distribution information corresponding to the personal information according to the personal information and the scene recognition result includes:
determining a preset scene relation matrix corresponding to the character information according to the character information and the scene recognition result; wherein the preset scene relationship matrix comprises: the role distribution information corresponding to a plurality of different preset persona information respectively;
and determining the role distribution information corresponding to the personal information in the preset scene relation matrix.
2. The method according to claim 1, wherein after the scene recognition is performed on the image to be recognized to obtain a scene recognition result, the method further comprises:
carrying out object recognition on the image to be recognized to obtain information of an object in the image to be recognized;
based on the information of the object, correcting the scene recognition result to obtain a corrected scene recognition result;
the determining, according to the personal information and the scene recognition result, the role distribution information corresponding to the personal information specifically includes:
and determining role distribution information corresponding to the character information according to the character information and the corrected scene identification result.
3. The method according to claim 2, wherein the performing object recognition on the image to be recognized to obtain information of an object in the image to be recognized comprises:
and carrying out object recognition on the image to be recognized, and determining the category of the object in the image to be recognized.
4. The method according to claim 3, wherein the performing object recognition on the image to be recognized to obtain information of an object in the image to be recognized further comprises:
determining a distance intersection ratio between the objects, and/or,
determining a distance intersection ratio between the person and the object.
5. The method of claim 4, wherein before the determining the character distribution information corresponding to the personal information according to the personal information and the scene recognition result, the method further comprises:
and when the distance intersection ratio between the person and the object is larger than a preset threshold value, taking the distance intersection ratio between the person and the object as the person information of the person.
6. The method of claim 1, wherein the extracting features of at least two persons in the image to be recognized respectively, and determining the personal information of the at least two persons based on the extracted features of the persons comprises:
respectively extracting the face characteristics of at least two persons in the image to be recognized, and confirming the attribute information of the persons based on the extracted face characteristics; and/or
And respectively extracting human body posture features of at least two people in the image to be recognized, and confirming the posture information of the people on the basis of the extracted human body posture features.
7. The method of claim 1, wherein determining the relationships between the persons according to the character distribution information comprises:
obtaining the role distribution information of the characters corresponding to the plurality of characters;
and determining the character relationship among the characters based on the corresponding angle values of the characters.
8. The method of claim 7, wherein obtaining the character values of the respective personas corresponding to the plurality of characters according to the character distribution information comprises:
and carrying out weighting processing on the role distribution information corresponding to each role to obtain the role values of each character corresponding to the plurality of roles.
9. The method of claim 7, wherein determining the character relationship between the characters based on the respective corresponding character values comprises:
determining the role of each character based on the corresponding role value of each character;
and determining the character relationship among the characters based on the characters of the characters and a preset character relationship matrix.
10. The method of claim 9, wherein determining the persona of each character based on the respective corresponding character's respective character hue value comprises:
if the number of the role values corresponding to any two characters is M and N respectively, wherein M and N are integers larger than 1;
combining one of the M angular color values and one of the N angular color values and inputting the combination into a pre-constructed objective function to obtain M multiplied by N function values; wherein the objective function is used for representing the degree of the character relationship between the two characters;
and taking the roles corresponding to the two angular color values corresponding to the maximum function value as the roles of the two characters.
11. The method of claim 9, wherein after the determining the person relationships between the persons, the method further comprises:
storing the scene recognition result and the related data of the person locally, so that when the scene identical to the scene recognition result is recognized in other images and the person is recognized, the person relationship between the person and other persons in other images is determined according to the related data;
wherein the relevant data comprises: the character's role, the character's role values, and the character relationships.
12. The method of claim 11, further comprising:
responding to a role modification instruction input by a user, modifying the role of the character, and updating the modified role to the local;
and responding to a character relation modification instruction input by a user, modifying the character relation, and updating the modified character relation to the local.
13. A personal relationship recognition apparatus, comprising:
the scene recognition module is used for carrying out scene recognition on the image to be recognized containing at least two characters to obtain a scene recognition result;
the figure information determining module is used for respectively extracting the features of at least two figures in the image to be recognized and determining the figure information of the at least two figures based on the extracted figure features;
the role distribution information determining module is used for determining role distribution information corresponding to the character information according to the character information and the scene recognition result; wherein the character distribution information is used for indicating the probability that each character belongs to a plurality of characters;
the character relation determining module is used for determining the character relation among all the characters according to the character distribution information;
the role distribution information determining module is specifically used for determining a preset scene relation matrix corresponding to the character information according to the character information and the scene recognition result; wherein the preset scene relationship matrix comprises: the role distribution information corresponding to a plurality of different preset persona information respectively; and determining the role distribution information corresponding to the personal information in the preset scene relation matrix.
14. An electronic device, comprising: a processor; and a memory configured to store computer-executable instructions that, when executed, cause the processor to implement the method of any of claims 1-12 above.
15. A computer storage medium having stored thereon computer-executable instructions that, when executed, implement the method of any of claims 1-12.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011317975.9A CN112132118B (en) | 2020-11-23 | 2020-11-23 | Character relation recognition method and device, electronic equipment and computer storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011317975.9A CN112132118B (en) | 2020-11-23 | 2020-11-23 | Character relation recognition method and device, electronic equipment and computer storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112132118A CN112132118A (en) | 2020-12-25 |
| CN112132118B true CN112132118B (en) | 2021-03-12 |
Family
ID=73852252
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011317975.9A Active CN112132118B (en) | 2020-11-23 | 2020-11-23 | Character relation recognition method and device, electronic equipment and computer storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112132118B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113343867A (en) * | 2021-06-15 | 2021-09-03 | 广东电网有限责任公司 | Machine room carrying equipment detection method, system, equipment and medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104778224A (en) * | 2015-03-26 | 2015-07-15 | 南京邮电大学 | Target object social relation identification method based on video semantics |
| CN107784273A (en) * | 2017-09-28 | 2018-03-09 | 陕西师范大学 | Classroom seat distribution forecasting method based on student's social modeling |
| CN109376581A (en) * | 2018-09-03 | 2019-02-22 | 腾讯科技（武汉）有限公司 | Object relationship recognition methods and device, storage medium and electronic device |
-
2020
- 2020-11-23 CN CN202011317975.9A patent/CN112132118B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104778224A (en) * | 2015-03-26 | 2015-07-15 | 南京邮电大学 | Target object social relation identification method based on video semantics |
| CN107784273A (en) * | 2017-09-28 | 2018-03-09 | 陕西师范大学 | Classroom seat distribution forecasting method based on student's social modeling |
| CN109376581A (en) * | 2018-09-03 | 2019-02-22 | 腾讯科技（武汉）有限公司 | Object relationship recognition methods and device, storage medium and electronic device |
Non-Patent Citations (3)
| Title |
|---|
| An End-to-End Network for Generating Social Relationship Graphs;Arushi等;《arxiv》;20190323;第1-9页 * |
| Dual-Glance Model for Deciphering Social Relationships;Junnan Li等;《arxiv》;20170802;第1-10页 * |
| 基于图像的社会关系识别研究综述;高建军等;《计算机工程与应用》;20191231;第22卷(第21期);第36-45页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112132118A (en) | 2020-12-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108076154B (en) | Application information recommendation method and device, storage medium and server | |
| US9348898B2 (en) | Recommendation system with dual collaborative filter usage matrix | |
| WO2021169637A1 (en) | Image recognition method and apparatus, computer device and storage medium | |
| CN112016315B (en) | Model training method, text recognition method, model training device, text recognition device, electronic equipment and storage medium | |
| US20200311306A1 (en) | Electronic device for obfuscating and decoding data and method for controlling same | |
| CN109145083B (en) | Candidate answer selecting method based on deep learning | |
| CN108255962A (en) | Knowledge Relation method, apparatus, storage medium and electronic equipment | |
| CN111737439B (en) | Question generation method and device | |
| Hong et al. | Selective residual learning for visual question answering | |
| CN111860101A (en) | Training method and device for face key point detection model | |
| CN112132118B (en) | Character relation recognition method and device, electronic equipment and computer storage medium | |
| CN115471898A (en) | Human face and gesture recognition method and product integrating attention mechanism | |
| CN113435531B (en) | Zero sample image classification method and system, electronic equipment and storage medium | |
| CN112308102B (en) | Image similarity calculation method, calculation device, and storage medium | |
| CN110929285B (en) | Method and device for processing private data | |
| JP7293658B2 (en) | Information processing device, information processing method and program | |
| CN117195046A (en) | Abnormal text recognition method and related equipment | |
| CN111507250A (en) | Image recognition method, device and storage medium | |
| CN111310590A (en) | Action recognition method and electronic equipment | |
| CN113254788B (en) | Big data based recommendation method and system and readable storage medium | |
| CN112016281B (en) | Method and device for generating wrong medical text and storage medium | |
| CN115204301A (en) | Video text matching model training method and device and video text matching method and device | |
| CN115115901A (en) | Method and device for acquiring cross-domain learning model | |
| CN112184505A (en) | Information processing method and device and computer storage medium | |
| CN112541069A (en) | Text matching method, system, terminal and storage medium combined with keywords |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |