JP7152651B2 - Program, information processing device, and information processing method - Google Patents

Description

The present invention relates to a program, an information processing apparatus, and an information processing method.

Conventionally, there are techniques for estimating the line-of-sight direction. For example, the line of sight of a person is estimated from an image of the person captured by a surveillance camera installed in an apparel shop or supermarket. As a result, for example, a manager or the like can grasp what kind of products the crowd is paying attention to, and it may be possible to formulate a sales strategy. In addition, for example, for city surveillance for security purposes, by estimating the line-of-sight direction from images of people captured by surveillance cameras installed near signs and signage, it is possible to investigate the effects of installing signs and understand crowd behavior. may be used for

Techniques for estimating the line-of-sight direction include, for example, the following. That is, based on the image frame at the current time taken by the photographing means, the three-dimensional position of the center of the eyeball of the specific person is estimated using the three-dimensional model of the eyeball, and the position of the iris of the specific person is detected, There is a technique for estimating the line-of-sight direction based on the center of the eyeball and the position of the iris.

According to this technology, it is possible to estimate and track the line-of-sight direction of the person to be measured at any position within the observation range in real time, using a relatively small number of cameras, by relaxing restrictions on the orientation of the face. be done.

JP 2012-216180 A

The above-described technique for estimating the line-of-sight direction based on the center of the eyeball and the position of the iris requires, for example, that a person's face is photographed by a plurality of photographing means. Therefore, in the case of an image of a person whose face is hidden, the above-described technique may not be able to estimate the line-of-sight direction of the person.

Therefore, one disclosure is to provide a program, an information processing apparatus, and an information processing method that enable estimation of a line-of-sight direction even in an image of a person whose face is hidden.

One disclosure is to estimate the position information of the parts of the person included in the input image data using the correct data about the parts of the person, and to estimate the position information of the parts of the face among the parts. A program for causing a computer to execute a process of estimating the line-of-sight direction of a person included in an image based on the position information of other parts that could be estimated when the estimation could not be performed.

According to one disclosure, it is possible to estimate the gaze direction even for an image of a person whose face is hidden.

FIG. 1 is a diagram showing a configuration example of an information processing system. FIG. 2 is a flow chart showing an operation example. FIG. 3 is a diagram showing examples of part numbers. FIG. 4 is a diagram showing an example of an image. FIG. 5 is a flowchart showing an example of posture estimation processing. FIG. 6A is a diagram showing a configuration example of a posture estimation unit, FIG. 6B is an example of image data, and FIG. 6C is a diagram showing an example of probability distribution of each part. 7A shows an example of the probability distribution of the right hand, FIG. 7B shows an example of the probability distribution of the right elbow, and FIG. 7C shows an example of the probability distribution of the degree of connection between the right hand and the right elbow. be. FIG. 8 is a flowchart showing an example of attention level calculation processing. FIG. 9A is a diagram showing an example of a part, and FIG. 9B is a diagram showing an example of a direction vector. 10A is an example of a part, FIG. 10B is an example of a vector, and FIG. 10C is an example of a direction vector. FIG. 11 is a flowchart showing an example of counting processing. FIG. 12 is a diagram showing an example of three-dimensional position coordinates. FIG. 13 is a diagram showing a configuration example of a posture estimation unit. FIG. 14 is a flow chart showing an operation example. FIG. 15A is a diagram showing an example of same person identification processing, and FIG. 15B is a diagram showing an example of similarity calculation processing. FIG. 16 is a flowchart showing an operation example of attention degree calculation processing. FIG. 17 is a diagram showing a configuration example of an information processing system. FIG. 18 is a flow chart showing an operation example. FIG. 19 is a flowchart showing an example of attention degree change detection processing. FIG. 20 is a diagram showing an example of a time series of attention level vectors. FIGS. 21A and 21C are examples of images, and FIGS. 21B and 21D are examples of interest vectors. FIG. 22 is a diagram showing a hardware configuration example of an information processing apparatus.

EMBODIMENT OF THE INVENTION Hereinafter, the form for implementing this invention is demonstrated. It should be noted that the following examples do not limit the technology disclosed. Further, each embodiment can be appropriately combined within a range in which the processing contents are not inconsistent.

[First embodiment]
<Configuration example of information processing system>
FIG. 1 is a diagram showing a configuration example of an information processing system 10. As shown in FIG.

The information processing system 10 includes an information processing device 100 and an imaging device 200 . The information processing apparatus 100 receives image data of an image captured by the imaging apparatus 200, and estimates the line-of-sight direction of a person appearing in the image from the input image data. In the first embodiment, the information processing apparatus 100 can estimate the line-of-sight direction from the image data of even an image in which the face is hidden.

Information processing apparatus 100 includes posture estimation section 110 , attention level calculation section 120 , spatial information storage section 130 , and attention level storage section 140 .

The posture estimation unit 110 receives the image data output from the imaging device 200, and estimates the posture p _i of the person i (i=1, 2, . . . , I) included in the image based on the input image data. . The posture pi includes position information of each part such as the "nose", "left eye", and "right eye" of the person _i . The details of the posture _pi will be described in an operation example.

In the first embodiment, posture estimating section 110 generates model data (or correct data, or teacher data) relating to parts of a person based on input image data, for example. ) is used to generate the pose p _i . The estimation processing of posture p _i in posture estimation section 110 will be described with an operation example. Posture estimation section 110 outputs the estimated posture p _i to attention degree calculation section 120 .

The attention degree calculation unit 120 estimates the line-of-sight direction of the person _i included in the image using the position information included in the posture pi. Specifically, for example, when the posture estimating unit 110 cannot estimate the position information of the parts of the face among the parts of the person i to be estimated by the posture estimating unit 110, the attention level calculating unit 120 estimates The line-of-sight direction of the person i included in the image is estimated based on the position information of other parts other than the face.

For example, the image captured by the imaging device 200 may not show a part such as the face of the person i due to obstacles, line-of-sight direction, or the like. Therefore, the pose p _i estimated by the pose estimation unit 110 may not include position information of facial parts (eg, eyes, nose, etc.). Attention degree calculation section 120 calculates the position information of the part of the face that could not be estimated by posture estimation section 110 using the position information of other parts that could be estimated by posture estimation section 110 . , the line-of-sight direction of the person i is estimated based on the position information.

Note that the attention level calculation unit 120 estimates the line-of-sight direction of the person i, for example, by calculating the direction vector q _i of the person i. In the first embodiment, the orientation vector _qi may be referred to as, for example, the attention level. Below, attention level, orientation vector q _i , and line-of-sight direction may be used without distinction. Details of the attention level calculation process will be described in an operation example. The attention degree calculation unit 120 stores the calculated attention degree in the attention degree storage unit 140 .

Also, the attention level calculation unit 120 reads the position information of each of the objects 300-1 and 300-2 from the space information storage unit 130. FIG. Then, based on the calculated orientation vector q _i and the position information of each of the objects 300-1 and 300-2, the attention level calculation unit 120 directs the line of sight to each of the objects 300-1 and 300-2. Count the number of persons i present. The attention level calculation unit 120 stores the counted count value in the attention level storage unit 140 . The details of the counting process will also be explained with an operation example.

The spatial information storage unit 130 is, for example, a memory and stores position information of each of the objects 300-1 and 300-2. As the positional information, for example, the installation points of the objects 300-1 and 300-2 and their surrounding areas may be represented by two-dimensional coordinates (x, y).

The attention level storage unit 140 is, for example, a memory, and stores the attention level calculated by the attention level calculation unit 120 .

The imaging device 200, for example, photographs one or more persons and outputs the photographed image to the information processing device 100 as image data. In the example of FIG. 1, the imaging device 200 captures an image including objects 300-1 and 300-2 and a plurality of persons facing the objects 300-1 and 300-2.

In the example of FIG. 1, the imaging device 200 is arranged outside the information processing device 100, but the imaging device 200 may be provided inside the information processing device 100 as an imaging unit, for example. good. Also, in the example of FIG. 1, the imaging device 200 represents an example of one unit, but there may be a plurality of units. An example of multiple units will be described in the second embodiment.

<Operation example>
FIG. 2 is a flow chart showing an operation example of the information processing system 10 .

When the imaging device 200 and the information processing device 100 start processing (S10), the imaging device 200 photographs the crowd (S11). For example, as shown in FIG. 1, the imaging device 200 captures an image of a plurality of people (hereinafter sometimes referred to as a “crowd”) and outputs the captured image data to the information processing device 100. .

Next, the information processing device 100 estimates the posture pi of the person _i appearing in the image captured by the imaging device 200 based on the image data (S12). The attitude p _i is represented by the following equation (1), for example.

In equation (1), x _j ⁱ represents the x-coordinate of part j of person i in the image, and y _j ⁱ represents the y-coordinate of part j of person i in the image. Also, v _j ⁱ is a parameter that takes a value of “1” when the part j of the person i is visible (or captured or included) in the image, and takes a value of “0” when it is not visible. .

FIG. 3 is a diagram showing an example of the relationship between part numbers and parts. As shown in FIG. 3, a part number is assigned in advance to each part j. In the example of FIG. 3, when part j=1, it represents "nose", and when part j=6, it represents "neck". FIG. 3 is an example and other numbers may be assigned.

FIG. 4 is a diagram showing an example of a captured image. In the example of FIG. 4, the right hand part (j=12) of person i=1 is shown in the image, but the left elbow part (j=9) is not shown in the image due to an obstacle. Therefore, the posture p _i =(... x ₉ ¹ y ₉ ¹ 0... x ₁₂ ¹ y ₁₂ ¹ 1...) can be obtained. For example, posture estimation section 110 estimates such posture p _i based on input image data. Details of the posture estimation processing (S12) will be described below. It should be noted that, hereinafter, the posture p _i and the posture vector p _i may be used without distinction.

<Posture estimation processing>
FIG. 5 is a flowchart showing an example of posture estimation processing (S12). FIG. 6A is a diagram showing a configuration example of posture estimation section 110. As shown in FIG. FIG. 6A will be described while describing each process in FIG. In addition, as shown in FIG. 6A, the posture estimation unit 110 includes a CNN (Convolutional neural network) processing unit 111 , a candidate point calculation unit 112 , and a grouping processing unit 113 .

As shown in FIG. 5, when posture estimation processing is started (S120), posture estimation section 110 calculates a probability distribution (or heat map) of part j (j=1, 2, . . . , J) (S121). . The CNN processing unit 111 calculates the probability distribution φ(X, W) by, for example, a known method using a convolutional neural network (hereinafter sometimes referred to as “CNN”). W represents a parameter used for CNN processing, for example. For example, the CNN processing unit 111 performs the following processing.

That is, the CNN processing unit 111 performs filtering processing (or convolution processing) on certain image data using the right-hand correct data. Then, the CNN processing unit 111 performs pooling processing for extracting a representative value (or maximum value) for each block containing a plurality of images from the data after filtering, and thereafter repeats the filtering processing and the pooling processing. Correct data of the right-hand (j=12) probability distribution are generated. Next, the CNN processing unit 111 inputs RGB (Red Green Blue) image data X as shown in FIG. 6B. The CNN processing unit 111 repeats the filtering process and the pooling process on the input image data X using the correct data of the right-hand (j=12) probability distribution, thereby obtaining the right-hand probability distribution φ(X, W ).

FIG. 6C is a diagram showing an example of the right-hand probability distribution φ(X, W). For example, the probability distribution φ(X, W) is expressed as a numerical value (or probability value) from "0" to "1" for each pixel (or each block containing a plurality of pixels).

In addition, below, the process which repeats a filtering process and a pooling process may be called CNN process, for example.

Similarly, the CNN processing unit 111 uses the correct data of the right elbow (j=10) to obtain the correct data of the probability distribution of the right elbow by CNN processing. Then, the CNN processing unit 111 obtains the probability distribution φ(X, W) of the right elbow from the input image data X by CNN processing using the correct data of the probability distribution of the right elbow.

Note that the CNN processing unit 111 calculates not only the probability distribution φ(X, W) of each part but also the probability distribution φ(X, W) representing the degree of connection of each part. The CNN processing unit 111 can obtain correct data in which each part is connected from the correct data of each part used for the CNN processing. Then, the CNN processing unit 111 performs CNN processing on certain image data using correct data in which each part is connected from certain image data, thereby obtaining a probability distribution of correct data in which each part is connected. By performing CNN processing on the input image X using this probability distribution, it is possible to calculate a probability distribution φ(X, W) representing the degree of connection between parts.

In this way, the CNN processing unit 111 uses the correct data of each part j to perform convolution processing or the like on the image data X, thereby calculating the probability distribution φ(X, W) of each part j. do.

The CNN processing unit 111 performs, for example, the following processing in S121. That is, the CNN processing unit 111 reads the correct data of each part j stored in the internal memory, uses this correct data, performs CNN processing on certain image data, and obtains the probability distribution of each part j. Correct answer data is calculated and stored in internal memory. The CNN processing unit 111 may calculate such correct data of the probability distribution of each part j before the process of S121 and store it in the internal memory. Then, the CNN processing unit 111 performs CNN processing on the input image data X using the correct data of the probability distribution of each part j read out from the internal memory, thereby obtaining the probability distribution φ(X , W).

The CNN processing unit 111 outputs the part number of each part j when using the correct data and the probability distribution φ(X, W) of the part j to the candidate point calculation unit 112 .

An example of calculation of the probability distribution φ(X, W) of each part j has been described above. As a method using such a convolutional neural network, for example, the method disclosed in Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh, "Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields", In CVPR 2017 There is The CNN processing unit 111 may use this to calculate the probability distribution φ(X, W) of each part j. As an example of calculation of the probability distribution φ(X, W) of each part, a known method using template matching may be used other than the convolutional neural network. Template matching is, for example, a method of calculating the probability distribution φ(X, W) of each part of the input image by comparing with model data of each part.

Returning to FIG. 5, next, posture estimation section 110 obtains candidate points for part positions (S122). For example, as shown in FIG. 6A, the candidate point calculator 112 calculates candidate points based on the probability distribution φ(X, W) of each part output from the CNN processor 111 .

FIG. 7A is a diagram showing an example of searching for candidate points A1 and A2 on the right hand with respect to the probability distribution φ(X, W) on the right hand (j=12). Candidate point calculation section 112 searches for position coordinates A1 and A2 at which the probability of a certain block is the maximum point in right-hand probability distribution φ(X, W). These position coordinates A1 and A2 are candidate points on the right hand side. Further, as shown in FIG. 7B, the candidate point calculation unit 112 searches for the position coordinates B1 and B2 at which the probability is the maximum point in the probability distribution φ(X, W) of the right elbow. . Position coordinates B1 and B2 are candidate points for the right elbow. Candidate point calculation section 112 outputs the part number of each part and its candidate points to grouping processing section 113 .

For example, when the local maximum point is equal to or greater than a threshold, posture estimation section 110 regards such a local maximum point as a candidate point for the part, and when the local maximum point is smaller than the threshold, it assumes that the part is not shown in the input image. judge. In the former case, posture estimation section 110 sets v _j ⁱ =1 (or the visible part), and in the latter case, v _j ⁱ =0 (or the non-visible part). Posture estimation section 110 performs subsequent processing in the former case, and terminates posture estimation processing (S12) here in the latter case.

Returning to FIG. 5, next, posture estimation section 110 performs grouping for each part position candidate point (S123), and identifies a part for each person i. For example, such grouping is performed in the grouping processing unit 113 of FIG. 6(A). The grouping processing unit 113 groups the candidate points by, for example, comparing the distances between the candidate points of the part j. Specifically, the grouping processing unit 113 performs processing using, for example, the probability distribution φ(X, W) indicating the degree of connection between parts, which has been subjected to CNN processing in the CNN processing unit 111 .

FIG. 7C is a diagram showing an example of the probability distribution φ(X, W) of the degree of connection between the right hand and the right elbow. Candidate point calculation section 112 obtains candidate points A1 and A2 for the right hand (FIG. 7A) and candidate points B1 and B2 for the right elbow (FIG. 7B). The grouping processing unit 113 determines whether A1 and B1 belong to the same person or whether A1 and B2 belong to the same person. The determination is made based on the probability distribution of the degree of connection between the right hand and the right elbow. For example, the grouping processing unit 113 uses the following formula to calculate the degree of connection of each combination.

In equations (2) and (3), y ₁ represents, for example, a set on a probability distribution indicating the degree of connection between the right hand and right elbow of person i=1, as shown in FIG. 7(C). Here, the degree of connection between the candidate points A1 and B1 represents, for example, the numerical value of the line integral connecting the candidate points A1 and _B1 with respect to the set y1. The degree of connection between the candidate points A1 and B2 represents, for example, the numerical value of the line integral connecting the candidate _points A1 and B2 with respect to the set y1. The grouping processing unit 113 selects, for example, the candidate points A1 and B1, which are larger than the numerical values of the equations (2) and (3), and groups the selected candidate points A1 and B1. Similarly, the grouping processing unit 113 acquires the probability distribution indicating the degree of connection between the right hand and the right elbow of the person i=2 from the CNN processing unit 111 via the candidate point calculation unit 112, and ), using the formula in which y ₁ is replaced with y ₂ , the larger value is selected and grouped. In this case, the grouping processing unit 113 groups A2 and B2.

In this way, the CNN processing unit 111 calculates a probability distribution indicating the degree of connection of each part, and the grouping processing unit 113 combines such a probability distribution with each candidate point calculated by the candidate point calculation unit 112. Calculate the line integral for Then, the grouping processing unit 113 groups the combination of candidate points with the largest calculation result. The grouping processing unit 113 can specify each part of the person i by each grouped candidate point.

The CNN processing unit 111 performs, for example, the following processing in S123. That is, the CNN processing unit 111 reads out the equations (2) and (3) from the internal memory, substitutes a set of probability distributions indicating the degree of connection of each part into the equations (2) and (3), and obtains each line segment Get the numerical value of the line integral of . Then, the CNN processing unit 111 groups the combination of candidate points having the largest numerical value into one.

Returning to FIG. 5, next, posture estimation section 110 acquires the posture p _i of each grouped person i (S124). For example, the grouping processing unit 113 in FIG. 6A substitutes the grouped candidate points (or position coordinates) of each part for each element of the posture _pi shown in Equation (1), thereby Get the pose p _i of i.

The posture estimation processing (S12) has been described above.

Returning to FIG. 2, next, the information processing apparatus 100 calculates the degree of attention for each person i in the crowd (S13). An example of attention degree calculation processing will be described below.

<Attention degree calculation processing>
FIG. 8 is a flowchart showing an example of attention level calculation processing. For example, it is a process performed by the attention degree calculation unit 120 .

When the attention degree calculation process is started (S130), the attention degree calculation unit 120 uses the posture _pi to determine whether or not the parts of the face, the neck, the left shoulder, and the right shoulder are visible (S131). For example, the attention degree calculation unit 120 uses the following formula for determination.

When v ⁱ is “1”, the attention level calculation unit 120 determines that all parts of the face (nose, left eye, right eye, left ear, right ear), neck, left shoulder, and right shoulder are visible. When ⁱ is "0", it is determined that any part of the face, neck, left shoulder, or right shoulder is not visible. For example, attention level calculation section 120 reads equation (4) from the internal memory, extracts v ₁ ⁱ to v ₈ ⁱ from posture p _i output from posture estimation section 110, and substitutes them into equation (4). Judge by

When the attention level calculation unit 120 determines that any part of the face, neck, left shoulder, or right shoulder is not visible (NO in S131), it calculates the position information of the invisible part by interpolation (S132). . The attention level calculation unit 120 uses, for example, the following equation to obtain the position vector (or position information) of the invisible part k of the person i. Hereinafter, the position vector and the position information may be used without distinction. ) a _k ⁱ =(x _k ⁱ y _k ⁱ ) ^T.

In _Equation (5 ⁾ , a _k1 ⁱ , a _k2 ⁱ , . A position vector, A _k ^M represents a matrix of 2 rows and 2 M columns, and b _k ^M represents a vertical vector of 2 rows and 1 column.

For example, the position vector a _k ⁱ of the _unseen part ^k is _obtained by using the position ^vectors a _k1 ⁱ , a _k2 ⁱ , . It means that it is calculated.

The matrix A _k ^M and the vertical vector b _k ^M in Equation (5) can be obtained by solving the following equation using the set P of postures s.

In Equation (6), s _k ⁱ =(x _k ⁱ y _k ⁱ ) ^T represents the position vector of the part with the part number k of the posture s. A set P of postures s is, for example, a set of position vectors of each part when a human body model created by 3D-CG software or the like is used as model data.

Equation (6) is, for example, a position vector s _k ⁱ of a part number k of a person created as such a human body model and a position vector a _k1 ⁱ of a visible part estimated by the posture estimation unit 110, A _{kM and b k M} ^that minimize the error from a _k2 ⁱ , . . . , a _kM ⁱ are ^shown . Equation (6) can be solved by a known method such as gradient descent.

In this way, for example, when posture estimation section 110 cannot estimate the position information of a part of the face, attention level calculation section 120 calculates the position information of other parts that could be estimated by posture estimation section 110. is used to calculate the position information of the parts of the face.

Specifically, the attention degree calculation unit 120 performs, for example, the following processing in S132. That is, attention level calculation section ^{120 extracts} position _vectors a _{k1 i} , a _k2 ⁱ , . Then, attention level calculation section 120 reads out equation (5) stored in the internal memory and substitutes it into equation (5) to obtain position vector a _k ⁱ of part k that is not visible in posture p _i . Calculate At that time, the attention level calculation unit 120 reads out the expression (6 ⁾ from the internal memory _, substitutes the position vectors a _k1 ⁱ , a _k2 ⁱ , . By performing calculations, A _k ^M and b _k ^M are obtained and substituted into equation (5).

Note that the neck, left shoulder, right shoulder, and nose may be included in the facial region. In this case, the attention degree calculation unit 120 may determine in S131 whether or not the part of the face including these parts is visible.

Next, the attention level calculation unit 120 calculates the orientation vector qi of the person _i (S133). For example, the attention level calculation unit 120 calculates the direction vector q _i using the following formula.

In equation (7), W represents a matrix of ₂ rows and 2 J columns, and w0 represents a vertical vector of 2 rows and 1 column. Also, a _j ⁱ represents the position vector of the part j of the person i. Matrix W and column vector _w0 can be obtained by solving the following equations as in equation (6).

In equation (8), s _k =(x _k y _k ) ^T represents the position vector of the part of pose s with part number k, and the set P of poses s is, for example, a 3D- It is a set of position vectors of each part in a human body model created by CG software or the like. Also, qs is, for example, the direction vector of the posture _s , and is defined as a vector orthogonal to the least-squares plane S _face of the parts of the face (left eye, right eye, left ear, right ear, nose, neck). . FIG. 9A shows an example of a person image, and FIG. 9B shows an example of a direction vector _qs .

For example, the attention degree calculation unit 120 performs the following processing in S133. That is, the attention level calculation unit 120 reads out the equation (7 ⁾ stored in the internal memory, and extracts the position _vectors a _{k1 i} , a _k2 ⁱ , ^. , and the position vector a _k ⁱ of the invisible site k calculated by interpolation are substituted into the equation (7). Then, the attention degree calculation unit 120 calculates the orientation vector q _i of the person i. At that time, the attention level calculation unit 120 reads out the expression (8), the direction vector q _s , and the position vector s _k of the part with the part number k from the internal memory, and substitutes them into the expression (8). Get W and _w0 . In this case, pre-calculated W and _w0 are stored in the internal memory, and the attention level calculation unit 120 reads out W and _w0 from the internal memory and substitutes them into equation (7). can be

Returning to FIG. 8, after calculating the orientation vector qi of the person _i , the attention level calculation unit 120 ends the attention level calculation process (S134).

On the other hand, when the attention level calculation unit 120 determines that all parts of the face, the neck, the left shoulder, and the right shoulder are visible (YES in S131), the attention level calculation unit 120 calculates the direction vector qi of the person _i (S133), The attention level calculation process is terminated (S134). In this case, the attention level calculation unit 120 _substitutes the position vectors a ₁ ⁱ , a ₂ ⁱ , ^. to calculate the orientation vector _qi .

In the above example, the example using equations (7) and (8) was described as an example of calculating the orientation vector qi of the person _i (S133). For example, the attention level calculation unit 120 may calculate the direction vector qi of the person _i using the following formula instead of the formulas (7) and (8).

As shown in Equation (9), the attention level calculation unit 120 calculates position vectors of the nose (j=1), neck (j=6), left shoulder (j=7), and right shoulder (j=8). A direction vector q _i is calculated using the position coordinates (x ₁ ⁱ , x ₆ ⁱ , x ₇ ⁱ , x ₈ ⁱ ) of a _k ⁱ in the x-axis direction. In equation (9), w ₁ and w ₂ are parameters, for example w ₁ =1.0 and w ₂ =0.5. FIGS. 10(A) to 10(C) are diagrams showing examples of the relationship between coordinates when w ₁ =1.0 and w ₂ =0.5.

The attention degree calculation unit 120 performs, for example, the following processing. That is, the attention level calculation unit 120 reads out the formula (9) stored in the internal memory. Then, attention level calculation section 120 calculates the x-coordinate of position vector a _k1 ⁱ of each part (j=1, 6, 7, 8) extracted from posture p _i or calculated by interpolation using equation (9). , the orientation vector q _i of the person i is calculated.

The above is an example of the attention level calculation process (S13). The above example describes an example of calculating the orientation vector q _i of the person i. For example, when attention level calculation section 120 receives posture pi+1 of another person ( _i +1) from posture estimation section 110, attention level calculation section 120 performs attention level calculation processing (S13) for this person (i+1), and performs attitude _pi+1. Calculate In this way, the attention level calculation unit 120 calculates the direction vectors qi of all the persons _i appearing in the image.

Returning to FIG. 2, next, the information processing apparatus 100 stores the calculated orientation vector qi of the person _i in the attention level storage unit 140 (S15).

Next, the information processing apparatus 100 determines whether or not to end (S16). For example, it is determined whether or not the user operating the information processing apparatus 100 has operated the end button or whether or not the end command has been input.

When the information processing apparatus 100 ends (YES in S16), the series of processes ends (S17). For example, the information processing apparatus 100 does not end (NO in _S16 ), The above processing may be repeated.

After storing the direction vector qi in the attention level storage unit 140, the information processing apparatus 100 may count the number of persons _i who are looking at the objects 300-1 and 300-2. .

FIG. 11 is a flowchart showing an example of counting processing. For example, it is performed by the attention degree calculation unit 120 .

When the process is started (S140), the attention level calculation unit 120 determines whether or not the direction vector _qi intersects the object m (S141). For example, the attention level calculation unit 120 multiplies the calculated orientation vector q _i by n (n>0), and the n-fold orientation vector q _i is the position coordinates of the installation points of the objects 300-1 and 300-2. or whether it intersects within a certain range around the position coordinates of the installation point. In this case, the attention level calculation unit 120 may determine, for example, whether or not a solution as a quadratic equation can be obtained by substituting the position coordinates of the installation point into the quadratic equation representing the direction vector _qi . good. Alternatively, the attention level calculation unit 120 may determine whether or not a solution can be obtained from a quadratic equation representing the direction vector q _i and a linear equation representing the region within a certain range. The attention degree calculation unit 120 performs such calculation using, for example, position information representing the set points of the objects 300-1 and 300-2 stored in the spatial information storage unit .

When the attention level calculation unit 120 determines that the direction vector q _i intersects the object m (YES in S141), it increases the count value for the object m (S142). For example, when the direction vector q _i multiplied by n intersects the positional coordinates of the installation point of the object 300-1 or the surrounding area, the attention level calculation unit 120 increments the count value of the object 300-1. do.

Next, the attention level calculation unit 120 determines whether to end the counting process (S143). The number is incremented (S145), and it is determined which object m the next person i is focused on (S141, S142).

On the other hand, when the attention level calculation unit 120 determines that the orientation vector _qi does not intersect the object m (No in S141), it performs end determination without increasing the count value for the object m (S143).

For example, when the counting process ends (S144), the attention level calculation unit 120 stores the count value for each object m stored in the internal memory or the like in the attention level storage unit 140, for example. By outputting the count value to the display device, the attention degree calculation unit 120 can notify the user of which objects 300-1 and 300-2 the crowd is paying attention to.

For example, consider a case where a surveillance camera is used to capture an image of a crowd in a city. In this case, the photographed image may include a person whose face is hidden due to obstacles, installation locations of surveillance cameras, or the like. In such a case, it may not be possible to obtain the positional information of the parts of the face.

Information processing apparatus 100 according to the first embodiment performs interpolation processing ( For example, in S132) of FIG. 8, the position information of the part of the face is calculated. Then, the information processing apparatus 100 calculates the direction vector q _i using the position information of the face and the position information of other parts. Therefore, the information processing apparatus 100 can estimate the line-of-sight direction of a person whose face is hidden in the image.

[Second embodiment]
In the first embodiment, an example has been described in which the posture p _i and the like are expressed as vectors in a two-dimensional space. In the second embodiment, an example in which postures p _i and the like are expressed as vectors in a three-dimensional space will be described. Vectors in such a three-dimensional space can be calculated using, for example, multiple cameras (or imaging devices 200).

FIG. 12 is a diagram showing an example of coordinate systems of two cameras and position coordinates in a three-dimensional space. Two cameras shown in FIG. 12 represent, for example, that there are two imaging devices 200 .

In FIG. 12, O represents the origin of the first camera, and O' represents the origin of the second camera. Also, t represents a translation vector from the first camera to the second camera, and (X, Y, Z) represents the three-dimensional position coordinates of the part viewed from the coordinate system of the first camera. Furthermore, R is a rotation matrix representing the rotation angle of the second camera as seen from the first camera, f is the focal length of the first camera (the distance from the origin O to the origin of the image coordinate system of the first camera ) and f′ represent the focal length of the second camera (the distance from the origin O′ to the origin of the image coordinate system of the second camera). Furthermore, (x, y) is the two-dimensional position coordinates of the part in the image of the first camera (or the image coordinate system of the first camera), and (x', y') is the image of the second camera. (or the image coordinate system of the second camera).

FIG. 13 is a diagram showing a configuration example of posture estimation section 110 in the second embodiment.

As shown in FIG. 13, posture estimation section 110 includes first and second CNN processing sections 111-1 and 111-2, first and second candidate point calculation sections 112-1 and 112-2, first and second grouping processing units 113-1 and 113-2. In addition, posture estimation section 110 includes same person identification processing section 114 , camera matrix calculation section 115 , and three-dimensional position calculation section 116 .

The first and second CNN processing units 111-1 and 111-2 perform CNN processing and the like on the image data output from the first and second cameras, and obtain a probability distribution φ(X , W). Each of the first and second CNN processing units 111-1 and 111-2 performs, for example, the same CNN processing as in the first embodiment on the image data from each camera, so that each camera output the probability distribution φ(X, W) for each part of the image captured by .

The first and second candidate point calculation units 112-1 and 112-2 are based on the probability distribution φ(X, W) output from the first and second CNN processing units 111-1 and 111-2. , and the candidate points for each part are calculated. Each of the first and second candidate point calculation units 112-1 and 112-2, for example, similarly to the first embodiment, searches for a local maximum point from the probability distribution φ(X, W), Calculate candidate points.

The first and second grouping processors 113-1 and 113-2 respectively group the candidate points output from the first and second candidate point calculators 112-1 and 112-2. Each of the first and second grouping processing units 113-1 and 113-2 performs grouping based on the distance of each candidate point, for example, as in the first embodiment.

The same person identification processing unit 114 determines whether or not the grouped candidate points output from the first and second grouping processing units 113-1 and 113-2 are candidate points of the same person. Identify using The same person identification processing unit 114 determines that a combination of candidate points with a high degree of similarity is candidate points of the same person, and outputs the candidate points. Details will be explained in an operation example.

The camera matrix calculator 115 calculates camera matrices P and P'. The camera matrix P represents a matrix for transforming the image coordinate system of the first camera into the coordinate system of the three-dimensional position, as shown in FIG. 12, for example. Also, the camera matrix P' represents, for example, a matrix for transforming the image coordinate system of the second camera into the coordinate system of the three-dimensional position. The camera matrix calculation unit 115 outputs each candidate point output from the same person identification processing unit 114 and the calculated camera matrices P and P′ to the three-dimensional position calculation unit 116 . A calculation example of the camera matrices P and P' will be described in an operation example.

The three-dimensional position calculation unit 116 converts the grouped candidate points (two-dimensional position coordinates) of each part into three-dimensional position coordinates using, for example, the camera matrices P and P′, and generates a three-dimensional position vector Output the pose p _i containing . Details will be explained in an operation example.

FIG. 14 is a flow chart showing an operation example in the second embodiment. For example, the information processing apparatus 100 performs processing according to the flowchart shown in FIG. 14 instead of the flowchart shown in FIG.

When the information processing apparatus 100 starts processing (S20), the first camera captures an image of the crowd (S21), and the second camera captures an image of the same crowd (S23). For example, there are two imaging devices 200, one imaging device 200 serves as a first camera, and the other imaging device 200 serves as a second camera, each of which photographs a crowd.

Next, the information processing apparatus 100 estimates the posture of each person included in the image captured by the first camera (S22), and also estimates the posture of each person included in the image captured by the second camera. (S24). For example, in the first CNN processing unit 111-1, the first candidate point calculation unit 112-1, and the first grouping processing unit 113-1, each person included in the image captured by the first camera Estimate pose. Further, for example, in the second CNN processing unit 111-2, the second candidate point calculation unit 112-2, and the second grouping processing unit 113-2, each Estimate the pose of a person.

Next, the information processing apparatus 100 performs same person identification processing on the images captured by the two cameras (S25).

FIG. 15A is a flowchart showing an example of same person identification processing. For example, it is performed in the same person identification processing unit 114 .

When starting the same person identification processing (S250), the same person identification processing unit 114 trims (or cuts out or cuts) the image of the person captured by the first camera (S251), and the image of the person captured by the second camera is trimmed (or cut out) (S251). Then, the image of the person who has been photographed is trimmed (S252). For example, the same person identification processing unit 114 performs the following processing.

That is, the same person identification processing unit 114 receives grouped candidate points from the first and second grouping processing units 113-1 and 113-2. Therefore, based on the candidate points, the same person identification processing unit 114 assigns the pixel values of the images surrounding the entire grouped candidate points within a certain range to the pixel values of the first and second images. Crop the image of the person by extracting from the image data. For example, the pixel value of each pixel in the image of a person is obtained by the first and second CNN processing units 111-1 and 111-2, the first and second candidate point calculation units 112-1 and 112-2, and the It is input to the same person identification processing unit 114 via the first and second grouping processing units 113-1 and 113-2.

Next, the same person identification processing unit 114 performs similarity calculation processing (S253).

FIG. 15B is a flowchart showing an example of similarity calculation processing.

When starting the similarity calculation process (S2530), the same person identification processing unit 114 trims the images of the parts of the person captured by the first and second cameras (S2531, S2533). Also in this case, for example, the same person identification processing unit 114 extracts the pixel values of the image around the candidate point within a certain range for each candidate point, thereby extracting the image of the part of the person. trim. The same person identification processing unit 114 trims the image of such a region, for example, for each image captured by the first and second cameras.

Next, the same person identification processing unit 114 calculates a color histogram for each trimmed part image (S2532, S2534). For example, the same person identification processing unit 114 performs the following processing.

That is, the same person identification processing unit 114 collects each pixel of the image of each part into a predetermined cell (for example, 8×8 pixels), and for each predetermined cell, calculates the number of appearances of each pixel value (or gradation value) of RGB. to get Such processing may be performed by a known method. For example, a color histogram may be calculated by calculating a CSS (Color Self-Similarity) feature amount, which is a local feature amount using color information. The same person identification processing unit 114 calculates a color histogram for each image captured by the first and second cameras.

Next, the same person identification processing unit 114 calculates the mean square error using the color histograms (S2532, S2534) and calculates the degree of similarity (S2535). For example, the same person identification processing unit 114 performs the following processing.

That is, the same person identification processing unit 114 creates a color histogram (S2532) corresponding to the image of a certain part taken by the first camera and a color histogram (S2532) corresponding to the image of the part taken by the second camera. S2534) to calculate the mean squared error. The color histogram is calculated as the number of appearances of a pixel value for each predetermined cell for images of regions captured by different cameras. Therefore, the same person identification processing unit 114 calculates the square of the error between the two appearance counts, and calculates the average value of the calculated values for the entire part. The same person identification processing unit 114 calculates the reciprocal of the calculated average value as the degree of similarity. The same person identification processing unit 114 calculates such a degree of similarity for each part.

After calculating the degree of similarity, the same person identification processing unit 114 ends the degree of similarity calculation processing (S2536).

Returning to FIG. 15A, next, the same person identification processing unit 114 searches for a combination with a high degree of similarity (S254). For example, the same person identification processing unit 114 determines that the person photographed by the first camera and the person photographed by the second camera are the same person when all of the plurality of similarities calculated for each part are equal to or greater than the similarity threshold. If not, it is determined that they are not the same person. Such a determination is an example, and the same person identification processing unit 114 determines that the person is the same person when the number of parts whose similarity is equal to or higher than the similarity threshold is equal to or higher than the number threshold. If not, it may be determined that they are not the same person. For example, the higher the degree of similarity, the higher the probability that the partial images are derived from the same person.

Then, the same person identification processing unit 114 ends the same person identification processing (S255).

Returning to FIG. 14, the information processing apparatus 100 then calculates the camera matrices P and P' (S26). Here, the camera matrices P and P' will be explained.

As shown in FIG. 12, first, the three-dimensional position coordinates (X, Y, Z) of the part viewed from the coordinate system of the first camera are converted to the two-dimensional position coordinates (x , y), the two-dimensional position coordinates (x′, y′) of the part in the image of the second camera, the translation vector t, and the rotation matrix R.

First, according to the investment projection model, the three-dimensional position coordinates (X, Y, Z) are obtained by the following equation (9- 1) or described by formula (9-2).

Similarly, the three-dimensional position coordinates (X', Y', Z') of the part viewed from the coordinate system of the second camera are described by the following formula (9-3) or formula (9-4). be done. Here, k and k' are real numbers that are not "0".

On the other hand, as shown in FIG. 12, the relationship between the coordinate system (X', Y', Z') of the second camera and the coordinate system (X, Y, Z) of the first camera is the translation vector t and the rotation matrix R can be described by the following equation (9-5) using

By simultaneously solving the above equations (9-1), (9-2), (9-3), (9-4), and (9-5), the three-dimensional position coordinates (X , Y, Z), two-dimensional position coordinates (x, y), and two-dimensional position coordinates (x', y') are described by the following four equations using a translation vector t and a rotation matrix R. I understand that.

Here, it is a matrix called a P, P' camera matrix. If the two-dimensional position coordinates (x, y), the two-dimensional position coordinates (x', y'), the translation vector t, and the rotation matrix R are known, the three-dimensional position coordinates (X, Y, Z) can be obtained by the above equation ( 9-6), equations (9-7), equations (9-8), and equations (9-9) can be solved in reverse.

Based on the above, the camera matrix calculation unit 115 calculates camera matrices P and P' using, for example, the following equations.

In equations (10) and (11), f ₀ is a parameter for adjusting the scale, for example f ₀ =1.

The camera matrix calculation unit 115 reads, for example, the equations (10) and (11) stored in the internal memory, the focal lengths f and f′ of the first and second cameras, the rotation matrix R, the first A vector t directed from the origin O of the camera to the origin O' of the second camera is substituted into the equations (10) and (11). The camera matrix calculator 115 also stores focal lengths f, f′, rotation matrix R, and vector t in the internal memory, for example, and reads out these values from the internal memory to obtain equations (10) and (11). should be substituted for

Next, the information processing device 100 calculates the three-dimensional position (X, Y, Z) of each part (S27). For example, the three-dimensional position calculation unit 116 calculates (X, Y, Z) by solving the following simultaneous equations.

For example, the 3D position calculator 116 performs the following processing. That is, the three-dimensional position calculation unit 116 reads out the equations (12) to (15) stored in the internal memory, the components of the camera matrices P and P′ calculated in S26, and the position coordinates (x , y) and (x', y') are substituted into equations (12) to (15). Then, the three-dimensional position calculation unit 116 obtains the three-dimensional position coordinates (X, Y, Z) of the part by solving the simultaneous equations (12) to (15). In this case, the three-dimensional position calculation unit 116 obtains the posture p _i expressed as a three-dimensional vector by substituting the calculated three-dimensional position coordinates of the part into Equation (1).

After that, the information processing apparatus 100 uses the orientation p _i to perform the processes of S13 to S17 to calculate the orientation vector q _i , as in the first embodiment, and ends the series of processes. (S28). In the processing from S13 to S17, the information processing apparatus 100 performs processing using, for example, the three-dimensional position coordinates, and obtains the orientation vector q _i expressed as a three-dimensional position vector.

As described above, in the second embodiment, the posture p _i and the orientation vector q _i can be expressed as three-dimensional position vectors. It becomes possible to obtain the orientation vector _qi .

[Third Embodiment]
In the first embodiment, in the attention degree calculation process (for example, S130 in FIG. 8), attention is paid to a certain part, and the process of interpolating a part that is not visible has been described. For example, when the number of unseen parts k of a certain person i is not equal to or less than a threshold, even if such parts k are calculated by interpolation, the position vector a _k ⁱ =(x _k ⁱ y _k ⁱ ) of the part k is ^T. may not be calculated with high accuracy.

The information processing apparatus 100 according to the third embodiment is based on empirical measurement that the posture of a person _i tends to resemble the posture of an adjacent person t. The position vector a _k ⁱ =(x _k ⁱ y _k ⁱ ) ^T of the person i is calculated using the pose p _t of t.

FIG. 16 is a flowchart showing an example of attention level calculation processing according to the third embodiment. However, the crowd is photographed by the imaging device 200 (S11 in FIG. 2), and the posture estimating unit 110 obtains the posture p _{i of the person i} and the posture p _t of the person t through the posture estimation process (S12 in FIG. 2). shall be provided.

Upon starting the attention level calculation process (S130), the attention level calculation unit 120 determines whether or not the number of unseen parts (especially facial parts) of a certain person i is equal to or less than a threshold (S135). For example, attention level calculation section 120 determines whether or not the number of v _j ⁱ =0 in posture p _i output from posture estimation section 110 is greater than or equal to a threshold. In this case, for example, the attention level calculation unit 120 focuses on a part of the face (j=1 to 5) and determines whether or not the number of v _j ⁱ of that part being “0” is equal to or greater than a threshold. may

When the number of unseen parts is greater than the threshold (NO in S135), the attention level calculation unit 120 calculates the position vector ak ⁱ =(x _k ⁱ y _k ⁱ ) ^T of the unseen part _k by interpolation. (S136). For example, the attention degree calculation unit 120 calculates the position vector a _k ⁱ using the following formula.

In ^Equation ( ^{16 )} , _{a k1} ^{i ,} _{a k2} ⁱ _{, .} Position vectors of parts visible to an adjacent person t are respectively represented. Also, A _k ^{M1 and M2} are matrices of 2M rows and 2M ₁ +2M ₂ columns . A _k ^{M1, M2} and b _k ^{M1, M2} are calculated using, for example, the set P (teaching data) of postures s using the following equations, similar to equation (6).

For example, the attention degree calculation unit 120 performs the following processing. That is, attention level calculation section 120 obtains the center coordinates of person i and the center coordinates of person t based on posture p _i and posture p _t output from posture estimation section 110 . The attention level calculation unit 120 determines that the person t is adjacent to the person i if the two center coordinates are within the threshold. If it is determined that they are adjacent, the attention level calculation unit 120 reads Equation (17) from the internal memory and calculates A _k ^{M1, M2} and b _k ^{M1, M2} . The attention level calculation unit 120 calculates the position vector v _j ⁱ =1 from the posture p _i , the position vector v _j ^t =1 from the posture p _t , and the calculated A _k ^{M1, M2} and b By substituting _k ^{M1 and M2} into the right side of the equation (16), the position vector a _k ⁱ of the invisible part k of the person i is calculated.

Next, attention level calculation section ¹²⁰ calculates position ^vectors a _k1 ⁱ , a _k2 ⁱ _{, .} Using this, the direction vector qi of the person _i is calculated (S137). For example, the attention level calculation unit 120 calculates the direction vector _qi using the equation (7) as in the first embodiment.

Then, the attention degree calculation unit 120 ends the attention degree calculation processing (S138).

On the other hand, when the number of unseen parts is equal to or less than the threshold (YES in S135), attention degree calculation section 120 calculates orientation vector _qi of person _i based on posture pi (S137). In this case, the posture p _i may not include part of the position coordinates of the body parts. Using equation (7) with W, the orientation vector q _i is calculated.

Thereafter, the information processing apparatus 100 performs the same processing (S15 to S17) as in the first embodiment.

As described above, in the third embodiment, when the number of invisible parts is larger than the threshold, the information processing apparatus 100 uses the position vector of the person t adjacent to the person i to be processed. , the position vector of the part of the person i is calculated by interpolation. Therefore, the information processing apparatus 100 according to the third embodiment can accurately calculate the position of the unseen part, and can accurately calculate the direction vector q _i as compared to the case where the interpolation process is not performed. It is also possible to calculate well.

[Fourth Embodiment]
Equations (5) and (7) used in the first embodiment are expressed as linear functions. Therefore, the approximation ability of the function to the object may be limited. Therefore, in the fourth embodiment, non-linear functions are used for equations (5) and (7). As a result, for example, the approximation ability for the object is enhanced compared to the case of using a linear function.

The information processing apparatus 100 according to the fourth embodiment uses the following equation instead of equation (5) to calculate the position vector a _k ⁱ of the invisible part k by interpolation.

In Equation (18), D _l1 ^M represents a matrix of l1 rows and 2M columns, D _l2 ^l1 represents a matrix of l2 rows and 11 columns, and D _k ^l2 represents a matrix of k rows and l2 columns. Also, a _k1 ⁱ , a _k2 ⁱ _, ^. Also, in equation (19), δ(x) is an activation function, and α and β respectively represent predetermined real numbers that satisfy α≠β.

The matrices D _l1 ^M , D _l2 ^l1 , and D _k ^l2 are, for example, matrices obtained by solving the following equations using the set P of postures s, like Equation (6).

For example, the attention level calculation unit 120 performs the following process as the interpolation process (S132 in FIG. 8). That is, the attention level calculation unit 120 _reads s _k ⁱ and s _k1 ⁱ , s _k2 ⁱ , ^. Substitute to obtain matrices D _l1 ^M , D _l2 ^l1 , and D _k ^l2 . Then, attention level calculation section 120 reads equation (18) stored in the internal memory from the internal memory, and obtains matrices D _l1 ^M , D _l2 ^l1 , and D _k ^l2 obtained by equation (20), and attitude s a _k1 ⁱ , a _k2 ⁱ _, ^. Thereby, the attention level calculation unit 120 calculates the position vector a _k ⁱ of the part k that is not visible.

Further, the information processing apparatus 100 according to the fourth embodiment calculates the orientation vector qi of the person _i using the following equation instead of the equation (7).

In equation (21), W _l1 ^J represents a matrix of l1 rows and 2J columns, W _l2 ^l1 represents a matrix of l2 rows and 11 columns, and W _k ^l2 represents a matrix of k rows and l2 columns.

The matrices W _l1 ^J , W _l2 ^l1 , and W _k ^l2 are, for example, matrices obtained by solving the following equations using the set P of postures s, like equation (8).

For example, the attention degree calculation unit 120 performs the following processing as the direction vector _qi calculation processing (S133 in FIG. 8). That is, the attention level calculation unit 120 ^reads _{q s} ⁱ and s ₁ ⁱ , s ₂ ⁱ , . is read from the internal memory and substituted into equation (22) to obtain the matrices W _l1 ^J , W _l2 ^l1 and W _k ^l2 . The attention level calculation unit 120 reads the expression (21) stored in the internal memory from the internal memory, and extracts from the matrix W _l1 ^J , W _l2 ^l1 , W _k ^l2 obtained by the expression (22) and the orientation s _{A 1} ⁱ , a ₂ ^{i ,} _.

Note that the example using the three matrices D _l1 ^M , D _l2 ^l1 , and D _k ^l2 in Equation (18) has been described. For example, attention level calculation section 120 may calculate equation (18) using two of these matrices. Also, for example, the attention level calculation unit 120 may calculate Equation (21) using two of these matrices instead of the three matrices W _l1 ^J , W _l2 ^l1 and W _k ^l2 .

[Fifth embodiment]
1st Embodiment demonstrated the example which calculates attention degree. In the fifth embodiment, an example of detecting a change in the calculated attention level will be described. By detecting such a change in the degree of attention in the information processing apparatus 100, it is possible to detect, for example, the occurrence of a sudden change in the direction in which the line of sight of the crowd is directed, and the occurrence of such a situation can be detected. It is also possible to detect the time when the

FIG. 17 is a diagram showing a configuration example of the information processing system 10 according to the fifth embodiment.

As shown in FIG. 17 , the information processing device 100 further includes a change detection section 150 . The change detection unit 150 reads attention levels from the attention level storage unit 140, and detects, for example, changes over time. The change detection unit 150 can output the detection result to, for example, an external display device to notify the user.

FIG. 18 is a flow chart showing an operation example of the information processing apparatus 100 . In FIG. 18, the processing from S11 to S15 is the same as in the first embodiment.

When the information processing apparatus 100 records the attention level (or orientation vector q _i ) of each person i in the attention level storage unit 140 (S15), the information processing apparatus 100 performs attention level change detection processing (S18).

FIG. 19 is a flowchart showing an example of attention degree change detection processing. Each process in FIG. 19 is performed by the change detection unit 150, for example.

When the attention level detection process is started (S180), the change detection unit 150 changes the attention level vector u _i ^t to the time (Tn)<t≦(Tm) and the time (Tm)<t. (S181).

Here, the interest vector u _i ^t is obtained by normalizing the direction vector q _i ^t of the person i at time t, for example, and is defined by the following equation.

Also, if the set of interest vectors at time (Tn)<t≦(Tm) (where n>m) is U _Tn<t≦Tm , then the set of interest vectors _UT-n<t≦Tm is defined, for example, by the following equation.

FIG. 20 is a diagram showing an example of the relationship between time t, time (Tn), and time (Tm). Assuming that one attention vector u is calculated at each time t, when time t is the current time T, n attention vector u are calculated from time t=Tn to time t=T. Calculated. Further, the number of interest vectors calculated from time (Tn) to time t=(Tm) is (nm). The number of interest vectors calculated up to time T is m. As shown in FIG. 20, the first half and the second half are separated at the time (Tm), and the set of attention level vectors U _T-n<t≦Tm is the first half, the time (Tm). to time ( _Tm ⁾ .

The change detection unit 150 performs, for example, the following process as the process of S181. That is, the change detection unit 150 reads out the n direction vectors q _i from the time (Tn) to the current time T from the attention level storage unit 140 . Then, the change detection unit 150 reads out the equation (22) stored in the internal memory, substitutes the orientation vector qi into the equation (22), and calculates n attention level vectors _{u i t} ^. The change detection unit 150 combines the n attention level vectors u _i ^t with a set of (nm) attention level vectors until the time (Tn)<t≦(Tm) and the time (T −m) < t into a set of m attention vectors. The former set of attention level vectors is represented, for example, by Equation (24).

Returning to FIG. 19, the change detection unit 150 selects a set of (nm) interest level vectors _UT-n<t≦Tm until time (Tn)<t≦(Tm). In contrast, the probability distribution p(u _i ^t ) of the attention vector u _i ^t is estimated (S182). In this process, it is assumed that the probability distribution p(u _i ^t ) is distributed along, for example, a mixed von Mises distribution (or a von Mises Fisher distribution). The mixed von Mises distribution, for example, assumes that the starting point of the attention vector u _i ^t is the origin in a d-dimensional (d is, for example, 2 or 3) space, the direction of the attention vector u _i ^t is It represents whether it is distributed stochastically to

FIG. 21A is a diagram showing an example of the probability distribution of the attention level vector u _i ^t for the input image, and FIG. 21B is a diagram showing an example of the probability distribution for the input image. As shown in FIGS. 21(A) and 21(B) , since the crowd mainly directs their line of sight in two directions in the image, the directions of the attention vector u _i ^t are also distributed mainly in two directions. ing. FIG. 21B shows an example of a mixed von Mises distribution.

The change detection unit 150 estimates a probability distribution p(u _i ^t ) for a set U _T-n<t≦Tm of attention level vectors u _i ^t using, for example, the following equation.

In Equation (25), M(u _i ^t |μ _j , σ _j ) is calculated using, for example, the following equation.

In Equation (26), I _ρ (γ) is calculated using, for example, the following equation.

In equations (25) to (27), d represents the number of dimensions (2 or 3) of the interest vector u _i ^t , and I _ρ (γ) represents the ρ-th order modified Bessel function of the first kind. Also, in equations (25) to (27), α _j , μ _j , and σ _j are parameters. The parameters α _j , μ _j , σ _j can be estimated using a known expectation maximization method, for example, using the set of interest vectors U _T-n<t≦Tm .

For example, the change detection unit 150 performs the following process in S182. That is, the change detection unit 150 applies the expected value maximization method or the like to the set of attention level vectors _UT-n<t≦Tm until the time (Tn)<t≦(Tm). to estimate the parameters α _j , μ _j , σ _j . Then, the change detection unit 150 reads out the equations (25) to (27) stored in the internal memory, and converts the estimated parameters α _j , μ _j , σ _j and the interest vector u _i ^t into the equation ( 25) into equation (27), the probability distribution p(u _i ^t ) is estimated.

Returning to FIG. 19, next, the change detection unit 150 calculates the degree of anomaly β of the attention vector set U _T−m<t at the time (T−m)<t (S183). The degree of abnormality β is calculated by, for example, the following formula.

As shown in equation (28), the degree of anomaly β is, for example, a probability distribution _p Based on (u _i ^t ), it represents how much the distribution of the set U _T-m<t of attention level vectors at time (Tm)<t deviates. When the distribution of the attention vector set U _T-m<t deviates from the probability distribution p(u _i ^t ) for the attention vector set U _T-n<t≦Tm , the value of the degree of abnormality β can take a large value, otherwise it can take a small value.

For example, the change detection unit 150 reads the equation (28) stored in the internal memory, and substitutes the probability distribution p(u _i ^t ) estimated in S182 into the equation (28), thereby obtaining the degree of abnormality β as calculate.

Next, the change detection unit 150 determines whether or not the degree of abnormality β is equal to or greater than a threshold (S184). When the degree of abnormality β is greater than or equal to the threshold (YES in S184), change detection unit 150 notifies the user of the change. Then, the change detection unit 150 ends the attention level change detection process (S186). On the other hand, when the degree of abnormality β is smaller than the threshold (NO in S184), the change detection unit 150 ends the attention level change detection process without performing the process of S184 (S186).

The change detection unit 150 performs, for example, the following processes. That is, the change detection unit 150 compares the degree of abnormality β calculated in S183 with the threshold value stored in the internal memory. For example, time t=(T−m)) is output to an external display device. On the other hand, when the degree of abnormality β is smaller than the threshold, the change detection unit 150 terminates the process without notifying the change.

FIGS. 21A to 21D show, for example, how the line-of-sight direction of the crowd changes at a certain time (Tm). As shown in FIGS. 21(A) and 21(C), when the line-of-sight direction changes, the direction vector q _i changes and the attention vector u _i ^t also changes. Therefore, when the probability distribution p(u _i ^t ) for the attention vector set U _T-n<t≦Tm is used as a reference (FIG. 21B), the attention vector set U _T-m<t The distribution deviates greatly (FIG. 21(D)), and the degree of anomaly β also increases.

The information processing apparatus 100 can notify the user of the occurrence of the change and the time at which the change occurs by outputting the detection result of such a change to an external display device. As a result, for example, in city surveillance for security purposes, it is possible to notify the user of the occurrence of an event in the line-of-sight direction, the time of occurrence, and the like.

[Other embodiments]
FIG. 22 is a diagram showing a hardware configuration example of the information processing apparatus 100. As shown in FIG.

The information processing apparatus 100 includes an interface section 180 , a memory 181 , a CPU (Central Processing Unit) 182 , a ROM (Read Only Memory) 183 , and a RAM (Random Access Memory) 184 .

The interface unit 180 outputs image data output from the imaging device 200 to the memory 181 and the CPU 182, for example.

The memory 181 corresponds to, for example, the spatial information storage unit 130 and the attention level storage unit 140 of the first embodiment. Also, the memory 181 corresponds to, for example, the internal memory in the posture estimation unit 110, the attention level calculation unit 120, and the change detection unit 150 of the fifth embodiment.

The CPU 182, for example, reads a program stored in the ROM 183, loads it into the RAM 184, and executes the loaded program, thereby realizing the functions of the posture estimation unit 110, the attention degree calculation unit 120, and the change detection unit 150. . The CPU 182 corresponds to, for example, the posture estimation unit 110, the attention level calculation unit 120, and the change detection unit 150.

In place of the CPU 182, a processor such as an MPU (Micro Processing Unit), a DSP (Digital Signal Processor), or an FPGA (Field Programmable Gate Array), a controller, or the like may be used.

The above is summarized as follows.

(Appendix 1)
estimating the positional information of the parts of the person included in the input image data using the correct data on the parts of the person;
estimating the line-of-sight direction of the person included in the image based on the position information of other parts that could be estimated when the position information of the part of the face could not be estimated among the parts;
A program characterized by causing a computer to execute processing.

(Appendix 2)
The program according to Supplementary Note 1, wherein the positional information of the parts of the face includes positional information of at least one of nose, left eye, right eye, left ear, right ear, neck, left shoulder, and right shoulder.

(Appendix 3)
For the input image data, using the correct data, the probability distribution of each part of the person is calculated, based on the probability distribution of each part, candidate points of each part are searched, The program according to Supplementary Note 1, wherein the position information of the body part of the person is estimated by grouping the candidate points of the body part for each person.

(Appendix 4)
Filtering processing is performed on image data of a predetermined image using the correct data, pooling processing is performed for extracting a maximum value in units of blocks including a plurality of pixels from the image data after filtering processing, and the filtering processing and the pooling are performed. are repeated to generate correct data of the probability distribution of each part of the person, perform the filtering process on the input image data using the correct data of the probability distribution of each part of the person, and perform the filtering process after the filtering process. The program according to Supplementary Note 3, wherein the pooling process for extracting the maximum value for each block is performed from the image data, and the filtering process and the pooling process are repeated to generate the probability distribution of each part. .

(Appendix 5)
3. The program according to claim 3, wherein a position coordinate of a local maximum point is set as a candidate point for each part based on the probability distribution of each part.

(Appendix 6)
When the local maximum point is smaller than a threshold, it is determined that the site is not included in the image, and when the local maximum point is greater than or equal to the threshold, it is determined that the site is included in the image. 5 program.

(Appendix 7)
A pooling process of filtering the input image data using correct data of the probability distribution connecting each part, and extracting the maximum value in units of blocks containing a plurality of pixels from the image data after the filtering process. and repeating the filtering process and the pooling process to generate a probability distribution indicating the degree of connection of each part, and based on the generated probability distribution and the candidate points of each part, the candidates for each person 3. The program according to appendix 3, wherein the points are grouped.

(Appendix 8)
Based on the input image data, using the correct data regarding the body part of the person, a posture vector containing the position information of the body part included in the image and a parameter indicating whether or not the body part is included in the image is generated. The program according to appendix 1, characterized by estimating.

(Appendix 9)
Supplementary note 8, wherein, based on the parameter indicating whether or not the facial part is included in the image, it is determined that the position information of the facial part could not be estimated among the facial parts. program.

（ただし、Ａ _ｋ ^Ｍ は２行２Ｍ列の行列、ｂ_ｋ ^Ｍは２行１列の縦ベクトル、Ｍは０＜Ｍ≦ｊを満たす整数をそれぞれ表し、Ａ _ｋ ^Ｍ とｂ_ｋ ^Ｍは、姿勢ｓの集合Ｐを用いて、以下の式（３０）を解くことで算出される。）

(Appendix 10)
^By substituting position _vectors a _k1 ⁱ , a _k2 ⁱ , . The program according to Supplementary Note 1, wherein the position vector a _k ⁱ representing the position information of the part of the face is calculated.

(where A _k ^M is a matrix of 2 rows and 2 M columns, b _k ^M is a column vector of 2 rows and 1 column, M represents an integer that satisfies 0 < M ≤ j, and A _k ^M and b _k ^M are It is calculated by solving the following equation (30) using the set P of postures s.)

（ただし、Ｗは２行２Ｊ列の行列、ｗ_０は２行１列の縦ベクトル、ｊは前記人物の部位をそれぞれ表し、姿勢ｓの集合Ｐと、姿勢ｓが持つ向きベクトルｑ_ｓ、及び姿勢ｓの部位番号ｋの部位の位置ベクトルｓ_ｋ＝（ｘ_ｋ，ｙ_ｋ）^Ｔを用いて、Ｗとｗ_０は、以下の式（３２）を解くことで算出される。）

(Appendix 11)
Position _vectors a ₁ ⁱ , a ₂ ⁱ , ^. The program according to Supplementary Note 1, wherein the direction vector qi representing the line-of-sight direction of the person _i included in the image is calculated by substituting (31).

(where W is a matrix of 2 rows and 2 J columns, w ₀ is a vertical vector of 2 rows and 1 column, j represents a part of the person, a set P of postures s, a direction vector q _s of postures s, W and w ₀ are calculated by solving the following equation (32) using the position vector _sk = (x _k , y _k ) ^T of the part with part number k in posture s.

（ただし、式（３３）において、ｗ_１とｗ_１はパラメータを表す） (Appendix 12)
2. The program according to appendix 1, wherein the direction vector q _i representing the line-of-sight direction of the person i included in the image is calculated based on the following equation (33) read out from the internal memory.

(where w ₁ and w ₁ represent parameters in equation (33))

(Appendix 13)
In the equation (33) read out from the internal memory, positional coordinates x in the x-axis direction of each part of the nose, neck, left shoulder, and right shoulder, which represent the positional information of the facial parts and the positional information of the other parts. 12. The program according to Supplementary note 11, wherein the orientation vector q _i representing the line-of-sight direction of the person i included in the image is calculated by substituting ₁ ⁱ , x ₆ ⁱ , x ₇ ⁱ , and x ₈ ⁱ .

(Appendix 14)
12. The program according to Supplementary Note 11, wherein the number of persons looking at the object is counted based on the orientation vector _qi and the positional coordinates of the object read from the spatial information storage unit.

(Appendix 15)
First image data of a first image captured by a first camera is represented as two-dimensional coordinates of a part of a person included in the first image using the correct data. 1 position coordinates are estimated, and based on the second image data of the second image captured by the second camera, the correct data is used to determine the part of the person included in the second image. estimating a second position coordinate represented as a two-dimensional coordinate;
converting the first position coordinates and the second position coordinates into three-dimensional position coordinates of a part;
The program according to Supplementary Note 1, wherein the three-dimensional position coordinates are used to calculate the position information of the part of the person's face, and to estimate the line-of-sight direction of the person included in the image.

(Appendix 16)
Based on the first number of appearances of each pixel value of the part in the first image and the second number of appearances of each pixel value of the part in the second image, in the first image determining whether or not the person and the person in the second image are the same person; 16. The program according to appendix 15, characterized by performing

(Appendix 17)
The following equations (34) and (35) read out from the internal memory are given by f representing the focal length from the origin of the first camera to the origin of the first position coordinates, f from the origin of the second camera f′ representing the focal length to the origin of the second position coordinates, R representing the rotation angle of the second camera viewed from the first camera, and direction from the first camera to the second camera Calculate the camera matrices P and P' by substituting t representing the translation vector,
(36) to (39) read out from the internal memory, each component of the camera matrices P and P′, (x, y) representing the first position coordinates, and the second position coordinates By substituting (x', y') representing and solving the simultaneous equations shown in equations (36) to (39), the transformation to the three-dimensional position coordinates of the part is performed. 15. The program according to 15 above.

(Appendix 18)
when the number of pieces of position information of facial parts among the parts of the first person that could not be estimated is greater than a threshold, position information of other parts that could be estimated in the first person; Calculating the position information of the part of the face that could not be estimated in the first person based on the position information of the part that could be estimated in the second person adjacent to the first person. The program according to appendix 1, characterized by:

（ただし、式（４０）において、δ（ｘ）は、以下の式（４１）に示す活性化関数であり、式（４０）において、行列Ｄ_ｌ１ ^Ｍ，Ｄ_ｌ２ ^ｌ１，及びＤ_ｋ ^ｌ２は、姿勢ｓの集合Ｐを用いて、以下の式（４２）を解くことで得られる行列である。）

(Appendix 19)
^By substituting position _vectors a _k1 ⁱ , a _k2 ⁱ , . The program according to Supplementary Note 1, wherein the position vector a _k ⁱ representing the position information of the part of the face is calculated.

(where, in equation (40), δ(x) is the activation function shown in equation (41) below, and in equation (40), matrices D _l1 ^M , D _l2 ^l1 , and D _k ^l2 are It is a matrix obtained by solving the following equation (42) using the set P of postures s.)

（ただし、式（４３）において、δ（ｘ）は、以下の式（４４）に示す活性化関数であり、式（４３）において、行列Ｗ_ｌ１ ^Ｊ，Ｗ_ｌ２ ^ｌ１，Ｗ_ｋ ^ｌ２は、姿勢ｓの集合Ｐを用いて、以下の式（４５）を解くことで得られる行列である。）

(Appendix 20)
_By substituting position ^vectors a ₁ ⁱ , a ₂ ⁱ , . , the program according to appendix 1, which calculates a direction vector q _i representing the line-of-sight direction of the person i included in the image.

(However, in equation (43), δ(x) is the activation function shown in equation (44) below, and in equation (43), matrices W _l1 ^J , W _l2 ^l1 , and W _k ^l2 are the orientation It is a matrix obtained by solving the following equation (45) using the set P of s.)

(Appendix 21)
The program according to Supplementary Note 1, further detecting a change in the estimated line-of-sight direction of the person and outputting a detection result.

(Appendix 22)
When the time t is the current time T, the attention level vector obtained by normalizing the direction vector representing the estimated line-of-sight direction of the person is obtained from the first Divided into a set of interest vectors and a set of second interest vectors obtained from time (T−m) to time t, and based on the first interest vector set, the second attention vector The degree of abnormality of the set of degree vectors is calculated, and when the degree of abnormality is equal to or greater than a threshold, a detection result indicating that the line-of-sight direction has changed at time (Tm) is output. The program according to Appendix 21.

前記内部メモリから読み出した以下の式（４７）に、注目度ベクトルｕ_ｉ ^ｔを代入して、時刻（Ｔ－ｎ）から時刻（Ｔ－ｍ）までに取得した第１の注目度ベクトルの集合における注目度ベクトルｕ_ｉ ^ｔの確率分布ｐ（ｕ_ｉ ^ｔ）を算出し、

前記内部メモリから読み出した以下の式（４８）に、確率分布ｐ（ｕ_ｉ ^ｔ）を代入することで、異常度βを算出することを特徴とする付記２１記載のプログラム。

（ただし、式（４６）は、以下の式（４９）と式（５０）を用いて算出され、α_ｊ，μ_ｊ，σ_ｊはパラメータを表す。）

(Appendix 23)
By substituting the direction vector q _i representing the line-of-sight direction of the person obtained from time (Tn) to time (Tm) into the following equation (46) read from the internal memory, the time (T- n) to the time (Tm) to obtain the interest vector u _i ^t included in the set of the first interest vectors,

A set of first attention vectors obtained from time (Tn) to time (Tm) by substituting attention vector u _i ^t into the following equation (47) read from the internal memory Calculate the probability distribution p(u _i ^t ) of the interest vector u _i ^t in

22. The program according to appendix 21, wherein the degree of abnormality β is calculated by substituting the probability distribution p(u _i ^t ) into the following equation (48) read out from the internal memory.

(However, Equation (46) is calculated using Equations (49) and (50) below, and α _j , μ _j , and σ _j represent parameters.)

(Appendix 24)
When the position information of the part of the face cannot be estimated among the parts, the position information of the part of the face is calculated using the position information of other parts that can be estimated, and the position information of the part of the face is calculated. The program according to Supplementary Note 1, wherein the line-of-sight direction of the person included in the image is estimated based on the positional information of the part of (1) and the positional information of the other parts.

(Appendix 25)
a posture estimation unit for estimating position information of human body parts included in an image by using correct data on human body parts for input image data;
Attention degree calculation for estimating the line-of-sight direction of a person included in an image based on the position information of other parts that could be estimated when the position information of the part of the face could not be estimated among the parts. An information processing apparatus comprising: a section;

(Appendix 26)
An information processing method in an information processing device having a posture estimation unit and an attention level calculation unit,
estimating the position information of the parts of the person included in the image by using the correct data regarding the parts of the person with respect to the input image data by the posture estimation unit;
When the attention level calculation unit cannot estimate the position information of the face part among the parts, the line of sight of the person included in the image is calculated based on the position information of other parts that could be estimated. An information processing method characterized by estimating a direction.

10: Information processing system 100: Information processing device 110: Posture estimation unit 111: CNN processing unit 111-1: First CNN processing unit 111-2: Second CNN processing unit 112: Candidate point calculation unit 112-1: First candidate point calculation unit 112-2: Second candidate point calculation unit 113: Grouping processing unit 113-1: First grouping processing unit 113-2: Second grouping processing unit 114: Same person identification processing unit 115: camera matrix calculation unit 116: three-dimensional position calculation unit 120: attention level calculation unit 130: spatial information storage unit 140: attention level storage unit 150: change detection unit 200: imaging devices 300-1, 300-2: target object

Claims (11)

estimating the positional information of the parts of the person included in the input image data using the correct data on the parts of the person;
estimating the line-of-sight direction of the person included in the image based on the estimated position information of the part other than the face when the position information of the part of the face cannot be estimated among the parts;
A program characterized by causing a computer to execute processing. 2. The program according to claim 1, wherein the positional information of said facial parts includes positional information of at least one of nose, left eye, right eye, left ear, right ear, neck, left shoulder, and right shoulder. . Based on the input image data, using the correct data regarding the body part of the person, a posture vector containing the position information of the body part included in the image and a parameter indicating whether or not the body part is included in the image is generated. 2. The program according to claim 1, wherein the estimation is performed. 2. The program according to claim 1, wherein the orientation vector qi representing the line-of-sight direction of the person _i included in the image is calculated based on the following equation (51) read out from the internal memory.

(In equation (51), w ₁ and w ₂ represent parameters. In equation (51), x _j ⁱ indicates the x-coordinate of the position of part j of person i. j=1 is the nose , j=6 for the neck, j=7 for the left shoulder, and j=8 for the right shoulder. ) First image data of a first image captured by a first camera is represented as two-dimensional coordinates of a part of a person included in the first image using the correct data. 1 position coordinates are estimated, and based on the second image data of the second image captured by the second camera, the correct data is used to determine the part of the person included in the second image. estimating a second position coordinate represented as a two-dimensional coordinate;
converting the first position coordinates and the second position coordinates into three-dimensional position coordinates of a part;
2. The program according to claim 1, wherein said three-dimensional positional coordinates are used to calculate positional information of parts of said person's face, and to estimate a line-of-sight direction of said person included in an image. when the number of pieces of position information of facial parts among the parts of the first person that could not be estimated is greater than a threshold, position information of other parts that could be estimated in the first person; Calculating the position information of the part of the face that could not be estimated in the first person based on the position information of the part that could be estimated in the second person adjacent to the first person. 2. The program according to claim 1, characterized by: 2. The program according to claim 1, further detecting a change in the estimated line-of-sight direction of said person and outputting a detection result. When the time t is the current time T, the attention level vector obtained by normalizing the direction vector representing the estimated line-of-sight direction of the person is obtained from the first Divided into a set of interest vectors and a set of second interest vectors acquired from time (T−m) to time t, and based on the first interest vector set, the second interest vector The degree of abnormality of the set of degree vectors is calculated, and when the degree of abnormality is equal to or greater than a threshold, a detection result indicating that the line-of-sight direction has changed at time (Tm) is output. 8. A program according to claim 7. When the position information of the part of the face cannot be estimated among the parts, the position information of the part of the face is calculated using the position information of other parts that can be estimated, and the position information of the part of the face is calculated. 2. The program according to claim 1, wherein the line-of-sight direction of the person included in the image is estimated based on the positional information of the part of and the positional information of the other part. a posture estimation unit for estimating position information of human body parts included in an image by using correct data on human body parts for input image data;
Attention level for estimating the line-of-sight direction of a person included in an image based on the position information of parts other than the face that can be estimated when the position information of the parts other than the face cannot be estimated among the parts. An information processing apparatus comprising: a calculation unit; An information processing method in an information processing device having a posture estimation unit and an attention level calculation unit,
estimating the position information of the parts of the person included in the image by using the correct data regarding the parts of the person with respect to the input image data by the posture estimation unit;
When the attention degree calculation unit cannot estimate the position information of the face part among the parts, the position information of the part other than the face that can be estimated is used to determine the position information of the person included in the image. An information processing method characterized by estimating a line-of-sight direction.

Images