JP2021064025A - Position estimation device, position learning device and program - Google Patents

Abstract

To estimate a position even if an object or the like is completely concealed, when estimating a position of the object or the like in an image.SOLUTION: A person position measurement unit 10 of a position estimation device 1-1 measures a position of each person from an image, and a person density map calculation unit 11 calculates a person density map from the position for each person. A person posture measuring unit 12 measures a posture for each person from the image, and a gaze-grade calculation unit 13 calculates a gaze-degree of the person for each position (coordinates) in a predetermined coordinate system from the position for each person and the posture for each person. A neural network unit 14-1 performs calculation of a neural network using a learned coefficient by using the person density map which is person density for each area and the gaze-degree for each position as input data, and acquires a position estimation value of play equipment as output data.SELECTED DRAWING: Figure 1

Description

The present invention relates to a position estimation device, a position learning device, and a program for estimating the position of an object or the like in an input image.

Conventionally, as a technique for estimating the position of an object, a tag such as a beacon or RFID (Radio Frequency IDentifier) or a reflector (retroreflective material, etc.) is attached to the object, and radio waves, infrared rays, visible light, etc. from the object are passively transmitted. Those that sense targetly or actively are known (see, for example, Patent Document 1).

Patent Document 1 discloses a technique for tracking a subject by using a process of detecting an invisible light marker attached to a subject from an invisible image and a process of detecting the subject from a visible image in combination. ..

Further, there is a technique of estimating the position of an object by extracting an image feature from an input image and determining whether or not the image feature matches a predetermined condition. In this technique, for example, a Haar-Like characteristic image feature is extracted from an input image, and the image feature is discriminated by a boosting method to extract a person's face region (for example, Non-Patent Document 1). reference). This technology is used for exposure of digital cameras, adjustment of focal position, and the like.

JP-A-2017-204757

P.Viola and M.Jones, “Rapid Object Detection using a Boosted Cascade of Simple Features”, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, Kauai, HI, USA, 2001, pp.I-I

In the object detection technique based on the image feature as in Non-Patent Document 1 described above, the position of the object can be detected when the object is concealed by another object or when the object exists in a shadow area not illuminated. It will be difficult.

Further, the technique of Patent Document 1 described above performs active sensing by invisible light, passive sensing by visible light, and online learning during normal operation of active sensing. Due to these synergistic effects, it is possible to detect the position of a robust object against changes in the concealed state and the lighting state of the object.

However, if the object is completely hidden, the invisible light marker cannot be observed from the invisible image, and the image feature of the subject cannot be extracted from the visible image, it is difficult to detect the position of the object. Become.

Therefore, the present invention has been made to solve the above-mentioned problems, and an object of the present invention is to estimate the position of an object or the like in an image even when the object or the like is completely concealed. An object of the present invention is to provide a position estimation device, a position learning device, and a program capable of estimating a position.

In order to solve the above problem, the position estimation device according to claim 1 detects a person based on the image features of the input image in the position estimation device that estimates the position of a predetermined object in the input image, and determines the predetermined coordinates. From the person position measuring unit that measures the position of each person in the system and the position of each person measured by the person position measuring unit, the person density is calculated for each area in the predetermined coordinate system, and for each area. A person density map calculation unit that generates the person density of the person as a person density map, and a neural network calculation using a preset learned coefficient using the person density map generated by the person density map calculation unit as input data. It is characterized in that it is provided with a neural network unit for obtaining a position estimated value indicating an estimated value of the position of the predetermined object as output data.

According to the position estimation device of claim 1, even if the predetermined object is not clearly visible due to interference with the person in the input image, the predetermined object exists based on the spatial distribution of the person. The location can be estimated with high accuracy.

Further, the position estimation device according to claim 2 is the position estimation device according to claim 1, further detecting the person based on the image features of the input image, and measuring the posture of each person. From the unit and the posture of each person measured by the person posture measuring unit, the degree to which all the persons gaze at each coordinate position in the predetermined coordinate system is calculated as the gaze degree for each position. The gaze calculation unit is provided, and the neural network unit inputs the person density map generated by the person density map calculation unit and the gaze degree for each position calculated by the gaze calculation unit as input data. It is characterized in that the calculation of the neural network using the learned coefficient is performed and the position estimated value is obtained as the output data.

According to the position estimation device of claim 2, even when a predetermined object is not clearly visible due to interference with a person in the input image, it can be obtained from the spatial distribution of the person and the posture of each person. It is possible to accurately estimate the location of a predetermined object based on the area of interest of the person.

Further, the position learning device according to claim 3 inputs an input image and a position true value indicating a true value of a position of a predetermined object corresponding to the input image as training data, and based on the training data, a neural network In the position learning device for obtaining the coefficient, a person is detected based on the image feature of the input image, and the position of each person in a predetermined coordinate system is measured by the person position measuring unit and the person position measuring unit. From the position of each person, the person density is calculated for each area in the predetermined coordinate system, and the person density for each area is generated as a person density map. The neural network unit that calculates the neural network using the coefficient using the person density map as input data and obtains the position estimated value indicating the estimated value of the position of the predetermined object as output data, and the input image. Based on the error calculation unit that calculates the error between the position true value corresponding to the above and the position estimation value obtained by the neural network unit, and the error calculated by the error calculation unit, the neural network. It is characterized in that it is provided with a coefficient updating unit for updating the above-mentioned coefficient.

According to the position learning device of claim 3, the learned coefficient used in the position estimation device of claim 1 can be obtained by using the image and the position true value of the predetermined object corresponding to the image as learning data.

Further, the position learning device according to claim 4 is the position learning device according to claim 3, further detecting the person based on the image features of the input image, and measuring the posture of each person. From the unit and the posture of each person measured by the person posture measuring unit, the degree to which all the persons gaze at each coordinate position in the predetermined coordinate system is calculated as the gaze degree for each position. The gaze calculation unit is provided, and the neural network unit receives the person density map generated by the person density map calculation unit and the gaze degree for each position calculated by the gaze calculation unit as input data. It is characterized in that the calculation of the neural network using the coefficient is performed and the position estimated value is obtained as output data.

According to the position learning device of claim 4, the learned coefficient used in the position estimation device of claim 2 can be obtained by using the image and the position true value of the predetermined object corresponding to the image as learning data.

Further, the program of claim 5 is characterized in that the computer functions as the position estimation device according to claim 1 or 2.

Further, the program of claim 6 is characterized in that the computer functions as the position learning device according to claim 3 or 4.

As described above, according to the present invention, when estimating the position of an object or the like in an image, it is possible to estimate the position even when the object or the like is completely concealed.

It is a block diagram which shows the structure of the position estimation apparatus of Example 1. FIG. It is a flowchart which shows the process of the position estimation apparatus of Example 1. FIG. It is a figure explaining the neural network part of Example 1. FIG. It is a block diagram which shows the structure of the position learning apparatus of Example 1. FIG. It is a flowchart which shows the process of the position learning apparatus of Example 1. FIG. It is a block diagram which shows the structure of the position estimation apparatus of Example 2. FIG. It is a flowchart which shows the process of the position estimation apparatus of Example 2. It is a figure explaining the neural network part of Example 2. It is a block diagram which shows the structure of the position learning apparatus of Example 2. It is a flowchart which shows the process of the position learning apparatus of Example 2.

Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings. The position estimation device of the first embodiment described below is used when estimating the position of play equipment (including a soccer ball, a tennis ball, an ice hockey pack, a badminton shuttle, a curling stone, etc.) used for a competition or the like in an image. In addition, the position of each person in the image is measured, the person density for each area is calculated, the posture of each person in the image is measured, and the gaze degree for each position (by all persons with respect to the position) is calculated. The neural network is calculated using the person density for each area and the gaze degree for each position as input data, and the estimated value of the position of the play equipment (estimated value of the position of the play equipment) is obtained as output data.

The position estimation device of the second embodiment measures the position of each person in the image, calculates the person density for each area, calculates the neural network using the person density for each area as input data, and performs the position estimation value of the play equipment. Is obtained as output data.

The position learning devices of Examples 1 and 2 are used in a neural network provided in the corresponding position estimation device using the image and the true value of the position of the playset corresponding to the image (the true value of the position of the playset) as learning data. Find the optimum coefficients (coupling weight and bias value).

This makes it possible to estimate the position of the playset even when the playset is completely concealed, such as when the playset is hidden by a person or when the playset is present in a shadow area that is not illuminated.

[Example 1]
First, Example 1 will be described. As described above, in the first embodiment, a neural network is used in which the person density for each area and the gaze degree for each position are calculated from the image, these data are used as input data, and the position estimated value of the playset is used as output data. , The position of the playset is estimated.

(Position Estimator / Example 1)
FIG. 1 is a block diagram showing the configuration of the position estimation device of the first embodiment, and FIG. 2 is a flowchart showing the processing. The position estimation device 1-1 includes a person position measurement unit 10, a person density map calculation unit 11, a person posture measurement unit 12, a gaze calculation unit 13, a neural network unit 14-1, and a memory 15-1.

The position estimation device 1-1 inputs an image, processes the input image, performs a neural network calculation, and outputs a position estimation value of the playset. The image is, for example, an image of a ball game such as soccer, hockey, badminton, or curling performed by a person who is a player using playsets such as balls, pucks, shuttles, and stones on a court set on the ground.

The person position measuring unit 10 inputs an image (step S201), extracts a person based on the image features of the input image, and measures the position (coordinates) of each person (step S202). Then, the person position measurement unit 10 outputs the position of each person to the person density map calculation unit 11 and the gaze calculation unit 13.

The number of persons may be one or a plurality of persons. Further, the position of the person may be the coordinates in which the person exists in the coordinate system of the input image (hereinafter referred to as "image coordinate system"), or the coordinate system set in the space in which the person exists (hereinafter referred to as "image coordinate system"). , "World coordinate system").

The person position measuring unit 10 may be, for example, a receiver of a global positioning satellite system (GNNS: Global Navigation Satellite System, for example, GPS (Global Positioning System)). Further, the person position measuring unit 10 extracts a person as a subject from an image by using a technique such as face image recognition, and estimates the position of the person based on the position and posture information of the camera that captured the image ( Positioning may be based on a stereo image, or positioning may be performed from a monocular image and restraint conditions (for example, the feet of the person who is the subject are in contact with a predetermined surface). May be good. Further, a wireless tag (RFID or the like) may be attached to a person for positioning.

The person density map calculation unit 11 inputs the position of each person from the person position measurement unit 10, and based on the position of each person, the population density (person density) for each local area in the image coordinate system or the world coordinate system. To generate a person density map (step S203). Then, the person density map calculation unit 11 outputs the person density map (person density for each area) to the neural network unit 14-1.

The person density map calculation unit 11 divides, for example, an area in which a person can exist (for example, a stadium such as a soccer field or a tennis court) into a vertical and horizontal grid pattern, and from the position of each person, the person existing in each grid. By finding the number, the person density for each area may be calculated. When the total number of grids is L, the person density map calculation unit 11 may generate an L-dimensional vector having the person density as an element for each grid, and use this as a person density map.

The grids may be arranged so as not to overlap each other, or the adjacent grids may be arranged so as to overlap each other, for example, a grid in which a 1 m square partial region is set at an interval of 0.5 m square may be arranged. ..

The person posture measuring unit 12 inputs an image, extracts a person's head based on the image features of the input image, and measures the posture (face orientation) of each person from the person's head (step S204). Then, the person posture measuring unit 12 outputs the posture of each person to the gaze calculation unit 13. The posture of the person may be measured in the image coordinate system or in the world coordinate system.

The human posture measuring unit 12 uses, for example, a technique of integrating the identification result based on the color histogram of the image and the identification result based on the feature amount other than the color histogram (for example, the gradient histogram) to estimate the orientation of the face. It may be. The technique for estimating the orientation of the face is known, and for example, refer to Japanese Patent Application Laid-Open No. 2018-22416.

The person posture measuring unit 12 may input the position of each person from the person position measuring unit 10 in order to specify the position of the person whose posture should be measured.

The gaze calculation unit 13 inputs the position of each person from the person position measurement unit 10, and also inputs the posture of each person from the person posture measurement unit 12. Then, the gaze calculation unit 13 gazes at each preset coordinate position in the image coordinate system or the world coordinate system based on the position of each person and the posture of each person (gaze degree). ), That is, the gaze degree for each position (coordinates) is calculated (step S205). The gaze calculation unit 13 outputs the gaze for each position to the neural network unit 14-1. The gaze level for each position is the total value of the gaze level of each person with respect to the position for each position in the image coordinate system or the world coordinate system.

_{Let X k} be the position of each person _{, P k} be the posture of each person, Y be the position in the image coordinate system or the world coordinate system, and G (Y) be the gaze degree of the position Y. k is an index for distinguishing people, and is an integer of 1 or more and K or less. Let it be the maximum value of k of K.

Here, the position X _k for each person is a three-dimensional position vector in world coordinates. Further, the posture P _{k for} each person is a direction vector fixed to a specific part of the person (for example, a face orientation vector fixed to the face, a line-of-sight vector fixed to the eyeball), and is used as a unit vector. The position Y is a position vector in world coordinates.

In this case, the gaze calculation unit 13 calculates the gaze G (Y) for each position Y by, for example, the following formula using the function f.

The function f shall take a scalar value that is larger (or not smaller) so that these two vectors point in the same direction with respect to the vector of the argument (Y-X _k ) and the posture P _{k of each person.} Is desirable. Further, it is desirable that the function f takes a larger (or not smaller) scalar value as the vector of _{(Y−Xk) becomes shorter.}

For example, the function f may calculate the inner product of the vectors (the cosine value (cos θ) of the angle θ formed by the two vectors) as in the following equation. Let the vector Q _k = Y-X _k .

Further, the function f may be calculated as in the following equation.

The neural network unit 14-1 reads the learned coefficients (coupling weight and bias value) from the memory 15-1. Further, the neural network unit 14-1 inputs a person density map (person density for each area) from the person density map calculation unit 11, and inputs the gaze degree for each position from the gaze degree calculation unit 13.

The neural network unit 14-1 uses the person density map and the gaze degree for each position as input data, performs a neural network calculation using the learned coefficients, and outputs an estimated value of the position of the playset (vector indicating the position of the playset). Obtained as data (step S206). Then, the neural network unit 14-1 outputs the position estimated value of the playset (step S207).

As the neural network unit 14-1, for example, any one such as a multi-layer perceptron, a convolutional neural network, a recurrent neural network, a pop field network, and a combination thereof can be used.

When the coordinate system of the position of each person output by the person position measuring unit 10 is the image coordinate system, the coordinate system of the position estimated value of the play equipment output by the neural network unit 14-1 is also typically the image coordinate system. However, it may be set to another coordinate system such as the world coordinate system. On the contrary, when the coordinate system of the position of each person output by the person position measuring unit 10 is the world coordinate system, the coordinate system of the position estimated value of the play equipment output by the neural network unit 14-1 is also the world coordinate system. This is typical, but other coordinate systems such as an image coordinate system may be used.

Let L (L is an integer of 2 or more) be the number of areas in the person density map, which is the person density for each area, and M (M is an integer of 0 or more) in the gaze degree for each position. Since M = 0 corresponds to the case of the second embodiment described later, M is an integer of 1 or more here.

For example, when the neural network unit 14-1 uses the multi-layer perceptron, the neural network unit 14-1 uses the person density for each L region and the gaze degree for each M position as input data, and the input layer of the multi-layer perceptron is used. Input to N (N = L + M) elements in.

FIG. 3 is a diagram for explaining the neural network unit 14-1 using the multi-layer perceptron in the first embodiment, and shows the case of L = 4, M = 4, N = 8. The neural network unit 14-1 is configured by using a three-layer perceptron, and has eight elements in the input layer, three elements in the intermediate layer, and two elements in the output layer.

The neural network unit 14-1 inputs the person densities (variables) x _{1 to x 4} to the four elements (1st to 4th elements) of the input layer for each of the four regions. Further, the neural network unit 14-1 _{inputs gaze (variables) x 5 to x 8} to the four elements (5th to 8th elements) of the input layer for each of the four positions.

_{The neural network unit 14-1 outputs variables x 1 to x 8 as they are from the 1st to} 8th elements of the input layer, and in the 9th to 11th elements (qth element) of the intermediate layer, the following equation is used. _{Then, the variables x 9 to x 11} (x _q ), which are the output values, are calculated using the variables x _{1 to x 8 and the like.}

Here, C _q indicates a set of element numbers connected to the input of the q-th element, and w _{p and q} are the coupling weights between the output of the p-th element and the input of the q-th element. Is shown. Further, b _q indicates a bias value given as an input of the qth element, and φ _q indicates an activation function of the qth element.

Neural network unit 14-1, and outputs the variable x ₉ ~x ₁₁ from 9 to 11 th element of the intermediate layer, in 12 and 13-th element of the output layer (q-th element), the formula (4) In, the position estimated value (X coordinate, Y coordinate) (x _q ) of the playset, which is the output value, is calculated using the variables x _{9 to x 11 and the like.} The neural network unit 14-1 outputs the position estimated values (X coordinate, Y coordinate) of the playset from the 12th and 13th elements of the output layer.

In the example of FIG. 3, C ₉ = C ₁₀ = C ₁₁ = {1, 2, 3, 4, 5, 6, 7, 8}, C ₁₂ = C ₁₃ = {9, 10, 11}, and the activity. Any conversion function φ _q can be used. For example, as the activation function φ _q , the ReLU function (Rectified Linear Unit: half-wave rectifier function: φ _q (x) = max {0, x}), the sigmoid function, and the hyperbolic _{rectifier function (φ q} (x) = tanhx) Can be used. The activation function φ _q may be a mixture of different functions for each element, or the same function may be used for all the elements.

Returning to FIG. 1, in the memory 15-1, for example, the optimum learned coefficients (coupling weights w _{p, q} and) obtained by learning the neural network unit 14-1 by the position learning device 2-1 described later. The bias value b _q ) is stored.

The learned coefficient stored in the memory 15-1 is read out by the neural network unit 14-1, and is used for the calculation from the element of the input layer of the neural network to the element of the output layer via the element of the intermediate layer.

The learned coefficient may be written to the memory 15-1 from the outside as needed, or may be set as data obtained in advance in the read-only memory 15-1.

As described above, according to the position estimation device 1-1 of the first embodiment, the person position measurement unit 10 measures the position of each person from the image, and the person density map calculation unit 11 measures the position of each person from the position of each person. Calculate the density map. Further, the person posture measuring unit 12 measures the posture of each person from the image, and the gaze calculation unit 13 calculates the gaze of each position in a predetermined coordinate system from the position of each person and the posture of each person. ..

The neural network unit 14-1 uses the person density map, which is the person density for each area, and the gaze degree for each position as input data, performs a neural network calculation using the learned coefficients, and outputs the position estimated value of the play equipment as output data. Ask as.

Thereby, the location where the playset exists can be estimated with high accuracy by the calculation of the neural network by using the spatial distribution of the person and the gaze degree for each position obtained from the posture of each person. That is, the playset is completely concealed due to interference with a person or the like, and even if it is not clearly visible, its position can be estimated.

(Position learning device / Example 1)
FIG. 4 is a block diagram showing the configuration of the position learning device of the first embodiment, and FIG. 5 is a flowchart showing the processing. The position learning device 2-1 includes a person position measurement unit 10, a person density map calculation unit 11, a person posture measurement unit 12, a gaze calculation unit 13, a neural network unit 14-1, a memory 20-1, and an error calculation unit 21. And the coefficient updating unit 22-1 is provided.

The position learning device 2-1 inputs an image and a pair of true positions of the play device, which is the position of the true play device corresponding to the image, as learning data one or more times (step S501). Then, the position learning device 2-1 learns the neural network by, for example, the back-propagation method _{, and obtains the optimum learned coefficients (coupling weights w p, q} and bias value b _q ).

Since the person position measurement unit 10, the person density map calculation unit 11, the person posture measurement unit 12, the gaze calculation unit 13, and the neural network unit 14-1 are the same as the components shown in FIG. 1, description thereof will be omitted. .. Further, since steps S502 to S506 shown in FIG. 5 are the same as steps S202 to S206 shown in FIG. 2, the description thereof will be omitted.

_{Here, the neural network unit 14-1 reads out the coefficients (coupling weights w p, q} and bias value b _q ) updated and stored by the coefficient updating unit 22-1 from the memory 20-1, and uses the coefficients. Neural network operations are performed from the elements of the input layer to the elements of the output layer via the elements of the intermediate layer. Then, the neural network unit 14-1 outputs the position estimated value of the playset to the error calculation unit 21.

The memory 20-1 is a rewritable memory (for example, RAM (Random Access Memory)), and the coefficient used by the neural network unit 14-1 is temporarily stored.

The coefficient stored in the memory 20-1 is read out by the neural network unit 14-1, and is used for the calculation from the element of the input layer of the neural network to the element of the output layer via the element of the intermediate layer.

At the time when the operation of the position learning device 2-1 starts, the initial value of the coefficient is stored in the memory 20-1. The initial value may be a preset numerical string or a random number (including a pseudo-random number).

The error calculation unit 21 inputs the true position value of the playset paired with the image input by the position learning device 2-1 and also inputs the position estimation value of the playset from the neural network unit 14-1. Then, the error calculation unit 21 calculates an error between the true position value and the estimated position value of the playset (step S507), and outputs the error to the coefficient update unit 22-1. The error is, for example, the difference vector between the vector of the estimated position of the playset and the vector of the true position of the playset.

The coefficient updating unit 22-1 reads the coefficient from the memory 20-1 and inputs an error from the error calculation unit 21. Then, the coefficient updating unit 22-1 updates the coefficient used by the neural network unit 14-1 based on the error (step S508), and the updated coefficient is used by the neural network unit 14-1 at the next time point. It is stored in the memory 20-1 as a coefficient to be used. The coefficient updating unit 22-1 changes the coefficient based on the error, for example, by the error propagation method.

The coefficient updating unit 22-1 is optimized by updating the coefficient each time the position learning device 2-1 inputs learning data of the true position value of the image and the play device, and the updated coefficient is stored in the memory 20-1. Store.

When the learning is not completed (step S509: N), the position learning device 2-1 proceeds to step S501, inputs a new image and learning data of the position true value of the play device, and steps S502 to S508. Repeat the process.

On the other hand, when the learning is completed (step S509: Y), the position learning device 2-1 reads the coefficient from the memory 20-1 and outputs it to the outside as a learned coefficient (step S510). The learned coefficient output from the position learning device 2-1 is stored in the memory 15-1 of the position estimation device 1-1 shown in FIG.

As described above, according to the position learning device 2-1 of the first embodiment, the image and the position true value of the play equipment corresponding thereto are input as learning data, and the person position is measured in the same manner as the position estimation device 1-1. The unit 10 measures the position of each person, and the person density map calculation unit 11 calculates the person density map. Further, the person posture measuring unit 12 measures the posture of each person, and the gaze calculation unit 13 calculates the gaze degree for each position.

The neural network unit 14-1 uses the person density map, which is the person density for each area, and the gaze degree for each position as input data, and performs the calculation of the neural network using the coefficient stored in the memory 20-1 to perform the calculation of the neural network of the playset. Obtain the position estimate value as output data.

The error calculation unit 21 calculates an error between the true position value and the estimated position value of the playset. The coefficient update unit 22-1 updates the coefficient used by the neural network unit 14-1 based on the error, for example, by the error propagation method, and the updated coefficient is used by the neural network unit 14-1 at the next time point. It is stored in the memory 20-1 as a coefficient to be used.

When the learning process of updating the coefficient is repeated and the learning is completed, the position learning device 2-1 reads the coefficient from the memory 20-1 and outputs the learned coefficient to the outside.

The learned coefficient thus obtained is stored in the memory 15-1 of the position estimation device 1-1 shown in FIG. 1, and is used when the neural network unit 14-1 obtains the position estimation value of the playset. ..

As a result, when the position estimation device 1-1 estimates the position of the playset in the image, the position is estimated even if the playset is completely concealed due to interference with a person or the like and cannot be clearly seen. It becomes possible to do.

[Example 2]
Next, Example 2 will be described. As described above, in the second embodiment, the position of the playset is determined by using a neural network in which the person density for each area is calculated from the image, the person density for each area is used as input data, and the position estimated value of the playset is used as output data. It is an estimate.

(Position Estimator / Example 2)
FIG. 6 is a block diagram showing the configuration of the position estimation device of the second embodiment, and FIG. 7 is a flowchart showing the processing. The position estimation device 1-2 includes a person position measurement unit 10, a person density map calculation unit 11, a neural network unit 14-2, and a memory 15-2.

Comparing the position estimation device 1-1 of the first embodiment shown in FIG. 1 with the position estimation device 1-2, both position estimation devices 1-1 and 1-2 have a person position measurement unit 10 and a person density map. It is common in that it includes a calculation unit 11.

On the other hand, the position estimation device 1-2 does not include the person posture measurement unit 12 and the gaze calculation unit 13 of the position estimation device 1-1, and the neural network unit 14-1 and the memory 15 of the position estimation device 1-1. It differs from the position estimation device 1-1 in that it has a neural network unit 14-2 and a memory 15-2, which are different from -1.

Since the person position measurement unit 10 and the person density map calculation unit 11 are the same as the constituent units shown in FIG. 1, the description thereof will be omitted. Further, since steps S701 to S703 shown in FIG. 7 are the same as steps S201 to S203 shown in FIG. 2, the description thereof will be omitted. Here, the person density map calculation unit 11 outputs the person density map (person density for each area) to the neural network unit 14-2.

The neural network unit 14-2 reads the learned coefficient from the memory 15-2. Further, the neural network unit 14-2 inputs the person density map from the person density map calculation unit 11.

The neural network unit 14-2 uses the person density map, which is the person density for each region, as input data, performs a neural network calculation using the learned coefficients, and obtains the position estimated value of the play equipment as output data (step S704). .. Then, the neural network unit 14-2 outputs the position estimated value of the playset (step S705).

As the neural network unit 14-2, for example, a multi-layer perceptron or the like can be used in the same manner as the neural network unit 14-1 shown in FIG.

When the coordinate system of the position of each person output by the person position measuring unit 10 is the image coordinate system, the coordinate system of the position estimated value of the play equipment output by the neural network unit 14-2 is also typically the image coordinate system. However, it may be set to another coordinate system such as the world coordinate system. On the contrary, when the coordinate system of the position of each person output by the person position measuring unit 10 is the world coordinate system, the coordinate system of the position estimated value of the play equipment output by the neural network unit 14-2 is also the world coordinate system. This is typical, but other coordinate systems such as an image coordinate system may be used.

FIG. 8 is a diagram illustrating a neural network unit 14-2 using a multi-layer perceptron in the second embodiment. The neural network unit 14-2 is configured by using a three-layer perceptron, and has four elements in the input layer, three elements in the intermediate layer, and two elements in the output layer.

The neural network unit 14-2 inputs the person densities (variables) x _{1 to x 4} to the four elements (1st to 4th elements) of the input layer for each of the four regions.

_{The neural network unit 14-2 outputs variables x 1 to x 4 as they are from the 1st to} 4th elements of the input layer, and in the 5th to 7th elements (qth element) of the intermediate layer, the above equation (4). ), The output values x _{5 to x 7} (x _q ) are calculated _{using the variables x 1 to x 4 and the like.}

The neural network unit 14-2 outputs variables x5 _{to x7 from the 5th to 7th} elements of the intermediate layer, and in the 8th and 9th elements (qth element) of the output layer, the above equation (4) In, the position estimated value (X coordinate, Y coordinate) (x _q ) of the playset, which is the output value, is calculated using the variables x _{5 to x 7 and the like.} The neural network unit 14-2 outputs the position estimated values (X coordinate, Y coordinate) of the playset from the 8th and 9th elements of the output layer. In the example of FIG. 8, C ₅ = C ₆ = C ₇ = {1, 2, 3, 4}, C ₈ = C ₉ = {5, 6, 7}.

Returning to FIG. 6, the memory 15-2 stores, for example, the optimum learned coefficient obtained by learning the neural network unit 14-2 by the position learning device 2-2 described later.

The learned coefficient stored in the memory 15-2 is read out by the neural network unit 14-2 and used for the calculation from the element of the input layer of the neural network to the element of the output layer via the element of the intermediate layer.

The learned coefficient may be written to the memory 15-2 from the outside as needed, or may be set as data previously obtained in the read-only memory 15-2.

As described above, according to the position estimation device 1-2 of the second embodiment, the person position measurement unit 10 measures the position of each person from the image, and the person density map calculation unit 11 measures the position of each person from the position of each person. Calculate the density map.

The neural network unit 14-2 uses the person density map, which is the person density for each area, as input data, performs a neural network calculation using the learned coefficients, and obtains the position estimated value of the play equipment as output data.

As a result, the location of the playset can be estimated with high accuracy by the calculation of the neural network using the spatial distribution of the person. That is, the playset is completely concealed due to interference with a person or the like, and even if it is not clearly visible, its position can be estimated.

(Position learning device / Example 2)
FIG. 9 is a block diagram showing the configuration of the position learning device of the second embodiment, and FIG. 10 is a flowchart showing the processing. The position learning device 2-2 includes a person position measurement unit 10, a person density map calculation unit 11, a neural network unit 14-2, a memory 20-2, an error calculation unit 21, and a coefficient update unit 22-2.

Comparing the position learning device 2-1 of the first embodiment shown in FIG. 4 with the position learning device 2-2, both position learning devices 2-1 and 2-2 have a person position measuring unit 10 and a person density map. It is common in that it includes a calculation unit 11 and an error calculation unit 21.

On the other hand, the position learning device 2-2 does not include the person posture measuring unit 12 and the gaze calculation unit 13 of the position learning device 2-1. It differs from the position learning device 2-1 in that it includes a neural network unit 14-2, a memory 20-2, and a coefficient updating unit 22-2, which are different from the -1 and the coefficient updating unit 22-1.

Since the person position measurement unit 10, the person density map calculation unit 11, and the error calculation unit 21 are the same as the configuration units shown in FIG. 4, description thereof will be omitted. Further, since steps S1001 to S1005 shown in FIG. 10 are the same as steps S501 to S503 shown in FIG. 5, steps S704 shown in FIG. 7, and step S507 shown in FIG. 5, the description thereof will be omitted. Here, the person density map calculation unit 11 outputs the person density map to the neural network unit 14-2, and the error calculation unit 21 outputs the error to the coefficient update unit 22-2.

Here, the neural network unit 14-2 reads out the coefficient updated and stored by the coefficient updating unit 22-2 from the memory 20-2, and outputs the coefficient from the element of the input layer to the element of the intermediate layer via the element of the intermediate layer. Perform neural network operations on the layer elements. Then, the neural network unit 14-2 outputs the position estimated value of the playset to the error calculation unit 21.

The memory 20-2 is a rewritable memory like the memory 20-1 shown in FIG. 4, and the coefficient used by the neural network unit 14-2 is temporarily stored.

The coefficients stored in the memory 20-2 are read out by the neural network unit 14-2 and used for the calculation from the element of the input layer of the neural network to the element of the output layer via the element of the intermediate layer.

At the start of operation of the position learning device 2-2, the initial value of the coefficient is stored in the memory 20-2. The initial value may be a preset numerical string or a random number (including a pseudo-random number).

The coefficient updating unit 22-2 reads the coefficient from the memory 20-2 and inputs an error from the error calculation unit 21. Then, the coefficient updating unit 22-2 updates the coefficient used by the neural network unit 14-2 based on the error (step S1006), and the updated coefficient is used by the neural network unit 14-2 at the next time point. It is stored in the memory 20-2 as a coefficient to be used. The coefficient updating unit 22-2 changes the coefficient based on the error, for example, by the error propagation method.

The coefficient updating unit 22-2 optimizes by updating the coefficient each time the position learning device 2-2 inputs the learning data of the position true value of the image and the play device, and the updated coefficient is stored in the memory 20-2. Store.

When the learning is not completed (step S1007: N), the position learning device 2-2 proceeds to step S1001, inputs a new image and learning data of the position true value of the play device, and steps S1002 to S1006. Repeat the process.

On the other hand, when the learning is completed (step S1007: Y), the position learning device 2-2 reads the coefficient from the memory 20-2 and outputs it to the outside as a learned coefficient (step S1008). The learned coefficient output from the position learning device 2-2 is stored in the memory 15-2 of the position estimation device 1-2 shown in FIG.

As described above, according to the position learning device 2-2 of the second embodiment, the image and the position true value of the play equipment corresponding thereto are input as learning data, and the person position is measured in the same manner as the position estimation device 1-2. The unit 10 measures the position of each person, and the person density map calculation unit 11 calculates the person density map.

The neural network unit 14-2 uses the person density map, which is the person density for each area, as input data, performs a neural network calculation using the coefficients stored in the memory 20-2, and outputs the position estimated value of the playset as output data. Ask as.

The error calculation unit 21 calculates an error between the true position value and the estimated position value of the playset. The coefficient update unit 22-2 updates the coefficient used by the neural network unit 14-2 based on the error, for example, by the error propagation method, and the updated coefficient is used by the neural network unit 14-2 at the next time point. It is stored in the memory 20-2 as a coefficient to be used.

When the learning process of updating the coefficient is repeated and the learning is completed, the position learning device 2-2 reads the coefficient from the memory 20-2 and outputs the learned coefficient to the outside.

The learned coefficient obtained in this way is stored in the memory 15-2 of the position estimation device 1-2 shown in FIG. 6, and is used when the neural network unit 14-2 obtains the position estimation value of the playset. ..

As a result, when the position estimation device 1-2 estimates the position of the playset in the image, the position is estimated even if the playset is completely concealed due to interference with a person or the like and cannot be clearly seen. It becomes possible to do.

Although the present invention has been described above with reference to Examples 1 and 2, the present invention is not limited to the above Examples 1 and 2, and can be variously modified without departing from the technical idea. In the first and second embodiments, the position estimation devices 1-1 and 1-2 estimate the position of the playset in the image, and the position learning devices 2-1 and 2-2 learn the positions of the image and the playset as learning data. Changed to perform processing.

On the other hand, the present invention is not limited to estimating the position of the play device and using the position of the play device as learning data, but estimating the position of an object other than the play device and using the position of the object as learning data. You may do so. Further, in the present invention, an object other than an object, for example, the position of a person may be estimated, and the position of the person may be used as learning data.

Further, the person position measurement unit 10, the person density map calculation unit 11, the person posture measurement unit 12, and the gaze calculation unit 13 of the first embodiment, and the person position measurement unit 10 and the person density map calculation unit 11 of the second embodiment. To find the position of each person for all the people participating in the ball game.

On the other hand, the person position measurement unit 10, the person density map calculation unit 11, the person posture measurement unit 12, and the gaze calculation unit 13 target the people of each team participating in the ball game, and for each team, for each person. You may try to find the position of. In this case, the neural network unit 14-1 calculates the neural network using the learned coefficient by using the person density map for each team and the gaze degree for each team and each position as input data. Further, the neural network unit 14-2 uses the person density map for each team as input data and performs the calculation of the neural network using the learned coefficients.

Further, the neural network unit 14-1 of the first embodiment processes the person density map and the gaze degree for each position as input data, and the neural network unit 14-2 of the second embodiment uses the person density map. Processed as input data.

On the other hand, the neural network units 14-1 and 14-2 input the position of the person measured by the person position measuring unit 10, the moving speed of the person measured by other constituent units, the moving direction of the person, and the like. It may be processed as data. In this case, the person speed measuring unit detects a person from the time-series images and measures the moving speed of each person, and the person direction measuring unit detects the person from the time-series images and determines the moving direction of each person. measure.

As the hardware configuration of the position estimation devices 1-1, 1-2 and the position learning devices 2-1 and 2-2 according to the first and second embodiments, a normal computer can be used. The position estimation devices 1-1, 1-2 and the position learning devices 2-1 and 2-2 are computers equipped with a volatile storage medium such as a CPU and RAM, a non-volatile storage medium such as a ROM, and an interface. Consists of.

The functions of the person position measurement unit 10, the person density map calculation unit 11, the person posture measurement unit 12, the gaze calculation unit 13, the neural network unit 14-1 and the memory 15-1 provided in the position estimation device 1-1 are Each of these is realized by causing the CPU to execute a program that describes these functions.

Further, each function of the person position measurement unit 10, the person density map calculation unit 11, the neural network unit 14-2, and the memory 15-2 provided in the position estimation device 1-2 also uses a program describing these functions in the CPU. It is realized by executing each.

Further, the person position measurement unit 10, the person density map calculation unit 11, the person posture measurement unit 12, the gaze calculation unit 13, the neural network unit 14-1, the memory 20-1, and the error calculation provided in the position learning device 2-1. Each function of the unit 21 and the coefficient updating unit 22-1 is also realized by causing the CPU to execute a program describing these functions.

Further, each function of the person position measurement unit 10, the person density map calculation unit 11, the neural network unit 14-2, the memory 20-2, the error calculation unit 21, and the coefficient update unit 22-2 provided in the position learning device 2-2. Is also realized by causing the CPU to execute a program that describes these functions.

These programs are stored in the storage medium, read by the CPU, and executed. In addition, these programs can be stored and distributed in storage media such as magnetic disks (floppy (registered trademark) disks, hard disks, etc.), optical disks (CD-ROM, DVD, etc.), semiconductor memories, etc., and can be distributed via a network. You can also send and receive.

1-1, 1-2 Position estimation device 2-1, 2-2 Position learning device 10 Person position measurement unit 11 Person density map calculation unit 12 Person posture measurement unit 13 Gaze calculation unit 14-1, 14-2 Neural network Units 15-1, 15-2, 20-1, 20-2 Memory 21 Error calculation unit 22-1,22-2 Coefficient update unit

Claims (6)

In a position estimation device that estimates the position of a predetermined object in an input image,
A person position measuring unit that detects a person based on the image features of the input image and measures the position of each person in a predetermined coordinate system.
A person density map calculation unit that calculates the person density for each area in the predetermined coordinate system from the position of each person measured by the person position measurement unit and generates the person density for each area as a person density map. ,
Using the person density map generated by the person density map calculation unit as input data, a neural network calculation is performed using preset learned coefficients, and a position estimated value indicating an estimated value of the position of the predetermined object is obtained. Neural network part to be obtained as output data and
A position estimation device characterized by being equipped with. In the position estimation device according to claim 1,
Further, a person posture measuring unit that detects the person based on the image features of the input image and measures the posture of each person, and a person posture measuring unit.
Gaze calculation that calculates the degree to which all the persons gaze at each coordinate position in the predetermined coordinate system as the gaze degree for each position from the posture of each person measured by the person posture measuring unit. With a department,
The neural network unit
Using the person density map generated by the person density map calculation unit and the gaze degree for each position calculated by the gaze calculation unit as input data, the neural network calculation using the learned coefficient is performed. A position estimation device, characterized in that the position estimation value is obtained as the output data. In a position learning device that inputs an input image and a position true value indicating a true value of a position of a predetermined object corresponding to the input image as training data and obtains a neural network coefficient based on the training data.
A person position measuring unit that detects a person based on the image features of the input image and measures the position of each person in a predetermined coordinate system.
A person density map calculation unit that calculates the person density for each area in the predetermined coordinate system from the position of each person measured by the person position measurement unit and generates the person density for each area as a person density map. ,
Using the person density map generated by the person density map calculation unit as input data, the neural network is calculated using the coefficient, and a position estimated value indicating an estimated value of the position of the predetermined object is obtained as output data. Neural network part and
An error calculation unit that calculates an error between the true position value corresponding to the input image and the position estimation value obtained by the neural network unit.
A coefficient update unit that updates the coefficient of the neural network based on the error calculated by the error calculation unit, and a coefficient update unit.
A position learning device characterized by being equipped with. In the position learning apparatus according to claim 3,
Further, a person posture measuring unit that detects the person based on the image features of the input image and measures the posture of each person, and a person posture measuring unit.
Gaze calculation that calculates the degree to which all the persons gaze at each coordinate position in the predetermined coordinate system as the gaze degree for each position from the posture of each person measured by the person posture measuring unit. With a department,
The neural network unit
Using the person density map generated by the person density map calculation unit and the gaze degree for each position calculated by the gaze calculation unit as input data, the neural network is calculated using the coefficients, and the operation is performed. A position learning device characterized in that a position estimated value is obtained as output data. A program for causing a computer to function as the position estimation device according to claim 1 or 2. A program for causing a computer to function as the position learning device according to claim 3 or 4.

Images