JP7373352B2 - Location estimation device, location learning device and program - Google Patents

Description

The present invention relates to a position estimation device, a position learning device, and a program for estimating the position of an object or the like in an input image.

Conventionally, as a technology to estimate the position of an object, a tag such as a beacon or RFID (Radio Frequency IDentifier) or a reflector (retroreflective material, etc.) is attached to the object, and radio waves, infrared rays, visible light, etc. from the object are passively emitted. There are known devices that perform passive or active sensing (for example, see Patent Document 1).

Patent Document 1 discloses a technique for tracking a subject by using both a process of detecting an invisible light marker attached to a subject from a non-visible image and a process of detecting the subject from a visible image. .

Furthermore, there is a technique for estimating the position of an object by extracting image features from an input image and determining whether the image features match a predetermined condition. This technology extracts a person's face area by extracting an image feature, such as a Haar-Like feature, from an input image and determining the image feature using a boosting method (for example, see Non-Patent Document 1). reference). This technology is used to adjust the exposure and focus position of digital cameras.

JP2017-204757A

P.Viola and M.Jones, “Rapid Object Detection using a Boosted Cascade of Simple Features”, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, Kauai, HI, USA, 2001, pp.I-I

In object detection technology based on image features such as the one described in Non-Patent Document 1, it is difficult to detect the position of an object when the object is hidden by another object or when the object exists in a shadow area that is not illuminated. It becomes difficult.

Further, the technology of Patent Document 1 described above performs active sensing using invisible light, passive sensing using visible light, and online learning during normal operation of active sensing. These synergistic effects make it possible to detect the position of an object robustly against changes in the object's concealment state and illumination state.

However, if the object is completely hidden, the invisible light marker cannot be observed from the invisible image, and the image features of the object cannot be extracted from the visible image, it may be difficult to detect the object's position. Become.

Therefore, the present invention was made to solve the above problem, and its purpose is to estimate the position of an object, etc. in an image, even if the object, etc. is completely hidden. An object of the present invention is to provide a position estimating device, a position learning device, and a program capable of estimating a position.

In order to solve the above problem, a position estimating device according to claim 1 is a position estimating device that estimates the position of a predetermined object in an input image, detects a person based on image features of the input image, and detects a person at predetermined coordinates. a person position measurement unit that measures the position of each person in the system; and a person position measurement unit that calculates a person density for each area in the predetermined coordinate system from the position of each person measured by the person position measurement unit; a person density map calculation unit that generates a person density map as a person density map; and a neural network calculation using preset learned coefficients using the person density map generated by the person density map calculation unit as input data. and a neural network unit that calculates a position estimate value representing an estimated value of the position of the predetermined object as output data , and the person density map calculation unit divides an area in which the person can exist into a grid pattern. , the density of people in each region is calculated by calculating the number of people present in each grid from the position of each person, and the grids are arranged so that adjacent grids overlap each other. It is characterized by the fact that

According to the position estimation device of claim 1, even if the predetermined object is not clearly visible due to interference with a person in the input image, the presence of the predetermined object can be determined based on the spatial distribution of the person. The location can be estimated with high accuracy.

The position estimating device according to claim 2 is a position estimating device that estimates the position of a predetermined object in an input image, detects a person based on image features of the input image, and detects a person based on image features of the input image, and A person position measurement unit that measures the position, and a person density calculated for each area in the predetermined coordinate system from the position of each person measured by the person position measurement unit, and the person density for each area calculated as the person density. a person density map calculation unit that generates a map; a person posture measurement unit that detects the person based on the image characteristics of the input image and measures the posture of each person; A gaze degree calculation unit that calculates the degree of gaze of all the people to each coordinate position in the predetermined coordinate system from the posture of each person as a gaze degree for each position, and the person density map calculation unit. Using the generated person density map and the gaze degree for each position calculated by the gaze degree calculation unit as input data, a neural network calculation is performed using preset learned coefficients to determine the predetermined object. The present invention is characterized by comprising a neural network unit that obtains a position estimate value indicating an estimated position value as output data .

According to the position estimating device of claim 2, even if a predetermined object is not clearly visible due to interference with a person in the input image, the position estimation device can be obtained from the spatial distribution of the person and the posture of each person. The location where the predetermined object is present can be estimated with high accuracy based on the area that the person is paying attention to.

Further, the position learning device according to claim 3 inputs an input image and a position true value indicating the true value of the position of a predetermined object corresponding to the input image as learning data, and performs a neural network based on the learning data. In a position learning device for calculating coefficients, a person position measuring unit detects a person based on image characteristics of the input image and measures the position of each person in a predetermined coordinate system; a person density map calculation unit that calculates a person density for each area in the predetermined coordinate system from the position of each person, and generates the person density map for each area as a person density map; a neural network unit that uses the calculated person density map as input data to perform calculations on the neural network using the coefficients, and obtains a position estimate value indicating an estimated value of the position of the predetermined object as output data; an error calculation section that calculates an error between the true position value corresponding to the position value and the estimated position value obtained by the neural network section; a coefficient updating unit that updates the coefficients of , and the person density map calculation unit divides the area where the person can exist into grids, and calculates the number of people existing in each grid from the position of each person. The density of people in each region is calculated by determining the density of people in each region, and the grid is characterized in that the grids are arranged so as to overlap each other with adjacent grids.

According to the position learning device according to the third aspect, learned coefficients used in the position estimating device according to the first aspect can be obtained by using an image and a true position value of a predetermined object corresponding to the image as learning data.

Further, the position learning device according to claim 4 inputs an input image and a true position value indicating the true value of the position of a predetermined object corresponding to the input image as learning data, and performs a neural network based on the learning data. In a position learning device for calculating coefficients, a person position measuring unit detects a person based on image characteristics of the input image and measures the position of each person in a predetermined coordinate system; a person density map calculation unit that calculates a person density for each region in the predetermined coordinate system from the position of each person, and generates the person density for each region as a person density map, based on image characteristics of the input image; a human posture measuring section that detects the person and measures the posture of each person; and a human posture measuring section that detects the person and measures the posture of each person, and from the posture of each person measured by the human posture measuring section, for each coordinate position in the predetermined coordinate system. a gaze degree calculation unit that calculates the degree of gaze of all the people as a gaze degree for each position; the person density map generated by the person density map calculation unit; and the person density map calculated by the gaze degree calculation unit. a neural network unit that uses the degree of gaze for each position as input data to perform calculations on the neural network using the coefficients, and obtains a position estimate value indicating an estimated value of the position of the predetermined object as output data; an error calculation section that calculates an error between the corresponding true position value and the estimated position value obtained by the neural network section; The present invention is characterized by comprising a coefficient updating unit that updates the coefficients .

According to the position learning device according to the fourth aspect, learned coefficients used in the position estimating device according to the second aspect can be obtained by using an image and the true position value of a predetermined object corresponding to the image as learning data.

Furthermore, the program according to claim 5 causes a computer to function as the position estimating device according to claim 1 or 2.

Furthermore, the program according to claim 6 causes a computer to function as the position learning device according to claim 3 or 4.

As described above, according to the present invention, when estimating the position of an object in an image, it is possible to estimate the position even if the object is completely hidden.

1 is a block diagram showing the configuration of a position estimating device according to a first embodiment. FIG. 5 is a flowchart illustrating processing of the position estimating device according to the first embodiment. FIG. 2 is a diagram illustrating a neural network section of Example 1. FIG. 1 is a block diagram showing the configuration of a position learning device of Example 1. FIG. 3 is a flowchart illustrating processing of the position learning device according to the first embodiment. FIG. 2 is a block diagram showing the configuration of a position estimating device according to a second embodiment. 7 is a flowchart illustrating the processing of the position estimating device according to the second embodiment. FIG. 7 is a diagram illustrating a neural network section of Example 2. FIG. 2 is a block diagram showing the configuration of a position learning device according to a second embodiment. 7 is a flowchart showing the processing of the position learning device according to the second embodiment.

DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments for carrying out the present invention will be described in detail using the drawings. The position estimation device of Example 1, which will be described below, is used to estimate the position of play equipment used for competitions (including soccer balls, tennis balls, ice hockey pucks, badminton shuttles, curling stones, etc.) in images. Measure the position of each person in the image, calculate the density of people in each area, measure the posture of each person in the image, calculate the degree of gaze (by all people for the position) for each position, A neural network calculation is performed using the density of people for each region and the degree of gaze for each position as input data, and an estimated value of the position of the play equipment (estimated value of the position of the play equipment) is obtained as output data.

The position estimation device of the second embodiment measures the position of each person in an image, calculates the density of people in each area, uses the density of people in each area as input data to perform neural network calculations, and calculates the estimated position of the play equipment. is obtained as output data.

The position learning devices of Examples 1 and 2 use an image and the true value of the position of the play equipment corresponding to the image (true position value of the play equipment) as learning data in a neural network provided in the corresponding position estimation device. Find the optimal coefficients (coupling weights and bias values).

This makes it possible to estimate the position of the play equipment even when the play equipment is completely hidden, such as when the play equipment is hidden by a person or the play equipment is present in a shadow area that is not illuminated.

[Example 1]
First, Example 1 will be described. As mentioned above, the first embodiment uses a neural network that calculates the density of people for each area and the degree of gaze for each position from the image, uses these data as input data, and uses the estimated position of the play equipment as output data. , which estimates the position of play equipment.

(Position estimation device/Example 1)
FIG. 1 is a block diagram showing the configuration of a position estimating device according to a first embodiment, and FIG. 2 is a flowchart showing its processing. This position estimation device 1-1 includes a person position measurement section 10, a person density map calculation section 11, a person posture measurement section 12, a gaze degree calculation section 13, a neural network section 14-1, and a memory 15-1.

The position estimating device 1-1 inputs an image, processes the input image, performs neural network calculations, and outputs an estimated position value of the play equipment. The image is, for example, an image of a ball game such as soccer, hockey, badminton, curling, etc., played by players using play equipment such as balls, pucks, shuttles, and stones in a court set on the ground.

The person position measurement unit 10 inputs an image (step S201), extracts a person based on the image characteristics of this input image, and measures the position (coordinates) of each person (step S202). Then, the person position measurement section 10 outputs the position of each person to the person density map calculation section 11 and the gaze degree calculation section 13.

The number of people may be one or more than one. Furthermore, the position of the person may be the coordinates where the person exists in the coordinate system of the input image (hereinafter referred to as "image coordinate system"), or the coordinate system set in the space where the person exists (hereinafter referred to as "image coordinate system"). , referred to as the "world coordinate system").

The person position measurement unit 10 may be, for example, a receiver of a Global Navigation Satellite System (GNNS), such as a GPS (Global Positioning System). In addition, the person position measurement unit 10 uses a method (such as a method) that extracts a person as a subject from an image using technology such as facial image recognition, and estimates the person's position based on the position and orientation information of the camera that captured the image. Positioning may be performed using stereo images, or may be performed using monocular images and constraint conditions (for example, the subject's feet are in contact with a predetermined surface). Good too. Furthermore, positioning may be performed by attaching a wireless tag (RFID or the like) to a person.

The person density map calculation unit 11 inputs the position of each person from the person position measurement unit 10, and calculates the population density (person density) for each local area in the image coordinate system or the world coordinate system based on the position of each person. is calculated to generate a person density map (step S203). Then, the person density map calculation unit 11 outputs the person density map (person density for each region) to the neural network unit 14-1.

For example, the person density map calculation unit 11 divides an area where people can exist (for example, a stadium such as a soccer field or a tennis court) into a horizontal and vertical grid, and calculates the number of people present in each grid based on the position of each person. By calculating the number, the density of people in each area may be calculated. When the total number of grids is L, the person density map calculation unit 11 may generate an L-dimensional vector whose element is the person density for each grid, and use this as the person density map.

The grids may be arranged so that they do not overlap each other, or they may be arranged so that adjacent grids overlap each other, for example, by arranging grids in which partial areas of 1 m square are set at intervals of 0.5 m square. .

The human posture measurement unit 12 inputs an image, extracts a human head based on the image characteristics of the input image, and measures the posture (face orientation) of each person from the human head (step S204). Then, the person posture measurement section 12 outputs the posture of each person to the gaze degree calculation section 13. The posture of a person may be measured in the image coordinate system or in the world coordinate system.

For example, the human posture measurement unit 12 integrates the identification result based on the color histogram of the image and the identification result based on the feature amount other than the color histogram (for example, a gradient histogram), and uses a technology to estimate the face orientation. You can also do this. This technique of estimating the direction of the face is known, and please refer to, for example, Japanese Patent Application Publication No. 2018-22416.

Note that the human posture measuring section 12 may input the position of each person from the human position measuring section 10 in order to specify the position of the person whose posture is to be measured.

The gaze degree calculation unit 13 receives the position of each person from the person position measurement unit 10 and the posture of each person from the person posture measurement unit 12. Then, based on the position of each person and the posture of each person, the gaze degree calculation unit 13 calculates the degree to which all the people gaze at each preset coordinate position in the image coordinate system or the world coordinate system (gazing degree). ), that is, the degree of gaze for each position (coordinate) is calculated (step S205). The gaze degree calculation unit 13 outputs the gaze degree for each position to the neural network unit 14-1. The degree of gaze for each position is the total value of the degree of gaze by each person for each position in the image coordinate system or the world coordinate system.

Let the position of each person be _Xk , the posture of each person be _Pk , the position in the image coordinate system or the world coordinate system be Y, and the degree of gaze at position Y be G(Y). k is an index for distinguishing people, and is an integer from 1 to K, inclusive. Let it be the maximum value of k of K.

Here, the position X _k of each person is assumed to be a three-dimensional position vector in world coordinates. In addition, the posture P _k of each person is a directional vector fixed to a specific part of the person (for example, a face direction vector fixed to the face, a line of sight vector fixed to the eyeballs), and is a unit vector. The position Y is a position vector in world coordinates.

In this case, the gaze level calculation unit 13 calculates the gaze level G(Y) for each position Y using the following equation using the function f, for example.

Assume that the function f takes on a scalar value that is larger (or not smaller) with respect to the argument (Y-X _k ) vector and the pose P _k of each person, the more these two vectors point in the same direction. is desirable. Further, it is desirable that the function f takes on a larger (or not smaller) scalar value as the vector of (Y−X _k ) is shorter.

For example, the function f may be one that calculates the inner product of vectors (the cosine value (cos θ) of the angle θ formed by two vectors), as in the following equation. Let vector Q _k =Y−X _k .

Further, the function f may be calculated as shown in the following equation.

The neural network unit 14-1 reads learned coefficients (coupling weights and bias values) from the memory 15-1. Further, the neural network unit 14-1 inputs a person density map (person density for each region) from the person density map calculation unit 11, and also inputs the degree of gaze for each position from the degree of gaze calculation unit 13.

The neural network unit 14-1 uses the person density map and the degree of gaze for each position as input data, performs neural network calculations using learned coefficients, and outputs the estimated position of the play equipment (vector indicating the position of the play equipment). It is obtained as data (step S206). Then, the neural network unit 14-1 outputs the estimated position of the play equipment (step S207).

As the neural network unit 14-1, for example, any one such as a multilayer perceptron, a convolutional neural network, a recurrent neural network, a pop field network, or a combination thereof can be used.

When the coordinate system of the position of each person outputted by the person position measurement unit 10 is an image coordinate system, the coordinate system of the estimated position of the play equipment outputted by the neural network unit 14-1 is typically also set to the image coordinate system. However, other coordinate systems such as the world coordinate system may be used. Conversely, if the coordinate system of the position of each person outputted by the person position measurement section 10 is the world coordinate system, the coordinate system of the estimated position of the play equipment outputted by the neural network section 14-1 is also set to the world coordinate system. Although this is typical, other coordinate systems such as an image coordinate system may be used.

Let the number of regions in the person density map, which is the person density for each region, be L (L is an integer greater than or equal to 2), and the number of positions in the degree of gaze for each location be M (M is an integer greater than or equal to 0). Note that since M=0 corresponds to the case of Example 2, which will be described later, M is an integer of 1 or more here.

For example, when the neural network unit 14-1 uses a multilayer perceptron, the neural network unit 14-1 uses the human density for each of L regions and the degree of gaze for each M position as input data, and uses the input layer of the multilayer perceptron as input data. input to N elements (N=L+M) at.

FIG. 3 is a diagram illustrating the neural network unit 14-1 using a multilayer perceptron in the first embodiment, and shows a case where L=4, M=4, and N=8. This neural network section 14-1 is configured using a three-layer perceptron, and has eight elements in the input layer, three elements in the intermediate layer, and two elements in the output layer.

The neural network unit 14-1 inputs the person densities (variables) x ₁ to x ₄ to four elements (first to fourth elements) of the input layer for each of the four regions. Further, the neural network unit 14-1 inputs the degrees of gaze (variables) x ₅ to x ₈ to four elements (fifth to eighth elements) of the input layer for each of the four positions.

The neural network unit 14-1 outputs the variables x ₁ to x ₈ as they are from the 1st to 8th elements of the input layer, and uses the following formula in the 9th to 11th elements (q-th element) of the intermediate layer. Then, variables x ₉ to x ₁₁ (x _q ), which are output values, are calculated using variables x ₁ to x ₈ and the like.

Here, C _q indicates a set of element numbers connected to the input of the q-th element, and w _p,q is the coupling weight between the output of the p-th element and the input of the q-th element. shows. Further, b _q indicates a bias value given as an input to the q-th element, and φ _q indicates an activation function of the q-th element.

The neural network unit 14-1 outputs variables x ₉ to x ₁₁ from the 9th to 11th elements of the intermediate layer, and in the 12th and 13th elements (q-th element) of the output layer, formula (4) , the estimated position value (X coordinate, Y coordinate) (x _q ) of the play equipment, which is an output value, is calculated using variables x ₉ to x ₁₁ and the like. The neural network unit 14-1 outputs the estimated position value (X coordinate, Y coordinate) of the play equipment from the 12th and 13th elements of the output layer.

In the example of FIG. 3, C ₉ =C ₁₀ =C ₁₁ ={1,2,3,4,5,6,7,8}, C ₁₂ =C ₁₃ ={9,10,11}, and the activity Any arbitrary function φ _q can be used. For example, as the activation function φ _q , the ReLU function (Rectified Linear Unit: half-wave rectification function: φ _q (x) = max {0, x}), the sigmoid function, the hyperbolic tangent function (φ _q (x) = tanhx) can be used. For the activation function φ _q , different functions may be mixed for each element, or the same function may be used for all elements.

Returning to FIG. 1, the memory 15-1 stores, for example, optimal learned coefficients (connection weights w _p,q and Bias value b _q ) is stored.

The learned coefficients stored in the memory 15-1 are read out by the neural network section 14-1 and used for calculations from the input layer elements of the neural network to the output layer elements via the intermediate layer elements.

The learned coefficients may be written into the memory 15-1 from the outside as necessary, or may be set as predetermined data in the read-only memory 15-1.

As described above, according to the position estimation device 1-1 of the first embodiment, the person position measurement unit 10 measures the position of each person from the image, and the person density map calculation unit 11 measures the position of each person from the position of each person. Compute the density map. Further, the person posture measurement unit 12 measures the posture of each person from the image, and the gaze degree calculation unit 13 calculates the degree of gaze for each position in a predetermined coordinate system from the position of each person and the posture of each person. .

The neural network unit 14-1 performs neural network calculations using learned coefficients using a person density map that is the density of people in each area and the degree of gaze for each position as input data, and outputs the estimated position value of the play equipment. Find it as.

Thereby, using the spatial distribution of people and the degree of gaze for each position obtained from the posture of each person, the location where the play equipment is present can be estimated with high accuracy by calculation of the neural network. In other words, even if the play equipment is completely hidden and cannot be clearly seen due to interference with a person, it is possible to estimate its position.

(Position learning device/Example 1)
FIG. 4 is a block diagram showing the configuration of the position learning device according to the first embodiment, and FIG. 5 is a flowchart showing its processing. This position learning device 2-1 includes a person position measurement section 10, a person density map calculation section 11, a person posture measurement section 12, a gaze degree calculation section 13, a neural network section 14-1, a memory 20-1, an error calculation section 21 and a coefficient updating section 22-1.

The position learning device 2-1 inputs the image and the true position value of the play equipment, which is the true position of the play equipment corresponding to the image, as a pair at least once as learning data (step S501). Then, the position learning device 2-1 learns the neural network using, for example, an error backpropagation method, and obtains optimal learned coefficients (coupling weights w _p,q and bias values b _q ).

The person position measurement section 10, the person density map calculation section 11, the person posture measurement section 12, the gaze degree calculation section 13, and the neural network section 14-1 are the same as the components shown in FIG. 1, so their explanation will be omitted. . Further, steps S502 to S506 shown in FIG. 5 are the same as steps S202 to S206 shown in FIG. 2, so the explanation will be omitted.

Here, the neural network unit 14-1 reads out the coefficients (coupling weight w _p,q and bias value b _q ) updated and stored by the coefficient update unit 22-1 from the memory 20-1, and uses the coefficients to Neural network operations are executed from the input layer elements to the output layer elements via the intermediate layer elements. The neural network section 14-1 then outputs the estimated position of the play equipment to the error calculation section 21.

The memory 20-1 is a rewritable memory (for example, RAM (Random Access Memory)), and temporarily stores coefficients used by the neural network unit 14-1.

The coefficients stored in the memory 20-1 are read out by the neural network unit 14-1 and used for calculations from the input layer elements of the neural network to the output layer elements via the intermediate layer elements.

At the start of operation of the position learning device 2-1, initial values of coefficients are stored in the memory 20-1. The initial value may be a preset numerical string or may be a random number (including pseudo-random numbers).

The error calculation unit 21 inputs the true position value of the play equipment that is paired with the image input by the position learning device 2-1, and also inputs the estimated position value of the play equipment from the neural network unit 14-1. Then, the error calculation unit 21 calculates the error between the true position value and the estimated position value of the play equipment (step S507), and outputs the error to the coefficient update unit 22-1. The error is, for example, a vector of differences between a vector of estimated position values of the play equipment and a vector of true position values of the play equipment.

The coefficient updating section 22-1 reads out the coefficients from the memory 20-1 and inputs the error from the error calculation section 21. Then, the coefficient update unit 22-1 updates the coefficients used by the neural network unit 14-1 based on the error (step S508), and the updated coefficients are used by the neural network unit 14-1 at the next time. It is stored in the memory 20-1 as a coefficient. The coefficient update unit 22-1 changes the coefficients based on the error, for example, using an error propagation method.

The coefficient update unit 22-1 performs optimization by updating the coefficients every time the position learning device 2-1 inputs learning data of the true position values of images and play equipment, and stores the updated coefficients in the memory 20-1. Store.

If the learning is not completed (step S509: N), the position learning device 2-1 moves to step S501, inputs a new image and the learning data of the true position value of the play equipment, and performs steps S502 to S508. Repeat the process.

On the other hand, when the learning is completed (step S509: Y), the position learning device 2-1 reads the coefficients from the memory 20-1 and outputs them to the outside as learned coefficients (step S510). The learned coefficients output from the position learning device 2-1 are stored in the memory 15-1 of the position estimating device 1-1 shown in FIG.

As described above, according to the position learning device 2-1 of the first embodiment, an image and the true position value of the play equipment corresponding thereto are input as learning data, and similarly to the position estimating device 1-1, the person position can be measured. A unit 10 measures the position of each person, and a person density map calculation unit 11 calculates a person density map. Further, the human posture measurement section 12 measures the posture of each person, and the gaze degree calculation section 13 calculates the gaze degree for each position.

The neural network section 14-1 performs neural network calculations using the coefficients stored in the memory 20-1, using the human density map that is the human density for each area and the degree of gaze for each position as input data, and calculates the play equipment. Obtain the position estimate as output data.

The error calculation unit 21 calculates the error between the true position value and the estimated position value of the play equipment. The coefficient update unit 22-1 updates the coefficients used by the neural network unit 14-1 based on the error by, for example, an error propagation method, and uses the updated coefficients at the next time point. It is stored in the memory 20-1 as a coefficient.

When the position learning device 2-1 repeats the learning process of updating the coefficients and learning is completed, the position learning device 2-1 reads the coefficients from the memory 20-1 and outputs them to the outside as learned coefficients.

The learned coefficients obtained in this way are stored in the memory 15-1 of the position estimating device 1-1 shown in FIG. 1, and are used when the neural network unit 14-1 calculates the estimated position of the play equipment. .

As a result, when the position estimation device 1-1 estimates the position of the play equipment in the image, it can estimate the position even if the play equipment is completely hidden due to interference with a person and cannot be clearly seen. It becomes possible to do so.

[Example 2]
Next, Example 2 will be explained. As mentioned above, in the second embodiment, the density of people in each region is calculated from the image, and the position of the play equipment is determined using a neural network that uses the density of people in each region as input data and the estimated position of the play equipment as output data. It is estimated.

(Position estimation device/Example 2)
FIG. 6 is a block diagram showing the configuration of a position estimating device according to the second embodiment, and FIG. 7 is a flowchart showing its processing. This position estimation device 1-2 includes a person position measurement section 10, a person density map calculation section 11, a neural network section 14-2, and a memory 15-2.

Comparing the position estimating device 1-1 of the first embodiment shown in FIG. They are common in that they include an arithmetic unit 11.

On the other hand, the position estimation device 1-2 does not include the human posture measurement unit 12 and the gaze degree calculation unit 13 of the position estimation device 1-1, and does not include the neural network unit 14-1 and the memory 15 of the position estimation device 1-1. The position estimation device 1-1 is different from the position estimation device 1-1 in that it includes a neural network unit 14-2 and a memory 15-2, which are different from the position estimation device 1-1.

The person position measurement unit 10 and the person density map calculation unit 11 are the same as the components shown in FIG. 1, and therefore their description will be omitted. Furthermore, steps S701 to S703 shown in FIG. 7 are the same as steps S201 to S203 shown in FIG. 2, so the explanation will be omitted. Here, the person density map calculation unit 11 outputs a person density map (person density for each region) to the neural network unit 14-2.

Neural network unit 14-2 reads learned coefficients from memory 15-2. Further, the neural network unit 14-2 receives the person density map from the person density map calculation unit 11.

The neural network unit 14-2 uses the human density map, which is the human density for each region, as input data, performs neural network calculations using learned coefficients, and obtains the estimated position of the play equipment as output data (step S704). . Then, the neural network unit 14-2 outputs the estimated position of the play equipment (step S705).

As the neural network unit 14-2, for example, a multilayer perceptron or the like can be used, similar to the neural network unit 14-1 shown in FIG.

When the coordinate system of the position of each person outputted by the person position measurement unit 10 is an image coordinate system, the coordinate system of the estimated position of the play equipment outputted by the neural network unit 14-2 is typically also set to the image coordinate system. However, other coordinate systems such as the world coordinate system may be used. Conversely, if the coordinate system of the position of each person outputted by the person position measurement section 10 is the world coordinate system, the coordinate system of the estimated position of the play equipment outputted by the neural network section 14-2 is also set to the world coordinate system. Although this is typical, other coordinate systems such as an image coordinate system may be used.

FIG. 8 is a diagram illustrating the neural network unit 14-2 using a multilayer perceptron in the second embodiment. This neural network unit 14-2 is configured using a three-layer perceptron, and the number of elements in the input layer is four, the number of elements in the intermediate layer is three, and the number of elements in the output layer is two.

The neural network unit 14-2 inputs the person densities (variables) x ₁ to x ₄ to four elements (first to fourth elements) of the input layer for each of the four regions.

The neural network unit 14-2 outputs the variables x ₁ to x ₄ as they are from the first to fourth elements of the input layer, and calculates the formula (4) in the fifth to seventh elements (q-th element) of the intermediate layer. ), variables x ₅ to x ₇ (x _q ), which are output values, are calculated using variables x ₁ to x ₄ , etc.

The neural network unit 14-2 outputs variables x ₅ to x ₇ from the 5th to 7th elements of the intermediate layer, and in the 8th and 9th elements (q-th element) of the output layer, formula (4) , the estimated position value (X coordinate, Y coordinate) (x _q ) of the play equipment, which is an output value, is calculated using variables x ₅ to x ₇ and the like. The neural network unit 14-2 outputs the estimated position value (X coordinate, Y coordinate) of the play equipment from the 8th and 9th elements of the output layer. In the example of FIG. 8, C ₅ =C ₆ =C ₇ ={1, 2, 3, 4}, and C ₈ =C ₉ ={5, 6, 7}.

Returning to FIG. 6, the memory 15-2 stores, for example, optimal learned coefficients obtained by learning the neural network section 14-2 using a position learning device 2-2, which will be described later.

The learned coefficients stored in the memory 15-2 are read out by the neural network unit 14-2 and used for calculations from the input layer elements of the neural network to the output layer elements via the intermediate layer elements.

The learned coefficients may be written into the memory 15-2 from the outside as necessary, or may be set as data obtained in advance in the read-only memory 15-2.

As described above, according to the position estimation device 1-2 of the second embodiment, the person position measurement unit 10 measures the position of each person from the image, and the person density map calculation unit 11 measures the position of each person from the position of each person. Compute the density map.

The neural network unit 14-2 uses the human density map, which is the human density for each region, as input data, performs neural network calculations using learned coefficients, and obtains the estimated position of the play equipment as output data.

Thereby, using the spatial distribution of people, the location where the play equipment is present can be estimated with high accuracy through neural network calculations. In other words, even if the play equipment is completely hidden and cannot be clearly seen due to interference with a person, it is possible to estimate its position.

(Position learning device/Example 2)
FIG. 9 is a block diagram showing the configuration of a position learning device according to the second embodiment, and FIG. 10 is a flowchart showing its processing. This position learning device 2-2 includes a person position measurement section 10, a person density map calculation section 11, a neural network section 14-2, a memory 20-2, an error calculation section 21, and a coefficient update section 22-2.

Comparing the position learning device 2-1 of the first embodiment shown in FIG. They are common in that they include a calculation section 11 and an error calculation section 21.

On the other hand, the position learning device 2-2 does not include the human posture measurement unit 12 and the gaze degree calculation unit 13 of the position learning device 2-1, and does not include the neural network unit 14-1 and the memory 20 of the position learning device 2-1. The position learning device 2-1 is different from the position learning device 2-1 in that it includes a neural network unit 14-2, a memory 20-2, and a coefficient update unit 22-2 that are different from the position learning device 2-1 and the coefficient update unit 22-1.

The person position measurement unit 10, the person density map calculation unit 11, and the error calculation unit 21 are the same as the components shown in FIG. 4, and therefore their description will be omitted. Furthermore, steps S1001 to S1005 shown in FIG. 10 are the same as steps S501 to S503 shown in FIG. 5, step S704 shown in FIG. 7, and step S507 shown in FIG. 5, and therefore the description thereof will be omitted. Here, the person density map calculation unit 11 outputs the person density map to the neural network unit 14-2, and the error calculation unit 21 outputs the error to the coefficient update unit 22-2.

Here, the neural network unit 14-2 reads out the coefficients updated and stored by the coefficient update unit 22-2 from the memory 20-2, and uses the coefficients to output the coefficients from the input layer elements via the intermediate layer elements. Perform neural network operations on the elements of the layer. The neural network section 14-2 then outputs the estimated position of the play equipment to the error calculation section 21.

The memory 20-2 is a rewritable memory similar to the memory 20-1 shown in FIG. 4, and temporarily stores coefficients used in the neural network unit 14-2.

The coefficients stored in the memory 20-2 are read out by the neural network unit 14-2 and used for calculations from the input layer elements of the neural network to the output layer elements via the intermediate layer elements.

At the start of the operation of the position learning device 2-2, initial values of coefficients are stored in the memory 20-2. The initial value may be a preset numerical string or may be a random number (including pseudo-random numbers).

The coefficient updating section 22-2 reads out the coefficients from the memory 20-2 and inputs the error from the error calculation section 21. Then, the coefficient update unit 22-2 updates the coefficients used by the neural network unit 14-2 based on the error (step S1006), and the updated coefficients are used by the neural network unit 14-2 at the next point in time. It is stored in the memory 20-2 as a coefficient. The coefficient updating unit 22-2 changes the coefficients based on the error, for example, using an error propagation method.

The coefficient update unit 22-2 performs optimization by updating the coefficients each time the position learning device 2-2 inputs the learning data of the image and the true position value of the play equipment, and stores the updated coefficients in the memory 20-2. Store.

If the learning is not completed (step S1007: N), the position learning device 2-2 moves to step S1001, inputs a new image and the learning data of the true position value of the play equipment, and performs steps S1002 to S1006. Repeat the process.

On the other hand, when the learning is completed (step S1007: Y), the position learning device 2-2 reads the coefficients from the memory 20-2 and outputs them to the outside as learned coefficients (step S1008). The learned coefficients output from the position learning device 2-2 are stored in the memory 15-2 of the position estimating device 1-2 shown in FIG.

As described above, according to the position learning device 2-2 of the second embodiment, an image and the true position value of the play equipment corresponding thereto are input as learning data, and similarly to the position estimation device 1-2, the position estimation device 2-2 performs human position measurement. A unit 10 measures the position of each person, and a person density map calculation unit 11 calculates a person density map.

The neural network unit 14-2 performs neural network calculations using coefficients stored in the memory 20-2 using the human density map, which is the human density for each area, as input data, and outputs the estimated position of the play equipment as output data. Find it as.

The error calculation unit 21 calculates the error between the true position value and the estimated position value of the play equipment. The coefficient update unit 22-2 updates the coefficients used by the neural network unit 14-2 based on the error using, for example, an error propagation method, and uses the updated coefficients at the next time point. It is stored in the memory 20-2 as a coefficient.

When the position learning device 2-2 repeats the learning process of updating the coefficients and the learning is completed, the position learning device 2-2 reads the coefficients from the memory 20-2 and outputs them to the outside as learned coefficients.

The learned coefficients obtained in this way are stored in the memory 15-2 of the position estimating device 1-2 shown in FIG. 6, and are used when the neural network unit 14-2 calculates the estimated position of the play equipment. .

As a result, when the position estimation device 1-2 estimates the position of the play equipment in the image, it can estimate the position even if the play equipment is completely hidden due to interference with a person and cannot be clearly seen. It becomes possible to do so.

Although the present invention has been described above with reference to Examples 1 and 2, the present invention is not limited to the above-mentioned Examples 1 and 2, and can be modified in various ways without departing from the technical concept thereof. In the first and second embodiments, the position estimation devices 1-1 and 1-2 estimate the position of the play equipment in the image, and the position learning devices 2-1 and 2-2 learn the position of the image and the play equipment as learning data. I started processing.

In contrast, the present invention is not limited to estimating the position of play equipment and using the position of the play equipment as learning data, but estimating the position of an object other than play equipment and using the position of the object as learning data. You can do it like this. Further, in the present invention, the position of an object other than an object, such as a person, may be estimated and the position of the person may be used as learning data.

Further, the person position measurement unit 10, the person density map calculation unit 11, the person posture measurement unit 12, and the gaze degree calculation unit 13 of the first embodiment, and the person position measurement unit 10 and the person density map calculation unit 11 of the second embodiment The system targeted all the people participating in the ball game and determined the position of each person.

On the other hand, the person position measurement section 10, the person density map calculation section 11, the person posture measurement section 12, and the gaze degree calculation section 13 target the people of each team participating in the ball game, and It is also possible to find the position, etc. In this case, the neural network unit 14-1 uses the person density map for each team and the degree of gaze for each team and position as input data, and performs neural network calculations using learned coefficients. Further, the neural network unit 14-2 uses the person density map for each team as input data and performs neural network calculations using learned coefficients.

Further, the neural network unit 14-1 of the first embodiment processes the person density map and the degree of gaze for each position as input data, and the neural network unit 14-2 of the second embodiment processes the person density map and the degree of gaze for each position. Processed as input data.

In response, the neural network units 14-1 and 14-2 input the person's position measured by the person position measuring unit 10, the person's moving speed, the person's moving direction, etc. measured by other components. It may also be processed as data. In this case, the person speed measurement unit detects people from the time-series images and measures the moving speed of each person, and the person direction measurement unit detects people from the time-series images and measures the movement direction of each person. measure.

Note that a normal computer can be used as the hardware configuration of the position estimating devices 1-1, 1-2 and the position learning devices 2-1, 2-2 according to the first and second embodiments. The position estimation devices 1-1, 1-2 and the position learning devices 2-1, 2-2 are computers equipped with a CPU, a volatile storage medium such as a RAM, a non-volatile storage medium such as a ROM, an interface, etc. Consisted of.

The functions of the person position measurement unit 10, person density map calculation unit 11, person posture measurement unit 12, gaze degree calculation unit 13, neural network unit 14-1, and memory 15-1 provided in the position estimation device 1-1 are as follows. Each of these functions is realized by having the CPU execute a program in which these functions are written.

In addition, the functions of the person position measurement unit 10, person density map calculation unit 11, neural network unit 14-2, and memory 15-2 provided in the position estimation device 1-2 are also implemented by implementing a program that describes these functions in the CPU. Each is realized by executing them.

Also included in the position learning device 2-1 are a person position measurement section 10, a person density map calculation section 11, a person posture measurement section 12, a gaze degree calculation section 13, a neural network section 14-1, a memory 20-1, and an error calculation section. The functions of the section 21 and the coefficient updating section 22-1 are also realized by causing the CPU to execute a program in which these functions are written.

In addition, each function of the person position measurement unit 10, person density map calculation unit 11, neural network unit 14-2, memory 20-2, error calculation unit 21, and coefficient update unit 22-2 provided in the position learning device 2-2. These functions are also realized by having the CPU execute a program in which these functions are written.

These programs are stored in the storage medium, and are read and executed by the CPU. Additionally, these programs can be stored and distributed on storage media such as magnetic disks (floppy (registered trademark) disks, hard disks, etc.), optical disks (CD-ROMs, DVDs, etc.), semiconductor memories, etc., and can be distributed via networks. You can also send and receive messages.

1-1, 1-2 Position estimation device 2-1, 2-2 Position learning device 10 Person position measurement section 11 Person density map calculation section 12 Person posture measurement section 13 Gaze degree calculation section 14-1, 14-2 Neural network Sections 15-1, 15-2, 20-1, 20-2 Memory 21 Error calculation section 22-1, 22-2 Coefficient update section

Claims (6)

In a position estimation device that estimates the position of a predetermined object in an input image,
a person position measurement unit that detects a person based on image characteristics of the input image and measures the position of each person in a predetermined coordinate system;
a person density map calculation unit that calculates a person density for each area in the predetermined coordinate system from the position of each person measured by the person position measurement unit, and generates the person density for each area as a person density map; ,
Using the person density map generated by the person density map calculation unit as input data, a neural network calculation is performed using preset learned coefficients to obtain a position estimate value indicating an estimated value of the position of the predetermined object. A neural network section for obtaining output data ,
The person density map calculation unit calculates the person density for each area by dividing the area where the person can exist into a grid pattern and calculating the number of people existing in each grid from the position of each person. A position estimating device , characterized in that the grids are arranged so as to overlap each other with adjacent grids . In a position estimation device that estimates the position of a predetermined object in an input image,
a person position measurement unit that detects a person based on image characteristics of the input image and measures the position of each person in a predetermined coordinate system;
a person density map calculation unit that calculates a person density for each area in the predetermined coordinate system from the position of each person measured by the person position measurement unit, and generates the person density for each area as a person density map; ,
a person posture measurement unit that detects the person based on image characteristics of the input image and measures the posture of each person;
A gaze degree calculation that calculates the degree of gaze of all the people to each coordinate position in the predetermined coordinate system as a gaze degree for each position from the posture of each person measured by the person posture measurement unit. Department and
Neural network calculation using preset learned coefficients using the person density map generated by the person density map calculation unit and the degree of gaze for each position calculated by the gaze degree calculation unit as input data. a neural network unit that calculates, as output data, an estimated position value indicating an estimated value of the position of the predetermined object;
A position estimation device comprising : In a position learning device that inputs an input image and a true position value indicating a true value of a position of a predetermined object corresponding to the input image as learning data, and calculates coefficients of a neural network based on the learning data,
a person position measurement unit that detects a person based on image characteristics of the input image and measures the position of each person in a predetermined coordinate system;
a person density map calculation unit that calculates a person density for each area in the predetermined coordinate system from the position of each person measured by the person position measurement unit, and generates the person density for each area as a person density map; ,
Using the person density map generated by the person density map calculation unit as input data, the neural network is operated using the coefficients, and a position estimate value indicating an estimated value of the position of the predetermined object is obtained as output data. Neural network department and
an error calculation unit that calculates an error between the true position value corresponding to the input image and the estimated position value obtained by the neural network unit;
a coefficient updating unit that updates the coefficients of the neural network based on the error calculated by the error calculation unit ,
The person density map calculation unit calculates the person density for each area by dividing the area where the person can exist into a grid pattern and calculating the number of people existing in each grid from the position of each person. A position learning device , characterized in that the grids are arranged so as to overlap with adjacent grids . In a position learning device that inputs an input image and a true position value indicating a true value of a position of a predetermined object corresponding to the input image as learning data, and calculates coefficients of a neural network based on the learning data,
a person position measurement unit that detects a person based on image characteristics of the input image and measures the position of each person in a predetermined coordinate system;
a person density map calculation unit that calculates a person density for each area in the predetermined coordinate system from the position of each person measured by the person position measurement unit, and generates the person density for each area as a person density map; ,
a person posture measurement unit that detects the person based on image characteristics of the input image and measures the posture of each person;
A gaze degree calculation that calculates the degree of gaze of all the people to each coordinate position in the predetermined coordinate system as a gaze degree for each position from the posture of each person measured by the person posture measurement unit. Department and
Using the person density map generated by the person density map calculation unit and the gaze degree for each position calculated by the gaze degree calculation unit as input data, the neural network is operated using the coefficients, and the neural network is calculated using the coefficients. a neural network unit that obtains, as output data, an estimated position value indicating an estimated value of the position of the predetermined object;
an error calculation unit that calculates an error between the true position value corresponding to the input image and the estimated position value obtained by the neural network unit;
a coefficient updating unit that updates the coefficients of the neural network based on the error calculated by the error calculation unit;
A position learning device characterized by comprising: A program for causing a computer to function as the position estimating device according to claim 1 or 2. A program for causing a computer to function as the position learning device according to claim 3 or 4.

Images