JP6577397B2 - Image analysis apparatus, image analysis method, image analysis program, and image analysis system - Google Patents

Description

  The present invention relates to an image analysis device, an image analysis method, an image analysis program, and an image analysis system.

  A system for detecting an unsteady state from an image is known. For example, Patent Literature 1 discloses that an unsteady state is specified from an optical flow attribute of an image. In Patent Literature 1, as an unsteady state, a reverse flow with respect to a main flow, a speed abnormality in which a motion of a different speed exists, and a state in which a turbulence has occurred are specified.

JP 2012-22370 A

  However, in the prior art, the types of unsteady states that can be specified are limited.

  The problem to be solved by the present invention is to provide an image analysis apparatus, an image analysis method, an image analysis program, and an image analysis system that can increase the types of unsteady states that can be specified.

  The density measuring apparatus according to the embodiment includes an extracting unit, a first calculating unit, a second calculating unit, a dividing unit, a third calculating unit, and a specifying unit. The extraction unit extracts a plurality of partial regions from the image. The first calculation unit calculates an optical flow of the partial area. The second calculation unit calculates the density of the object included in the partial region. The dividing unit divides the image into a plurality of blocks. The third calculation unit calculates, for each of the plurality of blocks, a statistic that combines the density and the amount of movement of the object indicated by the optical flow of each of the partial regions included in the block. The identifying unit identifies, among the plurality of the blocks, the block having the statistic equal to or greater than a threshold as a non-stationary block in a non-stationary state.

The block diagram which shows the functional structure of the image analysis system of this Embodiment. The figure which shows an example of the image of analysis object. Explanatory drawing of an example of a partial area | region. Explanatory drawing of optical flow and density calculation. Explanatory drawing of the division | segmentation of an image. The figure which shows an example of a response | compatibility of a partial area | region and a block. Explanatory drawing of a threshold value. Explanatory drawing which shows a specific example for every imaging | photography time. Explanatory drawing which shows a specific example for every imaging | photography time. Explanatory drawing of a threshold value. Explanatory drawing which shows a specific example for every imaging | photography time. Explanatory drawing which shows a specific example for every imaging | photography time. The schematic diagram which shows an example of a display image. The schematic diagram which shows an example of a display image. The flowchart which shows an example of the procedure of an image analysis process. The schematic diagram which shows an example of a display image. The block diagram which shows an example of a structure of a 2nd calculation part. Explanatory drawing of a correction image, a partial area | region, and a label. The block diagram which shows an example of a structure of a calculating part. Explanatory drawing of a label and a histogram. Explanatory drawing of a voting histogram. Explanatory drawing of a random tree. Explanatory drawing of random forest. Explanatory drawing of prediction of a representative label. Explanatory drawing of a random tree after representative label prediction. Explanatory drawing of prediction of a representative label. The flowchart which shows the procedure of a density calculation process. The flowchart which shows the procedure of a calculation process. Explanatory drawing of the learning using a nearest neighbor discriminator. The block diagram which shows an example of a hardware configuration.

  Exemplary embodiments of an image analysis apparatus, an image analysis method, an image analysis program, and an image analysis system will be described below in detail with reference to the accompanying drawings.

(First embodiment)
FIG. 1 is a block diagram showing a functional configuration of an image analysis system 10 according to the present embodiment. The image analysis system 10 includes an image analysis device 12, a storage unit 11, and a UI (user interface) unit 21. The storage unit 11 and the UI unit 21 are electrically connected to the image analysis device 12.

  The UI unit 21 includes a display unit 21A and an input unit 21B. The display unit 21A displays various images. The display unit 21A is, for example, a CRT (Cathode Ray Tube) display, a liquid crystal display, an organic EL (Electro Luminescence) display, a plasma display, or the like. The input unit 21B receives various instructions and information input from the user. The input unit 21B is, for example, a keyboard, a mouse, a switch, a microphone, or the like.

  The UI unit 21 may be a touch panel in which the display unit 21A and the input unit 21B are integrally configured.

  The storage unit 11 stores various data. In the present embodiment, the storage unit 11 stores an image to be analyzed. The storage unit 11 is, for example, at least a storage device that can store magnetically, optically, and electrically such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), a ROM (Read Only Memory), and a memory card. It can be realized by either.

  The image analysis device 12 analyzes the image and specifies a region in an unsteady state (details will be described later). The image to be analyzed by the image analysis device 12 includes one or more objects. The object may be any object that can be image-analyzed. In the present embodiment, as an example, a case where an object is a person will be described. However, the object is not limited to a person. For example, the object may be a vehicle, a cell, or the like.

  The image analysis device 12 is a computer including a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. The image analysis device 12 may be a circuit other than the CPU.

  The image analysis device 12 controls the entire image analysis system 10. The image analysis device 12 includes an acquisition unit 13, an extraction unit 14, a first calculation unit 15, a second calculation unit 16, a division unit 17, a third calculation unit 18, a specifying unit 19, and a display control unit. 20 and.

  Some or all of the acquisition unit 13, the first calculation unit 15, the second calculation unit 16, the division unit 17, the third calculation unit 18, the specifying unit 19, and the display control unit 20 are, for example, a processing device such as a CPU. That is executed by software, that is, may be realized by software, may be realized by hardware such as an IC (Integrated Circuit), or may be realized by using software and hardware in combination.

  The acquisition unit 13 acquires an analysis target image. In the present embodiment, the acquisition unit 13 acquires an image from the storage unit 11. Note that the acquisition unit 13 may acquire an image from an external device (not shown) or a known imaging device (not shown).

  In the present embodiment, the case where the acquisition unit 13 acquires a plurality of images taken in time series will be described as an example.

  FIG. 2 is a diagram illustrating an example of the image 33 to be analyzed. The image 33 includes the object 30. As described above, in the present embodiment, a case where the object 30 is a person will be described.

  Returning to FIG. 1, the extraction unit 14 extracts a plurality of partial regions from the image 33.

  The partial area is a partial image of the image 33 and includes at least one object 30. In the present embodiment, the case where the partial area is an image obtained by extracting a part of the image 33 into a rectangular shape will be described.

  FIG. 3 is an explanatory diagram of an example of the partial region 50. The extraction unit 14 extracts a plurality of partial regions 50 by moving a rectangular region to be extracted on the image 33.

  In the example illustrated in FIG. 3, the extraction unit 14 performs extraction so that each partial region 50 includes one object 30. However, the partial region 50 may include two or more objects 30.

  The shape of the partial region 50 is not limited to a rectangle, and may be an arbitrary shape. In addition, the size and shape of each of the partial areas 50 extracted from the image 33 may be the same or different between the partial areas 50. For example, the size of each partial area 50 may be a shape or a size corresponding to the area occupied by the object 30 included in each partial area 50.

  Further, at least a part of the plurality of partial regions 50 extracted from the image 33 may overlap each other. The number of partial regions 50 extracted by the extraction unit 14 from the image 33 may be two or more, and the number is not limited.

  Note that the extraction unit 14 may prepare a rectangular frame having a predetermined shape and size, and extract the partial region 50 having a shape and size corresponding to the frame from the image 33. The extraction unit 14 may specify each of the objects 30 included in the image 33 and extract the partial region 50 so as to include the specified object 30. Further, the position, size, and shape of each of the partial areas 50 extracted by the extraction unit 14 may be changeable by the user operating the input unit 21B.

  In the present embodiment, the extraction unit 14 extracts a plurality of partial regions 50 for each of a plurality of images (including the image 33) taken in time series.

  Returning to FIG. 1, the first calculation unit 15 calculates the optical flow of the partial region 50. The first calculation unit 15 calculates an optical flow for each of the plurality of partial regions 50 extracted by the extraction unit 14.

  The optical flow is a vector representing the motion of the object 30 in the image. That is, the optical flow represents the moving amount and moving direction of the object 30.

  In the present embodiment, the first calculation unit 15 is provided between each of the plurality of partial regions 50 in the analysis target image 33 and each of the plurality of partial regions 50 in the image captured before the image 33. Thus, the same object 30 is associated. The image photographed before the image 33 is preferably an image photographed immediately before the image 33 among a plurality of images that are continuous in time series.

  Then, the first calculation unit 15 calculates the optical flow of the object 30 included in each of the plurality of partial regions 50 in the image 33.

  The first calculation unit 15 calculates an optical flow using a known method. For example, the first calculation unit 15 calculates each optical flow of each partial region 50 using the Lucas Kanade method.

  The Lucas Kanade method is specifically described in Ref. 1 (BD Lucas and T. Kanade, “An Iterative Image registration in the fon ation in Stereo. pp. 674 to 667.).

  FIG. 4 is an explanatory diagram of optical flow and density calculation. FIG. 4A is an explanatory diagram of the optical flow 35. As shown in FIG. 4A, the first calculation unit 15 calculates an optical flow 35 for each partial region 50.

  Note that the partial region 50 may include a plurality of objects 30. In this case, the first calculation unit 15 calculates an optical flow for each of the plurality of objects 30 included in the partial region 50. Then, the first calculation unit 15 calculates a vector optical flow 35 in which the average movement amount of the movement amount and the average direction of the vector direction indicated by the calculated optical flow are the movement amount and the vector direction. The first calculation unit 15 uses this optical flow 35 as the optical flow 35 of the partial region 50.

  Returning to FIG. 1, the second calculation unit 16 calculates the density of the object 30 included in the partial region 50 for each of the partial regions 50. That is, the second calculation unit 16 calculates the density of the object 30 for each of the partial regions 50 extracted from the analysis target image 33.

  The second calculation unit 16 may calculate the density of the object 30 included in each partial region 50 using a known method. In addition, for the reason of calculating a density with high precision, it is preferable that the 2nd calculation part 16 calculates a density using the calculation method demonstrated in 2nd Embodiment (it mentions later in detail).

  FIG. 4B is an explanatory diagram of density calculation. For example, the second calculation unit 16 calculates the density of the object 30 for each of the partial areas 50 included in the image 33.

  Therefore, the optical flow 35 and the density are calculated for each of the plurality of partial regions 50 included in the image 33 by the first calculation unit 15 and the second calculation unit 16 (FIG. 4A). ), See FIG. 4B).

  Returning to FIG. 1, the dividing unit 17 divides the analysis target image 33 into a plurality of blocks. FIG. 5 is an explanatory diagram of the division of the image 33. For example, the dividing unit 17 divides the image 33 into a plurality of blocks P.

  FIG. 5 shows an example in which the dividing unit 17 divides the image 33 into 9 × 3 blocks P (block P1 to block P9). However, the dividing unit 17 may divide the image 33 into a plurality of (two or more) blocks P, and is not limited to nine.

  The dividing unit 17 preferably divides the image 33 so that at least one of the divided blocks P includes at least one partial region 50. FIG. 6 is a diagram illustrating an example of a correspondence between the partial region 50 and the block P (block P1 to block P9). As illustrated in FIG. 6, the dividing unit 17 preferably divides the image 33 so that at least one of the plurality of blocks P includes at least one partial region 50.

  Each of the plurality of blocks P (blocks P1 to P9) may have the same shape and size, or may have different shapes and sizes. For example, the dividing unit 17 may divide the image 33 into a plurality of blocks P so as to include at least a predetermined number of partial regions 50.

  Further, the size and the number of divisions of the block P may be changed according to the attribute of the image 33 to be analyzed. The attribute of the image 33 to be analyzed indicates the scene (location, time, time, number of objects included in one image, size of the object, etc.) of the image 33 taken. In this case, the acquisition part 13 should just acquire the image 33 to which the attribute was provided. Then, the dividing unit 17 may read the attribute assigned to the analysis target image 33 and divide the image 33 into blocks P having a size and the number of divisions corresponding to the read attribute.

  Returning to FIG. 1, the third calculator 18 calculates a statistic for each of the plurality of blocks P. The statistic is a value obtained by combining the density of the object 30 and the movement amount of the object 30 indicated by the optical flow 35 in each of the partial areas 50 included in the block P. In other words, the statistic is information indicating a property represented by the density and the moving amount of the object 30 in one or more partial regions 50 included in each of the blocks P.

  The statistic is, for example, at least one of the momentum of the object 30, the kinetic energy of the object 30, the residence degree of the object 30, and the potential energy of the object 30 included in the block P. That is, the statistic may be any one of the momentum of the object 30, the kinetic energy of the object 30, the staying degree of the object 30, and the potential energy of the object 30, or a combination of two or more of these. It may be. In the following description, the object 30 may be simply referred to as an object while omitting the reference numerals.

  The third calculation unit 18 combines one or more of the momentum of the object, the kinetic energy of the object, the staying degree of the object, and the potential energy of the object calculated from the one or more partial regions 50 included in the block P. Are calculated as statistics.

  For each of the plurality of blocks P, the third calculation unit 18 calculates a statistic using the density of the object and the amount of movement of the object in each of the partial regions 50 included in each block P. The amount of movement of the object is the amount of movement of the object indicated by the optical flow 35. In other words, the length of the vector of the optical flow 35 corresponds to the amount of movement of the object.

  Note that, as shown in FIG. 6, each of the partial areas 50 is not necessarily arranged so as to fit in any one area. That is, the partial region 50 may be arranged across adjacent blocks P. In this case, for the partial region 50 arranged across the plurality of blocks P, the third calculation unit 18 uses a value obtained by distributing the calculated density and movement amount according to the area ratio of the region overlapping each block P. Use.

  For example, it is assumed that the density of an object in a certain partial region 50 is “H” and the movement amount indicated by the optical flow 35 is “R”. Further, it is assumed that the partial area 50 is arranged across a plurality of blocks P (for example, block P1 and block P2). Further, in this partial region 50, it is assumed that a region of 1/3 of the entire area is located in the block P1 and a region of 2/3 is located in the block P2.

  In this case, the third calculation unit 18 may use a value obtained by multiplying the density “H” by 1/3 as the density of the object in the partial region 50 in the block P1. The third calculation unit 18 may use a value obtained by multiplying the movement amount “R” by 1/3 as the movement amount of the partial region 50 in the block P1.

  Similarly, the third calculation unit 18 may use a value obtained by multiplying the density “H” by 2/3 as the density of the object in the partial region 50 in the block P2. The third calculation unit 18 may use a value obtained by multiplying the movement amount “R” by 2/3 as the movement amount of the partial region 50 in the block P2.

  The specifying unit 19 specifies a block P having a statistic calculated by the third calculating unit 18 that is equal to or greater than a threshold among the plurality of blocks P (blocks P1 to P9) as a non-stationary block in an unsteady state. The threshold value used for specifying whether it is in the unsteady state or the steady state may be determined in advance.

  The unsteady state indicates a state that is not a steady state. The unsteady state indicates, for example, at least one kind of presence / absence of congestion of objects, presence / absence of retention of objects, presence / absence of collection of objects, and the like included in the block P.

  The presence / absence of congestion of objects indicates a state where a large number of objects are moving densely or a state where a small number of objects are moving. A large number of objects refers to a number of objects equal to or greater than a predetermined threshold. The small number of objects indicates a number of objects less than the threshold value. These threshold values (number of objects) may be determined in advance by adjusting the threshold values.

  The presence / absence of an object indicates a state in which a large number of objects have stayed or a state in which no objects have stayed. The retention of a large number of objects indicates that the moving speed of the large number of objects is equal to or less than a threshold value. This threshold may be determined in advance.

  The presence / absence of a set of objects indicates a state where a large number of objects are gathered or a state where no objects are gathered. The collection of objects means that the distance between the objects is equal to or less than a predetermined threshold value. This threshold may be determined in advance.

  Next, the calculation of the statistic and the specification of the unsteady state will be specifically described.

  First, a case where the amount of movement of an object is calculated as a statistic will be described.

  In this case, the third calculation unit 18 calculates, for each of the plurality of blocks P, the sum of multiplication values of the density of the object and the movement amount indicated by the optical flow 35 in each of the partial regions 50 included in the block P. Calculated as the momentum of the object.

  Specifically, the third calculation unit 18 calculates the amount of movement M of the object for each block P using Equation (1).

  In Expression (1), M represents the momentum of the object in the block P. i is an integer of 1 to n. mi indicates the density of the object in the partial region 50 at the position i in the block P. vi represents the movement amount (movement amount indicated by the optical flow 35) of the partial region 50 at the position i in the block P. One position i is assigned to each of the partial areas 50 in the block P. For this reason, the value of n in the formula (1) matches the number of partial areas 50 existing in the block P.

  That is, the third calculation unit 18 calculates, for each of the partial areas 50 in the block P, a multiplication value of the moving amount indicated by the optical flow 35 and the density of the object. Then, the third calculation unit 18 calculates the total sum (total value) of the multiplication values of the partial areas 50 included in the block P as the amount of movement M of the object.

  In this case, the specifying unit 19 unsteadyly selects a block P in which the momentum M of the object calculated by the third calculation unit 18 is equal to or greater than the first threshold among the plurality of blocks P (blocks P1 to P9) of the image 33. Identifies as a block. That is, the specifying unit 19 specifies a block P that satisfies the relationship of M ≧ T1 as an unsteady block. T1 indicates a first threshold value.

  The specifying unit 19 associates the number of partial regions 50 included in the block P (that is, the value of n in the expression (1)) with the first threshold value and stores them in the storage unit 11 in advance. And the specific | specification part 19 reads the 1st threshold value corresponding to the number of the partial areas 50 contained in the block P for every block P from the memory | storage part 11, and uses it for specification whether it is an unsteady block.

  Further, the specifying unit 19 specifies the presence / absence of object congestion by using the sum of the densities of the objects in the included partial region 50 for the block P satisfying the relationship of M ≧ T1. Specifically, the specifying unit 19 specifies the presence / absence of object congestion for each block P using Expression (2).

  In Expression (2), D represents the total sum of the densities of the objects in the partial area 50 included in the block P. In formula (2), n, i, and mi have the same meaning as in formula (1).

  Of the blocks P identified as being in an unsteady state using the momentum M, the identifying unit 19 has a block P in which the total sum D of the densities of the objects in the partial areas 50 included in the block P is equal to or greater than a fifth threshold value. Are identified as being in a state where a large number of objects are moving densely. That is, the specifying unit 19 specifies the block P satisfying the relations of M ≧ T1 and D ≧ T5 as an unsteady block in which a large number of objects are moving densely. T5 represents a fifth threshold value.

  In addition, the specifying unit 19 uses the momentum M to specify that the density D of the objects in each of the partial areas 50 included in the block P among the blocks P specified as being in an unsteady state is less than the fifth threshold value. Block P is identified as being in a state where a small number of objects are moving. That is, the specifying unit 19 specifies a block P that satisfies the relationship of M ≧ T1 and D <T5 as an unsteady block in which a small number of objects are moving.

  Note that the specifying unit 19 associates the number of partial regions 50 included in the block P (that is, the value of n in Expression (2)) with the fifth threshold value and stores them in the storage unit 11 in advance. Then, for each block P, the specifying unit 19 reads the fifth threshold value corresponding to the number of partial areas 50 included in the block P from the storage unit 11 and uses it to specify whether or not the object is congested.

  FIG. 7 is an explanatory diagram of the threshold when there is only one partial area 50 in the block P. In FIG. 7, a line 38A is a line indicating the first threshold value T1. Further, in FIG. 7, a line T5 is a line indicating the fifth threshold T5.

  In the diagram shown in FIG. 7, the space A in which the value of m × | v | (the multiplication value of the density and the absolute value of the movement amount) is equal to or greater than the line 38A is a space indicating an unsteady state. A space B having a value of m × | v | smaller than the line 38A is a space indicating a steady state. In the non-steady state space A, a space having a density (m) value equal to or higher than the fifth threshold T5 is a space indicating a state in which many objects are densely packed. In the non-steady state space A, the space where the density (m) is less than the fifth threshold is a space indicating a state in which a small number of objects are moving.

  As described above, the specifying unit 19 uses the momentum M of the object calculated by the third calculation unit 18 for each of the blocks P (blocks P1 to P9) included in the image 33, so that the non-steady state is set. Identify stationary blocks. Further, the identifying unit 19 uses the momentum M of the object to check whether or not the object of the unsteady block is congested (a state in which a large number of objects are moving densely or a state in which a small number of objects are moving). ).

  For each of the plurality of images taken in time series acquired by the acquisition unit 13, the extraction unit 14, the first calculation unit 15, the second calculation unit 16, the division unit 17, the third calculation unit 18, and the specifying unit By performing the above-described processing, the presence / absence of congestion of the objects in each block P can be specified for each shooting time.

  FIG. 8 is an explanatory diagram showing a specific example for each shooting time. In FIG. 8, a line 39A shows an example of the transition of the calculation result of the momentum M of a specific block P. In the example shown in FIG. 8, the momentum M exceeds the first threshold T1 at the photographing time t1. In this case, the specifying unit 19 specifies that the block P in the image 33 shot at the timing of the shooting time t1 is an unsteady block in an unsteady state. At this time, the specifying unit 19 performs the above-described specification using the fifth threshold value T5, so that the presence or absence of congestion of objects in the block P (a state in which a large number of objects are moving densely, or A state where a small number of objects are moving) can also be specified.

  Next, a case where the kinetic energy of an object is calculated as a statistic will be described.

  In this case, the third calculation unit 18 calculates, for each of the plurality of blocks P, the sum of the multiplication values of the density of the object and the square of the movement amount of each of the partial regions 50 included in the block P, and calculates the kinetic energy of the object. Calculate as

  Specifically, the third calculation unit 18 calculates the kinetic energy K of the object for each block P using Equation (3).

  In equation (3), K represents the kinetic energy of the object in the block P. i, mi, vi, and n have the same meaning as in formula (1).

  That is, the third calculation unit 18 calculates, for each of the partial areas 50 in the block P, a multiplication value of the square of the movement amount indicated by the optical flow 35 and the density of the object. Then, the third calculation unit 18 calculates the sum (total value) of the multiplication values of each of the partial areas 50 included in the block P as the kinetic energy K of the object.

  In this case, the specifying unit 19 selects a block P in which the kinetic energy K of the object calculated by the third calculation unit 18 is not less than the second threshold among the plurality of blocks P (blocks P1 to P9) of the image 33. Identifies as a stationary block. That is, the specifying unit 19 specifies a block P that satisfies the relationship of K ≧ T2 as an unsteady block in an unsteady state. T2 indicates a second threshold value.

  The specifying unit 19 associates the number of partial areas 50 included in the block P (that is, the value of n in Expression (3)) with the second threshold value, and stores them in the storage unit 11 in advance. And the specific | specification part 19 reads the 2nd threshold value corresponding to the number of the partial areas 50 contained in the block P for every block P from the memory | storage part 11, and uses it for specification whether it is an unsteady block.

  Further, the specifying unit 19 specifies the presence / absence of object congestion for the block P satisfying the relationship of K ≧ T2, using the sum of the densities of the objects in the included partial region 50. Specifically, the specifying unit 19 specifies the presence or absence of object congestion for each block P using the above equation (2).

  Specifically, the specifying unit 19 uses the kinetic energy K to specify that the density D of the objects in each of the partial regions 50 included in the block P among the blocks P specified as being in an unsteady state is the sixth A block P that is equal to or greater than the threshold is identified as a state in which a large number of objects are moving densely. That is, the specifying unit 19 specifies a block P that satisfies the relations K ≧ T2 and D ≧ T6 as an unsteady block in an unsteady state in which many objects are moving densely. T6 represents a sixth threshold value.

  In addition, the specifying unit 19 uses the kinetic energy K to specify that the density D of each object in the partial area 50 included in the block P is less than the sixth threshold among the blocks P specified as being in an unsteady state. Block P is identified as being in a state where a small number of objects are moving. That is, the specifying unit 19 specifies a block P that satisfies the relationship of K ≧ T2 and D <T6 as an unsteady block in an unsteady state in which a small number of objects are moving.

  Note that the specifying unit 19 associates the number of partial regions 50 included in the block P (that is, the value of n in Equation (2)) with the sixth threshold value and stores them in the storage unit 11 in advance. Then, for each block P, the specifying unit 19 reads a sixth threshold value corresponding to the number of partial areas 50 included in the block P from the storage unit 11 and uses it to specify whether or not the object is congested.

  Thus, the specifying unit 19 is in an unsteady state by using the kinetic energy K of the object calculated by the third calculation unit 18 for each of the blocks P (blocks P1 to P9) included in the image 33. Identify non-stationary blocks. Further, the specifying unit 19 uses the kinetic energy K of the object to check whether or not the object of the non-stationary block is congested (a state where a large number of objects are moving densely or a small number of objects are moving). State).

  For each of the plurality of images taken in time series acquired by the acquisition unit 13, the extraction unit 14, the first calculation unit 15, the second calculation unit 16, the division unit 17, the third calculation unit 18, and the specifying unit By performing the above-described processing, the presence / absence of congestion of the non-stationary block object can be specified for each photographing time.

  FIG. 9 is an explanatory diagram illustrating a specific example for each shooting time. In FIG. 9, a line 39 </ b> B shows an example of the transition of the calculation result of the kinetic energy K of a specific block P. In the example shown in FIG. 9, the kinetic energy K exceeds the second threshold value T2 during the period of the imaging time t2 to the imaging time t3 and during the period of the imaging time t4 to the imaging time t5. In this case, the specifying unit 19 determines that the block P in the image 33 photographed during the photographing time t2 to the photographing time t3 and the photographing time t4 to the photographing time t5 is an unsteady block in an unsteady state. Identify. At this time, the specifying unit 19 performs the above-described specification using the sixth threshold value T6, thereby confirming whether the object of the block P is congested (a state in which a large number of objects are moving densely, or A state where a small number of objects are moving) is also identified.

  Next, a case where the staying degree of an object is calculated as a statistic will be described.

  In this case, the third calculation unit 18 calculates, for each of the plurality of blocks P, the sum of the product values of the density of the object and the reciprocal of the movement amount of each of the partial regions 50 included in the block P, as the degree of object retention. Calculate as Note that the reciprocal of the movement amount may be a reciprocal of an addition value obtained by adding a constant to the movement amount.

  Specifically, the third calculation unit 18 calculates the staying degree S of the object for each block P using Expression (4).

  In the formula (4), S indicates the staying degree of the object in the block P. i, mi, vi, and n have the same meaning as in formula (1). In equation (4), g (vi) is a function represented by equation (5).

  In formula (5), α is an integer of 0 or more. An arbitrary value may be determined in advance for α.

  That is, the third calculation unit 18 calculates the product of the density of the object and the reciprocal of the movement amount for each of the partial regions 50 in the block P. Then, the third calculation unit 18 calculates the total sum (total value) of the multiplication values of the partial areas 50 included in the block P as the staying degree S of the object.

  In this case, the specifying unit 19 determines that the block P in which the staying degree S of the object calculated by the third calculation unit 18 is equal to or greater than the third threshold among the plurality of blocks P (blocks P1 to P9) of the image 33 Identifies as a stationary block. That is, the specifying unit 19 specifies a block P that satisfies the relationship of S ≧ T3 as an unsteady block. T3 represents a third threshold value.

  Note that the specifying unit 19 associates the number of partial regions 50 included in the block P (that is, the value of n in Equation (4)) with the third threshold value and stores them in the storage unit 11 in advance. And the specific | specification part 19 reads the 3rd threshold value corresponding to the number of the partial areas 50 contained in the block P for every block P from the memory | storage part 11, and uses it for specification whether it is an unsteady block.

  In the present embodiment, the specifying unit 19 specifies that an object stays in the block P in which the object staying degree S is equal to or greater than the third threshold (that is, a large number of objects are staying).

  FIG. 10 is an explanatory diagram of the threshold when there is only one partial region 50 in the block P (that is, n is “1” (n = 1) in equation (4)). In FIG. 10, a line 38C is a line indicating the third threshold T3.

  In FIG. 10, the space A in which the staying degree S represented by the density and the moving amount is equal to or greater than the line 38 </ b> C is a space indicating that the staying of the object is occurring. In addition, the space B in which the staying degree S represented by the density and the moving amount is less than the line 38C is a space indicating a steady state.

  As described above, the specifying unit 19 is in an unsteady state by using the object retention degree S calculated by the third calculation unit 18 for each of the blocks P (blocks P1 to P9) included in the image 33. Identify non-stationary blocks. Further, the specifying unit 19 uses the object retention degree S to specify that an object stays in the unsteady block.

  For each of the plurality of images taken in time series acquired by the acquisition unit 13, the extraction unit 14, the first calculation unit 15, the second calculation unit 16, the division unit 17, the third calculation unit 18, and the specifying unit By performing the above-described processing in 19, it is possible to specify whether or not the object of the unsteady block remains for each shooting time.

  FIG. 11 is an explanatory diagram showing a specific example for each shooting time. In FIG. 11, a line 39C shows an example of the transition of the calculation result of the staying degree S of a specific block P. In the example shown in FIG. 11, the staying degree S exceeds the third threshold T3 during the period from the imaging time t6 to the imaging time t7. In this case, the specifying unit 19 specifies that the block P in the image 33 shot at the timing of the shooting time t6 to the shooting time t7 is an unsteady block in an unsteady state. At this time, the specifying unit 19 specifies that an object stays in the unsteady block.

  Next, the case where the potential energy of an object is calculated as a statistic will be described.

  In this case, for each of the plurality of blocks P, the third calculation unit 18 includes the density of the objects in the partial area 50 included in the block P, the reciprocal of the amount of movement of the objects in the partial area 50, and the same block P. The product of the density of the object in the other partial area 50 is calculated. Then, the third calculation unit 18 calculates the sum obtained by dividing the multiplication value by the distance between the partial region 50 and the other partial region 50 as the potential energy of the object.

  Specifically, the third calculation unit 18 calculates the kinetic energy K of the object for each block P using Equation (6).

  In equation (6), U represents the potential energy of the object in the block P. i, mi, vi, and n have the same meaning as in formula (1). g (vi) is a function represented by the above equation (5).

  In formula (6), j is an integer of 1 or more and n or less. j indicates the position of each partial region 50 in the block P, and indicates a position different from i. That is, i ≠ j. In Expression (6), rij indicates the distance between the partial area 50 at the position i and the partial area 50 at the position j in the same block P. For example, rij indicates the shortest distance between the center of the partial region 50 at the position i and the center of the partial region 50 at the position j in the image 33.

  In this case, the specifying unit 19 determines that the block P in which the potential energy U of the object calculated by the third calculation unit 18 is equal to or greater than the fourth threshold among the plurality of blocks P (blocks P1 to P9) of the image 33 is not determined. Identifies as a stationary block. That is, the specifying unit 19 specifies a block P that satisfies the relationship U ≧ T4 as an unsteady block. T4 represents a fourth threshold value.

  Note that the specifying unit 19 associates the number of partial areas 50 included in the block P (that is, the value of n in Expression (6)) with the fourth threshold value and stores them in the storage unit 11 in advance. And the specific | specification part 19 reads the 4th threshold value corresponding to the number of the partial areas 50 contained in the block P for every block P from the memory | storage part 11, and uses it for specifying whether it is an unsteady block.

  In the present embodiment, the specifying unit 19 specifies a block P having an object potential energy U equal to or greater than a fourth threshold as an unsteady block in an unsteady state. Further, the specifying unit 19 specifies that the stay of an object and a set of a large number of objects are generated in the unsteady block.

  As described above, the specifying unit 19 is in an unsteady state by using the potential energy U of the object calculated by the third calculation unit 18 for each of the blocks P (blocks P1 to P9) included in the image 33. Identify non-stationary blocks. Further, the specifying unit 19 specifies that the object stays and a large number of objects are gathered in the non-stationary block by using the potential energy U of the object.

  For each of the plurality of images taken in time series acquired by the acquisition unit 13, the extraction unit 14, the first calculation unit 15, the second calculation unit 16, the division unit 17, the third calculation unit 18, and the specifying unit By performing the above-described processing in 19, it is possible to specify the presence or absence of non-stationary block objects and the collection of a large number of objects for each photographing time.

  FIG. 12 is an explanatory diagram illustrating a specific example for each shooting time. In FIG. 12, line 39D shows an example of the transition of the calculation result of the potential energy U of a specific block P. In the example shown in FIG. 12, the potential energy U exceeds the fourth threshold T4 after the imaging time t8. In this case, the specifying unit 19 specifies that the block P in the image 33 shot after the shooting time t8 is an unsteady block in an unsteady state. Further, at this time, the specifying unit 19 specifies that the stay of the object and the collection of a large number of objects are generated in the block P specified as the unsteady block.

  In the above description, the case where the third calculation unit 18 calculates each of the momentum M, the kinetic energy K, the staying degree S, and the potential energy U as statistical amounts has been described as an example. Moreover, the specific | specification part 19 demonstrated the form which specifies an unsteady block using each of the momentum M, the kinetic energy K, the residence degree S, and the positional energy U. FIG.

  However, the third calculation unit 18 combines a plurality of object momentum, object kinetic energy, object retention, and object potential energy calculated from one or more partial regions 50 included in the block P. May be calculated as a statistic.

  In this case, the specifying unit 19 selects a block P in which the statistic calculated by the third calculation unit 18 is greater than or equal to a predetermined threshold among the plurality of blocks P (blocks P1 to P9) in a non-steady state. What is necessary is just to identify as a stationary block. The identifying unit 19 may identify the type of the unsteady state according to the statistic value.

  Returning to FIG. 1, the display control unit 20 performs control to display an image on the display unit 21A. For example, the display control unit 20 displays a display image in which the unsteady block specified by the specifying unit 19 among the plurality of blocks P (blocks P1 to P9) included in the image 33 is emphasized on the display unit 21A. Take control.

  FIG. 13 is a schematic diagram illustrating an example of the display image 37A. For example, it is assumed that the specifying unit 19 specifies the block P5 and the block P7 as non-stationary blocks. Further, the identifying unit 19 determines that a large number of objects are moving densely for the block P5, and determines that a small number of objects are moving for the block P7. Assume.

  In this case, for example, the display control unit 20 displays the display image 37A shown in FIG. 13 on the display unit 21A. The display image 37A is an image obtained by superimposing a frame line 38A for emphasizing the block P5 identified as an unsteady block and a frame line 38B for emphasizing the block P7 identified as an unsteady block on the image 33. Note that the method for emphasizing the non-stationary block is not limited to a form in which the non-stationary block is surrounded by a frame line. For example, the display control unit 20 may display the unsteady block in a display form different from the block P in the steady state. Different display forms differ in at least one of color, blinking interval, luminance, saturation, and brightness.

  Moreover, it is preferable that the display control part 20 displays an unsteady block with the display form according to the kind of unsteady state. Specifically, the display control unit 20 displays the block P7 in which a large number of objects are moving densely and the block P5 in which a small number of objects are moving in different display forms. To do. In the example shown in FIG. 13, an example is shown in which the line types of the lines surrounding these blocks P5 and P7 are changed (line 41A, line 41B). In addition, it is not limited to the form which changes a line type.

  The display control unit 20 also superimposes information indicating an unsteady state on the unsteady block identified by the identifying unit 19 among the plurality of blocks P (blocks P1 to P9) included in the image 33. May be displayed on the display unit 21A.

  FIG. 14 is a schematic diagram illustrating an example of the display image 37B. For example, it is assumed that the specifying unit 19 specifies the block P5 and the block P7 as non-stationary blocks. Further, the identifying unit 19 determines that a large number of objects are moving densely for the block P5, and determines that a small number of objects are moving for the block P7. Assume.

  In this case, for example, the display control unit 20 displays the display image 37B illustrated in FIG. 14 on the display unit 21A. The display image 37 </ b> B is an image obtained by superimposing characters indicating “move multiple objects” on the block P <b> 5 identified as the unsteady block in the image 33 as information indicating the unsteady state. The display image 37B is an image obtained by superimposing a character indicating “movement of a small number of objects” as information indicating an unsteady state on the block P7 identified as an unsteady block in the image 33.

  Note that the display control unit 20 may superimpose information (for example, characters) indicating the shooting time of the image 33 in which the unsteady state has occurred on the block P identified as the unsteady block. Thereby, the occurrence time (shooting time) of the unsteady state can be provided to the user.

  Thus, the display control unit 20 may display the display image 37B on the display unit 21A.

  Next, the procedure of image analysis processing executed by the image analysis device 12 will be described. FIG. 15A is a flowchart illustrating an example of a procedure of image analysis processing executed by the image analysis device 12.

  First, the acquisition unit 13 acquires an image to be analyzed (step S100). Next, the extraction unit 14 extracts a plurality of partial regions 50 from the image acquired in step S100 (step S102).

  Next, the first calculation unit 15 calculates an optical flow 35 for each of the plurality of partial regions 50 extracted in step S102 (step S104). Next, the 2nd calculation part 16 calculates the density of the object contained in the partial area 50 about each of the some partial area 50 extracted by step S102 (step S106).

  Next, the dividing unit 17 divides the analysis target image into a plurality of blocks P (step S108). Next, the third calculation unit 18 calculates a statistic for each of the plurality of blocks P (step S110).

  Next, the identifying unit 19 sets a block P in which the statistic calculated in step S110 is equal to or greater than a predetermined threshold among the plurality of blocks P (block P1 to block P9) divided in step S108 to an unsteady state. (Step S112).

  Next, the specifying unit 19 associates the identification information of the specified non-stationary block, the type of the specified non-stationary state, and the identification information (for example, the shooting time) of the image 33 from which the non-stationary block is specified. Is stored in the storage unit 11 (step S114).

  Next, the display control unit 20 generates an image to be displayed on the display unit 21A (step S116). For example, the display control unit 20 emphasizes the unsteady block identified by the identifying unit 19 among the plurality of blocks P (blocks P1 to P9) included in the image 33 (see the display image 37A in FIG. 13). ) Is generated. Note that the display control unit 20 may generate the display image 37B (see FIG. 14).

  Then, the display control unit 20 performs control to display the generated display image on the display control unit 20 (step S118). Then, this routine ends.

  In addition, it is preferable that the image analysis apparatus 12 repeats the process of step S102-step S118 whenever it acquires a new image by step S100.

  As described above, the image analysis apparatus 12 according to the present embodiment includes the extraction unit 14, the first calculation unit 15, the second calculation unit 16, the division unit 17, the third calculation unit 18, and the specifying unit. 19. The extraction unit 14 extracts a plurality of partial regions 50 from the image 33. The first calculation unit 15 calculates the optical flow 35 of the partial region 50. The second calculation unit 16 calculates the density of objects included in the partial region 50. The dividing unit 17 divides the image 33 into a plurality of blocks P. For each of the plurality of blocks P, the third calculation unit 18 calculates a statistic that combines the density of each of the partial regions 50 included in the block P and the amount of movement of the object indicated by the optical flow 35. The specifying unit 19 specifies a block P having a statistic equal to or greater than a threshold value among the plurality of blocks P as a non-stationary block in an unsteady state.

  As described above, in the image analysis apparatus 12 according to the present embodiment, the unsteady block in the unsteady state is specified using the amount of movement of the object indicated by the optical flow 35 and the density of the object. For this reason, in the image analysis device 12, many types of unsteady-state blocks P are set as unsteady blocks as compared with the case where the unsteady-state unsteady blocks are identified from only the movement amount or only the density. Can be identified.

  Therefore, in the image analysis device 12 of the present embodiment, it is possible to increase the number of types of unsteady states that can be specified.

  The display control unit 20 may display a display image indicating the unsteady block on the display unit 21A. FIG. 15B is a schematic diagram illustrating an example of the display image A1. The display image A1 shown in FIG. 15-2 can be used for security operation support.

  In general, surveillance cameras are installed in various places in buildings and commercial facilities. In the separate room, the observer checks whether there is an abnormality while watching the video of the surveillance camera. When a suspicious person or a suspicious object is found in the video of the surveillance camera, the surveillance staff contacts a security company or a nearby security staff, and the security staff who has been contacted visits the site and responds. However, normally, as shown in FIG. 15B, since it is necessary to simultaneously monitor the images of many surveillance cameras, it is difficult to find a problem. If a watcher overlooks a problem or delays discovery, the security cannot be handled and the quality of the security deteriorates.

  On the other hand, according to the image analysis apparatus 12 of the present embodiment, the unsteady block in the unsteady state is specified using the movement amount of the object indicated by the optical flow 35 and the density of the object. The display control unit 20 performs display with emphasis on the unsteady block in the unsteady state (for example, the annotation A3 is displayed on the unsteady block in the unsteady state). An annotation A2 indicating the occurrence of abnormality is also displayed. Such a display makes it easier for the observer to focus on the problem, so that problems can be found immediately and the quality of the security can be improved.

(Second Embodiment)
In the present embodiment, an example of density calculation processing executed by the second calculation unit 16 in the image analysis system 10 will be described.

  FIG. 16 is a block diagram illustrating an example of the configuration of the second calculation unit 16 provided in the image analysis system 10.

  The second calculation unit 16 includes a first calculation unit 42, a calculation unit 43, a second prediction unit 44, and a density calculation unit 45.

  Some or all of the first calculation unit 42, the calculation unit 43, the second prediction unit 44, and the density calculation unit 45 may be realized by causing a processing device such as a CPU to execute a program, that is, by software. Alternatively, it may be realized by hardware such as an IC, or may be realized by using software and hardware together.

  The 1st calculation part 42 calculates each feature-value of the some partial area | region 50 extracted by the extraction part 14 (refer FIG. 1). The feature amount is a value indicating the feature of the partial region 50. The feature amount includes, for example, one obtained by discretizing the pixel values of the pixels constituting the partial area and arranging them one-dimensionally, or the difference between adjacent pixel values in the one-dimensional pixel values (ie, gradient). Use a normalized version. In addition, SIFT features (see D. Low “, Object recognition from local scale-invariant features,” Int. Conf. Comp. Vision, Vol. 2, pp. 1150-1157, 1999) are used as feature amounts. Also good. The SIFT feature is a histogram feature that is robust to minute changes.

  FIG. 17 is an explanatory diagram of an image 33, a partial area 50, and a label 51 (details will be described later).

  FIG. 17A is a schematic diagram illustrating an example of the image 33. As described in the first embodiment, the image 33 includes the object 30. FIG. 17B is a diagram illustrating an example of the partial region 50.

  The 1st calculation part 42 calculates each feature-value of the some partial area | region 50 extracted by the extraction part 14 (refer FIG. 1). Note that the greater the number of partial regions 50 that the extraction unit 14 extracts from the image 33, the more the second calculation unit 16 can learn a regression model that can calculate the density with high accuracy in the process described later.

  Returning to FIG. 16, the calculation unit 43 calculates the regression model and the representative label. FIG. 18 is a block diagram illustrating an example of the configuration of the calculation unit 43.

  The calculation unit 43 includes a search unit 43A, a voting unit 43B, a learning unit 43C, and a first prediction unit 43D. A part or all of the search unit 43A, the voting unit 43B, the learning unit 43C, and the first prediction unit 43D may be realized by causing a processing device such as a CPU to execute a program, that is, by software, You may implement | achieve by hardware, such as IC, and may implement | achieve combining software and hardware.

  The search unit 43A gives a label to each feature amount of the plurality of partial regions 50. The label represents a relative position between the object included in each partial region 50 and the first position in each partial region 50. Specifically, the search unit 43A first searches for an object included in each of the plurality of partial regions 50 extracted by the extraction unit 14. Then, the search unit 43A generates, for each of the partial areas 50, a vector representing a relative position between the first position in the partial area 50 and each of all the objects included in the partial area 50 as a label. Then, the search unit 43A gives the generated label to the feature amount of the corresponding partial region 50.

  The first position may be any predetermined position in the partial region 50. In the present embodiment, the first position will be described as being the center position in the partial region 50 (the center of the partial region 50).

  Returning to FIG. 17, FIGS. 17C and 17D are explanatory diagrams of the label 51. For example, the search unit 43A searches for an object included in each of the partial areas 50 shown in FIG. Then, the search unit 43A includes the center position P of the partial region 50 and each of all the objects included in the partial region 50 (three objects in the examples shown in FIGS. 17B and 17C). , Vectors L1, L2, and L3 indicating the relative positions are generated (see FIG. 17C). Then, the search unit 43A assigns a vector L, which is a set of these vectors L1, L2, and L3, to the feature amount of the partial region 50 as a label 51 (see FIG. 17D).

  Returning to FIG. 18, the voting unit 43 </ b> B calculates a histogram representing the distribution of the relative positions of the objects included in each partial region 50 for each of the plurality of partial regions 50.

  FIG. 19 is an explanatory diagram of the label 51 and the histogram 52. As illustrated in FIG. 19, the voting unit 43 </ b> B calculates a histogram 52 from the label 51.

  The histogram 52 is a set of bins arranged uniformly in the partial region 50. The size of the bin in the histogram 52 is determined by the relative position of the object included in the partial region 50. For example, the size of the bin at the position b in the partial region 50 is expressed by the following formula (7).

  B (b) = ΣN (b; oj, σ) (7)

  In equation (7), B (b) indicates the size of the bin at position b in the partial region 50. oj indicates the position of the object. In equation (7), N (b; oj, σ) is the value of the probability density function of the normal distribution of (center oj, variance σ) at position b.

  Returning to FIG. 18, next, the voting unit 43 </ b> B votes each of the histograms 52 calculated for each of the plurality of partial areas 50 in the parameter space. Accordingly, the voting unit 43B generates a voting histogram corresponding to each partial region 50 for each of the plurality of partial regions 50.

  FIG. 20 is an explanatory diagram of the voting histogram 54. The histogram 52 becomes a voting histogram 54 by voting to the parameter space 53. In FIG. 20, the parameter space is shown in a simplified manner in two dimensions.

  In the present embodiment, the parameter space is described as being a three-dimensional parameter space (x, y, s). (X, y) indicates a two-dimensional position (x, y) in the partial region. (S) indicates the size (s) of the object. The parameter space may be a multi-dimensional parameter space in which the posture of the object, the orientation of the object, and the like are added in addition to the above parameters.

  Returning to FIG. 18, the learning unit 43 </ b> C learns a regression model indicating the relationship between the feature amount of the partial region 50 and the relative position of the object included in the partial region 50. Specifically, the learning unit 43C divides the feature amount to which the label 51 is assigned corresponding to each of the plurality of partial regions 50 into a plurality of clusters so that the variation of the corresponding voting histogram becomes small. Learn the regression model.

  In this embodiment, a case where the regression model is one or a plurality of random trees will be described. The plurality of random trees is a random forest. In the present embodiment, a cluster means a leaf node that is a node at the end of a random tree.

  In the present embodiment, the learning unit 43C learns the regression model, which means that each node from the root node indicated by the random tree to the leaf node via the child node belongs to the leaf node. This means that the feature amount is determined. Note that this feature amount is the feature amount to which the label 51 is attached as described above.

  In the present embodiment, the learning unit 43C determines the division index of each node from the root node to the plurality of leaf nodes through the child nodes and the plurality of leaf nodes so that the variation of the voting histogram 54 is reduced. The regression model is learned by determining the feature quantities belonging to each.

  Note that the learning unit 43C preferably learns a plurality of regression models having different combinations of division indexes. In the present embodiment, the learning unit 43C learns a predetermined number (hereinafter referred to as “T”) of regression models by changing the combination of the division indices of each node.

  FIG. 21 is an explanatory diagram of the random tree 55.

  FIG. 21 shows a voting histogram 54 of the parameter space 53 simplified in two dimensions beside each node. In the example illustrated in FIG. 21, the voting histograms 54 </ b> A to 54 </ b> F are illustrated as the voting histograms 54 corresponding to the feature amounts of the plurality of partial regions 50. Hereinafter, the feature quantity of the partial region 50 may be referred to as a feature quantity v. As described above, a label is attached to the feature amount v.

  First, the learning unit 43C assigns all the feature values v to which labels are assigned, calculated by the first calculation unit 42 and the search unit 43A, to “S” which is the root node 55A.

  The learning unit 43C determines a division index for dividing “S” as the root node 55A into two “L” and “R” as two child nodes 55B. The division index is determined by the element vj of the feature quantity v and the threshold value tj.

  Specifically, the learning unit 43C determines the division index of the division source node so that the variation of the voting histogram in the division destination node (child node 55B or leaf node 55C) becomes small. A division | segmentation parameter | index is defined by the element vj of the feature-value v, and its threshold value tj.

  Specifically, the learning unit 43C sets the labeled feature quantity v satisfying the relationship of element vj <threshold value tj to “L” as the child node 55B (in the case of yes in FIG. 21), and the relationship of element vj <threshold value tj. A feature index v that does not satisfy is assigned to “R”, which is the child node 55B (in the case of no in FIG. 21), and a division index when temporarily allocated is determined (hereinafter referred to as a temporary allocation operation).

  At this time, the learning unit 43C determines the division index of the feature amount v so that the variation of the voting histogram 54 becomes small. The learning unit 43C determines a division index using, for example, the following equation (8).

G = Σ {H (l) −HL} ² + Σ {H (r) −HR} ² Formula (8)

  In Expression (8), H (l) indicates a voting histogram 54 that is divided from “S” that is the root node 55A to “L” that is the child node 55B. In Expression (8), H (r) represents the voting histogram 54 divided from “S” as the root node 55A to “R” as the child node 55B. In formula (8), HL is the average value of all H (l). HR is the average value of all H (r).

  Note that the equation used by the learning unit 43C to determine the division index is not limited to the equation (8).

  The learning unit 43C determines a division index for each node so that the variation of the voting histogram 54 is minimized, and repeats this temporary assignment operation from the root node 55A to the leaf node 55C via the child node 55B. That is, the learning unit 43C determines, for each node, a combination of the element vj and the threshold value tj as a division index so that the value of G in Equation (8) is the smallest, and determines the feature value v belonging to each node. Repeat splitting.

  Then, the learning unit 43C determines the node when the end condition is satisfied as the terminal leaf node 55C. The termination condition is, for example, at least one of a first condition, a second condition, and a third condition. The first condition is when the number of feature values v included in the node is smaller than a predetermined number. The second condition is when the depth of the tree structure of the random tree 55 is larger than a predetermined value. The third condition is when the value of the division index is smaller than a predetermined value.

  With this determination of the leaf node 55C, the learning unit 43C learns the feature quantity v belonging to the leaf node 55C.

  As described above, the learning unit 43C determines the division index of each node from the root node 55A to the leaf node 55C via the child node 55B, and the feature amount v belonging to the leaf node 55C, and randomly The tree 55 is learned. Further, the learning unit 43C learns a predetermined number T of random trees 55 by performing the above-described temporary allocation operation while changing the combination of the division indexes.

  Note that the number T of the random trees 55 learned by the learning unit 43C may be one, or may be any number greater than or equal to two. As the learning unit 43C learns a larger number of random trees 55 from the image 33, the image analysis system 10 can learn the random trees 55 whose density can be calculated with higher accuracy. That is, it is preferable that the learning unit 43 </ b> C learns a random forest that is a plurality of random trees 55.

FIG. 22 is an explanatory diagram of a plurality of learned random trees 55 (that is, random forests). Each of the random trees 55 ₁ to 55 _T has a different division index for each node. Thus, for example, assigned to the root node 55A, be all of the feature v granted the label 51 is the same, random tree 55 _1, and Random Trees 55 _T, the labels belonging to the leaf node 55C The attached feature quantity v may be different. In the example illustrated in FIG. 22, only the label 51 is illustrated in the leaf node 55 </ b> C, but actually, the feature quantity v to which the label 51 is assigned belongs to each leaf node 55 </ b> C.

  Returning to FIG. 18, the first prediction unit 43D predicts a representative label for each of the clusters divided by the learning unit 43C during learning. The first prediction unit 43D predicts a representative label from the label 51 assigned to one or a plurality of feature values v belonging to the cluster.

  As described above, in this embodiment, the cluster means the leaf node 55 </ b> C that is the terminal node of the random tree 55. For this reason, the first predicting unit 43D predicts the representative label of each leaf node 55C from the label 51 assigned to each feature quantity v belonging to the leaf node 55C.

  FIG. 23 is an explanatory diagram of representative label prediction. FIG. 23 illustrates one leaf node 55C as an example. First, the first prediction unit 43D reads the labels 51 assigned to all the feature values v belonging to the leaf node 55C. In the example illustrated in FIG. 23, the first prediction unit 43D reads labels 51C, 51D, 51E, 51G, and 51H. Next, the first prediction unit 43D generates an average histogram 56, which is an average of the voting histograms 54 (54C, 54D, 54E, 54G, 54H) corresponding to each of the labels 51C, 51D, 51E, 51G, 51H. calculate.

  Next, the first prediction unit 43D selects a voting histogram 54 that is close to the average histogram 56 among the plurality of voting histograms 54 (54C, 54D, 54E, 54G, 54H) belonging to the leaf node 55C. Note that the first prediction unit 43D preferably selects the voting histogram 54 closest to the average histogram 56 among the plurality of voting histograms 54 (54C, 54D, 54E, 54G, 54H) belonging to the leaf node 55C. In the example illustrated in FIG. 23, the first prediction unit 43D selects the voting histogram 54E that is closest to the average histogram 56. Then, the first prediction unit 43D predicts the label 51E, which is the label 51 corresponding to the voting histogram 54E, as the representative label of the leaf node 55C.

  The first prediction unit 43D performs the same process on all leaf nodes 55C in all the random trees 55 learned by the learning unit 43C, and predicts the representative label of each leaf node 55C.

  FIG. 24 is an explanatory diagram of the random tree 55 after representative label prediction.

As shown in FIG. 24, the first prediction unit 43D predicts the representative label for each leaf node 55C, whereby all the random trees 55 (random trees 55 _{1 to} 55 _T included in the random forest learned by the learning unit 43C. ), A representative label is predicted for all leaf nodes 55C of each random tree 55.

  Through the above processing, the calculation unit 43 calculates the regression model and the representative label.

  Returning to FIG. 16, next, the second prediction unit 44 acquires a random tree 55 as a regression model calculated by the calculation unit 43 and each representative label of the leaf node 55C. Then, the second prediction unit 44 substitutes the feature amount calculated from the partial region 50 for the variable of the random tree 55 acquired by the calculation unit 43. Thereby, the second prediction unit 44 predicts a representative label corresponding to each of the partial regions 50.

  Here, when there is one random tree 55 acquired by the calculation unit 43, the second prediction unit 44 predicts one representative label for each partial region 50 using one random tree 55. On the other hand, when there are a plurality of random trees 55 acquired by the calculation unit 43 (that is, in the case of a random forest), the second prediction unit 44 sets each of the plurality of random trees 55 for each partial region 50. A plurality of representative labels corresponding to 1 is obtained, and one of the plurality of representative labels is predicted as a representative label used for density measurement.

FIG. 25 is an explanatory diagram of representative label prediction performed by the second prediction unit 44. Assume that the random tree 55 and the representative label acquired by the calculation unit 43 are the random tree 55 (random trees 55 _{1 to} 55 _T ) and the representative label shown in FIG.

In this case, the second prediction unit 44 substitutes the feature amount of the partial region into each root node 55A of each random tree 55 (random trees 55 _{1 to} 55 _T ) included in the random forest. Then, the second prediction unit 44 changes from the root node 55A to the leaf node 55C via the child node 55B along the division index determined for each node of each random tree 55 (random trees 55 _{1 to} 55 _T ). Go down the tree structure until you reach it. Then, the representative label belonging to the reached leaf node 55C is read.

Accordingly, the second prediction unit 44 obtains a plurality of representative labels obtained for each random tree 55 (random trees 55 _{1 to} 55 _T ) as representative labels corresponding to the feature amounts of one partial region 50.

For example, it is assumed that assigned to the root node 55A feature amount v1 of a partial area 50 as a random tree 55 of _the variables. Then, following the child node _55B 1, 55B ₃ of the child node _55B 1 _{~55B 5,} and reaches the leaf node 55C ₁ of the leaf node _55C 1 _{~55C 7.} In this case, the feature quantity v1, representative label is determined by a random tree 55 ₁ becomes the label 51C _1.

Further, it is assumed that substituting the feature quantity v1 as a variable of the random tree 55 _T to the root node 55A. Then, following the child node 55B ₂ of the child node _55B 1 _{~55B 2,} and reaches the leaf node 55C ₃ of the leaf node _55C 1 _{~55C 4.} In this case, the feature quantity v1, representative label is determined by a random tree 55 _T becomes label _{51C 10.}

Next, the second prediction unit 44 predicts _{one of} the representative labels obtained for all the random trees 55 (random trees 55 _{1 to} 55 _T ) as a representative label used for density measurement. The second prediction unit 44 predicts a representative label used for density measurement in the same manner as the first prediction unit 43D.

That is, the second prediction unit 44 calculates an average histogram of the voting histograms 54 corresponding to the representative labels obtained for all the random trees 55 (random trees 55 _{1 to} 55 _T ). Then, the second prediction unit 44 selects a representative label corresponding to the voting histogram 54 closest to the average histogram among the plurality of representative labels obtained for all the random trees 55 (random trees 55 _{1 to} 55 _T ). Predicted as a representative label used for density measurement.

  Returning to FIG. 16, the density calculator 45 calculates the average density of the objects included in the image 33. The density calculation unit 45 performs the prediction of the second prediction unit 44 based on the relative position of the object indicated by the representative label corresponding to each of the partial regions predicted by the second prediction unit 44.

  The density calculation unit 45 includes a second calculation unit 45A, a third calculation unit 45B, and a fourth calculation unit 45C.

  The second calculation unit 45A calculates the density distribution of the object in each of the plurality of partial areas based on the relative position of the object indicated by the representative label corresponding to each of the plurality of partial areas. The second calculation unit 45A stores in advance the first position used by the calculation unit 43. This representative label is the above-described representative label used for density measurement.

  For example, the second calculation unit 45A calculates the density distribution Di (x) of the object in the partial region using the probability density function N () of the normal distribution.

  Di (x) = ΣN (x; lj, σ) (9)

  In formula (9), x represents an arbitrary position in the partial region 50. In equation (9), lj represents the predicted relative position of the object. In formula (9), σ represents dispersion.

  The third calculation unit 45B arranges the density distribution of the partial regions 50 at positions corresponding to the plurality of partial regions 50 in the image 33. The arrangement of the density distribution means that the density distribution of the corresponding partial region 50 is pasted at a position corresponding to each of the plurality of partial regions 50 in the image 33.

  Here, at least a part of the plurality of partial areas 50 extracted from the image 33 may overlap each other. For this reason, when the density distributions of the partial areas 50 extracted from the image 33 are arranged in the image 33, at least a part of the density distributions corresponding to the partial areas 50 may be overlapped.

  Therefore, the fourth calculation unit 45 </ b> C calculates the first average value of the density of the object for each pixel constituting the image 33 according to the overlapping frequency of the density distribution in the image 33.

  Then, the fourth calculation unit 45 </ b> C calculates, for each partial region 50 in the image 33, a further average value of the first average value of each pixel constituting the partial region 50. Specifically, the fourth calculation unit 45 </ b> C divides the addition value (the addition result) obtained by adding the first average values of the pixels constituting the partial area 50 by the number of pixels of the pixels constituting the partial area 50. The obtained value is used as the density of the object in the partial area 50.

  Thereby, the second calculation unit 16 calculates the density of the object for each of the plurality of partial regions 50 included in the image 33.

  Next, the procedure of density calculation processing executed by the second calculation unit 16 will be described. FIG. 26 is a flowchart illustrating a procedure of density calculation processing executed by the second calculation unit 16.

  First, the second calculation unit 16 calculates a feature amount for each of the plurality of partial regions 50 extracted from the image 33 by the extraction unit 14 (see FIG. 1) (step S200).

  Next, the calculation unit 43 calculates a random tree 55 as a regression model and a representative label (step S202) (details will be described later).

  Next, the second prediction unit 44 substitutes the calculated feature amount for the variable of the random tree 55 calculated by the calculation unit 43. Thereby, the second prediction unit 44 predicts a representative label corresponding to each of the partial regions 50 (step S204).

  Next, the second calculation unit 45A calculates the density distribution of the object in each of the plurality of partial regions 50 based on the relative position of the object indicated by the representative label (step S206).

  Next, the third calculation unit 45B arranges the density distribution of the corresponding partial region 50 at a position corresponding to each of the plurality of partial regions 50 in the image 33 (step S208). Next, the fourth calculation unit 45C calculates the density of the object for each partial region 50 included in the image 33 according to the frequency of density distribution overlap in the image 33 (step S210). Then, this routine ends.

  Next, calculation processing performed by the calculation unit 43 in step S202 of FIG. 26 will be described. FIG. 27 is a flowchart illustrating a procedure of calculation processing performed by the calculation unit 43.

  First, the search unit 43A of the calculation unit 43 assigns a label to each feature amount of the plurality of partial regions 50 calculated in step S200 (see FIG. 26) (step S300). The voting unit 43B calculates a histogram 52 from the label 51, and generates a voting histogram 54 by voting on the parameter space 53 (step S302).

  Next, the learning unit 43C learns a regression model indicating the relationship between the feature amount of the partial region 50 and the relative position of the object included in the partial region 50 (step S304). In the present embodiment, as described above, the random tree 55 is learned as a regression model.

  Next, the first prediction unit 43D predicts a representative label for each of the clusters (leaf nodes 55C) divided by the learning unit 43C during learning (step S306).

  Then, the calculation unit 43 outputs the random tree 55 as the learned regression model and the representative label of the cluster (leaf node 55C) to the second prediction unit 44, and this routine is terminated.

  As described above, in the second calculation unit 16 of the present embodiment, the search unit 43A searches for an object included in each of the plurality of partial regions 50 extracted from the image 33. In addition, the search unit 43A assigns the vector representing the relative position between a predetermined first position in the partial region 50 and each of all objects included in the partial region 50 to the feature amount of the partial region 50. . The learning unit 43C assigns the feature amount to which the label is assigned to each node to determine a division index for each node. Thereby, the learning unit 43C learns the regression model. The first prediction unit 43D predicts a representative label for each of the leaf nodes 55C in the regression model.

  The label is a vector indicating the relative position of the object and has a small data size. For this reason, it is possible to reduce the amount of data necessary for configuring the regression model. That is, by performing density calculation using the regression model of the present embodiment, the image analysis system 10 can perform density calculation of an object with low memory in addition to the effects of the first embodiment. .

  Further, the second calculation unit 16 learns a regression model without directly detecting an object from the image 33. For this reason, the second calculation unit 16 according to the present embodiment can perform the density calculation with high accuracy without reducing the measurement accuracy even if the objects included in the image 33 are small and overlap. You can learn the model.

  Therefore, in the image analysis system 10 of the present embodiment, the second calculation unit 16 performs the above-described processing described in the present embodiment, so that the density of the object is added to the effect of the first embodiment. Calculations can be performed with high accuracy and low memory.

<Modification>
In the second embodiment, the case where a random forest is used as the regression model has been described. However, the regression model learned by the second calculator 16 is not limited to a random forest. For example, the second calculator 16 may use a nearest neighbor classifier as the regression model.

  FIG. 28 is an explanatory diagram of learning using the nearest neighbor classifier. The learning unit 43C (see FIG. 18) divides the feature amount v assigned with the label 51 corresponding to each of the plurality of partial regions 50 into a plurality of clusters 560 so that the variation of the corresponding voting histogram becomes small. . By this division, the learning unit 43C learns the learning model.

  Specifically, the learning unit 43C uses the vector quantization method such as the k-average method to calculate all the clustered feature quantities v corresponding to each of the plurality of partial regions 50, using k clusters. Divide into 56.

  Specifically, the learning unit 43 </ b> C randomly assigns clusters 56 to an arbitrary label 51 and calculates an average value for each cluster 56. Next, the learning unit 43 </ b> C calculates the distance between each label 51 and the average value for each cluster 56. Then, the learning unit 43C reassigns each label 51 to the cluster 56 having the closest average value. If the assignment of all labels 51 to the clusters 56 has not changed in this series of processing, the processing ends. Otherwise, repeat the process.

  As a result, the feature quantity v assigned with the label 51 is divided into clusters 56 for each group of similar feature quantities v.

  The first prediction unit 43D calculates the average value of the labels 51 assigned to the feature amount v belonging to the cluster 56. Then, the first prediction unit 43D predicts, as a representative label, the label 51 that belongs to each cluster 56 and that is closest to the calculated average value among the labels 51 that are assigned to the feature quantity v.

  In addition, the second calculation unit 16 acquires the nearest neighbor classifier as a regression model. In this case, the second prediction unit 44 matches the feature amount calculated from the partial region with the predicted representative vector of each cluster by using the nearest neighbor classifier, and the feature amount calculated from the partial region has the longest distance. Select the nearest representative vector. Then, the second prediction unit 44, from the set of labels 51 assigned to the feature quantities belonging to the cluster to which the selected representative vector belongs, in the same manner as the first prediction unit 43D, the representative label corresponding to each of the partial areas. Just predict.

  FIG. 29 is a block diagram illustrating an example of a hardware configuration of the image analysis system 10 according to the embodiment and the modification. As illustrated in FIG. 29, the image analysis system 10 according to the above-described embodiment and modification includes a CPU 902, a RAM 906, a ROM 904 that stores programs, an HDD 908, an I / F 910 that is an interface with the HDD 908, an image, and the like. An I / F 912 that is an input interface and a bus 922 are provided, and a hardware configuration using a normal computer is employed. Note that the CPU 902, ROM 904, RAM 906, I / F 910, and I / F 912 are connected to each other via a bus 922.

  In the image analysis system 10 of the above embodiment and the modification, the CPU 902 reads out a program from the ROM 904 onto the RAM 906 and executes the program, whereby the above-described units are realized on the computer.

  It should be noted that a program for executing each of the processes executed by the image analysis system 10 of the above embodiment may be stored in the HDD 908. In addition, a program for executing each of the processes executed by the image analysis system 10 according to the embodiment may be provided by being incorporated in the ROM 904 in advance.

  A program for executing the above-described processing executed by the image analysis system 10 of the above-described embodiment is a CD-ROM, CD-R, memory card, DVD (installable format or executable format file). It may be stored in a computer-readable storage medium such as a digital versatile disk (FD) or a flexible disk (FD) and provided as a computer program product. Further, a program for executing the above-described processing executed by the image analysis system 10 of the above-described embodiment is provided by being stored on a computer connected to a network such as the Internet and downloaded via the network. May be. In addition, a program for executing the above-described processing executed by the image analysis system 10 of the above-described embodiment may be provided or distributed via a network such as the Internet.

  In addition, although embodiment and the modification of this invention were demonstrated above, these embodiment and the modification are shown as an example and are not intending limiting the range of invention. . The novel embodiment can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the spirit of the invention. This embodiment and its modifications are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

  For example, as long as each step in the flowchart of the above embodiment is not contrary to its nature, the execution order may be changed, a plurality of steps may be performed simultaneously, or may be performed in a different order for each execution.

DESCRIPTION OF SYMBOLS 10 Image analysis system 12 Image analysis apparatus 14 Extraction part 15 1st calculation part 16 2nd calculation part 17 Dividing part 18 3rd calculation part 19 Specification part 20 Display control part 21A Display part

Claims (13)

An extraction unit for extracting a plurality of partial regions from the image;
A first calculation unit for calculating an optical flow of the partial area;
A second calculation unit for calculating the density of an object included in the partial region;
A dividing unit for dividing the image into a plurality of blocks;
A third calculator that calculates, for each of the plurality of blocks, a statistic that combines the density and the amount of movement of the object indicated by the optical flow of each of the partial regions included in the block;
A identifying unit that identifies the block having the statistic equal to or greater than a threshold value as a non-stationary block in a non-stationary state among the plurality of blocks;
An image analysis apparatus comprising: The third calculator is
For each of the plurality of blocks, the sum of the multiplication values of the density and the amount of movement of each of the partial areas included in the block is calculated as the statistic indicating the amount of movement of the object,
The specific part is:
Among the plurality of blocks, the block whose statistic is greater than or equal to a first threshold is identified as the non-stationary block whose statistic is greater than or equal to the threshold.
The image analysis apparatus according to claim 1. The third calculator is
For each of the plurality of blocks, the sum of multiplication values of the density of each of the partial areas included in the block and the square of the moving amount is calculated as the statistic indicating the kinetic energy of the object,
The specific part is:
Among the plurality of blocks, the block whose statistic is greater than or equal to a second threshold is identified as the non-stationary block whose statistic is greater than or equal to the threshold.
The image analysis apparatus according to claim 1. The third calculator is
For each of the plurality of blocks, the sum of the multiplication values of the density of each of the partial areas included in the block and the reciprocal of the moving amount is calculated as the statistic indicating the staying degree of the object,
The specific part is:
Among the plurality of blocks, the block whose statistic is equal to or greater than a third threshold is identified as the non-stationary block whose statistic is equal to or greater than the threshold.
The image analysis apparatus according to claim 1. The third calculator is
For each of the plurality of blocks, for each of the partial areas included in the block, the density of the partial areas, the reciprocal of the amount of movement of the partial areas, and the other partial areas included in the same block The sum of the product of the density and the divided value divided by the distance between the partial area and the other partial area is calculated as the statistic indicating the potential energy of the object,
The specific part is:
Among the plurality of blocks, the block whose statistic is a fourth threshold or more is identified as the non-stationary block whose statistic is the threshold or more.
The image analysis apparatus according to claim 1. The non-steady state, the included in the block is, presence or absence of an object of congestion, presence or absence of an object in the residence, indicating the presence or absence of a set of objects, at least one of any one of claims 1 to 5 The image analysis apparatus described in 1. The image analysis apparatus according to any one of claims 1 to 5, further comprising a display control unit that performs control to display a display image in which the unsteady block is emphasized in the image on a display unit. In the image, the unsteady block, a display image obtained by superimposing the information indicating the non-steady state, with a display control unit that performs control to display on the display unit, any one of claims 1 to 5 The image analysis apparatus described in 1. The image analysis apparatus according to any one of claims 1 to 5, further comprising a display control unit configured to perform control for displaying a display image indicating the unsteady block on a display unit. For each of the plurality of blocks included in the image, a calculation unit that calculates a statistic that combines the density and the amount of movement of the object indicated by the optical flow of each of the partial regions included in the block;
A identifying unit that identifies the block having the statistic equal to or greater than a threshold value as a non-stationary block in a non-stationary state among the plurality of blocks;
An image analysis apparatus comprising: Extracting a plurality of partial regions from the image;
Calculating an optical flow of the partial region;
Calculating a density of an object included in the partial region;
Dividing the image into a plurality of blocks;
For each of the plurality of blocks, calculating a statistic that combines the density and the amount of movement of the object indicated by the optical flow of each of the partial regions included in the block;
Identifying the block whose statistic is greater than or equal to a threshold value among a plurality of the blocks as a non-stationary block in a non-stationary state;
An image analysis method including: Extracting a plurality of partial regions from the image;
Calculating an optical flow of the partial region;
Calculating a density of an object included in the partial region;
Dividing the image into a plurality of blocks;
For each of the plurality of blocks, calculating a statistic that combines the density and the amount of movement of the object indicated by the optical flow of each of the partial regions included in the block;
Identifying the block whose statistic is greater than or equal to a threshold value among a plurality of the blocks as a non-stationary block in a non-stationary state;
Image analysis program that causes a computer to execute. An extraction unit for extracting a plurality of partial regions from the image;
A first calculation unit for calculating an optical flow of the partial area;
A second calculation unit for calculating the density of an object included in the partial region;
A dividing unit for dividing the image into a plurality of blocks;
A third calculator that calculates, for each of the plurality of blocks, a statistic that combines the density and the amount of movement of the object indicated by the optical flow of each of the partial regions included in the block;
A identifying unit that identifies the block having the statistic equal to or greater than a threshold value as a non-stationary block in a non-stationary state among the plurality of blocks;
A display unit for displaying a display image in which the unsteady block is highlighted in the image;
An image analysis system comprising:

Images