JP2012175207A - Image processing apparatus, image processing method, and computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus which when generating a cut-out image of a display frame in a taken image, is capable of reducing stress caused by variation in the display frame, onto a user observing the cut-out image.SOLUTION: An image processing apparatus 100 comprises: an image generating part 101 which generates an image by photographing; a person detection part 102 which detects an area surrounding a moving object in the image as a detection frame; a display frame center position determination part 107 which determines a display frame corresponding to the detection frame so as to avoid positional fluctuation, on the basis of the history of the position of the detection frame; and a cut-out image generating part 110 which generates the cut-out image of the display frame determined by the display frame center position determination part 107.

Description

  The present invention relates to an image processing apparatus, an image processing method, and a computer program that perform image processing on a captured image, and in particular, an image processing apparatus, an image processing method, and an image processing apparatus that cut out a display frame in a captured image and generate a cut-out image. And a computer program.

  Conventionally, in a surveillance system in which an image is generated by shooting with a monitoring camera and the image is displayed on the monitor, in addition to a mode (all display mode) for displaying the entire image (taken image) obtained by shooting with the camera A mode (cutout display mode) in which the partial area is cut out and displayed is used.

  In addition, in the cut-out display mode, in particular, there is a mode in which a moving body is detected from a captured image by image processing, and the cut-out area is automatically moved so as to track the moving body. In such a cut-out display mode, a frame that surrounds a moving area in the captured image is used as a detection frame, a display frame that includes the detection frame and is larger than the detection frame is set, and an image in the display frame is displayed. The display frame can be set so that its center coincides with the center of the detection frame.

  However, simply setting the display frame so that its center always coincides with the center of the detection frame causes the following problems.

  First, if the moving object is a person, even if the person is not moving, it is affected by the shadow and noise of the person, the person turns around, stretches his hand, crouches, The detection frame fluctuates and the center position thereof also moves. In addition, even when the detection accuracy is not sufficiently high, the detection frame similarly varies.

  The variation in the detection frame due to such a cause is frequent and high-speed variation (hereinafter, such variation is referred to as “shake”) within a very small range. Even for such a fluctuation of the detection frame, if the display frame is set so that the center of the detection frame coincides with the center of the detection frame, the display frame also shakes. As a result, a cut-out image that appears to have shaken is displayed on the monitor, which causes stress on the user who observes the image.

  To solve this problem, there is a technique for reducing the fluctuation of the display frame by performing a smoothing process on the variation of the position of the detection frame by adopting, for example, an average position with the previous frame as the position of the detection frame. (For example, see Patent Document 1).

JP-A-5-68195

  However, even with the above-described prior art, there is a possibility that the width of the shake is reduced, but the shake itself is not eliminated, and it is not a fundamental solution for the shake. In addition, when a person is actually moving, if the figure moves back and forth within a certain relatively narrow range, the display frame will frequently move up and down, left and right, The user who observes this is stressed.

  The present invention has been made in view of the above problems, and in an image processing apparatus that generates a cutout image obtained by cutting out a display frame in a captured image, a change in the display frame that is given to a user who is observing the cutout image. An object of the present invention is to provide an image processing apparatus capable of reducing the stress caused by the above.

  In order to solve the above-described conventional problems, an image processing apparatus of the present invention includes an image generation unit that generates an image by photographing, a detection frame generation unit that detects a region surrounding the moving body in the image as a detection frame, Based on the history of the position of the detection frame, a display frame determining unit that determines a display frame so that the position does not move, and a cut-out image that generates a cut-out image of the display frame determined by the display frame determining unit And a generator.

  With this configuration, based on the history of the position of the detection frame, the display frame is determined so that the position is not shaken, and a cut-out image is generated with the display frame, so the display given to the user who is observing the cut-out image The stress due to the fluctuation of the frame can be reduced. The display frame may be determined so as to completely include the detection frame, may be determined so as to include at least a part of the region surrounded by the detection frame, and does not include the region surrounded by the detection frame. May be determined as follows. In addition, determining the display frame means determining at least one of the position and size of the display frame.

  In the image processing apparatus, the display frame determination unit may move the display mode according to the position history of the detection frame, the moving display mode in which the display frame is moved following the detection frame, or the detection. The display frame may be set to any one of the fixed display modes for fixing the display frame regardless of the change of the frame, and the position of the display frame may be determined according to the set display mode.

  With this configuration, by displaying the cut image in the fixed display mode, the cut image is displayed in the display frame that is not affected by the shake of the detection frame even if the detection frame is shaken.

  In the image processing apparatus, the display frame determination unit estimates whether the moving body is moving, stopped, or stays based on the history of the position of the detection frame, and the moving body When it is estimated that the moving body is moving, the display mode is set to the moving display mode, and when it is estimated that the moving body is stopped or staying, the display mode is set to the fixed display mode. It's okay.

  With this configuration, when it is estimated that the moving body is stopped or staying, the display frame does not follow the minute fluctuation of the detection frame, so the fluctuation of the detection frame is not accompanied by the movement of the substantial position of the moving body. Can prevent the display frame from being shaken.

  In the image processing apparatus, the display frame determination unit determines the display frame so that the display frame changes smoothly when the display mode shifts from the fixed display mode to the moving display mode. It's okay.

  When switching from the fixed display mode to the moving display mode immediately, the position and size of the display frame in the fixed display mode may change greatly. In this case, the user who is observing the cutout image switches the display mode. It may be difficult to understand the correlation between the cropped images before and after, but with this configuration, the display frame changes smoothly when switching from the fixed display mode to the moving display mode. It becomes easy to recognize the correlation.

  In the image processing apparatus, the display frame determination unit may determine the size of the display frame based on the size of the detection frame and the history of the position of the detection frame.

  With this configuration, when the detection frame swings, if the display frame is made larger than the detection frame, the display frame shake can be reduced and the display frame position can be reduced. Even when the detection frame is not allowed to follow, a cut-out image including an image in the detection frame can be generated with the display frame fixed.

  In the image processing apparatus, the display frame determination unit estimates whether the moving body is moving, stopped, or stays based on the history of the position of the detection frame, and the moving body The size of the display frame may be larger when it is estimated that the moving body is moving than when it is estimated that the moving body is moving.

  With this configuration, when the moving body is estimated to be stopped or staying, the display frame can be enlarged by reducing the display frame, or the display frame can be fixed.

  In the above image processing device, the display frame determination unit may determine the position of the display frame and the position of the display frame so as to include a range in which the detection frame moves within a predetermined period based on the history of the position of the detection frame. The size may be determined and fixed.

  With this configuration, when the moving body moves around a relatively narrow range in a complicated manner, the display frame is moved to a complicated position so that the position of the display frame can be moved. A cut-out image of the moving body can be obtained without doing so.

  In the image processing apparatus, the display frame determination unit may release the fixing of the display frame when the detection frame reaches the periphery of the fixed display frame.

  With this configuration, it is possible to avoid the inconvenience that the moving body does not appear in the cut-out image without following the movement of the moving body by fixing the display frame.

  In the image processing apparatus, the display frame determination unit obtains an average position of the detection frames within a predetermined period, and calculates a distance difference between the position of each detection frame and the average position within the predetermined period. When the sum of the distance differences is larger than a predetermined threshold, it is estimated that the moving body is moving. When the sum of the distance differences is smaller than the predetermined threshold, the moving body is stopped. Or it may be estimated that it is stagnant.

  With this configuration, it can be suitably estimated whether the moving body is moving, stopped, or staying.

  In the image processing apparatus, the image may be distorted, and the display frame determination unit may weight the distance difference according to the distortion of each detection frame.

  With this configuration, it is possible to reduce the estimation that the moving object is stopped or stayed when the moving object is actually moving due to the distortion in the image.

  The image processing apparatus may further include a moving body direction detecting unit that detects the direction of the moving body based on the image of the moving body, and the display frame determining unit may detect the position of the detection frame. When the moving direction of the detection frame based on the history is different from the direction of the moving body, it may be estimated that the moving body is stopped or staying.

  With this configuration, it can be suitably estimated whether the moving body is moving, stopped, or staying.

  In the image processing apparatus, the display frame determination unit refers to a constraint condition regarding a change in the position of the display frame so that the position of the display frame cannot be shaken, so that the constraint condition is satisfied. The display frame may be determined.

  In the case where the display frame determining unit determines the display frame according to a predetermined algorithm for determining the display frame so that the position does not move, even if the position of the display frame may actually move, this configuration Since the display frame is determined after imposing the constraint condition, the shake of the display frame can be reliably reduced or prevented.

  In the image processing apparatus, the constraint condition may be a condition that a moving speed of the display frame does not exceed a predetermined upper limit.

  With this configuration, it is ensured that the display frame does not move at a predetermined speed or higher, and the display frame can be reliably reduced or prevented from shaking.

  The constraint condition may be a condition for limiting the position of the display frame on a predetermined route.

  In a situation where an image generation unit is photographed and a person is detected as a moving body, if the movement route of the person is limited at that place due to an obstacle or the like, the position of the display frame May be changed to follow the movement route. Therefore, with this configuration, by restricting the position of the display frame to a predetermined route, the user who observes the cut-out image does not shake the display frame due to a change in the detection frame in a direction different from the route. Can reduce the stress caused by fluctuations in the display frame.

  Another aspect of the present invention is an image processing method. The image processing method includes an image generation step of generating an image by photographing, and a detection frame generation step of detecting a region surrounding the moving body in the image as a detection frame. And a display frame determining step for determining a display frame corresponding to the detection frame so that the position does not move based on the history of the position of the detection frame, and the display frame determined in the display frame determining step. A cut image generation step for generating a cut image.

  Also with this configuration, based on the history of the position of the detection frame, the display frame is determined so that the position is not shaken, and a cutout image is generated with the display frame, so it is given to the user who is observing the cutout image. Stress due to fluctuations in the display frame can be reduced.

  Yet another embodiment of the present invention is a computer program that causes a computer to execute the above-described image processing method.

  Since the present invention determines a display frame so that the position does not fluctuate based on the history of the position of the detection frame and generates a cut-out image with the display frame, the display given to the user who is observing the cut-out image The stress due to the fluctuation of the frame can be reduced.

1 is a block diagram showing a configuration of an image processing apparatus according to a first embodiment of the present invention. (A) The figure explaining the sum total (when the person is moving) of the distance difference from the average position from the time ts to the time t and the average position of each time in the 1st Embodiment of this invention. b) A diagram for explaining the sum of distance differences between the average position from time ts to time t and the average position at each time (when a person is stopped or staying) in the first embodiment of the present invention. (A) The figure which shows the relationship between the detection frame at the time of the fixed display mode start in the 1st Embodiment of this invention, and a display frame (b) The display frame while the person in the 1st Embodiment of this invention stagnates (C) The figure which shows the relationship between the detection frame and display frame of the transition period in the 1st Embodiment of this invention (d) This invention The figure which shows the relationship between the detection frame at the time of the transition period end in 1st Embodiment of this, and a display frame (A) The figure which shows the relationship between the detection frame and display frame in the 1st Embodiment of this invention (b) The figure which shows the boundary line in the 1st Embodiment of this invention (c) The detection frame is a boundary line A diagram showing the condition The flowchart which shows operation | movement of the image processing apparatus in the 1st Embodiment of this invention. The block diagram which shows the structure of the image processing apparatus in the 2nd Embodiment of this invention. (A) The figure which shows the case where the matching score of the rightward template in the 2nd Embodiment of this invention is high (b) The figure which shows the case where the matching score of the leftward template in the 2nd Embodiment of this invention is high The block diagram which shows the structure of the image processing apparatus in the 3rd Embodiment of this invention. The block diagram which shows the structure of the image processing apparatus in the 4th Embodiment of this invention. (A) The figure explaining the process which calculates | requires an average route from the locus | trajectory by the movement of the person in multiple times in the 4th Embodiment of this invention (b) The average movement in the 4th Embodiment of this invention The figure which shows the display frame restricted on the route The figure which shows the modification of the 4th Embodiment of this invention The block diagram which shows the structure of the image processing apparatus in the 5th Embodiment of this invention. (A) The figure which shows the case where the person in the 5th Embodiment of this invention moves to the circumferential direction of an omnidirectional image (b) The person in the 5th Embodiment of this invention is the radial direction of an omnidirectional image (C) The figure which shows the case where the person in the 5th Embodiment of this invention moves a comparatively short distance in the radial direction of an omnidirectional image

  Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.

(First embodiment)
FIG. 1 is a block diagram showing the configuration of the image processing apparatus according to the first embodiment of the present invention. As shown in FIG. 1, the image processing apparatus 100 includes an image generation unit 101, a person detection unit 102, a movement history holding unit 103, an average position calculation unit 104, a distance difference sum calculation unit 105, a display mode determination unit 106, a display frame. A center position determination unit 107, a display frame position determination parameter holding unit 108, a display frame position holding unit 109, and a cut-out image generation unit 110 are provided. The configuration including the average position calculation unit 104, the distance difference total calculation unit 105, the display mode determination unit 106, the display frame center position determination unit 107, and the display frame position determination parameter holding unit 108 is included in the display frame determination unit of the present invention. Equivalent to.

  The image generation unit 101 generates an image (hereinafter referred to as “captured image”) by capturing with a camera or the like. When the image processing apparatus 100 is applied to a monitoring system, the camera is installed at a monitoring target location and images the monitoring target location. The captured image is output to the person detection unit 102.

  The person detection unit 102 detects a person from the captured image and outputs the detection result. The person detection unit 102 compares the previous frame with the current frame, and detects a part where a difference in pixel values is equal to or greater than a predetermined threshold in the captured image as a part where a person is projected. The person detection unit 102 generates a rectangular area including a portion where the difference is equal to or greater than a threshold value as a detection frame, and outputs information on the generated detection frame (center coordinates, height and width of the detection frame) as a detection result. The person detection unit 102 corresponds to the detection frame generation unit of the present invention.

  The movement history holding unit 103 holds information on the detection frame output from the person detection unit 102 in association with the detection time. The average position calculation unit 104 refers to the position of the detection frame at each time held in the movement history holding unit 103, sets the current time as t, and sets the current time from time ts that is a time s before the current time. The average position of person positions (that is, the position of the center coordinates of the detection frame) up to t (the number of frames is X) is calculated.

  The distance difference sum calculation unit 105 obtains the sum of distance differences between the average position calculated by the average position calculation unit 104 and the person position at each time from time ts to the current time t. The sum of the distance differences increases when the person moves in one direction, and decreases when the person stops or stays at one place.

  FIG. 2 is a diagram for explaining the sum of the average position and the distance difference from time ts to time t. FIG. 2A is a diagram illustrating a difference in distance between the average position when the person is moving and the average position at each time. In the example of FIG. 2A, since the person is moving, the sum of the distance differences from time ts to time t becomes large. On the other hand, FIG.2 (b) is a figure which shows the distance difference from the average position in case a person has stopped or stayed, and the average position of each time. In this case, the distance difference from the average position from time ts to time t is small, and the sum is also small.

  The display mode determination unit 106 sets the display mode based on the detection frame position history. Specifically, the display mode determination unit 106 sets the “moving display mode” as the display mode when the sum of the distance differences calculated by the distance sum calculation unit 105 exceeds a predetermined threshold σ, and exceeds the threshold σ. If not, the “fixed display mode” is set as the display mode. The display mode determination unit 106 outputs the set display mode to the display frame center position determination unit 107.

  The display frame center position determination unit 107 determines the position of the display frame corresponding to the position of the detection frame. The display frame center position determination unit 107 refers to the display mode output from the display mode determination unit 106, and when the display mode is the “moving display mode”, the display frame center position determination unit 107 follows the position of the detection frame. Determine the position. Specifically, the position of the display frame is determined so that the center of the display frame matches the center of the detection frame.

  In this case, however, the position of the display frame is determined with reference to the display frame position determination parameter held in the display frame position determination parameter holding unit 108. The display frame position determination parameter holding unit 108 holds a restriction condition for moving the display frame as a display frame position determination parameter. In the present embodiment, the maximum speed of the display frame is held as the display frame position determination parameter.

  The display frame center position determination unit 107 refers to the display frame position of the previous frame held in the display frame position holding unit 109, and the display frame speed (the amount of movement of the display frame from the previous frame) is the maximum. If the speed is not exceeded, the position of the display frame is determined so that the center of the display frame matches the center of the detection frame of the current frame input from the movement history holding unit 103, and the maximum speed is exceeded. Determines the position of the display frame as long as the maximum speed is not exceeded.

  When the display mode is “fixed display mode”, the display frame center position determination unit 107 sets the display frame of the previous frame held in the display frame position holding unit 109 as the display frame of the current frame. That is, the display frame is fixed regardless of the position of the detection frame.

  In the fixed display mode, when the sum of the distance differences from the average position during the most recent time s in a certain frame exceeds the threshold value, the fixed display mode is switched to the moving display mode. However, in this case, in the moving display mode, the display frame is not made to immediately follow the detection frame, but gradually follows the detection frame.

  FIG. 3 is a diagram for explaining switching from the fixed display mode to the moving display mode. As shown in FIG. 3A, after the display frame is fixed so that the center position of the display frame coincides with the center of the detection frame and the fixed display mode is started, the person is moved as shown in FIG. If you move to the edge of the display frame while staying, and start moving from there, the center of the detection frame is shifted from the center of the display frame. When the position of the display frame is determined so as to match, the position of the display frame suddenly changes from the position of the fixed display frame, and it becomes difficult for the user to grasp the correlation between the cut-out images before and after switching the display mode.

Therefore, the center position (x, y) of the display frame is determined in the following manner during the transition period when shifting from the fixed display mode to the moving display mode. That is, the center coordinates of the display frame of a fixed display mode and (x _f, y _f), the center coordinates of the detection frame at time t of the moving display mode and (x _t, y _t), the movement from the fixed display mode the time to switch the display mode and t _a, the transient period of the switching Ri to the mobile display mode and n from the fixed display mode, the time t _a at the previous and the time t _a + n later, the center position of the display frame (x, y ), respectively (x _f, y _f), and (x _t, y _t), is from time t _a to time t _a + n, the center position of the display frame the formula (1) (x, y) .

FIG. 3 (c), at time t _a + n from time t _a, and shows a state where the center of the display frame of the detection frame gradually approaches, Fig. 3 (d), at time t _a + n, It shows a state where the center of the detection frame has caught up with the center of the display frame. After FIG. 3D, the normal moving display mode is set, and the position of the display frame is determined so that the center of the display frame coincides with the center of the detection frame.

  As described above, when the sum of the distance differences from the average position during the most recent time s does not exceed the threshold value, the display mode is set to “fixed display mode” and the display frame is fixed. . However, if the person slowly moves from the state where the person has stopped or stayed at the predetermined place, the total position of the distance does not exceed the threshold (that is, the fixed display mode is maintained), and the person position is fixed. May fall out of the display frame.

  Therefore, the display frame center position determination unit 107 changes the display mode to the moving display mode when the detection frame exceeds the boundary line. FIG. 4A is a diagram illustrating the relationship between the display frame and the detection frame. As shown in FIG. 4A, the display frame completely includes the detection frame, and the center of the detection frame and the display frame coincides when the fixed display mode is started. FIG. 4B is a diagram showing boundary lines. The boundary line is set as a frame that is slightly smaller than the display frame at the position of a fixed pixel (or a fixed ratio) inside the display frame.

  FIG. 4C is a diagram illustrating a state in which the detection frame is on the boundary line. As shown in FIG. 4C, when the detection frame is on the boundary line, the display frame center position determination unit 107 changes the display mode from the fixed display mode to the moving display mode. Also in this case, as described above, the center of the detection frame and the center of the display frame are not immediately matched, but the display frame gradually follows the detection frame over the transition period n.

  Note that the display frame center position determination unit 107 may change the display mode from the fixed display mode to the moving display mode when the center of the detection frame exceeds the boundary line, and the relationship between the detection frame and the boundary line. The display mode may be changed according to other criteria.

  Returning to FIG. 1, the cut-out image generation unit 110 cuts out an image in the display frame, generates a cut-out image, and displays it. When the captured image is an omnidirectional image captured using a fisheye lens, the cut-out image generation unit 110 performs distortion correction on the display frame to generate a cut-out image. The size of the cut-out image may be fixed, or the cut-out image generation unit 110 may set the size of the cut-out image so as to be proportional to the size of the person detection frame.

  FIG. 5 is a flowchart illustrating the operation of the image processing apparatus 100 according to the first embodiment. When the image generation unit 101 of the image processing apparatus 100 generates a captured image of a certain frame (step S501), the person detection unit 102 detects a person from the captured image (step S502) and generates a detection frame for the frame. Then, the information of the detection frame of the frame is stored in the movement history holding unit 103 (step S504).

  Next, it is determined whether or not the display mode is the moving display mode and is in a transition period of change from the fixed display mode (hereinafter referred to as “moving display mode (transition period)”) (step S505). ). When the display mode is not the movement display mode (transition period) (NO in step S505), the average position calculation unit 104 and the distance difference total calculation unit 105 respectively store the movement history of the past detection frames from the movement history holding unit 103. Reading (step S506), the average position calculation unit 104 calculates the average position (step S507), and the distance sum calculation unit 105 calculates the sum of distance differences between the average position and the person positions for the past X frames. (Step S508).

  The display mode determination unit 106 determines whether the sum of the distance differences is larger than the threshold σ (step S509). If the sum of the distance differences does not exceed the threshold σ (NO in step S509), the display mode is determined. The fixed display mode is set (step S510). Then, it is determined whether the detection frame is on the boundary line (step S511). If not (NO in step S511), the center of the display frame of the current frame and the center of the display frame of the previous frame are determined. Are determined so as to match (step S512), a cut-out image is generated and displayed, and the process returns to step S501 for the next frame.

  If the sum of the distance differences exceeds the threshold value σ in step S509 (YES in step S509), the display mode determination unit 106 sets the display mode to the moving display mode. It is determined whether or not the display mode is a fixed display mode (step S513). When the current display mode is the fixed display mode (YES in step S513), that is, when switching from the fixed display mode to the moving display mode, the display mode is set to the moving display mode (transition period) (step S514), The position of the display frame is determined by the transient processing described in (Step S515), and a cut-out image is generated and displayed.

  Then, it is determined whether or not the transition period has ended. If the transition period has not ended (NO in step S516), the display mode remains unchanged, that is, the display mode remains in the moving display mode (transition period). Return to step S501. On the other hand, when the transition period ends (YES in step S516), the display mode is set to the normal moving display mode (step S517), and the process returns to step S501 for the next frame.

  If the sum of the distances exceeds the threshold σ (YES in step S509) and the current display mode is not the fixed display mode (NO in step S513), the display mode determination unit 107 sets the display mode. The normal moving display mode is set (step S518), the display frame is set so that the center of the display frame coincides with the center of the detection frame of the current frame (step S519), and a cut-out image is generated and displayed. Return to step S501 for the next frame.

  If the display mode is set to the moving display mode (transition period) in step S514, the display mode is changed to the movement display mode (transition period) in step S505 after returning to step S501 for the next frame. It is determined that there is (YES in step S505). In this case, the display frame is immediately set by transient processing (step S515).

  In addition, when the display mode is changed to the fixed display mode in step S510, and the detection frame is placed on the boundary line due to the slow movement of the person (step S511), the display mode is the moving display mode (transition period). (Step S514), and the display frame is set in the transient process until the transition period ends.

  As described above, according to the image processing apparatus 100 of the present embodiment, when a person is stopped or staying at a certain place, even if the detection frame is shaken, the display mode is set to the fixed display mode. Therefore, the display frame does not follow the display frame. Therefore, it is possible to prevent the clipped image from being an image that causes camera shake, and no burden is imposed on the user who observes the clipped image.

  Also, when the display mode is switched from the fixed display mode to the moving display mode, the center of the display frame is gradually set by providing a certain transition period instead of immediately following the position of the display frame completely. Since the transient process is performed so as to approach the center of the detection frame, it is possible to avoid a situation in which the user cannot grasp the correlation between the cut-out images before and after switching due to the switching of the display mode.

  Further, when determining whether to set the fixed display mode or the moving display mode depending on whether the sum of the distance differences between the average position and the person position for a predetermined number of frames exceeds a predetermined threshold, If it moves slowly, the position of the person may deviate from the display frame in a state where the sum of the distance differences does not exceed the threshold. However, according to the image processing apparatus 100 of the present embodiment, even in the fixed display mode, Since the shift to the moving display mode occurs when the detection frame is on the boundary line, such a situation can be avoided.

(Second Embodiment)
FIG. 6 is a block diagram illustrating a configuration of the image processing apparatus 200 according to the second embodiment. In the image processing apparatus 200 of FIG. 6, the same components as those of the image processing apparatus 100 of the first embodiment are denoted by the same reference numerals, and the description thereof is omitted. In addition to the configuration of the image processing apparatus 100 according to the first embodiment, the image processing apparatus 200 according to the present embodiment further includes a person orientation detection unit 211 and a direction comparison unit 212. The image processing apparatus 200 according to the present embodiment includes a display mode determination unit 206 as a component corresponding to the display mode determination unit 106 according to the first embodiment, and the display according to the first embodiment. As a component equivalent to the frame center position determination unit 107, a display frame center position and display frame size determination unit 207 is provided.

  The person orientation detection unit 211 detects the orientation of the person in the captured image. Specifically, the direction of the person is detected by determining the corresponding template using the template of the person model for each direction. The person model is prepared in three directions, for example, front and rear, left and right. The direction indicated by the template having the highest matching score among the three types of templates may be detected as the direction of the person. Further, when the matching score between a person image and one of the three types of templates is higher than a predetermined threshold, it may be determined that the person is facing the direction indicated by the template having the higher matching score. . FIG. 7A is a diagram illustrating a case where the matching score of the rightward template is high, and FIG. 7B is a diagram illustrating a case where the matching score of the leftward template is high.

  The direction comparison unit 212 reads the movement history of the person position from the movement history holding unit 103 and obtains the movement direction of the person position (the direction of the person flow line). The direction comparison unit 212 further compares the obtained direction of the person flow line with the direction of the person detected by the person direction detection unit 211, determines whether or not both directions match, and determines the determination result as a display mode. The data is output to the unit 206. If the two directions do not match, the direction comparison unit 212 notifies the display frame center position and the display frame size determination unit 207 to that effect.

  In the example of FIG. 7A, since the direction of the person flow line is the right direction and the direction of the person is also the right direction, the direction comparison unit 212 determines that the two directions match. In the example of FIG. 7B, since the direction of the person flow line is the right direction and the direction of the person is the left direction, the direction comparison unit 212 determines that the two directions do not match. Specifically, the direction comparison unit 212 determines that the two directions match when the angle between the two directions is equal to or smaller than a predetermined threshold.

  The display mode determination unit 206 displays the direction of the person flow line when the sum of the distance differences does not exceed the threshold σ and the result output from the direction comparison unit 212 when the sum of the distance differences exceeds the threshold σ. If the result is that the direction of the person does not match the direction of the person, the display mode is set to the fixed display mode. When the sum of the distance differences exceeds the threshold σ and the result output from the direction comparison unit 212 is a result that the direction of the person flow line matches the direction of the person The display mode is the moving display mode.

  When the fixed display mode is set, the display frame center position and display frame size determination unit 207 sets the display frame size to α times the detection frame size (α> 1), and the moving display mode is set. In this case, the display frame is made the same size as the detection frame.

  The display frame center position and display frame size determination unit 207 is notified by the display mode determination unit 206 that the fixed display mode is set as the display mode, and the direction comparison unit 212 notifies the direction of the person flow line and the direction of the person. If the result is that they do not match, when setting the display frame in the fixed display mode, the size of the display frame is set so that the display frame becomes wider in the direction in which the person in the detection frame faces. To decide. Other configurations are the same as those of the first embodiment.

  As described above, according to the image processing apparatus 200 of the second embodiment, the direction of the person in the captured image is compared by the image processing, and the direction of the person flow line based on the movement history of the person position is the person's direction. If it matches the direction, it is determined that the person flow line is due to the movement of the person, and when the distance difference exceeds the threshold σ, the display mode is set to the movement display mode, and the direction of the person flow line and the person If the direction does not match, it is determined that the fluctuation of the detection frame is due to the fluctuation of the detection frame, and the display mode is set to the fixed display mode even when the distance difference exceeds the threshold σ. Therefore, it is possible to more appropriately determine whether the person is moving, stopped, or staying in consideration of the direction of the person.

(Third embodiment)
FIG. 8 is a block diagram illustrating a configuration of an image processing apparatus 300 according to the third embodiment. In the image processing apparatus 300 of FIG. 8, the same components as those of the image processing apparatus 100 according to the first embodiment are denoted by the same reference numerals, and description thereof is omitted. In addition to the configuration of the image processing apparatus 100 according to the first embodiment, the image processing apparatus 300 according to the present embodiment further includes a movement state calculation unit 311 and an intra-pattern weight holding unit 312. The image processing apparatus 300 according to the present embodiment includes a display mode determination unit 306 as a component corresponding to the display mode determination unit 106 according to the first embodiment, and the display according to the first embodiment. As a component equivalent to the frame center position determining unit 107, a display frame center position and display frame size determining unit 307 is provided.

  Although the person is moving, the movement area is within a certain range, and even if the display frame follows the detection frame, it does not cause frequent and high-speed fluctuation within a very small range like camera shake. When the display frame frequently moves up and down and left and right at a high speed within a certain range, the user who observes the cut-out image may be stressed. The image processing apparatus 300 according to the present embodiment can eliminate such stress on the user.

  The movement state calculation unit 311 refers to the movement history of the person position stored in the movement history holding unit 103 to determine whether or not the person is moving at a high speed within a predetermined size range. The result is output to the display mode determination unit 306 and the display frame center position. Whether or not a person is moving within a predetermined size range at a high speed is determined based on the detection frame of each frame for the most recent X frame (current time is t to time t-s to time t). It is possible to determine whether or not the parameter obtained by dividing the sum of the speeds (the movement distance from the previous frame) by the sum of the distance differences between the average position and the center position of the detection frame of each frame exceeds a predetermined threshold value. When the movement state calculation unit 311 determines that the person is moving at a high speed within a predetermined size range, the movement state calculation unit 311 notifies the display frame center position and the display frame size determination unit 307 to that effect. .

  The display mode determination unit 306 determines that the person is a predetermined value when the sum of the distance differences does not exceed the threshold σ and the result output from the movement state calculation unit 311 when the sum of the distance differences exceeds the threshold σ. If the result is that the robot is moving at a high speed within the size range, the display mode is set to the fixed display mode. The display mode determination unit 306 indicates that the sum of the distance differences exceeds the threshold σ, and the result output from the movement state calculation unit 311 indicates that the person is moving within a predetermined size range at a high speed. If the result is not, the display mode is set to the moving display mode.

  The display frame center position and display frame size determination unit 307 is notified by the display mode determination unit 306 that the fixed display mode has been set as the display mode, and the movement state calculation unit 311 has a predetermined size range. When a determination result indicating that the user is moving at a high speed is notified, the size of the display frame is determined so as to widen the display frame when determining the display frame in the fixed display mode. The display frame center position and display frame size determination unit 307 may determine the size of the display frame so as to include the range in which the person is moving in the latest X frame.

  The intra-pattern weight holding unit 312 holds the intra-pattern weight. The display frame center position and display frame size determination unit 307 refers to the weight held in the in-pattern weight holding unit 312 when determining where to set the center position when setting the size of the display frame to be large. Then, the weight of the face portion in the person pattern is increased. Other configurations are the same as those of the first embodiment.

  As described above, according to the image processing apparatus 300 of the third embodiment, it is determined whether or not the movement state of a person is moving within a predetermined size range at a high speed. When a person is moving within a predetermined size range at a high speed, the display frame is enlarged and fixed, so the display frame frequently moves up, down, left, and right within a certain range at a high speed. By doing so, it is possible to avoid stressing the user who is observing the cut-out image.

(Fourth embodiment)
FIG. 9 is a block diagram illustrating a configuration of an image processing apparatus 400 according to the fourth embodiment. In the image processing apparatus 400 of FIG. 9, the same components as those of the image processing apparatus 100 according to the first embodiment are denoted by the same reference numerals, and description thereof is omitted. In addition to the configuration of the image processing apparatus 100 according to the first embodiment, the image processing apparatus 400 according to the present embodiment further includes a movement route calculation unit 411 and a movement route holding unit 412. Further, the image processing apparatus 400 according to the present embodiment includes a display frame position determination unit 407 as a component corresponding to the display frame center position determination unit 107 according to the first embodiment.

  When a place where the image generation unit 101 is located is photographed, the movement route of the person may be limited at the place due to an obstacle or the like. In this case, the position of the display frame may be changed so as to follow the movement route. Therefore, in the present embodiment, the position of the display frame is limited on the moving route along which the person moves.

  The travel route calculation unit 411 calculates an average travel route from a person who has passed the same place multiple times. The movement route holding unit 412 holds the average movement route calculated by the movement route calculation unit 411. FIG. 10A is a diagram for explaining processing for obtaining an average route from a trajectory caused by a plurality of movements of a person. As shown in FIG. 10 (a), when a person is detected many times on substantially the same route, the movement route calculation unit 411 determines an average single curve based on the movement status of the detection frame. Is determined as the average travel route, and the travel route holding unit 412 holds this.

  When the movement display mode is set, the display frame position determination unit 407 fixes the display frame display range on the movement route held in the movement route holding unit 412. That is, the display frame position determination unit 407 determines only the front-rear direction on the average moving route to correspond to the position of the detection frame. FIG. 10B is a diagram showing the display frame determined in this way. The center of the display frame is always located on the movement route held by the movement route holding unit 407. The center of the display frame can be a point on the moving route that is closest to the center of the detection frame.

  FIG. 11 is a diagram illustrating a modification of the present embodiment. In this modification, the display frame position determination unit 407 determines the moving direction based on the movement history of the person, and the point on the moving route is shifted in the moving direction from the point on the moving route closest to the center of the detection frame. Is the center of the display frame. In FIG. 11, a solid line is a display frame centered on a point on the moving route closest to the center of the detection frame, and a dotted line is a moving direction on the point moving route on the moving route closest to the center of the detection frame. This is a display frame centered on a point shifted by a predetermined length.

  As described above, according to the image processing apparatus 400 of the fourth embodiment, an average movement route is obtained from the movement history of past persons, and the position of the display frame is limited on the movement route. The display frame does not shake due to the change of the detection frame in a direction different from the moving route, and the stress caused by the change of the display frame that is given to the user who observes the cut-out image can be reduced.

  Further, if the average movement route is obtained and the movement direction of the person is known, the movement destination of the person can be predicted. Therefore, the display frame can be set on the movement route in advance of the movement of the person.

(Fifth embodiment)
FIG. 12 is a block diagram illustrating a configuration of an image processing apparatus 500 according to the fifth embodiment. In the image processing apparatus 500 of FIG. 12, the same components as those of the image processing apparatus 100 of the first embodiment are denoted by the same reference numerals, and description thereof is omitted. The image processing apparatus 500 according to the present embodiment further includes an image distortion information holding unit 511 and a weighting unit 512 in addition to the configuration of the image processing apparatus 100 according to the first embodiment. Further, the image processing apparatus 500 according to the present embodiment includes a distance difference total calculation unit 505 as a component corresponding to the distance difference total calculation unit 105 according to the first embodiment.

  The image generation unit 101 according to the present embodiment generates an omnidirectional image as a captured image using a fisheye lens with a horizontal angle of view of 360 degrees. The omnidirectional image is a circular image, and the distortion increases from the center of the circle toward the circumference. The cut-out image generation unit 110 cuts out a display frame from a circular omnidirectional image, corrects distortion, generates a plane image, and displays it.

  The image distortion information holding unit 511 holds image distortion information for each position of the captured image. As described above, the distortion of the image is larger at the periphery of the image than at the center of the image. The weighting unit 512 weights the movement from the center of the image outward (that is, in the radial direction) more heavily than the movement along the circumference of the image. The distance difference sum calculation unit 505 obtains the sum of the distance differences weighted by the weighting unit 512 according to the position of the detection frame on the image and the moving direction.

  In the omnidirectional image, the position variation on the image is smaller when moving in the radial direction than when moving in the circumferential direction. For this reason, the weighting unit 512 reduces the weight given to the distance difference for the movement in the radial direction, and when moving in the radial direction, the movement display is performed only when the position largely fluctuates on the image. Make it a mode.

  FIG. 13 is a diagram illustrating weighting processing according to the position of the omnidirectional image. As shown in FIG. 13A, when the person moves along the circumferential direction of the circular omnidirectional image, the weighting unit 512 applies a heavy weight to the distance difference from the average position, and the distance difference A sum calculation unit 505 calculates the sum of the distance differences. As a result, the sum of the distance differences exceeds the threshold σ, and the display mode determination unit 106 sets the movement display mode.

  When the person moves in the radial direction of the circular omnidirectional image as shown in FIG. 13B, the weighting unit 512 gives a light weight to the distance difference from the average position, and the distance difference sum calculation unit 505. Calculates the sum of the distance differences. As a result, as shown in FIG. 13B, when moving over a sufficiently long distance, the sum of the distance differences exceeds the threshold σ, and the display mode determination unit 106 sets the movement display mode. .

  As shown in FIG. 13C, when the person moves in the radial direction of the circular omnidirectional image, the weighting unit 512 gives a light weight to the distance difference from the average position, and the distance difference sum calculation unit 505. Calculates the sum of the distance differences. As a result, as shown in FIG. 13C, when the distance is not sufficiently long, the sum of the distance differences does not exceed the threshold σ, and the display mode determination unit 106 sets the fixed display mode. The

  Furthermore, the weighting unit 512 changes the weight depending on whether the person is moving along the circumference of the image, near the center or near the periphery. In the omnidirectional image, the movement near the center portion has a larger positional variation on the image than the movement near the outer periphery. For this reason, the weighting unit 512 increases the weight in the case of movement near the center.

  As described above, according to the image processing apparatus 500 of the fifth embodiment, weighting is performed according to the characteristics of the omnidirectional image that distortion increases from the center toward the circumference, and the total distance difference is calculated. Since the calculation is performed, it can be reduced that the person is actually moving but is estimated to be stopped or stayed due to the distortion of the omnidirectional image.

  In the first to fifth embodiments, the display frame is set to be larger than the detection frame and to be included in the display frame. However, for example, a human face portion may be extracted from the detection frame, and a cut-out image may be generated for only the face portion. In this case, the display frame may be smaller than the detection frame. Therefore, even when the position of the display frame is made to follow the position of the detection frame in the moving display mode, the display frame and the detection frame need only have a certain relationship, and the display frame necessarily includes the detection frame and is larger than the detection frame. It doesn't have to be a thing.

  In the first to fifth embodiments, the person detection unit 102 detects a person in the image and generates a detection frame. However, a moving body other than a person may be detected.

  As described above, according to the present invention, the display frame is determined based on the history of the position of the detection frame so that the position does not move, and the cut image is generated with the display frame. Therefore, the cut image is observed. This is useful as an image processing apparatus or the like that can reduce stress on the user due to fluctuations in the display frame, performs image processing on the captured image, and generates a cut image obtained by cutting out the display frame in the captured image.

DESCRIPTION OF SYMBOLS 100 Image processing apparatus 101 Image generation part 102 Person detection part 103 Movement history holding part 104 Average position calculation part 105 Distance difference sum total calculation part 106 Display mode determination part 107 Display center position determination part 108 Display frame position determination parameter holding part 109 Display frame Position holding unit 110 Cutout image generation unit 200 Image processing device 207 Display frame center position and display frame size determination unit 211 Person orientation detection unit 212 Direction comparison unit 300 Image processing device 306 Display mode determination unit 307 Display frame center position and display frame size Determination unit 311 Movement state calculation unit 312 In-pattern weight holding unit 400 Image processing device 407 Display frame position determination unit 411 Movement route calculation unit 412 Movement route holding unit 500 Image processing device 505 Distance difference sum calculation unit 511 Image distortion information holding unit 512 Weighting part

Claims (16)

An image generation unit that generates an image by shooting;
A detection frame generation unit that detects a region surrounding the moving body in the image as a detection frame;
A display frame determining unit that determines a display frame based on a history of the position of the detection frame so that the position does not move;
A cut-out image generation unit that generates a cut-out image of the display frame determined by the display frame determination unit;
An image processing apparatus comprising:   The display frame determination unit is configured to display the display mode regardless of a movement display mode in which the display frame is moved following the detection frame or a change in the detection frame based on a history of the position of the detection frame. The image processing apparatus according to claim 1, wherein the image processing apparatus is set to any one of a fixed display mode for fixing a frame, and the position of the display frame is determined according to the set display mode.   When the display frame determining unit estimates whether the moving body is moving or stopped or stays based on the history of the position of the detection frame, and estimates that the moving body is moving Further, the display mode is set to the fixed display mode when the display mode is set to the moving display mode and it is estimated that the moving body is stopped or staying. The image processing apparatus described.   The said display frame determination part determines the said display frame so that the said display frame may change smoothly, when the said display mode transfers to the said movement display mode from the said fixed display mode. The image processing apparatus according to 3.   5. The display frame determination unit determines the size of the display frame based on the size of the detection frame and the history of the position of the detection frame. 6. The image processing apparatus according to item.   The display frame determination unit estimates whether the moving body is moving, stopped or stayed based on the history of the position of the detection frame, and estimates that the moving body is stopped or staying. The image processing apparatus according to claim 5, wherein the size of the display frame is made larger than when it is estimated that the moving body is moving.   The display frame determining unit determines and fixes the position and size of the display frame based on a history of the position of the detection frame so as to include a range in which the detection frame has moved within a predetermined period. The image processing apparatus according to claim 5.   8. The display frame determination unit according to claim 2, wherein the display frame determination unit releases the fixing of the display frame when the detection frame reaches the periphery of the fixed display frame. The image processing apparatus according to item.   The display frame determination unit obtains an average position of the detection frames within a predetermined period, obtains a sum of distance differences between the positions of the detection frames and the average position within the predetermined period, and sums the distance differences. Is greater than a predetermined threshold, it is estimated that the moving body is moving, and when the sum of the distance differences is smaller than the predetermined threshold, it is estimated that the moving body is stopped or staying. The image processing apparatus according to claim 3, wherein:   The image processing apparatus according to claim 9, wherein the image has distortion, and the display frame determination unit weights the distance difference according to distortion of each detection frame. A moving body direction detection unit that detects the direction of the moving body based on the image of the moving body;
The display frame determining unit estimates that the moving body is stopped or staying when the moving direction of the detection frame and the direction of the moving body are different based on the history of the position of the detection frame. The image processing apparatus according to claim 3 or 6.   The display frame determination unit determines the display frame so as to satisfy the constraint condition with reference to the constraint condition regarding the change in the position of the display frame so that the position of the display frame cannot be shaken. The image processing apparatus according to any one of claims 1 to 11.   The image processing apparatus according to claim 12, wherein the constraint condition is a condition that a moving speed of the display frame does not exceed a predetermined upper limit.   The image processing apparatus according to claim 12, wherein the constraint condition is a condition for limiting a position of the display frame on a predetermined route. An image generation step for generating an image by shooting;
A detection frame generation step of detecting a region surrounding the moving body in the image as a detection frame;
A display frame determining step for determining a display frame corresponding to the detection frame based on a history of the position of the detection frame so that the position does not move;
A cut image generation step for generating a cut image of the display frame determined in the display frame determination step;
An image processing method comprising:   A computer program for causing a computer to execute the image processing method according to claim 15.

Images