JP4996404B2 - Human behavior search device - Google Patents

Description

  The present invention relates to a person behavior search device, and more specifically, when analyzing a person's behavior from moving image data recording a predetermined monitoring place, a person key indicating a person's behavior is searched as a search key, and the person is recorded. The present invention relates to a human behavior search apparatus that can quickly reproduce the moving image data from a recording position.

Conventionally, a monitoring camera such as an automatic teller machine (ATM), convenience store, department store, condominium, house, etc. has been installed with a monitoring camera, and a captured image is recorded in a storage unit by a recording device. It is done to record human behavior. The moving image data recorded by the recording device is reproduced on the monitor by the reproducing device and used for analyzing the action of a specific person at the monitoring place.
JP 2006-331049 A Japanese Patent No. 3406587 Japanese Patent No. 3390426

  However, since the moving image data is recorded for a long time, it is necessary to perform operations such as fast-forwarding the playback device until a person is projected. At that time, in order to find a person who shows a specific behavior, a scene where the person's action can be easily judged is searched by rewinding or fast-forwarding, and sometimes it is confirmed by pausing or frame-by-frame. Moreover, when a plurality of people appear on the screen at the same time, it is necessary to rewind and confirm each individual person.

  It is very troublesome to find a person from the video data recorded in this way and analyze the action of the person. Originally, the video recording of the monitoring location is always for the purpose of grasping the behavior of the person, so that the behavior of the person can be easily and visually confirmed from the video data recorded for a long time. It is desirable to make it.

  The present invention has been made in view of the above circumstances, and a person who shows a specific behavior is recorded from moving image data in order to reduce the work burden required to analyze human behavior from moving image data recorded for a long time. It is an object of the present invention to provide a human behavior search apparatus that easily searches for a position and makes it possible to quickly reproduce the position.

In order to solve the above problems, the present invention provides a human behavior search apparatus for searching for a person who shows a predetermined behavior and reproducing the video data recorded by monitoring and monitoring the behavior of the person from a scene where the person appears. It is to provide. The means is as follows.
A recording device that causes the image recording unit to record the captured image of the monitoring area captured by the imaging device as moving image data;
Recognize each person on the captured images continuously sent from the imaging device or the image recording unit, track the movement of the virtual center of gravity of the person, and grasp the behavior of the person at a specific location in the monitoring area. In order to do so, the virtual center of gravity of the person appearing on the captured image in the measurement area preset within a certain range on the captured image enters the measurement area preset on the captured image and then moves outside the measurement area. An image processing device for acquiring the behavior information of the person until leaving,
From the behavior information of the person acquired by the image processing device, a behavior determination device that determines the behavior of the person based on a predetermined behavior determination criterion;
栞 data creation unit for creating 栞 data indicating the behavior of the person from the result of behavior determination by the behavior determination device,
A movement trajectory recording unit that records a tracking line indicating the movement trajectory of the virtual center of gravity of the person tracked by the image processing device;
Preset representative image extraction line at a position passing through the person in the monitoring area on the captured image, the virtual center of gravity is the representative image of the person appearing on the captured image from the captured image by the image processing apparatus The captured image when passing through the extraction line for the first time is extracted as a representative image in which the person's face is clearly shown, and the person's eyelid image obtained by combining the representative image with the tracking line recorded in the moving locus recording unit.栞 Image creation part to create
A recording position recording unit for recording the recording position of the moving image data in which the person is recognized;
A person key creating unit for associating the above-mentioned 栞 image, 栞 data and recording position by the person, and storing the person key in the 栞 storage unit;
A search unit for searching for the heel data of the person key, calling the heel image associated with the heel data from the heel storage unit, and displaying the list of the heel images on a display device;
A display editing section for causing the display device to reproduce the moving image data recorded by the recording device by selecting the thumbnail image displayed in the list from the recording position associated with the thumbnail image;
The above data is
(1) movement order indicating whether there is movement between the different measurement areas of the person,
(2) Movement speed indicating the average speed of the person in the measurement area,
(3) Stop time indicating the time the person was stopped in the measurement area, and
(4) Consists of all combinations of items (1), (2), (3), and (4) of the staying time indicating the time during which the person stayed in the measurement area.

The above heel data
(5) A companion showing whether the person was moving in parallel within the measurement area, and
(6) One of the items (5) and (6) of the number of staying persons indicating the number of people who were present in the measurement area at the same time including the person or all of these items (5) and (6) Can do.
The saddle image creation unit sets a representative image extraction line in advance in a position where a person in the monitoring area passes on the captured image, thereby representing the representative image from the captured image captured at a position where the person's face is clearly visible. May be extracted to create a bag image.
The image processing apparatus performs a fundamental wave Fourier transform on pixel values of each pixel on the circumference of a circle having the processing point as a center point for each processing point arranged at equal intervals in the captured image. The outline of the person image is recognized by a normal vector for each processing point from the obtained phase, and the person image is recognized as the virtual centroid point based on the normal vector, and the virtual centroid point is determined over time. A person recognition means for tracking may be included.

According to the above configuration, a captured image sent from an imaging device such as a monitoring camera is recorded as moving image data in a predetermined storage unit by the recording device, while being always processed by the image processing device, Individual persons are individually captured as virtual centroid points and tracked, and behavior information such as the position, speed, and time of the person in the measurement region is acquired. Then, the captured image when passing through the representative image extraction line for the first time is extracted as the representative image of the person, and the tracking line is recorded to create a habit image of the person.栞 Data is created, and the recording position of the moving image data in which the person is recognized is recorded. The person's wrinkle image, wrinkle data, and recording position are stored in the storage unit as person keys associated with each other.

When the moving image data recorded by the recording device is played back, first, the search unit causes the search data to be searched using the font data of the person key as a search key . Next, when a desired haze image is selected, the moving image data is reproduced on the display device from a scene where a person of the haze image selected by the display editing unit appears.
In this way, it is possible to easily search for the position of moving image data where a person exhibiting a specific behavior is recorded, and to easily and quickly reproduce the scene from which the person appears.

  As described above, according to the present invention, by creating a person key, it is possible to easily reproduce a scene in which a person showing a specific behavior appears from moving image data recorded for a long time. The work burden required for analyzing human behavior based on the moving image data recorded by the recording device can be greatly reduced.

Embodiments of the present invention will be described below with reference to the drawings.
In this embodiment, as an application example of the human behavior search apparatus, predetermined behavior is obtained from video data obtained by monitoring and recording the behavior of a person who is a user of an ATM machine (automated teller machine) installed in a store such as a bank. The person who showed the behavior is searched and can be quickly displayed. FIG. 1 schematically shows a state in which an ATM room S (monitoring area) in which three ATM machines 100 (M1 to M3) are installed side by side is always photographed from above by a single camera 9.

  In FIG. 1, T is the door of the ATM room S, A1 to A3 are measurement areas set in the captured image 1, and h is a person image h of the user of the ATM machine 100. G is the virtual center of gravity of the person image h, and R1 is a movement trajectory indicating the behavior of the person image h by the user of the right ATM machine 100 in FIG. R2 is a movement trajectory indicating the behavior of the person image h by the user of the central ATM machine 100 in FIG. 1 and indicating a normal behavior. L is a representative image extraction line set on the captured image 1.

  FIG. 2 shows a configuration diagram of the human behavior search device X. The human behavior search device X includes a camera (imaging device) 9, a recording device 2, an operation means 7, an image processing device 3, a behavior determination device 4, and a cocoon generation. A device 5, a bag storage unit 50, a reproduction control device 6, and a display device 8 are provided.

  The camera 9 is composed of, for example, a single CCD camera or the like, and is installed on the ceiling or wall at an angle at which the ATM machine 100 and the floor surface can be seen from above, and always images the inside of the ATM room S, which is the monitoring area S, from above. The captured image 1 is continuously sent to the image processing device 3. By installing the camera 9 at an angle at which the floor surface is reflected, it is possible to obtain information that makes it easy to grasp the course when the movement locus of the person image h is displayed. When the ceiling is low, the range of the movement trajectory of the person image h is narrowed, so the camera 9 equipped with a wide-angle lens may be used to widen the shooting range.

  The recording device 2 records the captured image 1 in the ATM room S captured by the camera 9 in a predetermined image recording unit 21 as moving image data. The recording device 2 uses a known recorder, a computer drive device, or the like, and the image recording unit 21 for storing moving image data includes various types of storage media such as a built-in hard disk, a removable DVD, and a DV tape. used.

  For example, the operation unit 7 includes an input device such as a keyboard and a mouse. The operation unit 7 sets the measurement area A on the captured image 1 captured by the camera 9 with respect to the image processing device 3 or the behavior determination device 4. Then, a behavior determination reference 41 for analyzing human behavior is set, or the playback control device 6 is operated to instruct the display device 8 to play back the moving image data recorded by the recording device 2. The operation means 7 also sets a representative image extraction line L that indicates a position on the captured image 1 where a representative image of a person imaged in the captured image 1 is extracted.

  The measurement area A is set in order to acquire action information of the person when the person who appears on the captured image 1 exists in the measurement area A. For example, when monitoring an unattended ATM room S, the measurement area A is set within a certain range of the front position of the ATM machine 100 on the captured image 1. As a result, the behavior of the user operating the ATM machine 100 can be grasped, and the ATM machine 100 can be disguised as a user of the ATM machine 100, or suspicious such as peeping at the operation of other users. The presence or absence of proper behavior is grasped. In the ATM room S shown in FIG. 1, the measurement area A is set in a square shape at the front surface of each of the three ATM machines 100. The measurement area A is positioned and sized according to the monitoring purpose. The shape and the like can be set as appropriate.

  The image processing device 3 recognizes each person on the captured image 1 continuously sent from the camera 9 and tracks the movement of the virtual center of gravity G of the person, and the person in the measurement area A The action information of the virtual center of gravity point G is acquired. The image processing apparatus 3 includes a person recognition unit 31 and a behavior measurement unit 32.

  The human recognition means 31 executes fundamental wave Fourier transform on each pixel value on the circumference of a circle having the processing point as the center point for each processing point arranged at equal intervals in the captured image 1 and obtained. From the phase, the outline of the person image h is recognized by a normal vector for each processing point, the virtual center of gravity G of the person image h is captured based on the normal vector, and the virtual center of gravity G is determined over time. The image processing technology (Patent No. 3406657, Patent No. 3406587, Patent No. 3390426) that tracks the image automatically is known.

  As the image processing of the vector focus method, first, as shown in FIG. 3, the density difference between the object image B and the background image W in the captured image 1 is used and arranged at equal intervals on the captured image 1. For each processing point P, a fundamental wave Fourier transform represented by the following formula (1) is performed on the pixel values of the pixels d1 to d12 on the circumference of the circle 11 having the processing point P as the center point ( Circular Fourier method). The fundamental wave means a sine wave and a cosine wave having a length corresponding to one cycle of the circumference centering on the processing point P.

（式（１）中、ＳＶは画素のピクセル値にフーリエ変換の正弦波を積和した積和値、ＣＶ
は画素のピクセル値にフーリエ変換の余弦波を積和した積和値、ＡＴＡＮはアークタンジェント、ｓｉｔａはフーリエ変換で得られる位相値を示す。）

(In the formula (1), SV is a product-sum value obtained by multiplying the pixel value of a pixel by a sine wave of Fourier transform, CV
Denotes a product-sum value obtained by multiplying the pixel value of a pixel by a cosine wave of Fourier transform, ATRAN is an arctangent, and sita is a phase value obtained by Fourier transform. )

  The phase obtained by the Fourier transform indicates the light and shade direction of the image existing in the circumference, and becomes the light and shade direction of the image at the pixel at the processing point P at the center point. That is, this grayscale direction is recognized as a normal vector with respect to the boundary between the object B and the surrounding background W in the two-dimensional captured image 1, that is, the contour portion of the object B.

  For example, when the circle 11 is on the object B as shown in FIG. 4A, it is rotated counterclockwise from a predetermined reference position, and Fourier transform is executed in a circle, and a sine wave and a cosine wave are circled. When the pixel values of the respective pixels on the circumference are integrated, a waveform peak appears near 90 degrees as shown in FIG. 4B. As a result, the phase obtained by the above circular Fourier method just indicates the normal direction with respect to the contour portion of the object B. In FIG. 4B, sin represents a sine wave, cos represents a cosine wave, s · p represents a waveform obtained by integrating the sine wave and the pixel value, and c · p represents a waveform obtained by integrating the cosine wave and the pixel value. ing.

  However, since the contour portion of the object image B is not included in the circumference and the image in the circumference has almost uniform density distribution, the above SV and the above formula (2) are excluded. When the square root value (pow) of the square sum of CVs is less than a certain value, the processing point P is treated as having no shading.

  In order to extract the person image h, as shown in FIG. 5A, first, the standard human figure image o is arranged with reference to an arbitrary arrangement point Q on the captured image 1, and the circular Fourier method is used. Thus, the normal vector group b in the contour portion of the standard human figure image o is obtained, and the position information from the arrangement point Q to the normal vector group b and the angle information of the normal vector group b are used as standard data of the arrangement point Q. Remember.

  Next, as shown in FIG. 5B, a normal vector group b ′ at the contour portion of the person image h is obtained by the circular Fourier method for the captured image 1 in which the person image h moving during monitoring is copied. Based on the standard data, a plurality of correct answer points corresponding to the arrangement point Q are obtained from the normal vector group b ′, and if the so-called vector focus G where these correct answer points are gathered is equal to or greater than a certain value, the person image Recognize that h is a person. This vector focus G becomes the virtual centroid G of the person, and the centroid G is tracked for each captured image 1 sent from the camera 9.

  Note that the position of the virtual center of gravity G can be set to an appropriate position on the person image h in accordance with the monitoring purpose, but in the present embodiment, for example, it is arranged at the head of the person image h. That is, when the ATM machine 100 is disguised as a user of the ATM machine 100, or when looking at other user's operations, the head movement becomes noticeable to look around. For this reason, in the present embodiment, it is preferable to arrange a virtual center of gravity G for capturing human behavior on the head of the person image h. Thereby, when monitoring the behavior of a person in the ATM room S, by observing the movement of the head, it is possible to reliably grasp the presence or absence of the suspicious behavior of the person, and the reliability of the human behavior analysis Can be improved.

  According to the image processing technique based on the vector focus method described above, the captured image 1 sent from the camera 9 can be continuously processed, and the human image h can be defined by a virtual center of gravity G called a vector focus for each processing. The movement of the virtual center of gravity point G can be tracked, and the instantaneous movement speed (including stop) of the virtual center of gravity point G can be measured by this tracking. For example, in the method of recognizing a difference image from the captured images 1 of the preceding and following frames in human recognition, if the person image h stops, the difference may not be obtained and the person may be lost. However, in the image processing technique using the vector focus method, There is no inconvenience.

  Therefore, according to the person recognition means 31, each person appearing in the captured image 1 can be recognized one by one. For example, regarding the two persons crossing the shoulder, the individual person can be distinguished and recognized. Can be recognized. Therefore, it is possible to grasp a companion that two people run in parallel. Then, the person recognition means 31 tracks the movement of the virtual center of gravity G of the person image h until the virtual center of gravity G of the person image h appears in the captured image 1 and disappears outside the captured image 1.

  The behavior measuring unit 32 measures and records behavior information from when the virtual center of gravity G of the person image h specified by the person recognizing unit 31 enters the measurement region A until it exits outside the measurement region A. This behavior information is, for example, the position, speed, time, etc. of the person in the measurement area A.

  The behavior determination device 4 determines the behavior of the person based on a predetermined behavior determination reference 41 from the behavior information of the person acquired by the image processing device 3. A behavior determination standard 41 is registered in the behavior determination device 4 (see FIG. 2). This behavior determination criterion 41 is a criterion for determining how a person is taking action in the monitoring area S. For example, (1) movement order, (2) companion, (3) movement speed , (4) stop time, (5) stay time, and (6) number of visitors, etc. are set (see FIG. 7).

  Here, the movement order of the above (1) determines whether or not the person has moved between a plurality of different measurement areas A. That is, a normal user of the ATM machine 100 does not normally move to the side during the operation of the ATM machine 100 (see R2 in FIG. 1). On the other hand, when a suspicious action such as peeking at the adjacent ATM machine 100 is shown, movement to the measurement area A set for the adjacent ATM machine 100 occurs (in FIG. 1, R1 is set to R1). reference). Therefore, it can be determined that a person who performs an action between a plurality of different measurement areas A can include a suspicious person.

  In the above (2), it is determined whether or not the person has moved in the measurement area A in parallel. In this case, it can be determined that a person who pretend to be a user of the ATM machine 100 and mischief the ATM machine 100 can be included.

  The moving speed of the above (3) determines the average speed in the measurement area A of the person. In this case, if the average speed is abnormally high, it can be understood that the surrounding area has been overlooked, and some of them include those who pretend to be a user of the ATM machine 100 and mischief the ATM machine 100. It is because it can be judged that it obtains.

  The stop time of the above (4) determines the time during which the person has stopped in the measurement area A. In this case, if the stop time is abnormally long, it can be determined that a person who pretend to be a user of the ATM machine 100 and mischief the ATM machine 100 can be included.

  The staying time of (5) above determines the time during which the person has stayed in the measurement area A. Also in this case, if the staying time is abnormally long, it can be determined that a person who pretends to be a user of the ATM machine 100 and mischief the ATM machine 100 can be included.

The number of stays in (6) above determines the number of people who were present in the measurement area A at the same time including the person. In this case, for example, if there are a large number of three or more, it can be determined that a person who pretends to be a user of the ATM machine 100 and mischief the ATM machine 100 can be included.
In addition, this behavior determination reference | standard 41 can determine the content of the behavior suitably not only according to each said behavior but according to the monitoring objective.

  The bag generating device 5 generates a person key for easily calling a scene in which a person showing a specific behavior appears from moving image data recorded in the recording device 2 for a long time. The bag generating device 5 includes a recording position recording unit 51, a movement locus recording unit 52, a bag image generating unit 53, a bag data generating unit 54, and a person key generating unit 55 (see FIG. 2).

  The recording position recording unit 51 records the recording position of moving image data in which individual persons recognized on the captured image 1 are captured. The recording position is, for example, an image frame number (start frame) at the time when a person in the captured image 1 first appears and an image frame number (one frame before the point at which the person disappears outside the captured image 1). End frame) (see FIG. 11). Note that this recording position is not limited to the above-described image frame number, and can also be configured with a recording time. Further, the recording position may be composed of only the start frame.

  The movement trajectory recording unit 52 records a tracking line indicating the movement trajectory of the virtual centroid point G of the person tracked by the image processing device 3. That is, the movement trajectory recording unit 52 individually shows the movement trajectory from when the virtual center of gravity G of the person image h appears in the captured image 1 until it disappears outside the captured image 1 in the captured image 1. Record as tracking line.

  When a person is recognized on the captured image 1 by the image processing device 3, the heel image creation unit 53 extracts a representative image of the person, and uses the tracking line recorded in the movement locus recording unit 52 as the representative image. Create a composite image of the person. The wrinkle image creation unit 53 extracts a representative image from the captured image 1 captured at a position where the person's face is clearly visible, and creates a wrinkle image. At this time, the representative image extraction position is on the captured image 1. Can be set arbitrarily. For example, when the inside of the ATM room S in FIG. 1 is monitored, the face of the person is relatively clearly shown at a position slightly entered from the door T at the entrance / exit of the ATM room S. Therefore, in this case, a representative image extraction line L is set at a position slightly entered from the entrance / exit of the ATM room S, and an image frame (captured image 1) at the moment of first passing through the line L is extracted as a representative image. A human eyelid image is created (see FIGS. 1 and 11).

  The heel data creation unit 54 creates heel data indicating the behavior of the person from the result of the behavior determination performed by the behavior determination device 4. This trap data is an item classified according to a person's behavior for searching for a person showing a specific behavior, and each item corresponds to each item of the person's behavior set in the behavior determination criterion 41. ing. Therefore, the items of the trap data include, for example, (1) movement order (whether the person has moved between a plurality of different measurement areas A), (2) sharing (the person is within the measurement area A in two persons) (3) Movement speed (average speed of the person in the measurement area A), (4) Stop time (time the person has been stopped in the measurement area A), (5) Stay time (time during which the person stayed in the measurement area A) and (6) Number of stayers (number of people who were present in the measurement area A including the person), etc. 7).

  The person key creation unit 55 creates a person key for associating the above-mentioned person's heel image, heel data, and recording position, and stores the person key in the heel storage section 50. The person key can be, for example, tabular data as shown in FIG. 8, and a person number (“person No.” in FIG. 8) is set for each individual. A person's habit image, habit data, and recording position are associated.

  As described above, the bag storage unit 50 stores the person key created by the bag generation device 5. The person key stored in the bag storage unit 50 is associated with the moving image data on which the person key is created.

The playback control device 6 causes the moving image data recorded for a long time in the recording device 2 to jump-play from a scene where a person showing a specific behavior appears with reference to the person key. The reproduction control device 6 includes a search unit 61 and a display editing unit 62 (see FIG. 2).
The search unit 61 searches the wrinkle data of the person key as a search key, calls the wrinkle image associated with the person key from the wrinkle storage unit 50, and causes the display device 8 to display a list of the wrinkle images.
The display editing unit 62 causes the display device 8 to reproduce the moving image data recorded by the recording apparatus 2 from the recording position associated with the haze image by selecting the haze images displayed in the list. That is, when a habit image is selected, the moving image data is jump-reproduced from the start frame indicated by the moving image position associated with the eclipse image, and the reproduction is stopped at the end frame indicated by the moving image position. As a result, the moving image data is reproduced on the display device 8 for the scene in which the person of the selected bag image appears.

  The display device 8 uses a known monitor device, and displays the picked-up image 1 picked up by the camera 9, displays a list of the saddle images, and reproduces and displays the moving image data.

Next, the operation of the person behavior search device X will be described.
First, the measurement unit A is set on the captured image 1 captured by the camera 9 with respect to the image processing apparatus 3 by the operation means 7 and the behavior determination reference 41 is set for the behavior determination apparatus 4. As shown in FIG. 1, the measurement area A is set so as to surround a certain range of the front position of the ATM machine 100 in a square shape so that the user operating the ATM machine 100 is completely surrounded. The As the behavior determination criteria 41, the above-mentioned items (1) order of movement, (2) companion, (3) movement speed, (4) stop time, (5) stay time, and (6) number of staying people are set. Is done. In addition, the operation means 7 draws a representative image extraction line L at a predetermined position on the captured image 1 to set a position for extracting each representative image captured by the camera 9. When the above setting is completed, a person key is created for each individual together with the recording of the captured image 1 in order to analyze the behavior of the user of the ATM machine 100.

The operation for creating the person key is performed according to the procedure shown in FIG.
That is, referring to FIG. 6, when the captured image 1 is continuously input from the camera 9 to the image processing device 3 (S1), it is recorded as moving image data in the image recording unit 21 by the recording device 2, while the image The processing device 3 recognizes whether a person exists in the captured image 1. When the presence of a person is recognized, the image frame number is recorded as a start frame indicating the time when the person first appears in the captured image 1 (S2). That is, as shown in FIG. 11, when a person is recognized on the captured image 1 for the first time, the image frame number at that time becomes a start frame indicating the recording position where the person appears in the recorded moving image data.

  Subsequently, the person recognized in the captured image 1 is regarded as a virtual center of gravity G, and the virtual center of gravity G is tracked over time on the continuously input captured image 1 (S3). The person recognition and the tracking of the virtual center of gravity G are performed using the image processing technique based on the vector focus method described above (see FIGS. 3 to 5). When the tracked virtual centroid G of the person passes through the representative image extraction line L, the captured image 1 (image frame) at that time is extracted as the representative image of the person (S4). That is, the captured image 1 that has been photographed when entering the ATM room S through the door T at the entrance / exit becomes a representative image of the person.

  When the tracked virtual centroid G of the person enters the measurement area A, the behavior information of the person from when the virtual centroid G enters the measurement area A and exits the measurement area A is obtained. Acquired by the behavior measuring means 32 of the image processing apparatus 3 (S5). That is, the position, speed, and time while a person in real space is operating the ATM machine 100 are acquired.

  The behavior information of the person acquired here is determined based on behavior determination criteria (S6), and wrinkle data is created from the behavior determination result (S7) (see also FIG. 11). That is, as described above, the behaviors of (1) movement order, (2) companion, (3) movement speed, (4) stop time, (5) stay time, and (6) number of staying people are determined, The result is created as bag data (see also FIG. 7).

  Next, when the virtual barycentric point G disappears from the captured image 1, the image frame number immediately before the image frame is recorded as an end frame indicating the time when the person leaves the ATM room S which is the monitoring area S. (S8). That is, as shown in FIG. 11, when the person disappears from the captured image 1, the recording immediately before the person disappears in the moving image data in which the image frame number immediately before the image frame number at that time is recorded. This is the end frame indicating the position. A pair of the end frame and the start frame is recorded as a recording position at which the person is captured in the captured image 1.

  The movement trajectory recording unit 52 traces the movement trajectory from when the virtual center of gravity G of the person image h appears in the captured image 1 until it disappears outside the captured image 1 for each individual person in the captured image 1. Record as. The heel image creation unit 53 creates a heel image of the person by combining the representative image extracted at the time of passing through the representative image extraction line L with the tracking line recorded in the movement locus recording unit 52 (S9). .

  Then, the person key creating unit 55 creates a person key associating the above-mentioned 栞 image, 栞 data, and recording position with the person as shown in FIG. 8, and this person key is stored in the 栞 storage unit 50. (S10).

  The captured image 1 sequentially sent from the camera 9 by the above operation is continuously processed and recorded as moving image data, while the image (contrast image) or action (action image) of the person shown in the captured image 1 is recorded. A person key indicating the position (recording position) of the moving image data in which the person appears is created and recorded.

Next, the operation for reproducing moving image data using the person key is performed according to the procedure shown in FIG.
That is, referring to FIG. 9, when the heel data of the person key is searched as a search key (S 20), all the heel images associated with the person key are called from the heel storage unit 50 and listed on the display device 8. It is displayed (S21). For example, as shown in FIG. 10, this list display can display a thumbnail image (person representative image with a tracking line) as a thumbnail.

  When a desired cocoon image is selected from the list-displayed cocoon images (S22), the moving image data recorded by the recording device 2 is reproduced on the display device 8 from the recording position associated with the cocoon image. (S23). In other words, when a habit image is selected, the moving image data is jumped and reproduced from the start frame indicated by the moving image position associated with the ivy image, and the reproduction is stopped at the end frame indicated by the moving image position (see FIG. 11). ). As a result, the moving image data is reproduced on the display device 8 for the scene in which the person of the selected bag image appears.

  As described above, according to the human behavior search apparatus X according to the present embodiment, the position of moving image data in which a person exhibiting a specific behavior is recorded can be easily searched, and playback from a scene where the person appears can be easily performed. And can be done quickly. In this way, by creating a person key, it is possible to easily reproduce a scene in which a person who shows a specific behavior appears from moving image data recorded for a long time. Therefore, it is possible to greatly reduce the work load required to analyze human behavior based on the moving image data.

The present invention is not limited to the above embodiment, and various design changes can be made.
For example, in the above-described embodiment, the captured image 1 from the camera 9 is input to the image processing device 3 and the person key is created by the bag generating device 5. However, the moving image data (imaging image) already recorded by the recording device 2 is used. The person 1 may be generated by inputting the image 1) into the image processing apparatus 3 and using the bag generating apparatus 5 as described above.

  The recording device 2 starts recording when a person appears in the captured image 1 and stops recording when the person goes out of the captured image 1 based on a command from the image processing device 3. A configuration may be adopted. In this case, the storage capacity of the image recording unit 21 that records moving image data can be reduced.

  In addition, although the above-described haze image is a still image extracted from one image frame, a plurality of continuous image frames may be extracted and used as a haze image that can be displayed as a moving image. In this case, for example, a moving image may be created by extracting a predetermined number of captured images 1 after passing through the representative image extraction line L.

  In addition, only the start frame is recorded at the recording position, and the playback of the video data by selecting the 栞 image is performed only by jump playback from this start frame, and is not stopped, and the playback is stopped by the operator's intention. You may do it.

  Moreover, the human behavior search device X according to the present invention is not limited to the ATM room S, but can be applied to human behavior observation in any monitoring place such as a convenience store, a department store, a condominium, and a house.

  In addition, the recording device 2, the operation means 7, the image processing device 3, the behavior determination device 4, the bag generation device 5, the bag storage unit 50, the playback control device 6, and the display device 8 may be configured by independent dedicated units. Alternatively, all or a part of these may be configured by a general-purpose computer such as a single personal computer.

It is a schematic diagram which shows a mode that the ATM room (monitoring area | region) which installed 3 ATM machines (M1-M3) side by side is always image | photographed from the upper direction with one camera. It is a block diagram which shows the structure of the person action search apparatus by this Embodiment. It is a schematic diagram for demonstrating the circular Fourier method used in the image processing of person recognition in the human action search device by this Embodiment. It is a figure for demonstrating that the shading direction of an image is shown by performing a circular Fourier method, The figure (a) shows a mode that fundamental wave Fourier transformation is performed to the outline part of a person image in circular shape. FIG. 5B shows a waveform graph when the fundamental wave Fourier transform is executed. It is a schematic diagram for demonstrating the vector focus method used at the time of the image processing of a human recognition, The figure (a) shows a mode when a standard human-type image is arrange | positioned on an image, The figure (b) is an image. The situation when the person to be identified is shown above is shown. It is a flowchart which shows operation | movement when producing a person key in the person action search device by this Embodiment. It is a schematic diagram for demonstrating operation | movement when creating eyelid data in the person action search device by this Embodiment. It is a schematic diagram which shows the person key produced in the person action search device by this Embodiment. It is a flowchart which shows operation | movement when reproducing | regenerating moving image data using a person key in the person action search apparatus by this Embodiment. It is a schematic diagram which shows a mode that the haze image was displayed as a list as a thumbnail, when searching with a person key as a search key in the person action search device by this Embodiment. It is a schematic diagram which shows the relationship between the captured image frame and the reproduction | regeneration image frame in the human action search device by this Embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Captured image 2 Recording device 3 Image processing device 4 Behavior determination device 5 栞 Generation device 6 Playback control device 7 Operation means 8 Display device 9 Camera (imaging device)
21 Image recording unit 31 Human recognition unit 32 Behavior measuring unit 41 Behavior determination reference 50 栞 storage unit 51 recording position recording unit 52 movement locus recording unit 53 栞 image creation unit 55 person key creation unit 54 栞 data creation unit 61 search unit 62 display Editing unit 100 ATM machine A Measurement area G Virtual center of gravity h Person image L Representative image extraction line S ATM room (monitoring area)

Claims (3)

A human behavior search apparatus for monitoring and recording video data of a person to search for a person who exhibits a predetermined behavior and to reproduce it from a scene where the person appears,
A recording device that causes the image recording unit to record the captured image of the monitoring area captured by the imaging device as moving image data;
Recognize each person on the captured images continuously sent from the imaging device or the image recording unit, track the movement of the virtual center of gravity of the person, and grasp the behavior of the person at a specific location in the monitoring area. In order to do so, the virtual center of gravity of the person appearing on the captured image in the measurement area preset within a certain range on the captured image enters the measurement area preset on the captured image and then moves outside the measurement area. An image processing device for acquiring the behavior information of the person until leaving,
From the behavior information of the person acquired by the image processing device, a behavior determination device that determines the behavior of the person based on a predetermined behavior determination criterion;
栞 data creation unit for creating 栞 data indicating the behavior of the person from the result of behavior determination by the behavior determination device,
A movement trajectory recording unit that records a tracking line indicating the movement trajectory of the virtual center of gravity of the person tracked by the image processing device;
Preset representative image extraction line at a position passing through the person in the monitoring area on the captured image, the virtual center of gravity is the representative image of the person appearing on the captured image from the captured image by the image processing apparatus The captured image when passing through the extraction line for the first time is extracted as a representative image in which the person's face is clearly shown, and the person's eyelid image obtained by combining the representative image with the tracking line recorded in the moving locus recording unit.栞 Image creation part to create
A recording position recording unit for recording the recording position of the moving image data in which the person is recognized;
A person key creating unit for associating the above-mentioned 栞 image, 栞 data and recording position by the person, and storing the person key in the 栞 storage unit;
A search unit for searching for the heel data of the person key, calling the heel image associated with the heel data from the heel storage unit, and displaying the list of the heel images on a display device;
A display editing section for causing the display device to reproduce the moving image data recorded by the recording device by selecting the thumbnail image displayed in the list from the recording position associated with the thumbnail image;
The above data is
(1) movement order indicating whether there is movement between the different measurement areas of the person,
(2) Movement speed indicating the average speed of the person in the measurement area,
(3) Stop time indicating the time the person was stopped in the measurement area, and
(4) A person behavior search device configured by all combinations of the items (1), (2), (3), and (4) of the stay time indicating the time during which the person stayed in the measurement area. In the human action search device according to claim 1,
The above heel data
(5) A companion showing whether the person was moving in parallel within the measurement area, and
(6) A person who has one of the items (5) and (6) or all of these items (5) and (6) indicating the number of people who were present in the measurement area at the same time, including that person. Behavior search device. In the human action search device according to claim 1 or 2,
The image processing apparatus performs a fundamental wave Fourier transform on pixel values of each pixel on the circumference of a circle having the processing point as a center point for each processing point arranged at equal intervals in the captured image. The outline of the person image is recognized by a normal vector for each processing point from the obtained phase, and the person image is recognized as the virtual centroid point based on the normal vector, and the virtual centroid point is determined over time. A human behavior search device including a human recognition means for tracking.

Images