JP6440749B2 - Surveillance image processing apparatus and surveillance image processing method - Google Patents

Description

  This embodiment relates to a monitoring image processing apparatus and a monitoring image processing method.

  A monitoring device using a monocular camera has been developed. A monitoring device has also been developed in which a 360 ° lens (fisheye lens) is attached to a monocular camera and the imaging field of view is enlarged. In addition, a monitoring apparatus that uses a fish-eye lens to capture, for example, a state of a conference room to acquire a wide-angle visual field monitoring image, expand the wide-angle visual field monitoring image into a panoramic image, and perform face detection using the expanded panoramic image Has also been developed.

JP, 2015-19162, A JP 2016-25516 A JP 2016-39539 A

  Further improvement is demanded for a monitoring apparatus that uses a 360 ° lens (fisheye lens) to expand the imaging field of view.

  For example, there is a demand for the tracking process of a specific person to be accurate and resistant (robust). The tracking process is relatively stable when the human face is facing the camera. This is because when a photographed person's facial feature pattern is compared with a comparison pattern recorded in the monitoring device (usually a facial feature pattern seen from the front is recorded), there is a high probability that they are similar. It is.

  However, there are cases where the face of the person to be monitored cannot be captured stably from the front. For example, this is a case where the person being tracked changes his / her direction and takes a picture of the person's face from the side or behind the camera. When such a state lasts for a relatively long time, the monitoring device cannot track a specific person. That is, the target person cannot be tracked (lost state).

  Therefore, according to the present embodiment, an object is to provide a monitoring image processing apparatus and a monitoring image processing method in which the tracking ability of an object to be tracked (for example, a person) is improved.

According to an apparatus which is an example of an embodiment, the apparatus includes an imaging unit, a data processing device, and a display device, and the imaging unit inputs wide-area image data obtained by capturing a space including an object to the data processing device. There,
The data processing device includes:
A first image processing unit that tracks individual features (faces) of the object, a second image processing unit that tracks movement features of the objects, and a first attention including an image of the individual features (faces) A noticeable image output unit that cuts out an image (face image) and supplies it to the display device,
When the first image processing unit detects the image of the individual feature (face), the first image of interest is extracted from the wide area image data based on rectangular information surrounding the image of the individual feature (face). Generating first image area information for cutting out an image of (face image);
The second image processing unit tracks a second attention image (moving body image) based on the motion detection information of the image when the first image processing unit loses tracking of the image of the individual feature (face). And generating second image area information capable of cutting out the second attention image (moving body image) from the wide area image data,
While the first image processing unit is tracking the individual feature (face), the second image processing unit is configured to extract the second image for cutting out the second attention image (moving body image). As the area information, the second image of interest (moving body image) is virtually tracked based on the first image area information generated by the first image processing unit,
The attention image output unit cuts out an image of the individual feature (face) from the wide area image data based on at least the first image area information and supplies the image to the display device.

According to another apparatus of the embodiment, the apparatus includes an imaging unit, a data processing device, and a display device, and the imaging unit inputs wide-area image data obtained by capturing a space including an object to the data processing device. And
The data processing device includes:
A first image processing unit that tracks individual features (faces) of the object, a second image processing unit that tracks movement features of the objects, and a first attention including an image of the individual features (faces) A noticeable image output unit that cuts out an image (face image) and supplies it to the display device,
When the first image processing unit detects the image of the individual feature (face), the first image of interest is extracted from the wide area image data based on rectangular information surrounding the image of the individual feature (face). Generating first image area information for cutting out an image of (face image);
The second image processing unit tracks a second attention image (moving body image) based on the motion detection information of the image when the first image processing unit loses tracking of the image of the individual feature (face). And generating second image area information capable of cutting out the second attention image (moving body image) from the wide area image data,
While the second image processing unit is tracking the motion feature, the first image processing unit uses the first image area information for cutting out a first attention image (face image) as the first image area information. The individual feature (face) is virtually tracked based on the second image area information generated by the second image processing unit,
The attention image output unit cuts out an image of the individual feature (face) from the wide area image data based on at least the first image area information and supplies the image to the display device.

FIG. 1 is a block diagram schematically showing an example of an omnidirectional monitoring image display processing system in which the monitoring image processing apparatus of the present embodiment is used. FIG. 2 is a block diagram illustrating an example in which the block configuration illustrated in FIG. 1 is further embodied. FIG. 3 is an explanatory diagram schematically showing the configuration and operation of the monitoring image processing apparatus of the present embodiment. FIG. 4 is a diagram illustrating an example of a state transition route when a tracking process based on face detection and a tracking process based on motion detection are performed in the monitoring image processing apparatus according to the present embodiment. FIG. 5 is a flowchart schematically showing the operation of the apparatus according to the present embodiment. FIG. 6 is a flowchart showing more specifically the tracking processing step for tracking the face rectangular image shown in FIG. FIG. 7 is a flowchart showing a continuation of the flowchart of FIG. FIG. 8 is a flowchart showing more specifically the tracking processing step for tracking the moving object feature shown in FIG. FIG. 9 is an explanatory diagram of an omnidirectional high-definition monitoring image and a panoramic image in the monitoring image processing apparatus of the present embodiment. FIG. 10 is a diagram illustrating an example in which an image is displayed on the display device by the monitoring image processing apparatus of the present embodiment. FIG. 11 is a diagram illustrating an example of an image displayed on the display device when tracking of individual features (faces) in the first image processing unit A1 is in a lost state.

  Hereinafter, embodiments will be described with reference to the drawings. FIG. 1 shows an omnidirectional monitoring (also referred to as wide-angle visual field monitoring) image display processing system to which a monitoring image processing apparatus according to an embodiment is applied. In the embodiment, a monocular camera (one ip camera) 101 (which may be referred to as an imaging unit) having a pixel number (high definition pixel) having a number of pixels of 1.3 mega (M) pixels or more is used. A 360 ° lens (fisheye lens) is attached to the monocular camera 101, and the monocular camera 101 can acquire an omnidirectional high-definition monitoring image.

  The acquired data stream of the omnidirectional high-definition monitoring image is received by the stream receiving unit in the data processing apparatus 200 </ b> A, captured by the capture 201 in units of frames, and input to the image buffer memory 202. When viewed on the display screen, the omnidirectional high-definition monitoring image is displayed with an imaging region arranged in an elliptical shape with respect to the image frame. The omnidirectional high-definition monitoring image is an image having a high resolution of 2560 × 1920 pixels or more.

  FIG. 1 is a basic block diagram of an embodiment. According to the embodiment, the image capturing unit 101, the data processing device 200A, and the display device 321 are devices that input wide-area image data obtained by capturing the space including the object to the data processing device 200A. Here, the data processing apparatus cuts out and displays a target image, a first image processing unit A1 that tracks individual features (faces) of the object, a second image processing unit A2 that tracks the movement features of the object, and the like. An attention image output unit A3 to be supplied to the device 321 is provided.

  Then, when the image of the individual feature (face) is detected, the first image processing unit A1 surrounds the image region where the image of the individual feature (face) is located while tracking the individual feature (face). First rectangular information (which may be referred to as face detection rectangular coordinate information or first image area information) is generated. In addition, when the first image processing unit A1 loses tracking of the individual feature (face), the second image processing unit A2 calculates the motion feature based on the motion detection information of the image existing at the position where the tracking is lost. While tracking, second rectangular information (also referred to as moving object detection rectangular coordinate information or second image area information) surrounding the image area where the motion feature is located is generated.

  The attention image output unit A3 cuts out the attention image for output from the wide area image data using at least the first rectangular information (face detection rectangular coordinate information or first image area information) and supplies the image to the display device 321. The second rectangular information (moving object detection rectangular coordinate information or second image area information) is not always required to be used.

  The attention image output unit A3 gives priority to the first rectangular information (face detection rectangular coordinate information or first image region information) when the first image processing unit A1 detects the image of the individual feature (face). To use.

  Various output display forms of the second attention image are possible. For example, when only the first attention image (face image) is cut out and output, and the second attention image (moving body image) can be cut out, for example, a message “lost” may be output to the display device 321. Alternatively, a moving body image (an image without a face or a pseudo face) may be displayed in a rectangular frame.

  The first image processing unit A1 refers to the second rectangular information generated by the second image processing unit A2 when the individual feature (face) is not tracked (lost). Then, the individual feature (face) may be tracked virtually. At this time, the first image processing unit A1 may perform virtual tracking using the first rectangular information indicating an area slightly enlarged from the area represented by the second rectangular information. The degree of enlargement is such that, for example, a rectangular image is enlarged several tens of pixels vertically and horizontally.

  Conversely, when the first image processing unit A1 is tracking the individual feature (face) (when it is not lost), the second image processing unit A2 refers to the first rectangular information. Thus, the moving object may be virtually tracked. Also at this time, when the first image processing unit A1 loses the tracking of the individual feature (face), the second image processing unit A2 generates the second rectangular information while continuously tracking the moving object. To do. At this time, the second image processing unit A2 may perform virtual tracking using the second rectangular information indicating a region slightly enlarged from the region represented by the first rectangular information. The degree of enlargement is such that, for example, a rectangular image is enlarged several tens of pixels vertically and horizontally.

  The data processing device 200A includes a blocking luminance expansion unit 320. The blocking luminance expansion unit 320 performs a pixel-based blocking luminance expansion process on the target image (for example, a face rectangular image obtained by extracting a region including a face in a rectangular shape), and performs a bright and dark pixel level. Make corrections. By the blocking luminance expansion processing by the blocking luminance expansion processing unit 320, a greatly different portion of brightness and darkness is corrected at the pixel level with respect to the face rectangular image to be monitored, so that the face rectangular image to be monitored becomes an image having a dynamic range that is easy to monitor. It is corrected. The details of this blocking luminance expansion processing technique are disclosed in Japanese Patent Nos. 5486963, 5866146, and 5888670 by the same applicant.

  The display device 321 displays an omnidirectional monitoring image captured from the data processing apparatus 200A, and a tracking face rectangular image is displayed in the omnidirectional monitoring image. Thus, the user can easily confirm the movement position of the target image (face image) in the imaging region.

  FIG. 2 is a block diagram showing the configuration of FIG. 1 more specifically.

  The omnidirectional high-definition monitoring image (which may be referred to as wide-area image data) in the image buffer memory 202 is converted into an image having a resolution of VGA: 640 × 480 pixels by the resizing unit 11, and the above-described blocking luminance expansion unit 320 receives it. Entered.

  Further, the omnidirectional high-definition monitoring image (wide-area image data) in the image buffer memory 202 is converted into an image having a maximum resolution of 1920 × 1080 pixels by the resizing unit 11 in the first image processing unit A1 to generate a panoramic image. Input to the unit 203.

  The image input to the panorama image generation unit 203 is converted into a panorama image in which the entire periphery is converted into a front image. In this case, the elliptical image area is divided into, for example, two vertically and displayed as the first panorama image and the second panorama image. These panoramic images are images having a maximum resolution of 1920 × 1080 pixels due to the previous resizing, and are smaller than the number of pixels of the omnidirectional high-definition monitoring image, thereby reducing the burden of data processing.

  The panorama image of the panorama image generation unit 203 is a material for performing a face detection process for detecting an object to be detected or an attention image (for example, a human face) and a tracking process for the detected face in the image tracking processing unit 204. .

  When the image tracking processing unit 204 detects the face of a person, the face image information, attribute information (estimated age, sex, presence / absence of mask, presence / absence of glasses, etc.), face rectangle coordinate information surrounding the detected face area with a rectangle Is transmitted to the object information buffer 206.

  The attribute information is extracted from the face image information. However, the present invention is not limited to this, and the attribute information includes various information such as estimated age, gender, presence / absence of mask, presence / absence of glasses, person's clothes, clothes shape, accessories, color, hairstyle, etc. (ID) may be associated.

  The detected face image information is stored in the buffer memory 205a, the attribute information is stored in the buffer memory 205b, and face rectangle coordinate information indicating the detected face area (area including the face) is stored in the buffer memory 205c.

Here, the face rectangle coordinate information indicating the detected face area is obtained from the panoramic image. In addition, the face image in the area surrounded by the face rectangle coordinate information has a resolution of 1920 × 1080 pixels at the maximum. Therefore, the face rectangular coordinate information and resolution at this stage do not match the face rectangular coordinate information and resolution for the omnidirectional high-definition monitoring image in the image buffer memory 202. Therefore, when the face image is stored in the buffer memory 205 , the coordinate conversion processing unit 206 converts the face image position coordinates on the panoramic image into the face image position coordinates of the omnidirectional high-definition monitoring image. Next, the detection rectangular coordinate information generation unit 207 performs face detection rectangular coordinate information (coordinate information of the frame surrounding the face) based on the face image position coordinates of the omnidirectional high-definition monitoring image generated by the coordinate conversion processing unit 206. Is generated. The face detection rectangular coordinate information is given to the attention area image creation unit 315 via the object tracking processing unit 314 or directly. The object tracking processing unit 314 selects any face or moving body detection rectangular coordinate information output from the first image processing unit A1 or the second image processing unit A2 according to the situation, and an attention area image generation unit 315.

  Now, it is assumed that the attention area image creation unit 315 selects the face detection rectangular coordinate information from the first image processing unit A1. Then, the attention area image creation unit 315 is detected by the image processing unit 204 from the omnidirectional high-definition monitoring image stored in the image buffer memory 202 based on the face detection rectangular coordinate information generated by the detection rectangular coordinate information generation unit 207. A high-definition face rectangular image corresponding to the face image (referred to as a face rectangular image or a noticed image) is cut out. Then, a high-definition face rectangular image cut out from the omnidirectional high-definition monitoring image is corrected. The correction process is to normalize a distorted image into an image with an aspect (4: 3 or 16: 9) of the display device 321. The corrected face rectangular image is stored in the object image buffer memory 316.

  Note that the face detection rectangular coordinate information generated by the detection rectangular coordinate information generation unit 207 is used as data for tracking the detection target surrounding the face of the attention area and generating a detection result image in the image buffer memory 202. The face detection rectangular coordinate information may be referred to in the second image processing unit A2.

  The high-definition face rectangular image stored in the object image buffer memory 316 is resized to a screen size suitable for the pixel configuration of the display device 321 by the resizing unit 14 and sent to the blocking luminance expansion processing unit 320. The face rectangular image subjected to the blocking luminance expansion processing by the blocking luminance expansion processing unit 320 is displayed on the display device 321 as a monitoring image.

  The face rectangular image displayed on the display device 321 is a high-density-based monitoring image that is cut out from the omnidirectional high-definition monitoring image and corrected to an image with a dynamic range that can be easily monitored by blocking luminance expansion processing.

  As described above, the face rectangular image to be monitored is displayed on the display device 321 as a high-quality image with a dynamic range that is easy to monitor (see 303 in FIG. 10).

  Furthermore, the omnidirectional high-definition monitoring image stored in the image buffer memory 202 is resized by the resizing unit 13 to a size that matches the screen of the display device 321. The resized omnidirectional monitoring image has a resolution of, for example, 640 × 480 pixels. This omnidirectional monitoring image is subjected to luminance expansion processing by the blocking luminance expansion unit 320 and is synthesized so as to overlap the previous face rectangular image. This omnidirectional high-definition monitoring image is effective for easily determining to which position the object being tracked (in this embodiment, a human face) has moved in the omnidirectional monitoring region ( (See 302 in FIG. 10).

  The face image and attribute information stored in the object information buffer 205 may be input to the display device 321 and additionally displayed (see 311 and 312 in FIG. 10). The face image information includes face image information stored in the buffer memory 205a, and attribute information stored in the buffer memory 205b (estimated age, sex, presence / absence of mask, presence / absence of glasses, etc.) are linked to each other in the object Accumulated in the information buffer 205. These accumulated data can be used in subsequent face image recognition processing and tracking processing.

  In the configuration of FIG. 2, the image buffer memory 202, the resizing unit 11, and the panoramic image generating unit 203 reduce the number of bits and resize the wide-angle visual field monitoring image obtained by imaging the space with a visual angle of 180 ° or more, It can be referred to as a first conversion means for converting the image into at least two and converting it into a first panorama image and a second panorama image. The image processing unit 204 and the object information buffer 205 can be referred to as detection and tracking means for detecting and tracking at least a subject of interest from the first panorama image or the second panorama image.

  Furthermore, the coordinate conversion processing unit 206, the detection rectangular coordinate information generation unit 207, the attention area image creation unit 315, and the object image buffer memory 316 convert the coordinate data on the panoramic image of the subject of interest into the coordinates on the wide-angle visual field monitoring image. It can be referred to as attention image cutout means for converting into data and cutting out the attention image of the subject of interest from the wide-angle visual field monitoring image. The resizing unit 14, the blocking luminance expansion unit 320, the display device 321 and the like can be referred to as a display unit that displays at least the target image cut out by the target image cutout unit.

  The apparatus further includes a second image processing unit A2. The omnidirectional high-definition monitoring image in the image buffer memory 202 is converted into an image having a resolution of 320 × 240 pixels by the resizing unit 12 and input to the image tracking processing unit 311. The image tracking processing unit 311 detects and tracks a moving object in the image, and generates moving object detection information. The moving object detection information is converted in the moving object detection rectangular coordinate information unit 312 into rectangular coordinate information (moving object detection rectangular coordinate information) surrounding the moving object being tracked. This moving object detection rectangular coordinate information is used in the image buffer memory 202 as data for tracking a detection target surrounding a moving attention area and generating a detection result image. That is, the moving body detection rectangular coordinate information is used when a moving body image with movement is cut out from the omnidirectional high-definition monitoring image in the image buffer memory 202 and stored in the object image buffer memory 316.

  Also in this case, the attention area image creation unit 315 corrects a high-definition face rectangular image cut out from the omnidirectional high-definition monitoring image. The correction process is to normalize a distorted image into an image with an aspect (4: 3 or 16: 9) of the display device 321. The corrected face rectangular image is stored in the object image buffer memory 316.

  Through the processing described above, the present apparatus can provide a monitoring image processing apparatus and a monitoring image processing method that improve the tracking ability of an object to be tracked (for example, a person).

  FIG. 3 is an explanatory diagram schematically showing the configuration and operation of the above-described apparatus. The data processing device 200A receives the high-definition monitoring image from the imaging unit 101 by the stream receiving unit (S401). The high-definition monitoring image received by the stream receiving unit is captured by the capture unit in units of frames (S403). Note that the high-definition monitoring image received by the stream receiving unit may be captured in units of frames (S403) after being temporarily stored in the storage medium or being stored (S402).

  Next, the following processing is executed in the data processing device 200A. That is, the captured high-definition monitoring image is preprocessed (S404). This preprocessing corresponds to the resizing and panoramic image generation described in FIG. The preprocessed panoramic image is subjected to tracking processing based on face detection by image processing and tracking processing based on motion detection (S404, S405). Information obtained in the tracking process based on face detection includes a face rectangle image, attribute information, face rectangle detection coordinate information, and the like. The face detection rectangular coordinate information of the panoramic image is coordinate-converted to detection rectangular coordinate information corresponding to the coordinates of the omnidirectional high-definition image. Also, attribute information is extracted.

  Further, in this apparatus, in addition to the tracking process based on the face detection, the second image processing unit A2 also executes a tracking process based on the moving object detection. Also in this tracking process, moving object detection rectangular coordinate information for cutting out the target image is generated (S406). This moving object detection rectangular coordinate information can be referred to in the first image processing apparatus A1 that performs face detection. With this reference, the first image processing apparatus A1 can virtually track the target image and can shorten the time from the actual face detection to the face determination. That is, the first image processing apparatus A1 has a robust face detection capability.

  The object tracking process is performed as described above (S407). Next, in the attention image output unit A3, attention is drawn from the wide area image data based only on the first rectangular information (face detection rectangular coordinate information), the second rectangular information (moving object detection rectangular coordinate information), or the first rectangular information. The image is cut out, and correction processing of the cut out attention image is performed. The corrected attention image is subjected to blocking luminance expansion processing (S409) and supplied to the display device 321. As described above, tracking processing of an object (attention image) is performed by face detection or moving object detection.

  Here, one of face detection rectangular coordinate information based on tracking processing based on face detection and moving body detection rectangular coordinate information based on tracking processing based on moving object detection is used for extracting the target image. A typical state transition will be described below. This example is an example and is not necessarily limited to this state transition.

  FIG. 4 is a diagram illustrating an example of a state transition route when a tracking process based on face detection and a tracking process based on motion detection are performed in the monitoring image processing apparatus according to the present embodiment.

  Assume that the face detection / object detection operation state J1 is in effect. Here, when a face (attention image) is detected, the process proceeds to a tracking operation state J2 with a face rectangle via a path P1. As long as a face is detected (path P2), this tracking operation state J2 is maintained.

  When the face is not detected in the state J2, the process proceeds to the lost operation state J3 via the path P3. Even in the lost operation state J3, when a face is detected, the operation proceeds to the tracking operation state J2 by the face rectangle. However, in the lost motion state J3, when a face is not detected and an object (moving body) is detected, the operation moves to a tracking motion state J4 using a moving body / stopping object rectangle. In this state J4, after detecting a moving object, even if the moving object has stopped for a certain period, it is determined as moving object detection, and the moving object tracking operation can be maintained.

  If a face is detected in the tracking operation state J4 using the moving object / stop object rectangle, the process proceeds to the tracking operation state J2 using the face rectangle via the path P6. However, when the moving object (object) cannot be detected in the tracking operation state J4 using the moving object / stopping object rectangle, the process proceeds to the lost operation state J3 via the path P7.

  The lost operation state J3 maintains the lost operation state J3 until a predetermined time passes through the path P8. However, if a certain time elapses without the face / object being detected, the process proceeds to the face detection / object detection operation state J1 via the path P9.

  In the face detection / object detection operation state J1, both the face detection operation and the moving object detection operation are performed in parallel. For face detection, rectangle information obtained by moving object detection may be coordinate-converted (that is, converted into a panoramic coordinate system) and used as face detection rectangle information. This use achieves face detection relatively early.

  Further, in this case, the area of the face detection rectangle for face detection is used by being expanded by several percent than the area of the face rectangle during the tracking process. Thereby, face detection can be facilitated even if the face moves left and right or up and down.

  If the face detection is performed from the lost operation state J3 to the face detection / object detection operation state J1, it is determined whether or not the same face as the detected face has been detected in the past frame. If the same face exists, detection data of the same face corresponding to the face image data (for example, face detection rectangular coordinate information, face probability score value, number of detections, latest detection frame number, number of tracking frames, previous detection Update the frame interval etc.). If the same face is not detected in past face detection, new face detection data (for example, initial detection frame number, face ID information, face detection rectangular coordinate information, score value of face likelihood) is created. . In other words, the identification data (ID) attached to the face (object: moving object) that has been tracked up to now is temporarily discarded, and a new ID is assigned to the new face (object) detected in the state J1. Managed.

  FIG. 5 is an operation flowchart describing a schematic operation of the apparatus according to the present embodiment. Processing of the omnidirectional high-definition monitoring image is started. First, tracking processing for tracking a face rectangular image based on face detection is executed (SA1). When the tracking of the face rectangular image is lost (SA2), it is determined whether or not there is object (moving object) detection (SA3). When the object is detected, the tracking process using the moving object / stop rectangle is executed (SA4).

  If any detection (face detection) is detected in the lost motion state and the object detection motion state, the state moves to the state J2 as described with reference to FIG.

  6 and 7 are flowcharts showing more specifically the tracking process (SA1 in FIG. 5) for tracking the face rectangular image. FIG. 8 shows in detail an operation flow when a moving object is detected and tracked (SA4 in FIG. 5) from the lost state.

  The tracking process is managed in three states: “no tracking”, “tracking”, and “lost”. During the lost, it is monitored whether or not the lost state continues for a specified time. If the lost state continues for a specified time, a “no tracking” state is set.

  First, it is determined whether or not the immediately preceding tracking state is “no tracking” (SB1). If it is “no tracking”, it is determined whether or not new face detection rectangle information based on face detection exists. If there is new face detection rectangle information, the new face detection rectangle information is stored as the current tracking position rectangle information.

  Next, it is determined whether or not the acquisition (or creation) of the current tracking position rectangle information is successful (SB4), and if successful, the object ID being tracked is stored in a predetermined table (SB10).

  If it is determined that tracking is in progress in the previous “no tracking” determination in SB1, face detection rectangular information having the same object ID as the object ID corresponding to the existing tracking position rectangular information is acquired (SB5). Next, it is determined whether or not the same face image as the acquired face image exists in the predetermined table based on the acquired face detection rectangle information (SB6).

  If the same face image exists, it is determined whether or not the acquisition (or creation) of the current tracking position rectangle information is successful (SB4). If successful, the object ID being tracked is stored in a predetermined table. (SB10). That is, after that, face images with the same object ID are tracked.

  If it is determined in the previous SB6 that the same face image does not exist, the existing tracking position rectangle information is adjusted, the rectangle position is designated left and right, up and down, the size is expanded, and the enlarged existing tracking position rectangle Acquire face detection rectangle information that overlaps the information. In other words, here, the face may be moved to a slightly shifted position even for the same face, or another face may exist near the existing face. Therefore, when a new face is recognized, the subsequent tracking process is performed using the face image. This enhances the tracking process.

  The object ID being tracked is stored in a predetermined table (SB10), and tracking is executed. Next, it is determined again whether or not the tracking state is “no tracking” (the same determination as the determination of SB11 = SB1). If there is no tracking, the current tracking position rectangle information (information obtained in SB4) is stored in the tracking position rectangle information, and the tracking state is determined as “tracking” (SB12, SB13). After the tracking state is confirmed as “tracking”, the tracking release time is reset (SB21) as shown in FIG.

  When it is determined in SB11 that tracking is in progress, position information obtained by multiplying the current tracking position rectangular information (information acquired in SB4) by a weight is used as existing tracking position rectangular information (in tracking). The tracking position rectangle information is updated. This process is executed, for example, every 10 frames or 5 frames per second, and the existing tracking position rectangle information is multiplied by, for example, 0.3 weight, and the current tracking position rectangle information is multiplied by, for example, 0.7 weight. Thereby, a smooth change from the tracking position rectangular information to the current tracking position rectangular information is obtained.

  Incidentally, in SB4, the current tracking position rectangle information may not be acquired. In this case, as shown in SB22 of FIG. 7, it is determined whether or not the tracking state is “tracking”. If it is “tracking”, the tracking state is set to “lost” (SB23), and the tracking release elapsed time is calculated (SB24). If it is determined in SB22 that the tracking state is not “tracking”, that is, “lost”, the tracking cancellation elapsed time is calculated (SB24).

  If the tracking cancellation elapsed time exceeds the specified time, the tracking state is set to “no tracking” and the processing ends. If the tracking cancellation elapsed time does not exceed the specified time, the processing ends.

  FIG. 8 shows in detail the operation flow when a moving object is detected from the lost state (SA4 in FIG. 5).

  An attempt is made to acquire moving object / stop object rectangle information having the same object ID as the object ID of the existing tracking position rectangle information (SC1). If the same object exists (Yes in SC2), it is determined whether or not the current tracking position rectangle information corresponding to the same object has been successfully acquired (SC4). When the same object does not exist (No in SC2), the existing tracking position rectangle information is enlarged by the specified size to the left, right, up, down, and the moving object / stop object detection rectangle information that overlaps the enlarged existing tracking position rectangle information is acquired. (SC3).

  Next, it is determined whether or not the acquisition (or creation) of the current tracking position rectangle information is successful (SC4), and if successful, the object ID being tracked is stored in a predetermined table (SC5). Next, a position obtained by multiplying the current tracking position rectangular information by the weight is added to the existing tracking position rectangular information to create (update) new rectangular information (SC6). Also in this case, the existing tracking position rectangle information is multiplied by, for example, a weight of 0.3, and the current tracking position rectangle information is multiplied by, for example, a weight of 0.7. Thereby, a smooth change from the tracking position rectangular information to the current tracking position rectangular information is obtained.

  Next, the tracking release time is reset and the process ends (SC8). In the previous SC4, if the current tracking position rectangle information could not be acquired (No in SC4), the tracking cancellation elapsed time is calculated (SC11), and it is determined whether the tracking cancellation time exceeds the specified time ( SC12). If the tracking release time exceeds the specified time, the tracking state is set to “no tracking” (SC13), and the process ends. If not, the process ends.

  FIGS. 9A and 9B are explanatory diagrams of an omnidirectional high-definition monitoring image and a panoramic image. FIG. 9A shows an example of the omnidirectional high-definition monitoring image 250 stored in the image buffer memory 202. The omnidirectional high-definition monitoring image 250 is an image of a room space and includes a person 253 to be displayed. For example, a rectangular area surrounded by a frame 254 is the attention area. When this omnidirectional high-definition monitoring image 250 is converted into a panoramic image, it is divided vertically, for example, at the position indicated by the dividing line 251, and as shown in FIG. 9B, the first panoramic image 261 and the second panoramic image 250 are divided. A panoramic image 262 is generated.

  FIG. 10 is a diagram illustrating an example in which an image is displayed on the display device 321 by the monitoring image processing apparatus of the present embodiment. The whole is the attention area image 303 and includes the face image 301. Further, the resized omnidirectional monitoring image 302 is inserted into the attention area image 303. A person image 302 a corresponding to the face image 301 exists in the omnidirectional monitoring image 302.

  By performing this display, the moving position of the face image 301 in the attention area can be easily confirmed by the omnidirectional monitoring image 302. Furthermore, a frame 302b may be added to the person image 302a included in the omnidirectional monitoring image 302. The frame 302b is generated by, for example, writing face detection rectangular coordinate information used when cutting out an attention image (face image) into the image buffer memory 202.

  Further, attribute information 312 may be displayed in the attention area image 303. Further, the resized face rectangular image 311 may be displayed together with the attribute information 312. The face rectangular image 311 is an image extracted from the object information buffer 206 of FIG.

  According to such a monitoring image processing apparatus, the attention area image 303 is cut out from the omnidirectional high-definition monitoring image stored in the image buffer memory 202 as a high-definition face rectangular image. For this reason, the image quality is higher than that of cutting out from the panorama converted image. Further, since the blocking luminance expansion processing is performed, the image quality is improved.

  Next, the resized omnidirectional monitoring image 302 is inserted into the attention area image 303, and the person image 302 a corresponding to the face image 301 exists in the omnidirectional monitoring image 302. For this reason, it is possible to easily confirm (monitor) which position the person of the face image 301 has moved in the monitoring space. Reliability and performance as a monitoring device are improved. In addition, since the attribute information is also displayed, the performance of the monitoring function is improved.

  In the image to be displayed, the attribute information 312 and the resized face rectangular image 311 are not necessarily displayed.

  FIG. 11 shows an example of an image displayed on the display device 321 when the attention image tracking in the first image processing unit A1 is in a lost state. In the attention area image 303 that has been displayed during the face image tracking so far, for example, a face image of a person is masked. Then, a state in which the moving object is being tracked by the second image processing unit A2 is indicated by a moving object tracking frame 302c (that is, a mark) in the omnidirectional monitoring image 302. At this time, an arrow indicating the moving direction of the moving object may be displayed.

  In the above embodiment, when face detection is not obtained, the movement of a person who seems to be a target person is tracked. However, the person who looks like the target person may be tracked while checking the attribute information (color, shape, accessory, etc.), and further, the person who seems to be the target person can be tracked by combining this attribute information and the motion information. May be. That is, when a moving object is tracked, not only motion information but also single or plural attribute information (cloth color, hat, etc.) may be used in combination. At this time, the attribute information used for tracking may be displayed by blinking or changing the color.

  Furthermore, a plurality of human faces may be detected. In this case, when tracking a specific person (one person), it is preferable to track by combining attribute information. For example, it is assumed that the attribute information of a specific person is information with long hair and wearing red clothes. When the face detection is lost and the moving object having the same attribute information is tracked on the motion detection side, the probability that the face is detected again increases when the face is detected again.

  In the embodiment, the individual feature is a face and the face image is tracked. However, the tracking target is not limited to a face, but can be applied to a specific mark, a national flag, a specific design, or a specific light emitting object.

  The embodiment has been described as an apparatus that acquires an image obtained by imaging 360 ° in all directions, but may be an apparatus that acquires an image obtained by imaging an orientation of 180 ° or more. Further, the image cutout and the coordinate information are described as being rectangular, but the present invention is not limited to this, and various shapes may be used. For example, when there are a plurality of persons to be detected, the shape of the cutout frame of each face image may be different.

  Although several embodiments of the present invention have been described, these embodiments have been presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof. Furthermore, in each constituent element of the claims, even when the constituent element is expressed in a divided manner, when a plurality of constituent elements are expressed together, or when they are expressed in combination, they are within the scope of the present invention. In addition, when the claims are expressed as control logic, when expressed as a program including instructions for causing a computer to execute, and when expressed as a computer-readable recording medium describing the instructions, the apparatus of the present invention is applied. is there.

A1 ... first image processing unit, A2 ... second image processing unit, A3 ... target image output unit, 11, 12, 13, 14 ... resize unit, 101 ... monocular camera (Imaging unit), 200A ... data processing device, 201 ... capture, 202 ... image buffer memory, 203 ... panoramic image generation unit, 204 ... image tracking processing unit, 205 ... object Information buffer 206: Coordinate conversion processing unit 207 ... Face detection rectangular coordinate information unit 311 ... Image tracking processing unit 312 ... Moving object detection rectangular coordinate information unit 314 ... Object tracking processing 315... Attention area image creation unit, 316... Object image buffer memory, 320... Blocking luminance expansion unit, 321.

Claims (15)

An apparatus comprising an imaging unit, a data processing device, and a display device, wherein the imaging unit inputs wide-area image data obtained by imaging a space including an object to the data processing device,
The data processing device includes:
A first image processing unit that tracks individual features (faces) of the object, a second image processing unit that tracks movement features of the objects, and a first attention including an image of the individual features (faces) A noticeable image output unit that cuts out an image (face image) and supplies it to the display device,
When the first image processing unit detects the image of the individual feature (face), the first image of interest is extracted from the wide area image data based on rectangular information surrounding the image of the individual feature (face). Generating first image area information for cutting out an image of (face image);
The second image processing unit tracks a second attention image (moving body image) based on the motion detection information of the image when the first image processing unit loses tracking of the image of the individual feature (face). And generating second image area information capable of cutting out the second attention image (moving body image) from the wide area image data,
While the first image processing unit is tracking the individual feature (face), the second image processing unit is configured to extract the second image for cutting out the second attention image (moving body image). As the area information, the second image of interest (moving body image) is virtually tracked based on the first image area information generated by the first image processing unit,
The attention image output unit cuts out the image of the individual feature (face) from the wide area image data based on at least the first image area information and supplies the image to the display device.
A monitoring image processing apparatus characterized by that. An apparatus comprising an imaging unit, a data processing device, and a display device, wherein the imaging unit inputs wide-area image data obtained by imaging a space including an object to the data processing device,
The data processing device includes:
A first image processing unit that tracks individual features (faces) of the object, a second image processing unit that tracks movement features of the objects, and a first attention including an image of the individual features (faces) A noticeable image output unit that cuts out an image (face image) and supplies it to the display device,
When the first image processing unit detects the image of the individual feature (face), the first image of interest is extracted from the wide area image data based on rectangular information surrounding the image of the individual feature (face). Generating first image area information for cutting out an image of (face image);
The second image processing unit tracks a second attention image (moving body image) based on the motion detection information of the image when the first image processing unit loses tracking of the image of the individual feature (face). And generating second image area information capable of cutting out the second attention image (moving body image) from the wide area image data,
While the second image processing unit is tracking the motion feature, the first image processing unit uses the first image area information for cutting out a first attention image (face image) as the first image area information. The individual feature (face) is virtually tracked based on the second image area information generated by the second image processing unit,
The attention image output unit cuts out the image of the individual feature (face) from the wide area image data based on at least the first image area information and supplies the image to the display device.
A monitoring image processing apparatus characterized by that . When the first image processing unit detects an image of the individual feature (face), the attention image output unit preferentially uses the first image area information.
The monitoring image processing apparatus according to claim 1 , wherein the monitoring image processing apparatus is a monitoring image processing apparatus. When the first image processing unit loses the tracking of the individual feature (face), the second image processing unit uses the first image region information based on the first image region information. Generating the second image region information having a large region area and starting virtual tracking,
The monitoring image processing apparatus according to claim 1 , wherein the monitoring image processing apparatus is a monitoring image processing apparatus. The data processing device includes:
The display device inputs and displays the first attention image (face image), and inputs and displays the image of the wide area image data resized on the first attention image (face image). ,
The monitoring image processing apparatus according to claim 1 , wherein the monitoring image processing apparatus is a monitoring image processing apparatus. The data processing device includes:
The display device further inputs and displays attribute information related to the first image of interest (face image) .
The monitoring image processing apparatus according to claim 1 , wherein the monitoring image processing apparatus is a monitoring image processing apparatus. The data processing device includes:
When the first image processing unit has lost tracking of the individual feature (face),
The display device inputs and displays the image of the wide area image data, and marks the second image of interest (moving body image) tracked by the second image processing unit as a mark. Display it over the image,
The monitoring image processing apparatus according to claim 1 , wherein the monitoring image processing apparatus is a monitoring image processing apparatus. The imaging unit acquires a high-resolution wide-angle visual field monitoring image obtained by imaging a space with a visual angle of 180 ° or more,
The first image processing unit includes:
A first converting means for reducing the number of bits to a resolution lower than that of the wide-angle visual field monitoring image and resizing the first panoramic image and converting it into at least two, and the first panoramic image or Detection and tracking means for tracking at least the individual features from the second panoramic image, and coordinate data on the first panorama image or the second panoramic image of the individual features, the first on the wide-angle visual field monitoring image. Coordinate conversion means for converting the image area information into the first image area information.
The monitoring image processing apparatus according to claim 1 , wherein the monitoring image processing apparatus is a monitoring image processing apparatus. A monitoring image processing method comprising an imaging unit, a data processing device, and a display device, wherein the imaging unit inputs wide-area image data obtained by imaging a space including an object to the data processing device,
The first image processing unit tracks individual features (faces) corresponding to the object, the second image processing unit tracks the movement features of the object, and the attention image output unit performs first attention image ( Cut out the face image) and supply it to the display device,
When the image of the individual feature (face) is detected, the first attention image (face image) is cut out from the wide area image data based on rectangular information surrounding the image of the individual feature (face) Generating first image region information;
When the first image processing unit loses tracking of the individual features (faces), the motion features are tracked based on the motion detection information of the images, and a second attention image (moving body image) is obtained from the wide area image data. Generating second image area information that can be cut out;
While the first image processing unit is tracking the individual feature (face), the second image processing unit is configured to extract the second image for cutting out the second attention image (moving body image). As the area information, the second image of interest (moving body image) is virtually tracked based on the first image area information generated by the first image processing unit,
Cutting out the first image of interest (face image) from the wide-area image data based on at least the first image area information and supplying it to the display device;
And a monitoring image processing method. A monitoring image processing method comprising an imaging unit, a data processing device, and a display device, wherein the imaging unit inputs wide-area image data obtained by imaging a space including an object to the data processing device,
The first image processing unit tracks individual features (faces) corresponding to the object, the second image processing unit tracks the movement features of the object, and the attention image output unit performs first attention image ( Cut out the face image) and supply it to the display device,
When the image of the individual feature (face) is detected, the first attention image (face image) is cut out from the wide area image data based on rectangular information surrounding the image of the individual feature (face) Generating first image region information;
When the first image processing unit loses tracking of the individual features (faces), the motion features are tracked based on the motion detection information of the images, and a second attention image (moving body image) is obtained from the wide area image data. Generating second image area information that can be cut out;
While the second image processing unit is tracking the motion feature, the first image processing unit uses the first image area information for cutting out a first attention image (face image) as the first image area information. The individual feature (face) is virtually tracked based on the second image area information generated by the second image processing unit,
Cutting out the first image of interest (face image) from the wide-area image data based on at least the first image area information and supplying it to the display device;
And a monitoring image processing method. When the image of the individual feature (face) is detected, the first image area information is preferentially used, and the first attention image (face image) is cut out from the wide area image data and supplied to the display device. To
The monitoring image processing method according to claim 9 or 10, wherein: The imaging unit acquires a high-resolution wide-angle visual field monitoring image obtained by imaging a space with a visual angle of 180 ° or more,
The first image processing unit includes:
The number of bits is reduced and resized to a resolution lower than that of the wide-angle visual field monitoring image, and divided into at least two and converted into a first panoramic image and a second panoramic image,
Tracking at least the individual features from the first panoramic image or the second panoramic image;
Generating the first image area information on the wide-angle visual field monitoring image, the coordinate data on the first panoramic image or the second panoramic image of the individual feature,
The monitoring image processing method according to claim 9 or 10, wherein: An imaging unit, a data processing device, and a display device, wherein the imaging unit includes a space including an object
An apparatus for inputting captured wide-area image data to the data processing apparatus,
The data processing device includes:
A first image processing unit that tracks individual features of the object, a second image processing unit that tracks movement features of the object, and an attention image that cuts out an image of the individual features and supplies the image to the display device With an output section,
The first image processing unit, when detecting the individual feature image, extracts the individual feature image from the wide area image data based on rectangular information surrounding the individual feature image. 1 image area information is generated,
The second image processing unit tracks a moving body image based on motion detection information of an image when the first image processing unit loses tracking of the image of the individual feature, and from the wide area image data, the Generating second image area information capable of cutting out the moving object image;
While the first image processing unit is tracking the individual feature, the second image processing unit uses the first image as the second image area information for cutting out the moving body image. Virtually tracking the moving object image based on the first image area information generated by the processing unit,
The attention image output unit cuts out the image of the individual feature from the wide area image data based on at least the first image area information and supplies the image to the display device .
A monitoring image processing apparatus characterized by that . An imaging unit, a data processing device, and a display device, wherein the imaging unit includes a space including an object
An apparatus for inputting captured wide-area image data to the data processing apparatus,
The data processing device includes:
A first image processing unit that tracks individual features of the object, a second image processing unit that tracks movement features of the object, and an attention image that cuts out an image of the individual features and supplies the image to the display device With an output section,
The first image processing unit, when detecting the individual feature image, extracts the individual feature image from the wide area image data based on rectangular information surrounding the individual feature image. 1 image area information is generated,
The second image processing unit tracks a moving body image based on motion detection information of an image when the first image processing unit loses tracking of the image of the individual feature, and from the wide area image data, the Generating second image area information capable of cutting out the moving object image;
While the second image processing unit tracks the motion feature, the first image processing unit uses the second image processing unit as the first image area information for cutting out the image of the individual feature. Virtually tracking the individual features based on the second image area information generated by the image processing unit,
The attention image output unit cuts out the image of the individual feature from the wide area image data based on at least the first image area information and supplies the image to the display device.
A monitoring image processing apparatus characterized by that . The imaging unit acquires a high-resolution wide-angle visual field monitoring image obtained by imaging a space with a visual angle of 180 ° or more,
The first image processing unit includes:
A first converting means for reducing the number of bits to a resolution lower than that of the wide-angle visual field monitoring image and resizing the first panoramic image and converting it into at least two, and the first panoramic image or Detection and tracking means for tracking at least the individual features from the second panoramic image, and coordinate data on the first panorama image or the second panoramic image of the individual features, the first on the wide-angle visual field monitoring image. Coordinate conversion means for converting the image area information into the first image area information.
The monitoring image processing apparatus according to claim 13 or 14, characterized in that:

Images