JP2013211741A - Image monitoring device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable monitoring without increasing a monitoring load even under an environment in which a number of cameras are installed.SOLUTION: An image monitoring device extracts an attention change region from an image obtained by imaging a monitoring site, generates a change region object obtained by projecting the display change region to an object equivalent in size to the display change region, arranges the change region object at a virtual space position on a site model obtained by expressing the monitoring site as a 3D virtual space, and generates and displays a virtual monitoring image obtained when the site model having the change region object arranged therein is viewed from a virtual camera.

Description

  The present invention relates to an image monitoring apparatus in which a monitor visually monitors captured images of a monitoring location captured by a plurality of monitoring cameras.

  In recent years, image monitoring apparatuses that centrally monitor abnormal states in a plurality of areas at a monitoring location by sequentially receiving captured images captured by a plurality of monitoring cameras and displaying them on a monitoring terminal used by a supervisor are used. ing. Conventionally, in such an image monitoring apparatus using a plurality of monitoring cameras, a technique for dividing a monitor display of a monitoring terminal into a plurality of display sections and individually displaying captured images captured by the plurality of monitoring cameras in each display section. (For example, Patent Document 1).

Japanese Patent Laid-Open No. 05-064200

  However, with the rapid increase in the number of installed surveillance cameras as in recent years, it has been difficult for the supervisor to properly monitor the abnormal state with the monitoring method as in the prior art. In other words, in the prior art, the monitoring person must visually check all the captured images that are divided and displayed to determine whether or not it is in an abnormal state. Therefore, the monitoring load on the monitoring person increases as the number of installed monitoring cameras increases. There was a problem of increasing. Therefore, instead of displaying all the captured images from a plurality of surveillance cameras, only the image area (for example, a target moving object such as a person image) that must be watched is extracted and extracted from the captured images. There has been a need to monitor efficiently by collecting and displaying so that the positional relationship of the image area at the monitoring location can be grasped.

  Therefore, the present invention satisfies the above-described needs by extracting an image area that has to be particularly watched from a captured image, and displaying the image area in an aggregated manner on a place model that represents a monitoring place as a three-dimensional virtual space. With the goal.

In order to achieve the above-described object, an image monitoring apparatus according to claim 1 of the present application includes:
An imaging unit that sequentially acquires captured images of a monitoring location, a location model that represents the monitoring location as a three-dimensional virtual space, and an imaging that associates the installation position, optical axis, and angle of view of the imaging unit with the virtual space A storage unit that stores condition information and virtual viewpoint information including a position, an optical axis, and an angle of view of a virtual camera in the virtual space, and a change area extracted from the captured image is arranged on the place model and the virtual An image monitoring apparatus comprising: an image processing unit that outputs a virtual monitoring image obtained by capturing the location model from a camera; and a display unit that displays the virtual monitoring image,
The image processing unit
Using the location model, the imaging condition information, and the position and size of the change area on the captured image, a virtual space position that is the position of the change area in the virtual space is obtained, and the change area at the virtual space position Object generation means for generating a change area object obtained by projecting the change area onto an object corresponding to the size of
The change area object corresponding to the change area is arranged at the virtual space position on the place model, and the virtual monitoring image is generated using the place model and the virtual viewpoint information and output to the display unit. Virtual surveillance image generation means.

  With this configuration, the image processing unit of the present invention extracts a change area from a captured image acquired from a plurality of imaging units installed at a monitoring location using a conventional technique such as background difference. And the object production | generation means of this invention calculates | requires the position (virtual space position) in the virtual space of each extracted change area | region. Further, the object generation means obtains an object (for example, a plate-like object) corresponding to the size of the change region at the virtual space position, and projects the change region on the object (for example, the change region on the surface of the plate-like object). Find the changed area object (texture mapped). Then, the virtual monitoring image generation means of the present invention arranges the change area object corresponding to each change area at the corresponding virtual space position in the virtual space (on the place model). Further, the virtual monitoring image generation means performs a three-dimensional computer graphics rendering process using the virtual viewpoint information on the place model in which the change area object is arranged, so that a virtual monitoring image when viewed from the virtual camera is obtained. A virtual monitoring image is generated. The display unit of the present invention displays and outputs the generated virtual monitoring image.

  As described above, the image monitoring apparatus of the present invention extracts an image region (for example, a target moving object such as a human image) that has to be particularly watched from a captured image, and displays the monitored place as a three-dimensional virtual space. By consolidating the image areas, it is possible to aggregate and display only the change areas, which are image areas that have to be particularly watched, among the captured images from the plurality of monitoring cameras in one virtual monitoring image. Therefore, since the supervisor can monitor the abnormal state in the monitoring place efficiently and in a bird's-eye view by visually monitoring the virtual monitoring image displayed and output from the display unit, a large scale Even in a monitoring place where a large number of surveillance cameras are installed, such as a shopping center, monitoring can be performed without increasing the monitoring load on the supervisor.

The image monitoring apparatus according to claim 2 of the present application is the image monitoring apparatus of claim 1,
The virtual monitoring image generation means includes
Placed virtual image obtained by placing the change area object in the corresponding virtual space position on the place model and picked up from the virtual camera, and picked up from the virtual camera by placing the change area object in the virtual space position Generating a location-free virtual image, and using the first area of the image region corresponding to the change region in the location-containing virtual image and the second area of the image region corresponding to the change region in the location-free virtual image A visibility rate calculation process for calculating a visibility rate that is a rate at which the change area can be visually recognized from the virtual camera for each virtual camera;
It is characterized in that a selection process is performed for selecting the virtual surveillance image with location generated by the virtual camera with the highest visual recognition rate as the virtual surveillance image.

  With this configuration, the virtual monitoring image generation unit according to the present invention performs the rendering process using the virtual viewpoint information of the virtual camera by arranging the change area object at the virtual space position on the place model as the visibility calculation process. Then, a virtual image including a place model (virtual image with place) is generated. Then, the change area object is arranged at the virtual space position, and rendering processing is performed using the virtual viewpoint information of the virtual camera, thereby generating a virtual image (placeless virtual image) that does not include the place model. Then, an area (first area) of the image area corresponding to the change area in the place-within virtual image and an area (second area) of the image area corresponding to the change area in the place-free virtual image are obtained, A visibility rate, which is a rate at which the change area can be visually recognized from the virtual camera, is calculated for each virtual camera from the ratio with the second area. Then, as the selection process, the virtual monitoring image generation unit determines the virtual camera having the largest viewing rate among the obtained viewing rates, and selects the location-based virtual monitoring image generated by the virtual camera as the virtual monitoring image.

  As described above, the image monitoring apparatus of the present invention extracts the target moving object to be watched from the captured image, displays the target moving object in a three-dimensional virtual space, and displays the captured image as one virtual image. Monitoring can be performed from a monitoring image in a bird's-eye view. However, depending on the positional relationship between the current position of the target moving object and the virtual viewpoint of the virtual camera, it is assumed that the target moving object image is hidden behind the place model when viewed from the current virtual viewpoint of the observer . In this case, in order to continue to monitor the target moving object, the monitor is required to manually move to an appropriate virtual viewpoint. Such a concealment state by the location model is more likely to occur as the location model becomes more complex and the number of target moving objects increases, so the observer needs to frequently move the virtual viewpoint, which is complicated. You will be forced to operate. Therefore, the monitoring load on the observer can be reduced by evaluating the concealment state of the target moving object by calculating the visibility rate and automatically setting the virtual viewpoint according to the visibility rate.

The image monitoring apparatus according to claim 3 of the present application is the image monitoring apparatus according to claim 1 or 2,
The first area is an area of an image region of a place-containing virtual image corresponding to the change region having an image characteristic of a predetermined target moving object, and the second area is an image characteristic of a predetermined target moving object. It is the area of the image area of the place-free virtual image corresponding to the change area having

  With this configuration, a change area that does not have an image feature of the target moving object is not considered when calculating the visibility rate. Therefore, when the image feature of “child” is set as the target moving object, for example, the monitor can change the virtual viewpoint so that the change area having the image feature of the child can be suitably viewed. Thus, the virtual viewpoint can be changed so that the target moving object intended by the observer can be monitored more appropriately.

The image monitoring apparatus according to claim 4 of the present application is the image monitoring apparatus according to claim 2 or 3,
It further has an input unit for receiving operation input by the user,
The virtual monitoring image generation means is characterized by performing the visibility rate calculation process when the operation input is received from the input unit.

  With this configuration, the supervisor can change to a virtual viewpoint with good visibility at the timing he / she intends.

  According to the image monitoring apparatus according to the present invention, by extracting an image area that has to be particularly watched from a captured image and collectively displaying it as a virtual monitoring image, the monitor can efficiently detect an abnormal state in the monitoring place. In addition, it is possible to monitor from a bird's-eye view.

Block configuration diagram showing the overall configuration of the image monitoring device Diagram explaining object generation processing A diagram representing a place model with a change area object A diagram showing an example of a virtual image with place Flow chart showing processing procedure of overall operation by image monitoring apparatus The flowchart which shows the specific process sequence of a virtual monitoring image generation process A diagram representing a place model with a change area object A diagram showing an example of a placeless virtual image A diagram showing an example of a virtual image with place

  Hereinafter, an embodiment in which the inside of a building is used as a monitoring place, and a supervisor such as a security guard monitors the state in the monitoring place with a captured image obtained by taking the monitoring place with a monitoring camera will be described in detail with reference to the accompanying drawings. To do.

  The image monitoring apparatus according to the present invention sequentially acquires captured images obtained by capturing the monitored locations, and performs image processing on the sequentially acquired captured images to generate a monitoring image (virtual monitoring image) when viewed from the virtual camera. By displaying, the monitor can monitor the state at the monitoring place from a virtual viewpoint. In particular, the present embodiment extracts an image area (for example, a target moving object such as a person image) to be watched by a supervisor from captured images from a plurality of surveillance cameras, and arranges the image area on a place model, An image monitoring apparatus including a function of performing image processing so as to generate a virtual monitoring image is provided. Furthermore, in this embodiment, even if the extracted image region is displayed in a state hidden behind the place model in the generated virtual monitoring image, without performing a complicated operation by the monitor, It is an object of the present invention to provide an image monitoring apparatus including a function of performing image processing so as to automatically change a setting to a virtual viewpoint where concealment does not easily occur.

(About the overall configuration of the image monitoring device)
As shown in FIG. 1, the image monitoring apparatus 1 according to the present embodiment includes a monitoring terminal 2 and an imaging apparatus 3.

  The imaging device 3 is a so-called surveillance camera including an imaging element such as a CCD element or a C-MOS element, an optical system component, and the like, and functions as an imaging unit in the present invention. The imaging device 3 is installed on the upper part or ceiling of the indoor wall, and is installed so as to take an image while looking down at the monitoring place. The imaging device 3 captures the monitoring location at predetermined time intervals and sequentially transmits the captured images to the monitoring terminal 2. The time interval at which the captured image is captured is, for example, 1/5 second. In the present embodiment, it is assumed that a plurality of imaging devices 3 are installed in the monitoring place.

  The monitoring terminal 2 is schematically configured to include a storage unit 21, an image processing unit 22, a communication unit 23, a display unit 24, and an input unit 25. The communication unit 23 is a communication interface such as a LAN or USB, and communicates with the imaging device 3. In the present embodiment, captured images transmitted from a plurality of imaging devices 3 are received via the communication unit 23. The input unit 25 is an information input device such as a keyboard, a mouse, a touch panel, or a portable storage medium reading device. The monitor or the like can use the input unit 25 to set information regarding various imaging conditions such as the installation position of each imaging device 3, for example.

  The storage unit 21 is an information storage device such as a ROM, RAM, or HDD. The storage unit 21 stores various programs and various data, and inputs and outputs such information to and from the image processing unit 22. The various types of data include the location model 211, the imaging condition information 212, the virtual viewpoint information 213, and other various types of information used for the processing of the image processing unit 22 (for example, the captured image for each frame acquired by the imaging device 3, the extraction unit) 221 stores a reference image, an extraction threshold value, and the like used to extract a change area from the captured image.

  The place model 211 is coordinate information representing a three-dimensional virtual space including three-dimensional shape data created by modeling an object such as a real world wall, floor, pillar, or fixture existing at the monitoring place. . The three-dimensional shape data in the place model 211 may be created by three-dimensional CAD based on the shape information of the monitoring place, or the data obtained by capturing the three-dimensional shape of the monitoring place using a three-dimensional laser scanner or the like. Also good. The location model 211 created in this way is stored in the storage unit 21 by being set and registered from the input unit 25 by a supervisor or the like.

  The imaging condition information 212 includes installation condition information related to the installation position and optical axis (attitude) of the imaging device 3 at the current time, and field angle condition information related to the focal length, the number of pixels, the pixel size, and the lens distortion. Individually set for each. The imaging condition information 212 is set as a value associated with coordinate information in the virtual space of the place model. Here, the installation condition information regarding the installation position represents the inside of the monitoring place (real space) as a three-dimensional orthogonal coordinate system, and the coordinate value of the reference point whose coordinates are known in the orthogonal coordinate system of the real space is changed from the reference point to This is information expressed as coordinate data calculated using a known technique such as measuring and correcting the relative distance and direction. The installation condition information regarding the optical axis is information regarding the rotation angle of the optical axis of the imaging device 3 with respect to the coordinate axis, and can be obtained from the so-called pan angle and tilt angle of the imaging device 3. The imaging condition information 212 is stored in the storage unit 21 by being set and registered from the input unit 25 by a monitor or the like at the time of initial setting.

  The virtual viewpoint information 213 is information including the position of the virtual camera, the optical axis, and the angle of view in the virtual space, and is stored in the storage unit 21 when the monitor is designated by the input unit 25. The virtual viewpoint information 213 is set as a value associated with coordinate information in the virtual space of the place model. Note that, in this embodiment, as described above, even when the extracted image region is concealed by the place model, the virtual monitoring image is output after the setting is automatically changed to the virtual viewpoint where concealment does not easily occur. It has a function. In order to realize the function, the storage unit 21 stores a plurality of virtual viewpoint information 213 fixedly set in advance, in addition to the virtual viewpoint information 213 set by a designated operation by the supervisor. Suppose that

  The image processing unit 22 is configured by a computer having a CPU and the like, receives an input of a digitized image from the imaging device 3, and performs a captured image reading process and extraction as a series of processes shown in FIGS. An extraction unit 221, an object generation unit 222, and a virtual monitoring image generation unit 223 are included to execute processing, object generation processing, virtual monitoring image generation processing, and output processing.

  The extraction unit 221 performs an extraction process for extracting an area having a luminance change from the captured image acquired by the imaging apparatus 3 as a change area. In the present embodiment, a past captured image in which no moving object exists for each imaging device 3 is stored in the storage unit 21 in advance as a reference image, and a difference value between luminance values of the latest acquired captured image and the reference image is determined. An area that is equal to or greater than a predetermined extraction threshold stored in the storage unit 21 is extracted as a change area. At this time, a captured image of the background of the monitoring place, a captured image acquired in the past, or the like can be appropriately selected and used as the reference image. Further, the extraction unit 221 performs a process of labeling the extracted change area as the extraction process. At this time, when attention is paid to an extraction pixel having a change area, the extraction pixels adjacent to the extraction pixel of interest are regarded as a group of extraction pixel areas. Assign a unique label.

  The object generation means 222 uses the imaging condition information 212 stored in the storage unit 21 and the position and size of the change area extracted by the extraction means 221 on the captured image, and the step of the change area on the place model (virtual space). An object generation process is performed for calculating a position and generating a change area object obtained by projecting the change area onto an object corresponding to the size of the change area at the foot position. Here, the foot position is the coordinate information in the virtual space calculated so as to correspond to the coordinate information of the lower end center of the change area on the captured image, and the change area is assumed to be in contact with the place model. This is the value obtained. Note that the foot position corresponds to the virtual space position in the present invention. Details of the object generation process will be described below with reference to FIG.

  FIG. 2 illustrates an example in which a change area object is generated based on a captured image of the imaging device 3 that captures the ground direction from above the monitoring location. In FIG. 2, reference numerals 211a, 211b, and 211c represent a part of the location model 211, of which 211a represents the floor surface of the building that is the monitoring location, 211b represents the wall surface, and 211c represents the monitoring It shall represent a safe placed in place. In generating the change area object, the object generation unit 222 first reads the installation position (X, Y, Z) of the imaging device 3 from the imaging condition information 212 of the storage unit 21, and the location model 211 corresponding to the position. An optical center O on (virtual space) is obtained. Further, the object generation unit 222 obtains the optical axis on the place model 211 of the imaging device 3 from the optical center O and the installation condition information regarding the optical axis (posture) read from the imaging condition information 212 of the storage unit 21. . Further, the object generation unit 222 reads the imaging condition information 212 such as the focal length f, the actual size of the CCD pixel, the number of vertical and horizontal pixels of the image, and specifications regarding lens distortion from the storage unit 11, and the location of the imaging device 3. A projection plane abcd perpendicular to the optical axis on the model 211 is obtained. A quadrangular pyramid Oa′b′c′d ′ passing through the four vertices of the projection surface abcd from the optical center O is a solid that forms the field of view of the imaging device 3. In FIG. 2, it is assumed that an image area denoted by reference numeral 30 on the captured image projected on the projection surface abcd is a change area extracted by the extraction unit 221.

  Next, the object generation unit 222 obtains the foot position of the change area object on the place model 211 (virtual space) based on the position and size of the change area 30 on the captured image. A straight line connecting the center position 30a of the lower side of the rectangle including the change area 30 and the optical center O is obtained, and an intersection 31a with the place model 211 is obtained on an extension line of the straight line. In other words, the intersection point 31a corresponds to the position of the change area object on the place model 211 (virtual space) corresponding to the position 30a on the captured image of the change area 30, and in this embodiment, this position is referred to as a foot position. In the example of FIG. 2, it can be seen that the change area object corresponding to the change area 30 is in contact with the place model of the safe 211c. In addition, a straight line connecting the center position 30b of the upper side of the rectangle including the change area 30 and the optical center O is obtained. An intersection 31b between the straight line and the straight line perpendicular to the horizontal plane (for example, the floor surface 211a) of the place model 211 from the foot position 31a is on the place model 211 corresponding to the position 30b on the captured image of the change area 30. It corresponds to a position, and in the present embodiment, this position is called a head position.

  Next, the height of the change area object on the place model 211 (virtual space) is obtained from the distance between the foot position 31a and the head position 31b. That is, here, the height of the change area object is obtained on the assumption that the change area object to be obtained stands vertically from the foot position. Then, the width of the change area object on the place model 211 (virtual space) is obtained from the height of the obtained change area object and the aspect ratio of the change area 30. By changing the texture of the image area of the captured image corresponding to the change area 30 to the rectangular plate-like object (thickness is arbitrary) formed of the height and width of the change area object, the change area object is obtained. Is generated. Reference numeral 32 in FIG. 2 represents a change area object corresponding to the change area 30.

  The virtual monitoring image generation unit 223 arranges the change area object 32 on the place model, and uses the place model 211 where the change area object 32 is arranged and the virtual viewpoint information stored in the storage unit 21 to generate a virtual monitoring image. A virtual monitoring image generation process to be generated is performed, and a process of outputting the generated virtual monitoring image to the display unit 24 is performed. Details of the virtual monitoring image generation process will be described below.

  First, in the virtual monitoring image generation process, a process of arranging the change area object 32 generated by the object generation unit 222 on the place model is performed. When the change area object 32 is arranged, the change area object 32 is arranged at a position on the place model 211 (virtual space) according to the foot position of each change area object 32. Note that the change area object 32 is arranged so as to be perpendicular to the location model 211 and the texture-mapped plane faces the direction of the imaging device 3 that has captured the texture-mapped image. FIG. 3 shows an example of the place model 211 in which the change area object 32 is arranged. As illustrated in FIG. 3, the change area objects 32v, 32x, and 32z are arranged on the place model 211. In FIG. 3, the objects represented by reference numerals 34 v to 34 z are objects representing the imaging device 3, and the object represented by reference numeral 35 is an object representing a virtual camera installed based on the virtual viewpoint information 213 stored in the storage unit 21. Each of them is displayed for convenience of explanation of this processing.

  Next, in the virtual monitoring image generation process, the place model 211 in which the change area object 32 is arranged is rendered using the virtual viewpoint information 213 stored in the storage unit 21 to generate a rendered image. . In the present specification, a rendered image generated by rendering the change area object 32 together with the place model 211 by a virtual camera is referred to as a “place-with-virtual image”. FIG. 4 shows an example of a place-within virtual image 42 obtained by imaging (rendering) the place model 211 in which the change area object 32 is arranged from the virtual camera 35 in FIG. In the virtual monitoring image 42 shown in FIG. 4, the image areas corresponding to the reference numerals 42v, 42x, and 42z correspond to the change area objects 32v, 32x, and 32z in FIG.

  Next, in the virtual monitoring image generation process, one location-based monitoring image is selected from the location-based virtual images captured (rendered) by one or a plurality of virtual cameras 34, and the location-based monitoring image is virtually monitored. Output as an image. By the way, in the present embodiment, as described above, even when the image region extracted in the generated virtual monitoring image is concealed by the location model, the setting is automatically changed to a virtual viewpoint that is difficult to conceal. And has a function of outputting a virtual monitoring image. Details of the virtual monitoring image generation processing for realizing the function will be described later.

  The display unit 24 is an information display device such as a display. The display unit 24 displays the virtual monitoring image 42 output from the image processing unit 22. As described above, the image monitoring apparatus according to the present embodiment can collectively display the change area extracted from the captured image as an image area that the monitor must watch in one virtual monitoring image. Therefore, the monitor can visually monitor the virtual monitoring image 42 displayed and output from the display unit 24, so that the abnormal state in the monitoring place can be monitored efficiently and overhead. Even in a monitoring place where a large number of monitoring cameras are installed such as a large shopping center, it is possible to monitor without increasing the monitoring load of the monitoring person.

(About processing executed by the image processing unit 22 of the monitoring terminal 2)
Hereinafter, an example of the flow of processing executed by the image processing unit 22 of the monitoring terminal 2 according to the image monitoring apparatus 1 of the present embodiment will be described in detail with reference to FIG. In the present embodiment, description will be made assuming that five imaging devices 3 are installed in a building that is a monitoring place.

  Prior to the operation, various initial settings such as setting of the imaging condition information 212, setting of the virtual viewpoint information 213, and registration of the location model 211 are performed by the monitor or the like using the input unit 25 of the monitoring terminal 2 (ST1). In the present embodiment, since it is assumed that five imaging devices 3 are installed in a building that is a monitoring place, the imaging condition information 212 is registered by default for each imaging device 3. Further, the following processing will be described on the assumption that the three-dimensional shape data of FIG.

  Next, the image processing unit 22 performs a captured image reading process of reading the captured image received from the imaging device 3 and stored in the storage unit 21 for each imaging device 3 (ST2). In the captured image reading process, the latest captured image (closest to the current time) is read for each imaging device 3.

  Next, extraction processing is executed by the extraction means 221 of the image processing unit 22 (ST3). In the extraction process, each change area is extracted from the captured image of each imaging device 3 acquired in the captured image reading process. In this embodiment, the change area is extracted by taking the difference between the captured image acquired this time and the reference image. In the extraction process, a labeling process for labeling the extracted change area is also performed.

  Next, object generation processing is executed by the object generation means 222 of the image processing unit 22 (ST4). In the object generation process, for each change area extracted for each imaging device 3 in the extraction process, a foot position on the place model 211 (virtual space) is calculated and a change area object is generated. Since the details of the object generation process have been described above, the description thereof is omitted here.

  Next, virtual monitoring image generation processing is executed by the virtual monitoring image generation means 224 of the image processing unit 22 (ST5). In the virtual monitoring image generation processing, the change area object 32 generated by the object generation means 222 is arranged on the place model, and processing for generating a virtual monitoring image when viewed from the virtual camera is performed. Details of the virtual monitoring image generation process will be described later.

  Next, the image processing unit 22 performs processing to display and output the virtual monitoring image generated by the virtual monitoring image generation processing on the display unit 24. Then, the image processing unit 22 repeatedly executes the processes of ST2 to ST6 until a termination operation is performed from the input unit 25 by a supervisor or the like.

(Details of virtual monitoring image generation processing)
Next, details of the virtual monitoring image generation processing executed by the virtual monitoring image generation means 223 of the image processing unit 22 in ST5 of the flowchart of FIG. 5 will be described with reference to FIG.

  In the virtual monitoring image generation process, first, it is determined whether or not an optimal viewpoint operation input has been performed by the monitoring person via the input unit 25 (ST20). Here, the optimal viewpoint operation input is an input operation performed when the monitor wants to visually recognize the change area of the target moving object more appropriately than the virtual viewpoint in the currently displayed virtual monitoring image. A command executed by a mouse or the like. For example, when the change area of the target moving object is hidden behind the image area 42 ′ of the object model and is difficult to visually recognize like the image areas corresponding to reference numerals 42v and 42z in FIG. The optimal viewpoint operation input is executed so as to switch to the virtual viewpoint.

  When it is determined in ST20 that the optimal viewpoint operation input has been performed by the supervisor (ST20-Yes), the virtual monitoring image generation unit 223 executes a visibility ratio calculation process. The view rate calculation process is executed for each virtual camera, and the view rate is calculated for each virtual camera. Therefore, loop 1 in FIG. 5 means that the processes of ST21 to ST24 are executed by the number of virtual cameras. FIG. 7 illustrates, as reference numerals 35a to 35h, virtual cameras including virtual viewpoints fixedly set in advance in the storage unit 21 with respect to the place model of FIG. Reference numeral 35 represents a virtual camera currently displayed). In the example of FIG. 7, the number of virtual cameras is nine virtual images obtained by adding 1 which is the number of virtual cameras currently displayed to the number of virtual viewpoint information fixed in advance in the storage unit 21 (eight). The viewing rate calculation process is executed for the camera. In the description of this process, the selected virtual camera refers to a virtual camera that is a processing target in loop 1.

  In the viewing rate calculation process, first, a process of reading the virtual viewpoint information of the selected virtual camera from the storage unit 21 is performed (ST21). Then, a virtual image with location is generated using the virtual viewpoint information of the selected selected virtual camera (ST22). Since the details of the process of generating the place-in-place virtual image have been described above, the description thereof is omitted here.

  Next, in the virtual monitoring image generation processing, processing for generating a placeless virtual image is performed (ST23). A placeless virtual image is a virtual image (rendered image by a virtual camera) that does not include a place model. The change area object 32 is arranged at the foot position in the virtual space in the same manner as in the case of the place-within virtual image, while the place model is obtained by rendering processing with the selected virtual camera without being arranged in the virtual space. It is a possible image. In other words, in the placeless virtual image, since the change area object 32 is not hidden by the object model, it can be said that the image can most visually recognize the change area of the target moving object. FIG. 8 shows an example of a placeless virtual image captured by the same virtual camera (corresponding to reference numeral 35 in FIG. 7) as in FIG. In FIG. 8, image areas indicated by reference numerals 43x, 43v, and 43z are image areas corresponding to reference signs 42v, 42x, and 42z in FIG.

  Next, in the virtual monitoring image generation process, a process for calculating the viewing rate is performed (ST24). In calculating the viewing rate, first, the virtual monitoring image generation means 223 obtains the area (first area) of the image area corresponding to the change area in the place-containing virtual image generated in ST22. That is, in FIG. 4, the sum of the areas of the image areas corresponding to the symbols 42v, 42x, and 42z is calculated as the first area. Then, the virtual monitoring image generation means 223 obtains the area (second area) of the image area corresponding to the change area in the placeless virtual image generated in ST23. That is, in FIG. 8, the sum of the areas of the image areas corresponding to the symbols 43v, 43x, and 43z is calculated as the second area. And the visual recognition rate which is a ratio which can visually recognize a change area | region from a virtual camera from the ratio (1st area / 2nd area) of a 1st area and a 2nd area is calculated, and the said visual recognition rate and a selection virtual camera are matched. Is stored in the storage unit 21. When the process of ST24 is completed, another virtual camera that has not yet been processed in loop 1 is changed to the selected virtual camera, and the process from ST21 is performed.

  When the processing for all the virtual cameras is completed in loop 1, the virtual monitoring image generation unit 223 ends the visibility ratio calculation processing, and advances the processing to ST25. In ST25, a selection process is performed in which the virtual camera having the largest visual recognition ratio among the visual recognition ratios of the respective virtual cameras obtained in the visual recognition ratio calculation processing is determined, and the place-in-place virtual monitoring image of the virtual camera is selected as the virtual monitoring image. Perform (ST25). For example, when the virtual camera that generated the currently displayed virtual monitoring image is the virtual camera denoted by reference numeral 35 in FIG. 7, it is assumed that the virtual monitoring image as shown in FIG. 4 has been obtained. Here, when the optimal viewpoint operation input by the supervisor is made and the visual recognition rates of the virtual cameras of reference numeral 35 and reference numerals 35a to 35h in FIG. 7 are calculated, respectively, the visual recognition ratio of the virtual camera of reference numeral 35f is the largest. FIG. 9 which is the virtual monitor image with location generated by the virtual camera with the sign 35f is selected as the virtual monitor image. As described above, the monitoring rate can be reduced to a virtual viewpoint with better visibility by the visibility rate calculation process and the selection process, so that the monitoring load can be reduced. When the selection process ends, the virtual monitoring image generation process ends, and the process proceeds to ST6 in FIG.

  In ST20, when it is determined that the optimal viewpoint operation input is not performed from the supervisor (ST20-No), the virtual monitoring image generation unit 223 performs processing for reading out the virtual viewpoint information of the currently displayed virtual camera from the storage unit 21. Perform (ST26). Then, a virtual image with location is generated using the virtual viewpoint information of the selected selected virtual camera (ST27). Since the details of the process of generating the place-in-place virtual image have been described above, the description thereof is omitted here. Then, the generated virtual image with location is set as a virtual monitoring image (ST28), the virtual monitoring image generation process is terminated, and the process proceeds to ST6 in FIG.

  By the way, the present invention is not limited to the above-described embodiment, and may be implemented in various different embodiments within the scope of the technical idea described in the claims. Further, the effects described in the embodiments are not limited to this.

  In the above embodiment, the imaging condition information 212 of the imaging device 3 is fixedly set in ST1 (initial setting) in FIG. However, the present invention is not limited to this, and the imaging condition information 212 is periodically received from the imaging device 3 via the communication unit 23 of the monitoring terminal 2, and the processes after ST2 are performed based on the received imaging condition information 212. Also good. Thereby, for example, even if the imaging condition information 212 is changed by panning, tilting, zooming, or the like with respect to the imaging apparatus 3, the monitor changes the state at the monitoring place while following the changed imaging condition information 212. Monitor from a virtual perspective.

  In the above embodiment, the object generation unit 222 texture-maps the image area of the captured image corresponding to the change area to the rectangular plate-like object generated based on the change area, thereby changing the change area object. Is generated. However, the present invention is not limited to this, and a pre-registered humanoid model (for example, a cylindrical model) is used to change the size of the humanoid model based on the image characteristics of the change area, and the change area The change area object may be generated by texture mapping the image area of the captured image corresponding to. In addition, the image area of the captured image corresponding to the change area is directly projected (texture mapping) onto the area corresponding to the size of the change area on the place model, and the projection area is changed to the change area. It may be an object.

  In the above embodiment, the virtual monitoring image generation unit 223 calculates the first area (or the second area) from the area sum of the image areas of the place-containing virtual image (or the place-free virtual image) corresponding to any change area. However, the present invention is not limited to this, but is limited to a change area having an image characteristic of a target moving object determined in advance by a monitor or the like, and an image area of a place-containing virtual image (or a place-free virtual image) corresponding to the change area The first area (or the second area) may be calculated from the sum of the areas. As a result, the change area that does not have the image feature of the target moving object is not taken into account when calculating the visibility, and the monitor sets the image feature of, for example, “children” as the target moving object. The virtual viewpoint can be changed so that the change area having the image characteristics of the child can be suitably viewed.

DESCRIPTION OF SYMBOLS 1 ... Image monitoring apparatus 2 ... Monitoring terminal 3 ... Imaging device 21 ... Memory | storage part 22 ... Image processing part 23 ... Communication part 24 ... Display part 25 ... Input part 211 ... Location model 212 ... Imaging condition information 213 ... Virtual viewpoint information 221 ... Extraction means 222 ... Object generation means 223 ... Virtual monitoring image generation means 30 ... Change area 31a .. Foot position 31b ... Head position 32 ... Change area object 35 ... Virtual camera 40 ... Captured image 41 ... Reference image 42 ... Virtual image with place 43 ... No place Virtual image

Claims (4)

An imaging unit that sequentially acquires captured images of the monitoring location;
A place model expressing the monitoring place as a three-dimensional virtual space, an installation position of the imaging unit, imaging condition information in which an optical axis and an angle of view are associated with the virtual space, a position of the virtual camera in the virtual space, A storage unit storing virtual viewpoint information including an optical axis and an angle of view;
An image processing unit that arranges a change area extracted from the captured image on the place model and outputs a virtual monitoring image obtained by capturing the place model from the virtual camera;
An image monitoring apparatus comprising a display unit for displaying the virtual monitoring image,
The image processing unit
Using the location model, the imaging condition information, and the position and size of the change area on the captured image, a virtual space position that is the position of the change area in the virtual space is obtained, and the change area at the virtual space position Object generation means for generating a change area object obtained by projecting the change area onto an object corresponding to the size of
The change area object corresponding to the change area is arranged at the virtual space position on the place model, and the virtual monitoring image is generated using the place model and the virtual viewpoint information and output to the display unit. And a virtual monitoring image generating means. The storage unit stores the virtual viewpoint information about a plurality of the virtual cameras,
The virtual monitoring image generation means includes
Placed virtual image obtained by placing the change area object in the corresponding virtual space position on the place model and picked up from the virtual camera, and picked up from the virtual camera by placing the change area object in the virtual space position Generating a location-free virtual image, and using the first area of the image region corresponding to the change region in the location-containing virtual image and the second area of the image region corresponding to the change region in the location-free virtual image A visibility rate calculation process for calculating a visibility rate that is a rate at which the change area can be visually recognized from the virtual camera for each virtual camera;
The image monitoring apparatus according to claim 1, wherein a selection process is performed to select the place-in-place virtual monitoring image generated by the virtual camera having the highest visibility rate as the virtual monitoring image.   The first area is an area of an image region of a place-containing virtual image corresponding to the change region having an image characteristic of a predetermined target moving object, and the second area is an image characteristic of a predetermined target moving object. The image monitoring apparatus according to claim 1, wherein the image monitoring device has an area of an image region of a place-free virtual image corresponding to the change region. It further has an input unit for receiving operation input by the user,
The image monitoring apparatus according to claim 2, wherein the virtual monitoring image generation unit performs the visibility ratio calculating process when the operation input is received from the input unit.

Images