JP5983259B2 - Image processing apparatus, image processing method, and image processing program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and an image processing program.

  2. Description of the Related Art Conventionally, there is a technique for combining and displaying a model of an object captured by a camera at a corresponding position of a three-dimensional model in which a viewpoint position and a line-of-sight direction are set (for example, Patent Document 1). . Patent Document 1 discloses a technique for displaying a camera image of a region where a certain event has occurred together with the above-described three-dimensional model.

JP 2000-33279 A

  However, the conventional technique has a problem that it is difficult to intuitively grasp the area imaged by the camera. In the prior art, since the viewpoint position and line-of-sight direction in the three-dimensional model do not match the viewpoint position and line-of-sight direction in the actual camera image, it is difficult to intuitively grasp the area captured by the camera It is. This is the same when, for example, an image captured by a camera is recorded and reproduced later. The present invention has been made in view of the above, and an image processing device, an image processing method, and an image processing that make it possible to intuitively grasp an area captured by a camera when recording and reproducing a surveillance image The purpose is to provide a program.

  In order to solve the above problems, the present invention provides the following apparatus, method, and program.

1) 3D model data indicating a model of a 3D area, a 3D model storage unit that stores the position of a camera arranged in the 3D area as a camera position in the 3D model, a captured image captured by the camera, An acquisition unit that acquires an imaging direction at the time of imaging, an image storage unit that stores the captured image acquired by the acquisition unit in association with the imaging direction, and an operation unit that receives an instruction to reproduce the captured image stored in the image storage unit When the operation unit receives a reproduction instruction, the viewpoint for the three-dimensional model is determined according to the camera position, and the three-dimensional model is displayed on the two-dimensional surface according to the determined viewpoint and the imaging direction stored in the image storage unit. A combining unit that generates a projected two-dimensional projection image and generates a combined image by combining the captured image stored in the image storage unit with a predetermined region on the generated two-dimensional projection image; The image processing apparatus characterized by comprising an output section for outputting a composite image.

2) Three-dimensional model data indicating a model of the three-dimensional area; a storage step of storing the position of the camera arranged in the three-dimensional area as a camera position in the three-dimensional model; a captured image captured by the camera; An acquisition step for acquiring the imaging direction, an image storage step for storing the captured image acquired in the acquisition unit step in association with the imaging direction, and an operation step for receiving a reproduction instruction for the captured image stored in the image storage step When a playback instruction is received in the operation step, the viewpoint for the three-dimensional model is determined according to the camera position, and the three-dimensional model is converted into the two-dimensional according to the determined viewpoint and the imaging direction stored in the image storage step. A two-dimensional projection image projected onto the surface is generated and stored in a predetermined area on the generated two-dimensional projection image in the image storage step. Image processing method characterized by comprising the synthetic steps of generating a synthesized image by synthesizing the captured image, and outputting a composite image.

3) 3D model data indicating a model of the 3D area, a storage function for storing the position of the camera arranged in the 3D area in the storage unit as a camera position in the 3D model, a captured image captured by the camera, An acquisition function for acquiring the imaging direction at the time of imaging, an image storage function for storing the captured image acquired by the acquisition unit function in association with the imaging direction in the image storage unit, and a captured image stored in the image storage unit An operation function that accepts a reproduction instruction, and when a reproduction instruction is accepted by the operation function, a viewpoint for the three-dimensional model is determined according to the camera position, and according to the determined viewpoint and the imaging direction stored in the image storage unit A two-dimensional projection image obtained by projecting a three-dimensional model onto a two-dimensional surface is generated, and a captured image stored in an image storage unit in a predetermined region on the generated two-dimensional projection image Synthesis and synthetic function of generating a composite image by image processing program for realizing the output function on the computer to output a composite image.

  According to one aspect of the present invention, there is an effect that an area captured by a camera can be intuitively grasped when a surveillance image is recorded and reproduced.

FIG. 1 is a diagram illustrating a configuration example of a monitoring system according to the first embodiment. FIG. 2 is a diagram illustrating a configuration example of the imaging apparatus according to the first embodiment. FIG. 3 is a diagram illustrating a configuration example of the image processing apparatus according to the first embodiment. FIG. 4 is a diagram illustrating an example of information stored by the three-dimensional model storage unit. FIG. 5 is a diagram for explaining the three-dimensional model data. FIG. 6 is a diagram illustrating an example of information stored in the viewpoint information storage unit. FIG. 7 is a flowchart illustrating an example of the flow of overall processing according to the first embodiment. FIG. 8 is a flowchart illustrating an example of the flow of the initial image generation process according to the first embodiment. FIG. 9 is a diagram illustrating an example of an initial image. FIG. 10 is a flowchart illustrating an example of the flow of image processing according to the first embodiment. FIG. 11 is a diagram illustrating an example of a two-dimensional projection image. FIG. 12 is a diagram illustrating an example of a composite image. FIG. 13 is a diagram illustrating an example of changing the position of the predetermined area. FIG. 14A is a diagram illustrating an example of a two-dimensional projection image before transmitting a predetermined object. FIG. 14B is a diagram illustrating an example of a two-dimensional projection image after transmitting a predetermined object. FIG. 15A is a diagram illustrating an example of a two-dimensional projection image before adjusting the size of the pop-up. FIG. 15B is a diagram illustrating an example of a two-dimensional projection image after adjusting the size of the pop-up. FIG. 16 is a diagram illustrating a configuration example of an image processing apparatus according to the fourth embodiment. FIG. 17A is a diagram illustrating an example of a composite image before a drag operation is performed. FIG. 17B is a diagram illustrating an example of a composite image when a drag operation is performed in the left direction. FIG. 17C is a diagram illustrating an example of a composite image when a downward drag operation is performed. FIG. 18 is a flowchart illustrating an example of the flow of overall processing according to the fourth embodiment. FIG. 19 is a flowchart illustrating an example of the flow of image processing according to the fourth embodiment. FIG. 20 is a flowchart illustrating an example of the flow of image processing according to a modification of the fourth embodiment. FIG. 21 is a flowchart illustrating an example of the flow of image processing according to the fifth embodiment. FIG. 22 is an image diagram showing an example of determining a base point according to the sixth embodiment. FIG. 23 is an image diagram showing an example of setting the camera operating speed according to the drag distance according to the sixth embodiment. FIG. 24 is a flowchart illustrating an example of the flow of image processing according to the sixth embodiment. FIG. 25 is an image diagram showing an example of determining a base point according to the seventh embodiment. FIG. 26 is an image diagram showing an example of setting the camera operation speed according to the drag speed according to the seventh embodiment. FIG. 27 is a flowchart illustrating an example of the flow of image processing according to the seventh embodiment. FIG. 28 is a diagram illustrating an example of information stored in the three-dimensional model storage unit. FIG. 29 is an explanatory diagram of the hierarchical structure of the monitoring area. FIG. 30 is a diagram illustrating an example of information stored in the viewpoint information storage unit. FIG. 31 is an explanatory diagram showing monitoring areas or camera icons that can be selected from the selected image. FIG. 32 is a flowchart illustrating an example of the flow of overall processing according to the eighth embodiment. FIG. 33 is a flowchart illustrating an example of the flow of selected image generation processing according to the eighth embodiment. FIG. 34 is a diagram for explaining processing of the image processing apparatus according to the ninth embodiment. FIG. 35 is a diagram illustrating an example of information stored in the three-dimensional model storage unit. FIG. 36A is a diagram illustrating a two-dimensional projection image drawn when the viewpoint position is changed. FIG. 36B is a diagram showing a two-dimensional projection image drawn when the viewpoint position is changed. FIG. 36C is a diagram showing a two-dimensional projection image drawn when the viewpoint position is changed. FIG. 37 is a diagram for explaining the viewpoint range with the y-axis as the rotation axis for all monitoring areas. FIG. 38A is a diagram illustrating a two-dimensional projection image drawn when the viewpoint position is changed. FIG. 38B is a diagram showing a two-dimensional projection image drawn when the viewpoint position is changed. FIG. 39 is a flowchart of the image processing apparatus. FIG. 40 is a diagram illustrating a configuration example of an image processing device according to the tenth embodiment. FIG. 41 is a diagram showing an example of a captured image management table held by the image storage unit shown in FIG. FIG. 42 is a diagram illustrating a configuration example of the sensor according to the first to eleventh embodiments. FIG. 43 is a diagram showing an example of an event management table held by the event storage unit shown in FIG. FIG. 44 is a diagram illustrating an example of a sensor management table managed by the three-dimensional model storage unit in the tenth embodiment. FIG. 45 is a flowchart showing the recording operation of the control unit according to the tenth embodiment. FIG. 46 is a flowchart showing the event recording operation of the control unit according to the tenth embodiment. FIG. 47 is a flowchart showing an example of a recording / playback operation by the control unit according to the tenth embodiment. FIG. 48 is a diagram illustrating an example of a composite image according to the tenth embodiment. FIG. 49 is a diagram showing an example of the operation screen shown in FIG. FIG. 50 is a diagram illustrating an example of a composite image according to the eleventh embodiment. FIG. 51 is a diagram illustrating that an image processing program is realized using a computer.

  Embodiments of an image processing apparatus, an image processing method, and an image processing program according to the present invention will be described below with reference to the accompanying drawings. In addition, this invention is not limited by the following embodiment. Moreover, each embodiment can be combined suitably as long as the content is not contradicted.

(Embodiment 1)
[Configuration of Monitoring System According to Embodiment 1]
The configuration of the monitoring system according to the first embodiment will be described with reference to FIG. FIG. 1 is a diagram illustrating a configuration example of a monitoring system according to the first embodiment.

  As shown in FIG. 1, in the monitoring system 1, an image processing device 10, an imaging device 20a1, a sensor 40a1, an imaging device 20b1, and a sensor 40b1 are connected to a network 30. Such a monitoring system 1 is used for monitoring a predetermined area. The image processing apparatus 10 may be installed in, for example, a management room in a store or a security room of a security company. Hereinafter, the entire area to be monitored by the monitoring system 1 may be referred to as “all monitoring areas”.

  The imaging device 20a1 and the sensor 40a1 are arranged in an area 50a that is a part of the entire monitoring area. Similarly, the imaging device 20b1 and the sensor 40b1 are arranged in an area 50b that is a part of the entire monitoring area. In addition to the imaging device 20a1 and the sensor 40a1, a plurality of imaging devices and sensors may be arranged in the area 50a. In addition to the imaging device 20b1 and the sensor 40b1, a plurality of imaging devices and sensors may be arranged in the area 50b. Hereinafter, the imaging device 20a1 and the imaging device 20b1 may be referred to as “imaging device 20”. Hereinafter, the sensor 40a1 and the sensor 40b1 may be referred to as “sensor 40”. Hereinafter, the area 50a1 and the area 50b1 may be referred to as “area 50”.

  Among these, the imaging device 20 includes a camera that captures an image of a subject within the imageable range included in the area 50. Then, the imaging device 20 transmits a captured image obtained by compressing an image captured by the camera to the image processing device 10. At this time, the imaging device 20 also transmits zoom information and imaging direction information, which are information on the focal length of the zoom lens in the camera at the time of imaging, to the image processing device 10. Such imaging direction information is, for example, pan information that is the horizontal angle of the imaging direction and tilt information that is the vertical angle of the imaging direction.

  The image processing apparatus 10 stores three-dimensional model data representing a three-dimensional model of all monitoring areas. Such three-dimensional model data is information indicating the shape, size, and layout (position information) of an object (object) such as a building in all monitoring areas. The three-dimensional model data is generated in advance by, for example, rendering using information such as the shape and viewpoint of an object obtained by imaging with the imaging device 20. Alternatively, it may be generated from a floor plan of all monitoring areas. In addition, the image processing apparatus 10 stores a camera position that is a position in a three-dimensional model of a camera (corresponding to the imaging apparatus 20) arranged in all the monitoring areas.

  Then, the image processing apparatus 10 acquires the captured image captured by the imaging apparatus 20, the focal length of the zoom lens at the time of imaging, the imaging direction, and the like from the imaging apparatus 20. Subsequently, the image processing apparatus 10 determines a viewpoint for generating a two-dimensional image from the three-dimensional model based on the camera position, and a two-dimensional projection image corresponding to the determined viewpoint, imaging direction, and focal length of the zoom lens. Is generated. Thereafter, the image processing apparatus 10 generates a composite image by combining the captured image with a predetermined region on the generated two-dimensional projection image, and outputs the generated composite image.

  The sensor 40 is, for example, a human sensor or an opening / closing sensor that is appropriately disposed in the area 50 and detects an opening / closing of a person or a door and issues an alarm, and transmits detection data to the image processing apparatus 10. .

  That is, the image processing apparatus 10 synthesizes and outputs the captured image captured by the imaging apparatus 20 to a predetermined region of the two-dimensional projection image generated according to the imaging direction of the imaging apparatus 20 and the like. The person can intuitively grasp which area in the entire monitoring area is the area captured by the camera. Detailed configurations of the image processing device 10 and the imaging device 20 will be described later.

[Configuration of Imaging Device According to Embodiment 1]
Next, the configuration of the imaging device 20 according to Embodiment 1 will be described with reference to FIG. FIG. 2 is a diagram illustrating a configuration example of the imaging device 20 according to the first embodiment.

  As shown in FIG. 2, the imaging device 20 includes an imaging unit 201, a compression unit 202, a communication processing unit 203, a control unit 204, a pan head driving unit 205, an angle sensor 206, and a zoom driving unit 207. And a zoom sensor 208. The imaging device 20 is a network camera connected to the image processing device 10 via the network 30 and can change the imaging direction according to a control signal received via the network 30.

  The imaging unit 201 captures a subject within the imageable range and outputs the captured image to the compression unit 202. The captured image may be a still image or a moving image. The compression unit 202 compresses the captured image captured by the imaging unit 201 according to a standard such as JPEG (Joint Photographic Experts Group) or MPEG-4 (Moving Picture Experts Group phase 4), and communicates the compressed digital image as a captured image. The data is output to the processing unit 203. The compression unit 202 also receives pan information, tilt information, and zoom information from the control unit 204 and adds them to the header or footer of the captured image. As a result, a captured image to which pan information, tilt information, and zoom information are added is sent to the communication processing unit 203.

  The communication processing unit 203 transmits the captured image to which the pan information, tilt information, and zoom information output from the compression unit 202 are added to the image processing apparatus 10 via the network 30. Further, the communication processing unit 203 outputs a control signal received from the external device to the control unit 204. Such a control signal is, for example, a control signal related to pan, tilt, and zoom. That is, the communication processing unit 203 receives a control signal from an external device in order to perform camera control, as in a general network camera. The communication processing unit 203 according to the present embodiment receives a control signal from the image processing apparatus 10.

  The control unit 204 performs control for causing the pan / tilt head driving unit 205 to perform panning and tilting and causing the zoom driving unit 207 to zoom the zoom lens. This control is based on a control signal input via the communication processing unit 203. As another example, the control unit 204 may perform the control according to a preset algorithm. Further, the control unit 204 receives the pan and tilt angles detected by the angle sensor 206 and sends them to the compression unit 202. Further, the control unit 204 receives the focal length of the zoom lens detected by the zoom sensor 208 and sends it to the compression unit 202.

  The pan / tilt head driving unit 205 changes the imaging direction of the imaging unit 201 in accordance with, for example, control from the control unit 204. The pan head drive unit 205 is provided with an angle sensor 206 that detects an angle in the pan direction or the tilt direction. The angle sensor 206 detects, for example, the pan and tilt angles and outputs them to the control unit 204. The zoom drive unit 207 is attached to the zoom lens of the imaging unit 201. For example, the zoom driving unit 207 changes the focal length of the zoom lens according to the focal length of the zoom lens instructed by the control unit 204. The zoom lens is provided with a zoom sensor 208 that detects the focal length of the zoom lens. For example, the zoom sensor 208 detects the focal length of the zoom lens and outputs it to the control unit 204. Based on the detection results of the angle sensor 206 and the zoom sensor 208, the control unit 204 controls these while appropriately checking the imaging direction of the imaging unit 201 and the focal length of the zoom lens.

[Configuration of Image Processing Device According to Embodiment 1]
Next, the configuration of the image processing apparatus 10 according to the first embodiment will be described with reference to FIG. FIG. 3 is a diagram illustrating a configuration example of the image processing apparatus 10 according to the first embodiment.

  The image processing apparatus 10 includes a three-dimensional model storage unit 111, a viewpoint information storage unit 112, an operation unit 121, a communication processing unit 122, an acquisition unit 123, a control unit 124, a synthesis unit 125, and an output unit 126. And have. The image processing apparatus 10 is an information processing apparatus such as a PC (Personal Computer) connected to a network camera such as the imaging apparatus 20 or the sensor 40 via the network 30.

The 3D model storage unit 111 stores 3D model data and the like. FIG. 4 is a diagram illustrating an example of information stored in the three-dimensional model storage unit 111. As shown in FIG. 4, the three-dimensional model storage unit 111 associates three-dimensional model data representing a three-dimensional model of all monitoring areas, a device ID (identifier), an icon type ID, and an icon position. Remember. Here, the device ID is device-specific identification information for identifying a device arranged in the three-dimensional model represented by the three-dimensional model data. The icon type ID is a type of icon that represents a device in the three-dimensional model. In the image processing apparatus 10 according to the present embodiment, a different type of icon is assigned for each type of device, and the icon type ID is information for identifying the type of icon. The icon position is a position (coordinate) where the icon is arranged in the three-dimensional model.

  For example, as shown in FIG. 4, the 3D model storage unit 111 has 3D model data “all monitoring areas”, a device ID “# 01” indicating the camera 1, and an icon indicating a camera as an icon type. The type ID “A001” and the icon position “(x1, y1, z1)” are stored in association with each other. Therefore, by referring to the three-dimensional model storage unit 111, the arrangement position of each device can be specified. For example, in the example illustrated in FIG. 4, the coordinates “(x1, y1, z1)” in the three-dimensional model of “all monitoring areas” are used as the arrangement positions of “camera 1”, “camera 2”, and “sensor 1”. , “(X2, y2, z2)”, “(x3, y3, z3)” can be specified.

  Further, as illustrated in FIG. 4, the three-dimensional model storage unit 111 stores a three-dimensional part model of various icons in association with the icon type ID. Note that the three-dimensional part model of the icon corresponding to the imaging device 20 has a portion corresponding to a lens. When the three-dimensional part model of the imaging device 20 is arranged in the three-dimensional model, the lens direction of the three-dimensional part model of the imaging device 20 is arranged according to the actual lens direction (imaging direction) of the imaging device 20. can do.

  FIG. 5 is a diagram for explaining the three-dimensional model data. FIG. 5 shows an overhead view of the three-dimensional model observed from the obliquely upper viewpoint of the three-dimensional model. As shown in FIG. 5, the three-dimensional model is a model in which objects such as buildings, grounds, and trees in the entire monitoring area are laid out according to their sizes, and the three-dimensional model data includes the shapes of these objects. Or data indicating size, position, etc. Note that a two-dimensional projection image that is a two-dimensional image obtained by projecting a three-dimensional model onto a two-dimensional surface, as shown in an overhead view shown in FIG. 5, by determining a viewpoint, a gazing point, and a field angle for the three-dimensional model data. Can be generated.

  The viewpoint information storage unit 112 stores a viewpoint position used when a synthesis unit 125 described later generates a two-dimensional projection image from the three-dimensional model data. FIG. 6 is a diagram illustrating an example of information stored in the viewpoint information storage unit 112. Specifically, the viewpoint information storage unit 112 generates the two-dimensional projection image of the area observed from the viewpoint position used when generating the two-dimensional projection image of the entire monitoring area and the camera arrangement position. Are stored in association with the viewpoint position ID for identifying the viewpoint position.

  For example, as illustrated in FIG. 6, the viewpoint information storage unit 112 stores the viewpoint position ID “B001” indicating the entire monitoring area and the viewpoint position “(x10, y10, z10)” in association with each other. . The viewpoint information storage unit 112 stores a viewpoint position ID “B011” indicating the camera 1 and a viewpoint position “(x11, y11, z11)” in association with each other. Here, the viewpoint position associated with the viewpoint position ID indicating the camera is the camera position. Note that the viewpoint position ID of the viewpoint position indicating the camera position may be the same value as the device ID of the camera (hereinafter referred to as camera ID). Thereby, the viewpoint position can be specified directly from the camera ID.

  The operation unit 121 includes an input device such as a mouse or a touch panel, and accepts various instructions by a user operation of the user of the monitoring system 1. For example, the operation unit 121 receives a camera selection instruction from the user. In addition, when a predetermined camera (imaging device 20) is selected by a user operation, the operation unit 121 receives an instruction to output a captured image of the selected imaging device 20. In addition, the operation unit 121 receives pan, tilt, and zoom setting instructions for a predetermined imaging device 20 by a user operation. Note that the pan, tilt, and zoom setting instructions are transmitted to the imaging apparatus 20 via the control unit 124 and the communication processing unit 122 as control signals related to the above-described pan, tilt, and zoom.

  The communication processing unit 122 controls communication with the imaging device 20 and the sensor 40 connected via the network 30. For example, the communication processing unit 122 receives a captured image and pan information, tilt information, and zoom information from the imaging device 20. In addition, the communication processing unit 122 transmits the pan, tilt, and zoom setting instructions received by the operation unit 121 to the imaging device 20 as control signals related to pan, tilt, and zoom.

  The acquisition unit 123 acquires panned information, tilt information, and zoom information as a captured image and an imaging direction from the imaging device 20 via the communication processing unit 122. The acquisition unit 123 acquires detection data from the sensor 40. The detection data may include information related to the detection direction of the sensor 40 (pan information, tilt information, etc.).

  The control unit 124 controls the entire image processing apparatus 10. For example, the control unit 124 outputs the captured image acquired by the acquisition unit 123, pan information at the time of imaging, tilt information, and zoom information to the synthesis unit 125 in accordance with an instruction received by the operation unit 121. The control unit 124 also acquires the three-dimensional model and the viewpoint position from the three-dimensional model storage unit 111 according to the instruction received by the operation unit 121, and outputs these to the synthesis unit 125.

  At this time, the control unit 124 also acquires the camera icon corresponding to each imaging device 20 and the image of the sensor icon corresponding to the sensor 40 and the icon position of these icons from the three-dimensional model storage unit 111, The icon position of the icon is output to the composition unit 125. Further, the control unit 124 receives not only the captured image captured by the imaging device 20 corresponding to the camera icon selected by the user but also the captured image corresponding to each camera icon from the acquisition unit 123, and each captured image. Is also output to the combining unit 125.

  The synthesizing unit 125 generates a two-dimensional projection image based on the viewpoint and the imaging direction of the three-dimensional model, and a captured image that is a two-dimensional image captured by the imaging device 20 in a predetermined region on the generated two-dimensional projection image. Is synthesized. The predetermined area is an area having a predetermined size set at a predetermined position on the two-dimensional projection image. For example, the synthesizing unit 125 generates a two-dimensional projection image according to the viewpoint position read out from the three-dimensional model storage unit 111 by the control unit 124 and the pan, tilt, and zoom of the imaging device 20 that captures the captured image to be synthesized. Generate. Subsequently, the synthesizing unit 125 generates a synthesized image by synthesizing the captured image output from the control unit 124 with a predetermined area such as near the center of the generated two-dimensional projection image.

  At this time, the combining unit 125 also arranges icons such as camera icons and sensor icons output by the control unit 124 in the two-dimensional projection image based on the icon positions. Furthermore, the synthesis unit 125 synthesizes the captured images captured by the respective imaging devices 20 in areas (for example, pop-ups) set in the vicinity of the respective camera icons corresponding to the respective imaging devices 20. Note that the image pop-up displayed for each camera icon may be a thumbnail of the captured image. The area set in the vicinity of each camera icon is, for example, an area smaller than the predetermined area described above. As described above, the combining unit 125 displays not only the captured image corresponding to the predetermined area but also each captured image captured by the imaging device 20 in order to pop up the captured image in the area set near each camera icon. Accept from the control unit 124. Also, in this embodiment and other embodiments, comments such as camera numbers and names, places to be imaged are stored in the 3D model storage unit 111, and the comments are superimposed on the captured image displayed in the pop-up. Or you may display only the comment as a popup.

  The output unit 126 outputs a composite image. For example, the output unit 126 may display an icon combined with the two-dimensional projection image by the combining unit 125, a captured image combined with an area near the camera icon, or a captured image combined with a predetermined region of the two-dimensional projection image. Output the composite image including it. The output unit 126 may be a display device that displays a composite image, or may output the composite image to a display device connected to the image processing device 10.

[Overall Processing Flow According to Embodiment 1]
Next, the overall processing flow according to the first embodiment will be described with reference to FIG. FIG. 7 is a flowchart illustrating an example of the flow of overall processing according to the first embodiment.

  For example, as illustrated in FIG. 7, when the operation unit 121 of the image processing apparatus 10 receives an instruction to display an initial image (Yes in step S101), the synthesis unit 125 generates an initial image (step S102). And the output part 126 displays the initial image produced | generated by the synthetic | combination part 125 on a display screen (step S103). On the other hand, when the operation unit 121 has not received an instruction to display an initial image (No in step S101), the process waits for an instruction to display an initial image.

  FIG. 9 is a diagram illustrating an example of an initial image. As shown in FIG. 9, the initial image is a two-dimensional projection image of the entire monitoring area. The initial image includes icons of various devices stored in the 3D model storage unit 111 in association with the 3D model data of all monitoring areas. The two-dimensional projection image shown in FIG. 9 includes a camera icon corresponding to each imaging device 20 as a device. Further, in the two-dimensional projection image shown in FIG. 9, the captured image captured by each imaging device 20 is displayed in a pop-up near the camera icon. In FIG. 9, each camera icon is arranged in the air, but the camera is actually attached to a pole several meters above the ground. When monitoring is in a building, the camera is actually mounted at a high place such as a ceiling. That is, by arranging the camera icon in the air in a two-dimensional projection image such as an initial image, the display of the camera icon is suppressed from being hidden by a shielding object such as a building, thereby selecting the camera icon. Because of the improved operability. Although not shown in FIG. 9, when icons of various devices other than the camera such as the sensor 40 are included, the icon of the sensor 40 is also displayed.

  Returning to FIG. 7, when the camera icon is selected in the initial image by the user operation and the operation unit 121 accepts the selection instruction (Yes at Step S <b> 104), the image synthesizing unit 125 selects the selected camera. A composite image is generated by synthesizing the captured image of the imaging device 20 corresponding to the selected camera icon with the two-dimensional projection image with the icon arrangement position as a viewpoint (step S105). Then, the output unit 126 displays the composite image on the display screen (step S106). On the other hand, when the operation unit 121 does not select a camera icon on the initial screen (No in step S104), the camera unit waits for selection of the camera icon.

[Initial Image Generation Processing Flow According to Embodiment 1]
Next, the flow of initial image generation processing according to the first embodiment will be described with reference to FIG. FIG. 8 is a flowchart illustrating an example of the flow of the initial image generation process according to the first embodiment. The initial image generation process according to the first embodiment refers to the process in step S102.

  As illustrated in FIG. 8, the composition unit 125 of the image processing apparatus 10 acquires the three-dimensional model data of all monitoring areas from the three-dimensional model storage unit 111 (Step S201). The synthesizing unit 125 then displays the arrangement positions of the devices associated with the three-dimensional models in all the monitoring areas in the three-dimensional model storage unit 111, that is, the icons and the icon positions of the icons. Obtained from 111 (step S202). Subsequently, the synthesis unit 125 acquires the imaging direction transmitted by each imaging device 20 via the communication processing unit 122, the acquisition unit 123, and the control unit 124 (step S203).

  Thereafter, the combining unit 125 identifies the direction of each camera icon from the acquired imaging direction (step S204). Then, the synthesizing unit 125 synthesizes each device icon at each icon position of the three-dimensional model (step S205). At this time, for the camera icon, the combining unit 125 arranges the camera icon in the three-dimensional model with the camera icon lens facing the direction specified in step S204. However, for a device having no orientation such as the sensor 40, the icon is synthesized at the corresponding icon position in the three-dimensional model without special consideration of the orientation.

  Then, the synthesizing unit 125 acquires the viewpoint positions associated with the viewpoint position IDs of all monitoring areas from the viewpoint information storage unit 112 (step S206). Subsequently, the synthesizing unit 125 performs two-dimensional projection by performing rendering using an arbitrary method such as projecting a three-dimensional model on a projection plane based on the acquired viewpoint position, a preset gazing point, and a field angle. An image is generated, and the generated two-dimensional projection image is set as an initial image (step S207). Note that, as shown in FIG. 9, the initial image is combined with a captured image captured by the imaging device 20 corresponding to each camera icon in a pop-up area set in the vicinity of each camera icon in the two-dimensional projection image. Has been.

[Image Processing Flow According to Embodiment 1]
Next, the flow of image processing according to the first embodiment will be described with reference to FIG. FIG. 10 is a flowchart illustrating an example of the flow of image processing according to the first embodiment. Note that the image processing according to Embodiment 1 refers to the processing in step S105.

  For example, when a camera icon is selected in the initial image by a user operation on the operation unit 121, the acquisition unit 123 captures a captured image captured by the imaging device 20 corresponding to the camera icon and an imaging direction ( Pan information, tilt information) and zoom information are acquired (step S301). Further, the synthesizing unit 125 acquires the captured image, pan information, tilt information, and zoom information acquired by the acquiring unit 123 via the control unit 124 and sets the three-dimensional model, icon, and icon position (camera position). Obtained from the three-dimensional model storage unit 111. Then, the synthesis unit 125 determines the viewpoint, the gazing point, and the angle of view of the three-dimensional model based on the camera position, pan information, tilt information, and zoom information. Then, the synthesizing unit 125 generates a two-dimensional projection image from the three-dimensional model data based on the determined viewpoint, gazing point, and angle of view (Step S302). At this time, the synthesizing unit 125 arranges each icon in the two-dimensional projection image based on the icon position, and the captured image captured by the imaging device 20 corresponding to the camera icon is displayed in a pop-up set near the camera icon. Is synthesized.

  FIG. 11 is a diagram illustrating an example of the two-dimensional projection image generated by the process of step S302. As described above, the two-dimensional projection image includes a camera icon arranged in the three-dimensional model and an image captured by the imaging device 20 corresponding to the camera icon.

  Returning to FIG. 10, the synthesizing unit 125 synthesizes the captured image with a predetermined region on the two-dimensional projection image generated in step S302 to generate a synthesized image (step S303).

  FIG. 12 is a diagram illustrating an example of a composite image generated by the process of step S303. As described above, in the composite image, the captured image actually captured by the imaging device 20 is superimposed on the two-dimensional projection image. Here, the processing in step S303 will be described in detail with reference to FIG. When synthesizing the captured image on the two-dimensional projection image, the synthesis unit 125 superimposes the captured image on the two-dimensional projection image so that the center position of the captured image coincides with the gazing point of the two-dimensional projection image. . In the example shown in FIG. 12, the center position of the captured image and the gazing point of the two-dimensional projection image both coincide with the center position of the synthesized image.

  Furthermore, in the composite image shown in FIG. 12, the captured image and the two-dimensional projection image are displayed at the same magnification in the composite image. As described above, the combining unit 125 adjusts the angle of view when generating the two-dimensional projection image according to the zoom information so that the captured image and the two-dimensional projection image are displayed at the same magnification. Thereby, the viewer of a composite image can grasp | ascertain intuitively which area in the whole monitoring area the captured image in a composite image imaged.

  In the present embodiment, the combining unit 125 matches the center position of the captured image with the point of interest of the two-dimensional projection image in the composite image, and further displays the captured image and the two-dimensional projection image at the same magnification. In this way, the angle of view of the two-dimensional projection image is adjusted. However, it is sufficient that at least the imaging direction of the captured image and the direction (observation direction) connecting the viewpoint and the gazing point of the two-dimensional projection image match. That is, as another example, the synthesis unit 125 may generate a synthesized image in which the center position of the captured image does not match the gaze point of the two-dimensional projection image. For example, in the synthesized image shown in FIG. 12, the captured image is arranged at the center of the synthesized image, but may be arranged at a lower right position instead. Further, the magnification of the captured image and the two-dimensional projection image may be different. Thus, if the imaging direction and the observation direction match, the imaging direction of the captured image does not match the gaze point of the two-dimensional projection image, or the magnification of the captured image and the two-dimensional projection image is different. Even so, the viewer can intuitively grasp the correspondence between the captured image and the two-dimensional projection image.

  A button-like image (button image) is further displayed at the lower left of the composite image shown in FIG. This button image is for “returning to the previous image”. When the button-like image is selected by a user operation, the button image is returned to the selection screen for an initial image or the like.

  The acquisition unit 123 according to the present embodiment acquires the imaging direction and zoom information from the imaging device 20 corresponding to the selected camera icon each time a camera icon is selected. As another example, The acquisition unit 123 may acquire the changed imaging direction or zoom information from the imaging device 20 each time the camera direction or zoom of the imaging device 20 is changed, and store the acquired imaging direction or zoom information. In this case, since the acquisition unit 123 always stores the imaging direction and zoom information corresponding to the actual state of the imaging device 20, in step S302, the imaging direction and zoom information stored in the acquisition unit 123 are stored. Will be used.

  As described above, the image processing apparatus 10 acquires a captured image in the imaging apparatus 20 and an imaging direction at the time of imaging, generates a two-dimensional projection image based on the viewpoint and the imaging direction of the three-dimensional model, and generates the two-dimensional projection image. The captured image is synthesized and output in a predetermined area of the two-dimensional projection image. As a result, the viewer of the composite image can intuitively grasp the area captured by the camera. In other words, the image processing apparatus 10 generates a two-dimensional projection image in accordance with the captured image actually captured by the imaging apparatus 20 and the imaging direction at the time of imaging, and places the captured image in a predetermined region of the two-dimensional projection image. Compared with the prior art in which the viewpoint position and line-of-sight direction in the three-dimensional model do not match the viewpoint position and line-of-sight direction in the actual camera image, the viewer of the composite image is captured by the camera. Can intuitively grasp the area.

[Modification of Embodiment 1]
In the first embodiment, the case where the predetermined area is superimposed near the center of the two-dimensional projection image has been described. However, the position of the predetermined area can be arbitrarily changed. FIG. 13 is a diagram illustrating an example of changing the position of the predetermined area. For example, as shown in FIG. 13, the position of the predetermined area on the two-dimensional projection image may be arranged at the lower right position of the screen. In this way, by arranging the predetermined area in a place different from the vicinity of the center, the corresponding camera icon appears and the direction of the camera can also be understood, so that the area captured by the camera can be grasped intuitively. The effect that can be maintained.

(Embodiment 2)
[Transmit through the specified object]
In the first embodiment, a case has been described in which a building or tree model, icons of various devices, captured images of predetermined areas, pop-ups, and the like are combined and displayed on a two-dimensional projection image. It may be displayed in a transparent manner.

  FIG. 14A is a diagram illustrating an example of a two-dimensional projection image before transmitting a predetermined object. FIG. 14B is a diagram illustrating an example of a two-dimensional projection image after transmitting a predetermined object. For example, in the two-dimensional projection image shown in FIG. 14A, a camera icon “camera 4” and a pop-up of the camera icon are hidden behind a building located near the center. In such a state, when the user selects an icon, it may be difficult to select, or even not selectable.

  Therefore, as illustrated in FIG. 14B, the combining unit 125 transmits a model such as a building, a tree, or the ground that is not selected, and facilitates selection of the camera icon “camera 4”. Further, the transmission of the predetermined object may be performed in the initial image. Further, what is to be transmitted is not limited to the above, and for example, a predetermined area on a pop-up or a two-dimensional projection image may be transmitted. As a result, it is possible to improve the visibility of icons and pop-ups that may be hidden in buildings, predetermined areas, etc., and to improve the operability of camera icon selection operations.

(Embodiment 3)
[Adjust popup size]
In the first embodiment, the case where each icon is combined with the two-dimensional projection image according to the icon position of each icon has been described. However, the popup set near the icon combined with the two-dimensional projection image according to the icon position. You may adjust the size of. Specifically, since the icons and pop-ups are combined with the two-dimensional projection image according to the icon position of the icon, the size thereof is smaller as the icons and pop-ups located in the back of the two-dimensional projection image become smaller. Visibility and operability may decrease with icons and pop-ups. Therefore, the size of the pop-up set near the icon synthesized with the two-dimensional projection image is adjusted according to the icon position.

  FIG. 15A is a diagram illustrating an example of a two-dimensional projection image before adjusting the size of the pop-up. FIG. 15B is a diagram illustrating an example of a two-dimensional projection image after adjusting the size of the pop-up. For example, in the two-dimensional projection image shown in FIG. 15A, since the icons are combined with the two-dimensional projection image according to the icon position, the pop-up of the camera icon “camera 3” is displayed particularly small. The pop-up displays a picked-up image in the image pickup device 20 corresponding to “camera 3”, but if the display is too small, it is not preferable in terms of visibility.

  Therefore, as illustrated in FIG. 15B, the combining unit 125 adjusts the size of the pop-up of the camera icons “Camera 3” and “Camera 4” according to the pop-up of the camera icon “Camera 1”. Adjust the size to ensure the sex. In addition, the size of the pop-up can be adjusted according to the size of the pop-up positioned closest to the front in the two-dimensional projection image, or can be adjusted according to the size determined appropriately according to the display screen size. good. Also, the size of the pop-up may be adjusted in the initial image. Although the size of the camera icon may be adjusted, it is not necessary to adjust the camera icon so that the positional relationship of the camera in the two-dimensional projection image can be understood. As a result, the visibility of the pop-up can be improved.

(Embodiment 4)
In the first embodiment, the case has been described in which the composite image is generated based on the captured image captured by the imaging device 20 and the imaging direction. However, the imaging direction of the imaging device 20 is determined according to a predetermined user operation on the composite image. It is also possible to generate a composite image based on the captured image captured by the imaging device 20 and the imaging direction received according to the instruction. Therefore, in the fourth embodiment, the imaging direction of the imaging device 20 is instructed according to a predetermined user operation on the composite image, and the captured image captured by the imaging device 20 and the imaging direction received in response to the instruction are displayed. A case where a composite image is generated based on FIG.

[Configuration of Image Processing Apparatus According to Embodiment 4]
The configuration of the image processing apparatus according to the fourth embodiment will be described with reference to FIG. FIG. 16 is a diagram illustrating a configuration example of an image processing apparatus according to the fourth embodiment. In FIG. 16, detailed description of the configuration having the same function as that of the image processing apparatus 10 according to Embodiment 1 may be omitted.

  For example, as illustrated in FIG. 16, the image processing apparatus 10 includes a three-dimensional model storage unit 111, a viewpoint information storage unit 112, an operation unit 121, a communication processing unit 122, an acquisition unit 123, and a control unit 124. A synthesizing unit 125, an output unit 126, and a switching instruction unit 127.

  The switching instruction unit 127 calculates the imaging direction or the focal length of the zoom lens according to a predetermined user operation on the composite image, and instructs the imaging apparatus 20 to switch to the calculated imaging direction or the focal length of the zoom lens. Do. Here, in order to describe the processing by the switching instruction unit 127, the processing by the operation unit 121 and the control unit 124 related to the processing by the switching instruction unit 127 will be described.

The operation unit 121 receives a drag operation with a mouse or a touch panel in a state where a camera icon is selected by a user operation, that is, in a state where a composite image is displayed.
For example, the drag operation refers to a drag operation by a mouse as a pointing device and a wheel operation by a mouse wheel from an arbitrary position in a two-dimensional projection image or a predetermined region of the two-dimensional projection image. Alternatively, the drag operation refers to a drag operation or a pinch operation on the touch panel from an arbitrary position in a two-dimensional projection image or a predetermined area of the two-dimensional projection image. Further, the control unit 124 outputs the direction and distance of the drag operation received by the operation unit 121 to the switching instruction unit 127.

  Based on the direction and distance of the drag operation output by the control unit 124, the switching instruction unit 127, the focal length of the zoom lens of the imaging device 20 corresponding to the selected camera icon, the moving direction and movement of the imaging direction Calculate the angle. The moving direction of the imaging direction is the direction of the drag operation on the composite image. The focal length of the zoom lens is obtained by converting the wheel operation or pinch operation on the composite image into the focal length of the zoom lens. Further, the movement angle in the imaging direction is obtained by converting the distance (pixel) of the drag operation on the composite image into the movement angle (degree) in the imaging direction. As an example, the movement angle in the imaging direction is 0.05 degrees when one pixel of the composite image is converted into the movement angle in the imaging direction.

  Then, the switching instruction unit 127 corresponds to the selected camera icon via the communication processing unit 122 with an instruction to switch according to the calculated focal length of the zoom lens and the moving direction and moving angle of the imaging direction. To the imaging device 20 to be performed. After the instruction by the switching instruction unit 127, a two-dimensional projection image corresponding to the pan information, tilt information, and zoom information received from the imaging device 20 is generated as in the first embodiment, and a predetermined region of the two-dimensional projection image is generated. The synthesized image is generated by synthesizing the captured image.

[Screen transition according to Embodiment 4]
Next, screen transitions caused by a drag operation according to Embodiment 4 will be described with reference to FIGS. 17A to 17C. FIG. 17A is a diagram illustrating an example of a composite image before a drag operation is performed. FIG. 17B is a diagram illustrating an example of a composite image when a drag operation is performed in the right direction. FIG. 17C is a diagram illustrating an example of a composite image when a drag operation is performed in the upward direction.

  For example, as illustrated in FIG. 17A, the composite image before the drag operation is performed as a two-dimensional projection image obtained by the imaging device 20 corresponding to the camera icon “camera 2” described in the first embodiment. Are synthesized in a predetermined area. Hereinafter, in the composite image shown in FIG. 17A, screen transition when the user performs a drag operation in the right direction is shown in FIG. 17B, and screen transition when the user performs a drag operation in the upward direction is shown in FIG. 17C.

  In FIG. 17A, when a drag operation is performed in the right direction, the switching instruction unit 127 displays the zoom lens of the imaging device 20 corresponding to the camera icon “camera 2” based on the direction (right direction) of the drag operation and the distance. The focal length, the moving direction and moving angle in the imaging direction are calculated, and the imaging device 20 is instructed to set the focal length of the zoom lens of the camera, the camera orientation to the left, and the moving angle. As described above, the focal length, moving direction, and moving angle of the zoom lens can be obtained by converting the direction and distance of the drag operation, the wheel operation, or the pinch operation, respectively. The direction of the drag operation is opposite to the direction of camera movement. Thereafter, the synthesizing unit 125 generates a two-dimensional projection image corresponding to the new pan information, tilt information, and zoom information received from the imaging device 20, and applies the generated two-dimensional projection image to a predetermined region at this time. By synthesizing the captured images, a synthesized image shown in FIG. 17B is generated.

  In FIG. 17A, when a drag operation is performed in the upward direction, the switching instruction unit 127 captures an image of the image capturing apparatus 20 corresponding to the camera icon “camera 2” based on the drag operation direction (upward direction) and the distance. The movement direction and the movement angle of the direction are calculated, and the imaging apparatus 20 is instructed to move the camera to the down direction, the movement angle, and the like. In this case, since there is no operation relating to the focal length of the zoom lens, no instruction to change the focal length of the zoom lens is given. As described above, the moving direction and the moving angle are obtained by converting the direction and distance of the drag operation, the wheel operation or the pinch operation, respectively. Thereafter, the synthesizing unit 125 generates a two-dimensional projection image corresponding to the new pan information, tilt information, and zoom information received from the imaging device 20, and applies the generated two-dimensional projection image to a predetermined region at this time. By synthesizing the captured images, a synthesized image shown in FIG. 17C is generated.

[Overall Processing Flow According to Embodiment 4]
Next, the overall processing flow according to the fourth embodiment will be described with reference to FIG. FIG. 18 is a flowchart illustrating an example of the flow of overall processing according to the fourth embodiment. In FIG. 18, description of the same processing as the overall processing according to Embodiment 1 shown in FIG. 7 may be omitted. Specifically, the processes in steps S401 to S406 are the same as the processes in steps S101 to S106.

  For example, as illustrated in FIG. 18, when the operation unit 121 of the image processing apparatus 10 accepts an operation for switching the imaging direction of the camera on the composite image (Yes in step S407), the switching instruction unit according to the drag operation. After the movement direction of the imaging direction calculated by 127, the focal length of the zoom lens, and the movement angle of the imaging direction are transmitted to the imaging device 20, the combining unit 125 receives the pan information, tilt information, and the received information from the imaging device 20. A two-dimensional projection image is generated based on the zoom information, and a composite image is generated by combining the captured image of the imaging device 20 corresponding to the selected camera icon in a predetermined region of the generated two-dimensional image (step S408). . Then, the output unit 126 displays the composite image on the display screen (step S409). On the other hand, when the operation unit 121 has not received an operation for switching the imaging direction of the camera on the composite image (No in step S407), the process is terminated.

[Image Processing Flow According to Embodiment 4]
Next, the flow of image processing according to the fourth embodiment will be described with reference to FIG. FIG. 19 is a flowchart illustrating an example of the flow of image processing according to the fourth embodiment. Note that the image processing according to the fourth embodiment mainly refers to the processing in step S408.

  For example, as illustrated in FIG. 19, when a drag operation or a pinch operation (wheel operation) is received in a state where a camera icon is selected by a user operation on the operation unit 121, the switching instruction unit 127 is a drag operation. Is determined (step S501). At this time, when the operation is a drag operation (Yes at Step S501), the switching instruction unit 127 calculates the movement direction and movement angle of the imaging direction with respect to the imaging device 20 according to the drag distance on the composite image (Step S502). . On the other hand, when the operation is a pinch operation (No in step S501), the switching instruction unit 127 calculates the focal length of the zoom lens with respect to the imaging device 20 according to the pinch distance on the composite image (step S503). The moving direction of the imaging direction is opposite to the direction of the drag operation. The movement angle in the imaging direction can be obtained by converting the distance (pixel) of the drag operation into the movement angle (degree) in the imaging direction as one aspect. Further, the focal length of the zoom lens can be obtained by converting a pinch operation or a wheel operation into a focal length of the zoom lens as one aspect.

  Then, the switching instruction unit 127 transmits the focal length of the zoom lens, the moving direction and moving angle of the imaging direction, and the like as control signals to the imaging device 20 via the communication processing unit 122, thereby focusing the zoom lens. An instruction to switch between the distance and the imaging direction is given to the imaging apparatus 20 (step S504). Thereafter, when the captured image, pan information, tilt information, and zoom information are received by the communication processing unit 122 from the imaging device 20 corresponding to the selected camera icon (Yes in step S505), the combining unit 125 The captured image, pan information, tilt information, and zoom information acquired by the acquisition unit 123 are acquired via the control unit 124, and the 3D model, icon, and icon position (camera position) are acquired from the 3D model storage unit 111. get. On the other hand, when the captured image, pan information, tilt information, and zoom information are not received from the image capturing apparatus 20 (No in step S505), the communication processing unit 122 waits for reception of the captured image, pan information, tilt information, and zoom information. It becomes a state.

  Then, the synthesis unit 125 determines a viewpoint, a gazing point, and an angle of view for the three-dimensional model based on the camera position, pan information, tilt information, and zoom information. Subsequently, the synthesis unit 125 generates a two-dimensional projection image from the three-dimensional model data based on the determined viewpoint, gazing point, and angle of view (step S506). At this time, the synthesizing unit 125 arranges each icon in the two-dimensional projection image based on the icon position, and the captured image captured by the imaging device 20 corresponding to the camera icon is displayed in a pop-up set near the camera icon. Is synthesized. Thereafter, the combining unit 125 generates a combined image by combining the captured image with a predetermined area on the generated two-dimensional projection image (step S507).

  As described above, the image processing apparatus 10 instructs the focal length and the imaging direction of the zoom lens of the imaging apparatus 20 by a drag operation by the user on the composite image, and the zoom lens of the imaging by the imaging apparatus 20 according to the instruction. A two-dimensional projection image corresponding to the focal length and the imaging direction is generated, and the captured image is synthesized with a predetermined region of the generated two-dimensional projection image and output. As a result, the viewer of the composite image can switch the camera of the imaging device 20 with a simple user operation, and can intuitively grasp the area imaged by the camera.

[Modification of Embodiment 4]
In the fourth embodiment, a case in which an imaging direction or the like of the imaging device 20 is instructed according to a drag operation by the user on the composite image, and a composite image is generated based on the imaging direction or the like received from the imaging device 20 is described. As described above, it is also possible to generate a two-dimensional projection image in response to a drag operation by the user on the composite image and display it before the imaging direction and the like are received from the imaging device 20. Specifically, a two-dimensional projection image is generated according to a drag operation by the user on the composite image, the generated two-dimensional projection image is displayed, and then the imaging direction received from the imaging device 20 is changed. The captured image is synthesized and displayed in a predetermined area of the two-dimensional projection image generated accordingly.

  FIG. 20 is a flowchart illustrating an example of the flow of image processing according to a modification of the fourth embodiment. For example, as illustrated in FIG. 20, when a drag operation or a pinch operation (wheel operation) is received in a state where a camera icon is selected by a user operation on the operation unit 121, the switching instruction unit 127 is a drag operation. It is determined whether or not (step S601). At this time, in the case of a drag operation (Yes in step S601), the switching instruction unit 127 calculates the movement direction and movement angle of the imaging direction with respect to the imaging device 20 according to the drag distance on the composite image (step S602). ). On the other hand, when the operation is a pinch operation (No at Step S601), the switching instruction unit 127 calculates the focal length of the zoom lens with respect to the imaging device 20 according to the pinch distance on the composite image (Step S603). The moving direction of the imaging direction is opposite to the direction of the drag operation. The distance in the imaging direction can be obtained by converting the distance (pixel) of the drag operation into the moving angle (degree) in the imaging direction as one aspect. Further, the focal length of the zoom lens can be obtained by converting a pinch operation or a wheel operation into a focal length of the zoom lens as one aspect.

  Here, the switching instruction unit 127 instructs the synthesizing unit 125 to generate a two-dimensional projection image corresponding to the calculated focal length of the zoom lens and the moving direction and moving angle of the imaging direction. Thereby, the composition unit 125 generates a two-dimensional projection image according to the focal length of the zoom lens and the moving direction and moving angle in the imaging direction (step S604). At this time, the two-dimensional projection image generated by the synthesis unit 125 is displayed on the screen by the output unit 126.

  Then, the switching instruction unit 127 transmits the focal length of the zoom lens, the movement direction and the movement angle of the imaging direction, and the like as control signals to the imaging device 20 via the communication processing unit 122, so that the imaging direction and zoom An instruction to switch the focal length of the lens is given to the imaging device 20 (step S605). Further, in the user operation on the operation unit 121, when the drag operation or the pinch operation is not performed for a predetermined time (Yes in step S606), the switching instruction unit 127 is connected to the imaging device 20 via the communication processing unit 122. A request for transmitting the captured image, pan information, tilt information, and zoom information is made (step S607). On the other hand, when the user operation (drag operation or pinch operation) on the operation unit 121 is performed within a predetermined time (No at Step S606), the process at Step S601 is performed again. Fine adjustments may be made in user operations such as a drag operation or pinch operation by the user. If the drag operation is not performed continuously even after a predetermined time has elapsed, the adjustment at that point is confirmed. As a result, the subsequent processing is executed. In other words, while the user operation such as the drag operation or the pinch operation is being performed, it is not determined from which direction the user wants to monitor. Therefore, if the user operation is not performed after a predetermined time has elapsed. Assuming that the user wants to monitor from the current direction, the subsequent processing is executed.

  Subsequently, the switching instruction unit 127 sends a transmission request for the captured image, pan information, tilt information, and zoom information to the imaging device 20 corresponding to the selected camera icon (step S607). At this time, the imaging device 20 is operating based on the information transmitted in step S605. Thereafter, when the captured image, pan information, tilt information, and zoom information are received from the imaging device 20 corresponding to the selected camera icon by the communication processing unit 122 (Yes in step S608), the combining unit 125 The captured image, pan information, tilt information, and zoom information acquired by the acquisition unit 123 are acquired via the control unit 124, and the 3D model, icon, and icon position (camera position) are acquired from the 3D model storage unit 111. get. On the other hand, when the captured image, pan information, tilt information, and zoom information are not received from the imaging device 20 (No in step S608), the communication processing unit 122 waits for reception of the captured image, pan information, tilt information, and zoom information. It becomes a state.

  Then, the synthesis unit 125 determines the viewpoint, the gazing point, and the angle of view of the three-dimensional model based on the camera position, pan information, tilt information, and zoom information. Subsequently, the synthesizing unit 125 generates a two-dimensional projection image from the three-dimensional model data based on the determined viewpoint, gazing point, and angle of view (step S609). At this time, the synthesizing unit 125 arranges each icon in the two-dimensional projection image based on the icon position, and the captured image captured by the imaging device 20 corresponding to the camera icon is displayed in a pop-up set near the camera icon. Is synthesized. Thereafter, the combining unit 125 generates a combined image by combining the captured image with a predetermined area on the generated two-dimensional projection image (step S610).

  That is, the image processing apparatus 10 generates a two-dimensional projection image corresponding to a drag operation by the user on the composite image before generating the two-dimensional projection image based on the information from the imaging apparatus 20. Since the display is performed, the camera switching position of the imaging apparatus 20 can be matched to a position desired by the user to some extent, and the operability related to the camera switching can be improved.

(Embodiment 5)
In the second embodiment, a case has been described in which a camera icon or the like that is hidden behind a building or the like can be selected by transmitting a predetermined object. However, a camera that is selected in a state where the camera icon is selected. It is also possible to select a camera icon or the like that appears by rotating the composite image with the icon as an axis. Therefore, in the fifth embodiment, a case will be described in which, when a camera icon is selected, a camera icon or the like that appears by rotating the composite image with the selected camera icon as an axis can be selected.

[Image Processing Flow According to Embodiment 5]
FIG. 21 is a flowchart illustrating an example of the flow of image processing according to the fifth embodiment. Note that the image processing according to the fifth embodiment mainly refers to the processing in step S408.

  For example, as illustrated in FIG. 21, the composition unit 125 rotates the composite image according to the user's drag operation on the operation unit 121 with the selected camera icon as an axis (step S <b> 701). However, the rotation range of the composite image may be limited. The rotated composite image is generated and displayed as appropriate according to the drag direction, drag distance, and the like. In addition, in the user operation on the operation unit 121, when another camera icon different from the currently selected camera icon is not selected within a predetermined time (No in step S702), the switching instruction unit 127 displays the composite image. The distance in the imaging direction with respect to the imaging device 20 corresponding to the drag distance is calculated. The switching instruction unit 127 also calculates the focal length of the zoom lens with respect to the imaging device 20 according to the pinch distance on the composite image even when a pinch operation (wheel operation) is performed. Since the calculation of the distance in the imaging direction and the focal length of the zoom lens may be performed in the same manner as in the fourth embodiment, detailed description thereof is omitted here. However, since the distance in the imaging direction, the focal length of the zoom lens, and the like are limited in the actual camera (imaging device 20), they are changed to suitable values according to the limitation.

  Then, the switching instruction unit 127 transmits the focal length of the zoom lens, the movement direction and the movement angle of the imaging direction, and the like as control signals to the imaging device 20 via the communication processing unit 122, thereby obtaining the imaging direction. An instruction to switch the focal length of the zoom lens is given to the imaging device 20 (step S703). On the other hand, when another camera icon different from the currently selected camera icon is selected within the predetermined time on the operation unit 121 (Yes in step S702), the process of step S701 is executed again.

  Thereafter, when the captured image, pan information, tilt information, and zoom information are received by the communication processing unit 122 from the imaging device 20 corresponding to the selected camera icon (Yes in step S704), the combining unit 125 The captured image, pan information, tilt information, and zoom information acquired by the acquisition unit 123 are acquired via the control unit 124, and the 3D model, icon, and icon position (camera position) are acquired from the 3D model storage unit 111. get. On the other hand, when the captured image, pan information, tilt information, and zoom information are not received from the image capturing apparatus 20 (No in step S704), the communication processing unit 122 waits for reception of the captured image, pan information, tilt information, and zoom information. It becomes a state.

  Then, the synthesis unit 125 determines the viewpoint, the gazing point, and the angle of view of the three-dimensional model based on the camera position, pan information, tilt information, and zoom information. Subsequently, the composition unit 125 generates a two-dimensional projection image from the three-dimensional model data based on the determined viewpoint, gazing point, and angle of view (step S705). At this time, the synthesizing unit 125 arranges each icon in the two-dimensional projection image based on the icon position, and the captured image captured by the imaging device 20 corresponding to the camera icon is displayed in a pop-up set near the camera icon. Is synthesized. Thereafter, the combining unit 125 generates a combined image by combining the captured image with a predetermined region on the generated two-dimensional projection image (step S706).

  As described above, since the image processing apparatus 10 rotates around the camera icon on the composite image in accordance with the drag operation, the viewer of the composite image selects a camera icon that does not appear in the composite image before the rotation. Can be easily performed. Further, in the rotation of the composite image, the image processing apparatus 10 also performs the imaging direction of the camera and the focus of the zoom lens when no other camera icon is selected even after a predetermined time has elapsed. Since switching of the distance is instructed, the viewer of the composite image can intuitively grasp the area captured by the camera.

(Embodiment 6)
In the fourth embodiment, the case where the imaging direction of the imaging device 20 and the focal length of the zoom lens are set by a drag operation has been described. However, the operation speed of the camera toward the imaging direction is further set at a speed according to the drag distance. You can also. Therefore, in the sixth embodiment, a case will be described in which the operation speed of the camera directed in the imaging direction is further set at a speed according to the drag distance.

[Screen transition according to Embodiment 6]
The screen transition according to the sixth embodiment will be described with reference to FIGS. FIG. 22 is an image diagram showing an example of determining a base point according to the sixth embodiment. FIG. 23 is an image diagram showing an example of setting the camera operating speed according to the drag distance according to the sixth embodiment.

  As illustrated in FIG. 22, when a mouse operation is performed on a composite image in a user operation on the operation unit 121, the switching instruction unit 127 uses the position on the composite image that is mouse-down or touch-down as a base point. Determine as. When a drag operation is performed in the user operation on the operation unit 121 after the mouse down or the touch down, as illustrated in FIG. 23, the switching instruction unit 127 sets the direction opposite to the drag direction as the moving direction of the image capturing direction and performs image capturing. The moving speed when operating the camera in the moving direction is set to a speed corresponding to the drag distance.

  For example, as shown in FIG. 23, a range in which the panning and tilting operations of the camera are accelerated stepwise is set in advance according to the distance from the base point. Then, the switching instruction unit 127 determines the moving speed when the camera is operated in the moving direction of the imaging direction based on the speed set stepwise according to the drag distance from the base point determined by the mouse down or the touch down. Set. In the example shown in FIG. 23, the range of the moving speed that is set is set so as not to move if it is in the vicinity of the base point, and after passing through the vicinity of the range, “low speed”, “medium speed”, It is set to be “fast”. The moving speed of the camera corresponding to each of “low speed”, “medium speed”, and “high speed” may be set arbitrarily. Further, the speed range shown in FIG. 23 can be arbitrarily changed, and the display may not be performed.

[Image Processing Flow According to Embodiment 6]
Next, the flow of image processing according to the sixth embodiment will be described with reference to FIG. FIG. 24 is a flowchart illustrating an example of the flow of image processing according to the sixth embodiment. Note that image processing according to the sixth embodiment mainly refers to processing in step S408.

  For example, as illustrated in FIG. 24, when the mouse is down or down on the composite screen by a user operation on the operation unit 121, the switching instruction unit 127 displays the position on the composite image that is mouse down or touched down. The base point is determined (step S801). When a drag operation is performed on the operation unit 121, the switching instruction unit 127 determines a moving speed according to the determined drag distance from the base point (step S802).

  Here, the switching instruction unit 127 determines whether or not the determined movement direction and movement speed of the imaging direction are different from the instruction currently being given to the imaging device 20 (step S803). At this time, when the switching instruction unit 127 determines that the determined moving direction and moving speed in the imaging direction are different from the currently performed instruction (Yes in step S803), the switching instruction unit 127 determines via the communication processing unit 122. By transmitting the moving direction and moving speed of the image capturing direction, the image capturing apparatus 20 is instructed to switch the image capturing direction (step S804). On the other hand, when the switching instruction unit 127 determines that the determined moving direction and moving speed in the imaging direction are the same as the instruction that is currently being performed (No in step S803), the switching instruction unit 127 executes the process of step S802 again. .

  Thereafter, when the captured image, pan information, and tilt information are received from the imaging device 20 corresponding to the selected camera icon by the communication processing unit 122 (Yes in step S805), the combining unit 125 controls the control unit 124. In addition, the captured image, pan information, and tilt information acquired by the acquisition unit 123 are acquired, and the 3D model, icon, and icon position (camera position) are acquired from the 3D model storage unit 111. On the other hand, when the captured image and the imaging direction are not received from the imaging device 20 (No in step S805), the communication processing unit 122 enters a state of waiting for reception of the captured image and the imaging direction.

  Then, the combining unit 125 determines the viewpoint and the gazing point of the three-dimensional model based on the camera position, pan information, and tilt information. Subsequently, the synthesizing unit 125 generates a two-dimensional projection image from the three-dimensional model data based on the determined viewpoint, gazing point, and preset angle of view (step S806). At this time, the synthesizing unit 125 arranges each icon in the two-dimensional projection image based on the icon position, and the captured image captured by the imaging device 20 corresponding to the camera icon is displayed in a pop-up set near the camera icon. Is synthesized. Thereafter, the combining unit 125 generates a combined image by combining the captured image with a predetermined area on the generated two-dimensional projection image (step S807).

  As described above, the image processing apparatus 10 instructs the imaging direction of the imaging apparatus 20 and the speed of the camera operation by a drag operation by the user on the composite image, and responds to the imaging direction of imaging by the imaging apparatus 20 according to the instruction. The two-dimensional projection image is generated, and the captured image is synthesized and output to a predetermined area of the generated two-dimensional projection image. As a result, the viewer of the composite image can switch the camera of the imaging device 20 with a simple user operation, and can intuitively grasp the area imaged by the camera.

(Embodiment 7)
In the fourth embodiment, the case where the focal length of the zoom lens of the imaging apparatus 20 and the moving direction and moving speed of the imaging direction are set by a drag operation has been described, but the moving direction in the imaging direction is directed at a speed corresponding to the drag speed. You can also set the camera speed. Therefore, in the seventh embodiment, a case will be described in which the moving speed of the camera toward the moving direction of the imaging direction is further set according to the drag speed.

[Image processing according to Embodiment 7]
Image processing according to the seventh embodiment will be described with reference to FIGS. 25 and 26. FIG. FIG. 25 is an image diagram showing an example of determining a base point according to the seventh embodiment. FIG. 26 is an image diagram showing an example of setting the camera operation speed according to the drag speed according to the seventh embodiment.

  As shown in FIG. 25, when the mouse operation or touchdown is performed on the composite image in the user operation on the operation unit 121, the switching instruction unit 127 uses the position on the composite image that is mousedown or touchdown as a base point. Determine as. When a drag operation is performed by a user operation on the operation unit 121 after the mouse down or touch down, the switching instruction unit 127 sets the direction opposite to the drag direction as the movement direction of the imaging direction, and moves the camera in the movement direction of the imaging direction. Set the movement speed for operation to the speed corresponding to the drag speed. For example, as shown in FIG. 26, when dragging like an arrow is performed, the “base point”, “reference point”, and “current point” of the drag operation are set as shown in FIG. When a drag operation is performed, the switching instruction unit 127 sets the moving direction of the imaging direction based on the direction from the base point to the current point, and performs imaging based on the drag speed calculated from the distance from the reference point to the current point. Set the movement speed when operating the camera in the direction of movement.

[Image Processing Flow According to Embodiment 7]
Next, the flow of image processing according to the seventh embodiment will be described with reference to FIG. FIG. 27 is a flowchart illustrating an example of the flow of image processing according to the seventh embodiment. Note that image processing according to the seventh embodiment mainly refers to processing in step S408.

  For example, as shown in FIG. 27, when the mouse is down or touched down on the composite screen by a user operation on the operation unit 121, the switching instruction unit 127 displays the position on the composite image that is mouse down or touched down. The base point is determined (step S901). In addition, the switching instruction unit 127 calculates a position on the composite image every predetermined time, and sets the latest position as the current point and the previous position as the reference point. Then, the switching instruction unit 127 determines the moving direction of the imaging direction according to the direction from the base point to the current point. Further, the speed of the drag is calculated by dividing the distance between the reference point and the current point by a predetermined time, and the moving speed is set according to the speed of the drag (step S902).

  Here, the switching instruction unit 127 determines whether or not the determined movement direction and movement speed in the imaging direction are different from the instruction currently being given to the imaging device 20 (step S903). At this time, when the switching instruction unit 127 determines that the set moving direction and moving speed in the imaging direction are different from the currently performed instruction (Yes in step S903), the switching instruction unit 127 sets the setting via the communication processing unit 122. By transmitting the moving direction and moving speed of the imaging direction, the imaging device 20 is instructed to switch the imaging direction (step S904). On the other hand, when the switching instruction unit 127 determines that the set movement direction and movement speed in the imaging direction are the same as the instruction that is currently being made (No at step S903), the switching instruction unit 127 executes the process at step S902 again. .

  Thereafter, when the captured image, pan information, and tilt information are received by the communication processing unit 122 from the imaging device 20 corresponding to the selected camera icon (Yes in step S905), the combining unit 125 controls the control unit 124. In addition, the captured image, pan information, and tilt information acquired by the acquisition unit 123 are acquired, and the 3D model, icon, and icon position (camera position) are acquired from the 3D model storage unit 111. On the other hand, when the captured image, pan information, and tilt information are not received from the image capturing apparatus 20 (No in step S905), the communication processing unit 122 enters a state of waiting for reception of the captured image, pan information, and tilt information.

  Then, the combining unit 125 determines the viewpoint and the gazing point of the three-dimensional model based on the camera position, pan information, and tilt information. Subsequently, the synthesizing unit 125 generates a two-dimensional projection image from the three-dimensional model data based on the determined viewpoint, gazing point, and preset angle of view (step S906). At this time, the synthesizing unit 125 arranges each icon in the two-dimensional projection image based on the icon position, and the captured image captured by the imaging device 20 corresponding to the camera icon is displayed in a pop-up set near the camera icon. Is synthesized. Thereafter, the synthesizing unit 125 synthesizes the captured image with a predetermined region on the generated two-dimensional projection image to generate a synthesized image (step S907).

  As described above, the image processing apparatus 10 instructs the moving direction and moving speed of the image capturing apparatus 20 by a drag operation by the user on the composite image, and the pan information sent from the image capturing apparatus 20 according to the instruction. A two-dimensional projection image corresponding to the tilt information is generated, and the captured image is synthesized with a predetermined region of the generated two-dimensional projection image and output. As a result, the viewer of the composite image can switch the imaging direction of the camera of the imaging device 20 with a simple user operation, and can intuitively grasp the area captured by the camera.

(Embodiment 8)
In the first embodiment, a camera icon is arranged on a two-dimensional projection image of all monitoring areas, and when a camera icon is selected, a captured image captured by a camera (imaging device) corresponding to the selected camera icon is synthesized. A composite image is displayed. On the other hand, in the present embodiment, a plurality of monitoring areas are arranged in a hierarchy, and a monitoring area of another hierarchy can be selected from a two-dimensional projection image of any monitoring area and is selected. When the camera icon of only the camera associated with the monitoring area is arranged in the two-dimensional projection image of the monitored area and the camera icon is selected, the camera corresponding to the selected camera icon has taken an image. A composite image obtained by combining captured images is displayed. In the following, the configuration of the monitoring system and the configuration of the imaging apparatus are the same as those in the first embodiment, and thus description thereof is omitted.

[Configuration of Image Processing Apparatus According to Embodiment 8]
Next, the configuration of the image processing apparatus 10 according to the eighth embodiment will be described with reference to FIG. 3 in the first embodiment. The image processing apparatus 10 includes a three-dimensional model storage unit 111, a viewpoint information storage unit 112, an operation unit 121, a communication processing unit 122, an acquisition unit 123, a control unit 124, a synthesis unit 125, and an output unit 126. And have. The image processing apparatus 10 is an information processing apparatus such as a PC connected to a network camera such as the imaging apparatus 20 or the sensor 40 via the network 30 (see FIG. 1). Here, since the communication processing unit 122 and the control unit 124 are the same as those in the first embodiment, description thereof is omitted.

  The three-dimensional model storage unit 111 stores a three-dimensional model and the like. FIG. 28 is a diagram illustrating an example of information stored in the three-dimensional model storage unit 111. As shown in FIG. 28, the three-dimensional model storage unit 111 stores the hierarchy to which the monitoring area belongs, the three-dimensional model data of each monitoring area, the device ID, the icon type ID, and the icon position in association with each other. . Here, the details of the device ID, icon type ID, and icon position are the same as those in the first embodiment.

  The three-dimensional model data indicates a three-dimensional model of each monitoring area (three-dimensional region) described above (see FIG. 5). In this embodiment, the monitoring area has a hierarchical structure in which a plurality of monitoring areas are configured in a hierarchical manner, and a plurality of three-dimensional model data corresponding to each of the plurality of monitoring areas is stored.

  The hierarchy indicates a hierarchy to which each monitoring area belongs. Here, the hierarchical structure of the monitoring area in the present embodiment will be described. FIG. 29 is an explanatory diagram of the hierarchical structure of the monitoring area. As shown in FIG. 29, in this embodiment, as a monitoring area, all monitoring areas indicating all monitoring areas, a mall area indicating a mall area that is a building in all monitoring areas, and each floor in the mall A central area, a west second floor area, a west first floor area, an east second floor area, and an east first floor area are provided. These monitoring areas are composed of three layers, all monitoring areas belong to the first layer, the mall area belongs to the second layer, which is the layer immediately below the first layer, and immediately below the second layer. The central area, the west second floor area, the west first floor area, the east second floor area, and the east first floor area belong to the third hierarchy which is a hierarchy.

  Returning to FIG. 28, for example, the 3D model storage unit 111 includes a hierarchy “first hierarchy”, 3D model data “all monitoring areas”, a device ID “# 01” indicating the camera 1, and an icon. An icon type ID “A001” indicating a camera as a type and an icon position “(x1, y1, z1)” are stored in association with each other. Therefore, by referring to the three-dimensional model storage unit 111, the arrangement position of each device can be specified as in the first embodiment. In the present embodiment, by referring to the three-dimensional model storage unit 111, it is possible to determine which hierarchy each monitoring area belongs to.

  Further, as shown in FIG. 28, the three-dimensional model storage unit 111 stores a three-dimensional part model of various icons in association with the icon type ID. The details are the same as those in the first embodiment, and thus the description thereof is omitted.

  The viewpoint information storage unit 112 is used when the synthesis unit 125 generates a two-dimensional projection image from the three-dimensional model data, or when a user generates a selection image to be displayed for selecting a monitoring area, a camera icon, or the like. Memorize the viewpoint position. FIG. 30 is a diagram illustrating an example of information stored in the viewpoint information storage unit 112. Specifically, the viewpoint information storage unit 112 generates two-dimensional projection images of the entire monitoring area, mall area, central area, west second floor area, west first floor area, east second floor area, and east first floor area. The viewpoint position used at the time and the viewpoint position used when generating the two-dimensional projection image of the monitoring area observed from the camera arrangement position are stored in association with the viewpoint position ID for identifying the viewpoint position. To do.

  Here, the selected image is a two-dimensional projection image obtained by projecting a three-dimensional model of any monitoring area onto a two-dimensional surface, and belongs to a layer different from the projected monitoring area, for example, a layer directly below or directly above. It is an image from which a monitoring area can be selected. When a camera is installed in the monitoring area, the selected image is a two-dimensional projection image of the monitoring area in which camera icons corresponding to only the cameras associated with the monitoring area among the cameras are arranged. This is an image in which a camera (imaging device) can be selected by selecting.

  Returning to FIG. 30, for example, the viewpoint information storage unit 112 associates the viewpoint position ID “B001” indicating all the monitoring areas belonging to the first hierarchy with the viewpoint position “(x10, y10, z10)”. Remember. Further, the viewpoint information storage unit 112 stores a viewpoint position ID “B003” indicating a central area belonging to the third hierarchy and a viewpoint position “(x14, y14, z14)” in association with each other. The viewpoint information storage unit 112 stores a viewpoint position ID “B011” indicating the camera 1 and a viewpoint position “(x11, y11, z11)” in association with each other.

  The operation unit 121 includes an input device such as a mouse or a touch panel, and accepts various instructions by a user operation of the user of the monitoring system 1. For example, the operation unit 121 receives a camera selection instruction (selection of a camera icon) from the user from the selected image. In addition, when a predetermined camera (imaging device 20) is selected by a user operation, the operation unit 121 receives an instruction to output a captured image of the selected imaging device 20. In addition, the operation unit 121 receives pan, tilt, and zoom setting instructions for a predetermined imaging device 20 by a user operation. Note that the pan, tilt, and zoom setting instructions are transmitted to the imaging apparatus 20 via the control unit 124 and the communication processing unit 122 as control signals related to the above-described pan, tilt, and zoom. In addition, the operation unit 121 receives an end instruction indicating that the process is to be ended from the user.

  In addition, the operation unit 121 accepts selection of a monitoring area belonging to a layer (other layer) different from the monitoring area displayed in the selected image, from the selected image on which the monitoring area is displayed. Specifically, the operation unit 121 accepts selection of a monitoring area belonging to a hierarchy immediately below or immediately above a hierarchy to which the monitoring area displayed in the selection image belongs, from a selection image on which the monitoring area is displayed. Furthermore, the operation unit 121 accepts selection of a monitoring area belonging to a lower layer or an upper layer than a hierarchy to which the monitoring area displayed in the selected image belongs, from the selected image in which the monitoring area is displayed.

  In addition, when a camera icon is displayed on the selected image, the operation unit 121 accepts selection of the camera icon from the selected image. For example, when a selection image that is a two-dimensional projection image of the entire monitoring area is displayed, the operation unit 121 displays, from the selection image, a mall area that belongs to the immediately lower layer, a central area that belongs to the lower layer, and the west second floor. Selection of camera icons of cameras associated with areas and all monitoring areas is accepted. The operation unit 121 corresponds to an area selection reception unit and a camera selection reception unit.

  FIG. 31 is an explanatory diagram showing monitoring areas or camera icons that can be selected from the selected image. As shown in FIG. 31, when there is a mall area in all the monitoring areas and cameras 1 to 5 are associated with each other, from the selected image that is a two-dimensional projection image of all the monitoring areas, the mall areas, cameras 1 to 5 are displayed. The camera icon can be selected. In addition, if there are a central area, a west second floor area, a west first floor area, an east second floor area, and an east first floor area in the mall area, and no camera is associated with these areas, two-dimensional projection of the mall area From the selected image, which is an image, the central area, the west second floor area, the west first floor area, the east second floor area, and the east first floor area can be selected.

  Further, when the cameras 6 and 7 are associated with the central area, the camera icons of the cameras 6 and 7 can be selected from the selection image that is a two-dimensional projection image of the central area. The same applies to the other second monitoring areas, the west second floor area, the west first floor area, the east second floor area, and the east first floor area.

  When the operation unit 121 receives a selection of a camera icon from the selected image, the acquisition unit 123 captures an image captured by the camera (imaging device 20) corresponding to the selected camera icon, pan information as an imaging direction, and tilt. Information and zoom information are acquired from the imaging device 20 via the communication processing unit 122. The acquisition unit 123 acquires detection data from the sensor 40. The detection data may include information related to the detection direction of the sensor 40 (pan information, tilt information, etc.).

  When the operation unit 121 accepts selection of a camera icon from the selected image, the combining unit 125 receives the viewpoint of the three-dimensional model of the monitoring area projected on the selected image and the camera (the imaging device 20) corresponding to the selected camera icon. ) Is generated in accordance with the imaging direction. The synthesizing unit 125 synthesizes a captured image, which is a two-dimensional image captured by the camera (imaging device 20) corresponding to the selected camera icon, in the two-dimensional image region of the generated two-dimensional projection image. At this time, the combining unit 125 also arranges icons such as camera icons and sensor icons output by the control unit 124 in the two-dimensional projection image based on the icon positions. The details of the composition of the captured image and the arrangement of the icons are the same as in the first embodiment.

  Further, when the combining unit 125 accepts selection of a monitoring area from the selected image by the operation unit 121, icon positions of camera icons, sensor icons, etc. in the three-dimensional model of the selected monitoring area (arranged in the selected monitoring area) The camera icon or sensor icon is placed at the camera position or the sensor position of the sensor. The synthesizing unit 125 acquires the viewpoint (viewpoint position) of the three-dimensional model of the monitoring area (selected monitoring area) stored in the viewpoint information storage unit 112, and a camera icon or a sensor icon according to the viewpoint. A selection image that is a two-dimensional projection image obtained by projecting a three-dimensional model of the monitoring area (selected monitoring area) on the two-dimensional surface is generated. At this time, if no camera or sensor is associated with the selected monitoring area, the camera icon or sensor icon is not arranged.

  The output unit 126 outputs the combined image generated by the combining unit 125. For example, the output unit 126 may include various icons combined with the two-dimensional projection image by the combining unit 125, a captured image combined with a region near the camera icon, or a captured image combined with the superimposed two-dimensional image region. A composite image including is output. The output unit 126 outputs the selected image generated by the combining unit 125. Note that the output unit 126 may be a display device that displays a composite image or a selected image, or may output the composite image or the selected image to a display device connected to the image processing device 10.

[Overall Processing Flow According to Embodiment 8]
Next, the flow of overall processing according to Embodiment 8 will be described using FIG. FIG. 32 is a flowchart illustrating an example of the flow of overall processing according to the eighth embodiment.

  Here, steps S2001 to S2009 are the same as the processing of FIG. 18 in the fourth embodiment, and thus the description thereof is omitted (see steps S401 to S409). At that time, in steps S2001 to S2003, the initial image in the fourth embodiment is replaced with the selected image. If the determination in step S2007 is negative, the process proceeds to step S2013.

  In step S2004, when the operation unit 121 does not accept selection of a camera icon from the selected image (No in step S2004), the control unit 124 determines whether the operation unit 121 accepts selection of a monitoring area from the selected image. (Step S2010).

  When the selection of the monitoring area is received from the selected image (Yes at Step S2010), the composition unit 125 generates a selection image that is a two-dimensional projection image of the selected monitoring area (Step S2011). Then, the output unit 126 displays the selected image on the display screen (step S2012). On the other hand, when the selection of the monitoring area is not received from the selected image (No at Step S2010), the process proceeds to Step S2013.

  Next, the control unit 124 determines whether or not an end instruction indicating that the process is to be ended is received by the operation unit 121 (step S2013). If an end instruction has been accepted (Yes at step S2013), the process ends. On the other hand, if an end instruction has not been accepted (No at step S2013), the process returns to step S2004. Note that, for example, an icon or the like for the user to input an end instruction is displayed on the selected image.

  Further, the initial image described in the fourth embodiment is one of the selected images. When the entire monitoring area is selected as the monitoring area by the user in step S2010, the initial image is displayed in step S2011. A selection image which is a two-dimensional projection image of all corresponding monitoring areas is displayed. By repeating the selection of the monitoring areas in this way, it is possible to sequentially switch and display the hierarchical monitoring areas.

  In addition, a selectable icon or the like indicating that the selected image returns to the selected image in the previous process is displayed, and when the selection of the icon is accepted, the selected image in the previous process is displayed ( Return). In addition, a selectable icon or the like indicating each monitoring area is displayed at the corner of the display screen where the selected image is displayed, and the selection is made when selection of one of the displayed monitoring area icons is accepted. A selection image that is a two-dimensional projection image of the monitoring area corresponding to the icon is generated, and the generated selection image is displayed. Thereby, it is possible to select not only the monitoring area immediately below the monitoring area displayed in the selected image, but also the monitoring area immediately above, the upper layer, and the lower layer.

[Initial Image Generation Processing Flow According to Embodiment 8]
Next, the flow of selected image generation processing according to Embodiment 8 will be described using FIG. FIG. 33 is a flowchart illustrating an example of the flow of selected image generation processing according to the eighth embodiment. The selected image generation process according to the eighth embodiment refers to the process in step S2011.

  As illustrated in FIG. 33, the composition unit 125 of the image processing apparatus 10 acquires the three-dimensional model data of the selected monitoring area from the three-dimensional model storage unit 111 (step S2021). The synthesizing unit 125 refers to the three-dimensional model storage unit 111 and determines whether there is a device (camera or sensor) associated with the three-dimensional model of the selected monitoring area (step S2022). If there is no associated device (No at step S2022), the process proceeds to step S2028 without combining icons.

  On the other hand, when there is a device associated (Yes in step S2022), the composition unit 125 refers to the three-dimensional model storage unit 111 and determines each device associated with the three-dimensional model of the selected monitoring area. An arrangement position, that is, each icon and the icon position of each icon are acquired (step S2023).

  Next, the composition unit 125 determines whether there is a camera (imaging device 20) in the associated device (step S2024). If there is no camera in the associated device (No at step S2024), the process proceeds to step S2027.

  On the other hand, when there is a camera in the associated device (Yes in step S2024), the synthesizing unit 125 sets the imaging direction of the camera transmitted by each imaging device 20 to the communication processing unit 122, the acquisition unit 123, and the control unit 124. (Step S2025).

  Thereafter, the combining unit 125 identifies the direction of each camera icon from the acquired imaging direction (step S2026). The synthesizing unit 125 synthesizes the icon of each device (only the camera, or the camera and the sensor) at each icon position of the three-dimensional model (step S227). At this time, for the camera icon, the combining unit 125 arranges the camera icon in the three-dimensional model with the lens of the camera icon directed in the direction specified in step S2026. However, for a device having no orientation such as the sensor 40, the icon is synthesized at the corresponding icon position in the three-dimensional model without special consideration of the orientation.

  Then, the synthesizing unit 125 acquires the viewpoint position associated with the viewpoint position ID of the selected monitoring area from the viewpoint information storage unit 112 (step S2028). Subsequently, the synthesizing unit 125 performs rendering by an arbitrary method such as projecting a three-dimensional model onto a projection plane (two-dimensional plane) based on the acquired viewpoint position, a preset gazing point, and a field angle. Thus, a two-dimensional projection image is generated, and the generated two-dimensional projection image is set as a selection image (step S2029). As shown in FIG. 9, the selected image is combined with a captured image captured by the imaging device 20 corresponding to each camera icon in a pop-up area set near each camera icon in the two-dimensional projection image. Has been.

  As described above, in the image processing apparatus according to the eighth embodiment, each monitoring area is configured in a hierarchical manner, and selection of a monitoring area in another hierarchy from a selection image that is a two-dimensional projection image of any monitoring area. Is possible. Moreover, since only the camera icons of the cameras associated with the selected monitoring area are displayed, many camera icons are not displayed at the same time. Furthermore, when a camera icon is selected from a selected image that is a two-dimensional projection image of the selected monitoring area, a combined image obtained by combining captured images captured by the camera corresponding to the selected camera icon is displayed. Therefore, it is easy to intuitively grasp the position of the camera installed in the monitoring area which is a three-dimensional space with a two-dimensional image, and the operability can be improved.

(Embodiment 9)
Next, an image processing apparatus 10 according to the ninth embodiment will be described. When the image processing apparatus 10 according to the first to eighth embodiments generates a two-dimensional projection image as a selected image (initial image), the viewpoint information storage unit 112 corresponds to a monitoring area to be processed. The attached viewpoint position is identified, and based on the viewpoint position and the preset gazing point and angle of view, a two-dimensional projection image is generated based on the three-dimensional model data of the monitoring area to be processed. The image processing apparatus 10 according to the present embodiment further appropriately changes the viewpoint position used when generating a two-dimensional projection image from the three-dimensional model data of the monitoring area to be processed in accordance with an instruction from the user. Can do.

  FIG. 34 is a diagram for explaining the processing of the image processing apparatus 10 according to the ninth embodiment. FIG. 34 is a two-dimensional projection image of the entire monitoring area corresponding to the initial image. Here, the vertical direction of the three-dimensional model of all the monitoring areas is defined as the z-axis, the predetermined direction of the horizontal plane that is horizontal to the vertical direction is defined as the x-axis, and the direction perpendicular to the x-axis is defined as the y-axis.

  Although the viewpoint and the position of the gazing point with respect to the three-dimensional model can be set freely, in the image processing apparatus 10 according to the ninth embodiment, a position that can be set to limit the range of positions that can be set as the viewpoint. It is assumed that a viewpoint range that is a range of is previously set in the three-dimensional model storage unit 111. The viewpoint range set in the image processing apparatus 10 according to the present embodiment is a range of an arc corresponding to a predetermined rotation angle range centered on a rotation axis arranged on the three-dimensional model and having a radius as a predetermined value. It is.

  For example, for all the monitoring areas, the image processing apparatus 10 has a two-dimensional projection image in a state where the z-axis is the rotation axis and the viewpoint position is changed along the arc within a predetermined rotation angle range, as shown in FIG. Can be generated and output, and a two-dimensional projection image can be generated and output in a state where the y-axis is the rotation axis and the viewpoint position is changed along the arc within a predetermined rotation angle range. .

  In the image processing apparatus 10 according to the ninth embodiment, the operation unit 121 receives an instruction to change the viewpoint position in response to a user operation. That is, the operation unit 121 functions as a change instruction receiving unit. The synthesizing unit 125 is based on the three-dimensional model stored in the three-dimensional model storage unit 111, the viewpoint position stored in the viewpoint information storage unit 112, and the viewpoint position indicated in the change instruction received by the operation unit 121. The viewpoint position is changed based on, and a two-dimensional projection image in which the three-dimensional model is drawn is generated based on the changed viewpoint. That is, the synthesizing unit 125 functions as a viewpoint position changing unit and a generating unit.

  FIG. 35 is a diagram illustrating an example of information stored in the three-dimensional model storage unit 111 of the image processing apparatus 10. As shown in FIG. 35, the three-dimensional model storage unit 111 of the present embodiment further stores a viewpoint range in association with the three-dimensional model data. That is, the three-dimensional model storage unit 111 corresponds to a three-dimensional model storage unit and a viewpoint range storage unit. Here, the viewpoint range includes a rotation axis position, a radius, and a rotation angle range with respect to the z axis and the y axis, respectively. The rotation axis position is the position of the rotation axis in the three-dimensional model. The radius is a value corresponding to the distance between the viewpoint and the gazing point. The rotation angle range is a rotation angle around the rotation axis corresponding to the viewpoint range. In the present embodiment, the rotation angle range is determined based on the viewpoint position associated with each monitoring area in the viewpoint information storage unit 112 described with reference to FIG. 30 in the eighth embodiment. The direction and the negative direction are set.

  As described above, the viewpoint range can be specified based on the rotation axis position, the radius, the rotation angle range, and the viewpoint position. That is, the three-dimensional model storage unit 111 stores a rotation axis position, a radius, and a rotation angle range as a viewpoint range.

  Note that the three-dimensional model storage unit 111 stores a plurality of coordinates of settable viewpoint positions in the viewpoint range as another example, instead of storing the coefficient for determining the viewpoint range in this way. It is good as well. When the change of the distance from the origin of the viewpoint position is permitted by the pinch operation of the operation unit 121 with no camera selected, the 3D model storage unit 111 changes the radius around the rotation axis, The range may be further stored as a viewpoint range. As described above, the three-dimensional model storage unit 111 only needs to store, as a viewpoint range, information that can specify coordinates that can be set as the viewpoint position, and specific information on the viewpoint range is limited to the embodiment. It is not a thing.

  The rotation axis position stored in the viewpoint information storage unit 112 is generated and displayed as a 2D projection image that is difficult for the user to grasp the 3D model, such as a 2D projection image including a blank area where no 3D model exists. It is a value that never happens. From this point of view, the rotation axis position is determined in advance by a designer or the like and registered in the viewpoint information storage unit 112.

  In addition, the rotation angle range stored in the viewpoint information storage unit 112 is observed from a position that is not supposed to be viewed by the user, such as the back side of the ground of the three-dimensional model of all monitoring areas. This is a range excluding a rotation angle at which a three-dimensional projection image is generated and displayed. It is assumed that the rotation angle range is determined in advance by a designer or the like and registered in the viewpoint information storage unit 112.

  As described above, in the image processing apparatus 10 according to the present embodiment, a viewpoint in which a two-dimensional projection image projected in an observation direction that is not assumed to be viewed by a user of a 3D model designer or the like is not generated. Since the range is set in advance, only the two-dimensional projection image assumed by the designer or the like can be provided to the user.

  36A to 36C are diagrams illustrating two-dimensional projection images drawn when the viewpoint position is changed. 36A to 36C, the viewpoint positions of the two-dimensional projection images of all the monitoring areas shown in FIG. 34 are based on the z axis stored in the three-dimensional model storage unit 111 in association with all the monitoring areas. It is a two-dimensional projection image drawn at a viewpoint position within the viewpoint range.

  FIG. 36A shows a state in which the viewpoint position of the two-dimensional projection image shown in FIG. 34 is rotated 90 ° clockwise around the z axis while the three-dimensional model is viewed from above the three-dimensional model in the z-axis direction. Is a two-dimensional projection image in which a three-dimensional model is drawn at the viewpoint position. FIG. 36B is a two-dimensional projection image in which the three-dimensional model is drawn at the viewpoint position after the viewpoint position of the two-dimensional projection image shown in FIG. 36A is further rotated 90 ° clockwise. FIG. 36C is a two-dimensional projection image in which a three-dimensional model is similarly drawn at the viewpoint position after the two-dimensional projection image shown in FIG. 36B is rotated by 90 °.

  FIG. 37 is a diagram for explaining the viewpoint range with the y-axis as the rotation axis for all monitoring areas. As shown in FIG. 37, the rotation angle range corresponding to the viewpoint range of the y-axis is set to an upper limit of 40 ° in the positive z-axis direction and a lower limit of 20 ° in the negative z-axis direction of the three-dimensional model. ing. In order to grasp the positional relationship of the monitoring area as if looking into from the lower side, the viewpoint position may be rotated in the minus direction around the y axis, but if it is greatly rotated in the minus direction, it will be in a state of looking up from the basement On the contrary, it is difficult to grasp the positional relationship. Therefore, the rotation angle range in the minus direction is set to a value smaller than the rotation angle range in the plus direction.

  38A and 38B are diagrams illustrating a two-dimensional projection image drawn when the viewpoint position is changed in the viewpoint range with the y axis as the rotation axis. When the y-axis shown in FIG. 38A is rotated 20 ° in the minus direction with the rotation axis as the rotation axis, a two-dimensional projection image as shown in FIG. 38B is obtained.

  In the image processing apparatus 10 according to the present embodiment, for convenience of explanation, the viewpoint range for any monitoring area is a range having the z-axis and the y-axis as rotation axes. The direction, position and number are not limited to those in the embodiment. For example, a rotation axis extending in a direction other than the xyz three axes may be set. Further, the number of rotating shafts may be one, or may be three or more.

  FIG. 39 is a flowchart of the image processing apparatus 10 according to the ninth embodiment. Note that the processing from step S3001 to step S3013 shown in FIG. 39 is the same as the processing from step S2001 to step S2013 according to the eighth embodiment.

  In the image processing apparatus 10 according to the ninth embodiment, for example, in a state where a selection image such as a two-dimensional projection image of all the monitoring areas shown in FIG. 34 is displayed, the user changes the viewpoint position by operating the operation unit 121. Can be entered. Specifically, for example, the direction of changing the viewpoint position is designated by moving the cursor displayed on the display screen with a mouse or the like. Furthermore, the moving distance of the viewpoint position is designated at a speed corresponding to the moving speed.

  When the operation unit 121 accepts an instruction to change the viewpoint position in response to a user operation (No at Step S3004, No at Step S3010, Yes at Step S3020), the synthesis unit 125 first selects a viewpoint position within the viewpoint range according to the change instruction. Identify. Specifically, for example, when the user moves the cursor on the display screen, the synthesis unit 125 specifies the change amount of the viewpoint position within the viewpoint range from the movement amount of the cursor, and changes based on the specified change amount. Specify a later viewpoint position. Note that it is only necessary to determine the relationship between the input method of the change instruction, the input information that has been input, and the viewpoint position that is determined from the input information, and the process for specifying the viewpoint position from the input method and the input information is limited to the embodiment. Is not to be done.

The synthesizing unit 125 further performs a two-dimensional projection image from the three-dimensional model data of the three-dimensional model corresponding to the display selection image displayed when the change instruction is received based on the changed viewpoint position, that is, after the viewpoint is changed. The selected image is generated (step S3021). In step S3021, if the device is associated with the 3D model of the display selection image in the 3D model storage unit 111, the synthesis unit 125 extracts the icon of the associated device, The extracted icon is arranged in the three-dimensional model, and a selection image is generated from the three-dimensional model data of the arranged three-dimensional model. Next, the output unit 126 outputs the selection image after changing the viewpoint, the selection image output on the display screen is displayed (step S3022), and the process proceeds to step S3013. That is, when a camera icon is selected in the selected image displayed in step S3022, the captured image captured by the imaging device 20 corresponding to the selected camera icon is synthesized by the processing in steps S3004 to S3006. Synthesized image generation and display.

  In the image processing apparatus 10 according to the present embodiment, when the viewpoint position is changed according to a change instruction from the user, the viewpoint position can be changed according to a preset rotation axis and rotation angle range. Therefore, a two-dimensional projection image unnecessary for the user is not displayed, and the user can display a two-dimensional projection image in which a desired area can be observed with a simple operation.

  Other configurations and processes of the image processing apparatus 10 according to the ninth embodiment are the same as the configurations and processes of the image processing apparatus 10 according to the other embodiments.

  Although the image processing apparatus 10 according to the present embodiment stores a plurality of monitoring areas and viewpoint ranges used for displaying the selection screen in association with each other, the viewpoint position of the selection screen can be changed. As an example, the image processing apparatus 10 further associates a viewpoint position and a viewpoint range for generating a confirmation image for selecting a camera by the user and for the user to confirm the direction of the lens of the camera, and the three-dimensional model. May be further stored and displayed so that the viewpoint position of the confirmation image can be changed.

  As a result, a two-dimensional projection image obtained by observing the three-dimensional model from a viewpoint that allows easy confirmation of the direction of the lens of the camera is generated and displayed as a confirmation image, and a three-dimensional model expressed in the two-dimensional projection image is further displayed. It is possible to display a two-dimensional projection image (confirmation image) that is rotated according to an instruction from the user.

(Embodiment 10)
In the embodiment described above, a captured image (hereinafter also referred to as a monitoring image) acquired by the imaging device 20 is displayed live (hereinafter referred to as live streaming playback) on the image processing device 10 using a live streaming technique or the like. However, it is not limited to such a form.
For example, imaging information or event information acquired by the image processing apparatus 10 from the imaging apparatus 20 or the sensor 40 via the network 30 is stored in a storage device provided in the image processing apparatus 10 or a storage device arranged on the network 30. It may be recorded and used for the later reproduction of the monitoring image (hereinafter referred to as recording reproduction in order to distinguish it from live streaming reproduction). Here, a mode in which the image processing apparatus 10 includes a storage device that stores imaging information and event information will be described as a tenth embodiment.

  FIG. 40 is a diagram illustrating a configuration example of an image processing device 4010 according to the tenth embodiment. As shown in FIG. 40, the image processing device 4010 includes an image storage unit 4127 and an event storage unit 4128 in addition to the same configuration as the image processing device 10 shown in FIG. Each of the image storage unit 4127 and the event storage unit 4128 may use various storage devices such as a hard disk built in a personal computer that implements the image processing apparatus 4010 and a hard disk externally attached to the personal computer.

  In addition to the above-described captured image, pan information, tilt information, and zoom information, the imaging information transmitted from the imaging device 20 includes the device ID of the imaging device 20 that acquired the captured image and the imaging time when the captured image was acquired. And are included. The device ID and the imaging time are sent from the control unit 204 of the imaging apparatus 20 to the compression unit 202, and are added to the header or footer of the captured image together with pan information, tilt information, and zoom information. In the following description, pan information, tilt information, and zoom information are collectively referred to as PTZ information.

  The acquisition unit 123 that acquired the imaging information from the network 30 via the communication processing unit 122 identifies the captured image, device ID, imaging time, and PTZ information included in the imaging information, and inputs these to the control unit 124.

  The control unit 124 uses the captured image, device ID, imaging time, and PTZ information acquired by the acquisition unit 123 to perform live streaming reproduction of the monitoring image with the two-dimensional projection image according to each embodiment described above as a background. At the same time, the captured image, device ID, captured time, and PTZ information are stored in the image storage unit 4127. Note that the captured image, device ID, imaging time, and PTZ information may be stored in the image storage unit 4127 from the control unit 124 even when live streaming playback of the monitoring image is not performed.

  The control unit 124 specifies the device ID of the imaging device 20 selected by the user when storing the captured image, device ID, imaging time, and PTZ information in the image storage unit 4127. Therefore, when the device ID of the selected imaging device 20 matches the device ID to be stored, the control unit 124 sets a predetermined selection flag in the data set to be stored (captured image, device ID, imaging time, and PTZ information). In addition, these are stored in the image storage unit 4127. The predetermined selection flag is flag information indicating that the associated captured image is a captured image acquired by the imaging device 20 while the imaging device 20 is selected. When the device ID of the selected imaging device 20 does not match the device ID to be stored, the control unit 124 is a captured image acquired by the imaging device 20 in a state where the corresponding captured image is not selected by the user. A selection flag indicating the presence may be added to the data set to be stored and stored in the image storage unit 4127. The selection flag at this time may be null data. Hereinafter, for clarity of explanation, a selection flag indicating that the image is acquired while the imaging device 20 is in a selected state is referred to as a selection flag ('1'). A selection flag indicating that the captured image is acquired during non-selection is referred to as a non-selection flag ('0').

  In the image storage unit 4127, the captured image, device ID, imaging time, PTZ information, and selection flag are distinguished and stored for each imaging device 20. A device ID is used for this distinction. In addition, for example, a table management method is used for data management in the image storage unit 4127.

  FIG. 41 is a diagram illustrating an example of a captured image management table held by the image storage unit 4127. As shown in FIG. 41, in the captured image management table, each captured image is associated with the device ID of the image capturing apparatus 20 (camera) that captured the captured image, the capturing time, PTZ information, and a selection flag. be registered. Note that FIG. 41 shows an example of a captured image management table that can be used for live streaming playback when the monitoring image (captured image) is a continuous still image for the sake of simplicity. It is not something that can be done. For example, when a captured image is compressed in a moving image compression format including a plurality of frames such as MPEG-4, a series of captured images acquired by one shooting is managed as one moving image file. In that case, PTZ information, selection flags, and the like that change for each time zone may be added to the corresponding time zone of the captured image (operation) using a metadata description method such as MPEG-7.

  On the other hand, the event information transmitted from the sensor 40 includes, in addition to the event detection data, the device ID of the sensor 40 that detected the event and the event occurrence time. Here, FIG. 42 shows a configuration example of the sensor 40. As shown in FIG. 42, the sensor 40 includes, for example, a sensor unit 4201, a pan head drive unit 4205, an angle sensor 4206, a control unit 4204, and a compression unit 4202, similarly to the imaging device 20 shown in FIG. And a communication processing unit 4203. Event detection data obtained by the sensor unit 4201, the device ID specified by the control unit 4204, and the event occurrence time are input to the compression unit 4202. When the sensor unit 4201 has directivity, pan information and tilt information (hereinafter referred to as PT information) obtained by the angle sensor 4206 is input to the compression unit 4202 via the control unit 4204. Also good. The detection data, device ID, and occurrence time input to the compression unit 4202 are compressed together with PT information as necessary, and are sent from the communication processing unit 4203 to the image processing apparatus 4010 via the network 30 as event information.

  The acquisition unit 123 that acquired the event information from the network 30 via the communication processing unit 122 identifies the detection data, device ID, and occurrence time (and PT information) included in the event information, and inputs them to the control unit 124. . In addition, the acquisition unit 123 inputs the occurrence time included in the event information when the detection of the same event is completed to the control unit 124. The control unit 124 specifies the occurrence time specified from the event information when the detection of the event is ended as the event end time, and uses the end time as the detection data, device ID, and occurrence time (and PT) of the corresponding event. Information) and the event storage unit 4128.

  The control unit 124 uses the event detection data, device, generation time, and end time (and PT information) acquired by the acquisition unit 123 for the live streaming reproduction of the monitoring image described above, and the detection data, device, and generation time. The end time (and PT information) is stored in the event storage unit 4128. Even when the live streaming reproduction of the monitoring image is not executed, the detection data, device ID, generation time and end time (and PT information) may be stored from the control unit 124 into the event storage unit 4128.

  Further, when storing the detection data, device, occurrence time and end time (and PT information) in the event storage unit 4128, the control unit 124 generates an event ID unique to each event, and stores this event ID. Together with the data set (detection data, device, occurrence time and end time (and PT information)) to be stored in the event storage unit 4128.

  The event storage unit 4128 stores the event ID, device ID, detection data, event occurrence time and end time (and PT information) separately for each sensor 40. A device ID is used for this distinction. For data management in the event storage unit 4128, for example, a table management method is used.

  FIG. 43 is a diagram illustrating an example of an event management table held by the event storage unit 4128. As shown in FIG. 43, in the event management table, event detection data is registered in association with the device ID, event ID, occurrence time, and end time (and PT information) of the sensor 40 that detected this event. Yes. Each record registered in the event management table may be associated with a confirmation flag indicating whether the user has confirmed each event. This confirmation flag is triggered by a predetermined event such as a user operation or completion of reproduction of the corresponding event during recording and reproduction of the captured image and event information stored in the image storage unit 4127 and event storage unit 4128. 4128 may be stored.

  In the tenth embodiment and each of the embodiments described above, there may be a sensor 40 associated with the imaging device 20. For example, in the case of a human sensor, the sensor 40 may be associated with the imaging device 20 that images the human sensing range of the sensor 40. If it is a door opening / closing sensor, the sensor 40 and the imaging device 20 that images the door detected by the sensor 40 may be associated with each other. Alternatively, motion detection may be performed using a captured image of the imaging device 20, and the imaging device 20 itself may be used as the sensor 40.

  The association between the sensor 40 and the imaging device 20 is managed in the three-dimensional model storage unit 111, for example. For this management, for example, a table management method can be used. FIG. 44 is a diagram illustrating an example of a sensor management table managed in the three-dimensional model storage unit 111. As shown in FIG. 44, the sensor management table associates and manages the device ID of the sensor 40 and the device ID (hereinafter referred to as a cover camera ID) of the imaging device 20 associated with the device ID. By using such a sensor management table, when any of the sensors 40 registered in the sensor management table is selected by the user, it is possible to specify the imaging device 20 associated with the sensor 40. Also, the sensor management table may manage the importance of the sensor 40 corresponding to each device ID and the sensor type indicating the sensor type (human sensor, thermal sensor, etc.) in association with each device ID. Good.

  Next, the operation of the control unit 124 when storing the imaging information and the event information in the image processing apparatus 4010 will be described in detail with reference to the drawings. Here, as an example of an operation when recording, a case where a recording operation is executed in a time slot reserved for preset recording is illustrated. However, the present invention is not limited to this, and various modifications can be made such as continuous recording during operation.

  FIG. 45 is a flowchart showing the recording operation of the control unit 124 according to the tenth embodiment. As shown in FIG. 45, after starting, the control unit 124 waits for the main recording operation until the recording start timing comes (No in step S4001). The recording start timing is not to record all images sent from the camera, but to record by thinning out in units of frames. In the recording operation, in order to record for a long time, only the necessary frames are recorded by counting the frames. When it is time to start recording (Yes at Step S4001), the control unit 124 acquires the device ID of the imaging device 20 that is currently selected by the user (Step S4002). The device ID of the imaging device 20 currently selected by the user may be managed by, for example, a CPU (Central Processing Unit) cache memory that implements the control unit 124. Further, when there is no imaging device 20 that is currently selected by the user, the control unit 124 may acquire, for example, null data as a device ID.

  Next, the control unit 124 stands by until imaging information is received from the imaging device 20 on the network 30 via the communication processing unit 122 (No in step S4003). When the imaging information is received from any of the imaging devices 20 (Yes at Step S4003), the control unit 124 uses the acquisition unit 123 to specify the captured image, device ID, imaging time, and PTZ information from the received imaging information. This is acquired from the acquisition unit 123 (step S4004). Next, the control unit 124 determines whether or not the acquired device ID matches the device ID specified in step S4002 (step S4005).

  As a result of the determination in step S4005, if both device IDs match (Yes in step S4005), the control unit 124 associates the captured image, device ID, imaging time, and PTZ information identified in step S4004 with the selection flag. And stored in the image storage unit 4127 (step S4006). On the other hand, if both device IDs do not match (No in step S4005), the control unit 124 associates the captured image, device ID, imaging time, and PTZ information specified in step S4004 with the non-selected flag, and stores the image storage unit. It stores in 4127 (step S4007).

  Thereafter, the control unit 124 determines whether or not the preset recording end time has come (step S4008). When the end time is reached (Yes at step S4008), the control unit 124 ends the recording operation. On the other hand, when it is not the end time (No at Step S4008), the control unit 124 returns to Step S4002 and executes the subsequent operations.

  Through the recording operation as described above, the captured image is stored in the image storage unit 4127 in association with the device ID, the imaging time, the PTZ information, and the selection flag.

  FIG. 46 is a flowchart showing the event recording operation of the control unit 124 according to the tenth embodiment. As shown in FIG. 46, the control unit 124 waits for this event recording operation until the preset recording start time comes after activation (No in step S4011). When the preset recording start time is reached (Yes at Step S4011), the control unit 124 waits until event information is received from the sensor 40 on the network 30 via the communication processing unit 122 (No at Step S4012). When event information is received from any of the sensors 40 (Yes in step S4012), the control unit 124 uses the acquisition unit 123 to specify detection data, device ID, and occurrence time (and PT information) from the received event information. This is acquired from the acquisition unit 123 (step S4013). Next, the control unit 124 waits until the input of the same event information is finished (No at Step S4014), and when finished (Yes at Step S4014), specifies the end time (Step S4015).

  Next, the control unit 124 determines whether PT information is included in the event information (step S4016). When PT information is included in the event information (Yes in S4016), the control unit 124 associates the detection data, device ID, generation time, and PT information specified in step S4013 with the end time specified in step S4015. And stored in the event storage unit 4128 (step S4017). At this time, the control unit 124 may generate an event ID and store it in the event storage unit 4128 together with the detection data, device ID, occurrence time, end time, and PT information. On the other hand, when the PT information is not included in the event information (No in step S4016), the control unit 124 associates the detection data, device ID, and occurrence time specified in step S4013 with the end time specified in step S4015. And stored in the event storage unit 4128 (step S4018). At this time, the control unit 124 may generate an event ID and store it in the event storage unit 4128 together with the detection data, device ID, occurrence time, and end time.

  Thereafter, the control unit 124 determines whether or not the recording end timing has come (step S4019). When the end timing is reached (Yes at Step S4019), the control unit 124 ends the event recording operation. On the other hand, when it is not the end timing (No at Step S4019), the control unit 124 returns to Step S4012 and executes the subsequent operations.

  By the event recording operation as described above, the detection data of the event detected by the sensor 40 is stored in the event storage unit 4128 in association with the event ID, device ID, occurrence time, end time (and PT information). The

  Next, the operation when recording and reproducing the captured image stored in the image storage unit 4127 will be described in detail with reference to the drawings. FIG. 47 is a flowchart illustrating an example of a recording / playback operation performed by the control unit 124 according to the tenth embodiment. As shown in FIG. 47, after starting, the control unit 124 waits until a recording data playback instruction is input from the user (No in step S4031). Note that an instruction to reproduce recorded data may be input by the user using the operation unit 121 of the image processing apparatus 4010.

  When receiving an instruction to reproduce the recorded data (Yes at Step S4031), the control unit 124 reads the recording data to be reproduced (Step S4032). The recorded data to be read includes the captured image, device ID, imaging time, PTZ information, and selection flag stored in the image storage unit 4127, detection data, event ID, device ID, stored in the event storage unit 4128, The generation time and end time (and PT information) and a confirmation flag are included.

  Next, the control unit 124 determines whether there is an imaging device 20 being selected by referring to a selection flag associated with the first captured image in the read recording data (step S4033). . Note that the imaging device 20 being selected here is an imaging device in a state selected by the user at the start of recording.

  When the selected imaging device 20 is present (Yes at Step S4033), the control unit 124 associates the selected position on the three-dimensional model (camera icon position) of the selected imaging device 20 with the captured image to be reproduced. Based on the PTZ information, a viewpoint position for generating a two-dimensional projection image from the three-dimensional model data is determined (step S4036). In addition, the control unit 124 generates an operation screen for the user to input an operation related to recording / playback (step S4037). This operation screen includes a predetermined area for recording and reproducing the captured image (hereinafter referred to as a reproduction area). This reproduction area may correspond to, for example, a predetermined area on the two-dimensional projection image in the above-described embodiment.

  On the other hand, when the imaging device 20 being selected does not exist (No at Step S4033), the control unit 124 determines the viewpoint position of the initial condition regarding the corresponding area from the viewpoint information storage unit 112 described above with reference to FIG. Obtain (step S4034). In addition, the control unit 124 generates an operation screen for the user to input an operation related to recording and reproduction (step S4035). This operation screen may or may not include a playback area for generating a captured image.

  Thereafter, the control unit 124 generates a two-dimensional projection image from the three-dimensional model data based on the viewpoint position determined in step S4036 or the viewpoint position acquired in step S4034 (step S4038). The PTZ information of the imaging device 20 and the PT information of the sensor 40 read out in step S4032 are reflected on all the camera icons and sensor icons included in the three-dimensional model data at this time. Accordingly, the three-dimensional model data at the time of reproduction is a reproduction of the movement (change, etc.) of the three-dimensional model data during the recording period as it is at the time of the live streaming reproduction according to the above-described embodiment. At this time, each camera icon may be reproduced by pop-up display of a captured image (or a thumbnail thereof) acquired by the imaging device 20 associated with each camera icon, as in the above-described embodiment.

  Next, the control unit 124 combines the captured image read out in step S4032 with the operation screen generated in step S4037 (step S4039). Note that the captured image combined with the operation screen sequentially transitions along a time axis from a series of captured images in time series. As a result, the captured image is reproduced in the reproduction area of the operation screen.

  Next, the control unit 124 combines the operation screen in which the captured image is combined in step S4039 with the two-dimensional projection image generated in step S4038 (step S4040). Subsequently, the control unit 124 outputs the two-dimensional projection image combined with the operation screen to the output unit 126 such as a monitor (step S4041), and displays it.

  Next, the control unit 124 determines whether or not to end the reproduction of the recorded data (step S4042). This determination may be made based on, for example, whether or not a playback stop button has been input from the operation screen, whether or not playback of recorded data has been completed to the end. If it is determined that the playback has ended (Yes at step S4042), the control unit 124 ends the recording / playback operation. On the other hand, if it is determined not to end the reproduction operation (No at Step S4042), the control unit 124 determines whether, for example, an operation input for changing the viewpoint position is received from the user (Step S4043). The operation input method for changing the viewpoint position may be the same as the operation input method for the pan direction, the tilt direction, and the focal length of the zoom lens of the imaging device 20 in the above-described embodiment.

  When there is an operation input for changing the viewpoint position (Yes at Step S4043), the control unit 124 determines a new viewpoint position from the operation data input to the operation unit 121 (Step S4044). Note that the user operation at the viewpoint position in step S <b> 4044 may be the same as the user operation on the imaging device 20 in the above-described embodiment. In this case, the operation data input from the operation unit 121 is not transmitted to the imaging apparatus 20 as a control signal, but is used to calculate the viewpoint position change amount with respect to the three-dimensional model. Thereafter, the control unit 124 returns to step S4038 and executes the subsequent operations.

  On the other hand, when there is no operation input for changing the viewpoint position (No in step S4043), the control unit 124 determines whether or not the PTZ information of the selected imaging device 20 specified in step S4033 is changed in the recording data read in step S4032. Is determined (step S4045). When there is a change in the PTZ information (Yes at Step S4045), the control unit 124 returns to Step S4036 and performs the subsequent operations based on the changed PTZ information.

  On the other hand, when there is no change in the PTZ information (No at Step S4045), the control unit 124 determines whether or not the user has performed a camera icon selection operation on the two-dimensional projection image (Step S4046). When the camera icon is selected (Yes at step S4046), the control unit 124 returns to step S4036, and based on the position of the selected camera icon on the three-dimensional model and the PTZ information corresponding to the device ID thereof. Execute the following operations.

  On the other hand, if no camera icon has been selected (No at step S4046), the control unit 124 determines whether the user has performed a sensor icon selection operation on the two-dimensional projection image (step S4047). When a sensor icon is selected (Yes at step S4047), the control unit 124 selects a cover camera ID (device ID) associated with the device ID of the sensor icon selected from the sensor management table shown in FIG. (Step S4048). Thereafter, the control unit 124 returns to step S4036 and performs the subsequent operation based on the position on the three-dimensional model of the imaging device 20 corresponding to the newly selected device ID and the PTZ information corresponding to this device ID. Run.

  By performing the recording / reproducing operation as described above, it is possible to reproduce the live streaming reproduction of the monitoring image at the time of recording as it is. In the recording / playback operation shown in FIG. 47, the selected imaging device 20 or sensor 40 is switched as necessary, or the viewpoint position with respect to the three-dimensional model is moved regardless of the selected camera icon position. Is possible. Therefore, the user can more easily and in detail check the past monitoring image.

  Next, an example of a composite image displayed on the output unit 126 of the image processing apparatus 4010 by the above reproduction operation will be described in detail below with reference to the drawings.

  The monitoring image recording / playback according to the tenth embodiment may be a form that reproduces the live streaming playback of the monitoring image according to any of the above-described embodiments. FIG. 48 is a diagram showing an example of the composite image generated in step S4040 of FIG. As shown in FIG. 48, in the composite image 4000, as in the case of the composite image at the time of live streaming reproduction in the above-described embodiment, for example, the operation screen 4300 with a two-dimensional projection image including camera icons 4001, 4002, and 4003 as a background. Is synthesized. Note that the arrangement of the operation screen 4300 is not limited to the lower right of the two-dimensional projection image, and may be a central portion or the like. The operation screen 4300 may display various information in addition to the captured image to be displayed as described above.

  Here, FIG. 49 shows an example of the operation screen 4300. As shown in FIG. 49, the operation screen 4300 includes a captured image reproduction area 4310 for recording and reproducing captured images, and an event display area 4320 for displaying information on various events. Further, the operation screen 4300 records the identification information (camera ID or camera name) 4311 of the imaging device 20 that acquired the captured image displayed in the captured image reproduction area 4310, that is, the selected imaging device 20, and preset recording. A preset recording / playback button 4302 for playing back the recorded data, a live button 4301 for switching the display on the output unit 126 to live streaming playback of a captured image (monitoring image), playback / stop / skip / fast-forward during recording / playback, etc. An operation button group 4330 for the user to perform the above operation and a recording interval button 4331 for setting a recording time interval may be displayed.

  In the event display area 4320, the imaging time (may be the date and time) of the captured image being displayed in the captured image reproduction area 4310, a time-sequential list of event IDs, and the event corresponding to each event ID are displayed. The occurrence time, the cover camera ID (device ID) of the imaging device 20 that covers the sensor 40 corresponding to each event ID, the importance level associated with each event or the sensor 40 that detected each event, and each event A confirmation flag indicating whether or not the user has confirmed may be displayed. In the event display area 4320, a time bar indicating where in the recorded data is currently being played back may be displayed. An index indicating the time of occurrence of each event may be added to the time bar.

  When the user selects the preset recording / playback button 4302 on the operation screen as described above, the output unit 126 displays, for example, a list of recording data files that have been preset recorded. When the user selects any file from the list, the control unit 124 executes an operation for recording and reproducing the selected file. This recording / playback operation is the operation described above with reference to FIG. On the other hand, when the user selects the live button 4301 on the operation screen, the synthesized image displayed on the output unit 126 is a synthesized image (for example, FIG. 9 and FIG. 12).

  In the operation screen, the recording interval button 4331 is, for example, a pull-down menu button. When the user selects the recording interval button 4331, a list of recording intervals that can be set (for example, full frame (30 frames / second), 15 frames / second, 10 frames / second, 5 frames / second, 1 frame / second, etc.) Is displayed. When the user selects one of the listed recording intervals (for example, 10 frames / second), the selected recording interval (for example, 10 frames / second) is set in the control unit 124, for example. The control unit 124 repeatedly executes the recording operation shown in FIG. 45 and the recording operation shown in FIG. 46 at each recording interval set in this way.

  By providing the configuration and operation as described above, the user can intuitively grasp the area captured by the camera during recording / playback as well as during live streaming playback of the monitoring image according to each of the above-described embodiments. Can be made. Since other configurations, operations, and effects may be the same as those in any of the above-described embodiments, detailed description thereof is omitted here. However, in the tenth embodiment, the case where the image processing apparatus 4010 acquires the imaging information and the event information from the imaging apparatus 20 and the sensor 40 via the network 30 is illustrated, but the present invention is not limited to this. For example, each of the imaging device 20 and the sensor 40 may include a storage device, and the image processing device 4010 may be configured to acquire imaging information and event information stored in the storage device online or offline. Alternatively, the imaging device 20 and the sensor 40 directly transmit imaging information and event information to the storage device on the network 30, and the image processing device 4010 transmits the imaging information and event information stored in the storage device via the network 30. It may be configured to obtain.

(Embodiment 11)
In the tenth embodiment and the other embodiments described above, the icon of the sensor 40 or the imaging device 20 that detected the event may be highlighted on the two-dimensional projection image. For example, when each imaging device 20 performs motion detection on a captured image and each imaging device 20 also functions as the sensor 40, an event is selected from the camera icons corresponding to these imaging devices 20, as shown in FIG. Camera icons 4003a and 4005a corresponding to the detected imaging device 20 may be highlighted. FIG. 50 is a diagram illustrating an example of a composite image according to the eleventh embodiment. Various display methods such as blinking display, display with a color different from the background (for example, red), and enlarged display can be used for highlighting various icons.

  Various icons can be highlighted by temporarily replacing, for example, the icon image associated with the device ID of the device to be highlighted (sensor 40 or imaging device 20) with another icon image for highlighting. In that case, since an icon image for highlighting is incorporated in the 3D model data instead of the normal icon image, the icon image in the 2D projection image generated from the 3D model data is also highlighted.

  Note that, here, as in the above-described embodiment, an example has been described in which an icon in a two-dimensional projection image generated from three-dimensional model data is highlighted, but the present invention is not limited to this. For example, instead of the three-dimensional model data, a two-dimensional model obtained by mapping a layout similar to this on a two-dimensional plane parallel to the ground (or the floor surface) may be used. In this case, the two-dimensional model as it is is combined with the captured image and the operation screen and displayed on the output unit 126. At that time, highlighting icons are used for various icons mapped in the two-dimensional model. Accordingly, as in the case of the above-described three-dimensional model, various icons mapped in the two-dimensional model are highlighted as necessary. Since other configurations, operations, and effects may be the same as those of the above-described embodiment, detailed description thereof is omitted here.

[program]
The configuration of the image processing apparatus 10 is realized by, for example, a hardware CPU (Central Processing Unit), a memory, and other LSI (Large Scale Integration) as hardware. Similarly, the configuration of the image processing apparatus 10 can be realized by, for example, a program loaded into a memory as software. The above embodiment has been described as a functional block realized by cooperation of these hardware or software. That is, these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof.

  FIG. 51 is a diagram illustrating that an image processing program is realized using a computer. For example, as shown in FIG. 51, a computer 1000 as an image processing apparatus 10 includes a control device such as a CPU (Central Processing Unit) 1001 connected via a bus 1009, a ROM (Read Only Memory) 1002, and a RAM (Random). A storage device such as an access memory (1003), an external storage device such as a hard disk drive (HDD) 1004 and a disk drive 1005, a display device such as a display 1006, and input devices such as a keyboard 1007 and a mouse 1008. It has a hardware configuration using a normal computer. In the storage device or the external storage device, various types of information of the above-described 3D model storage unit 111 and viewpoint information storage unit 112 are stored.

  An image processing program executed by the image processing apparatus 10 may be installed in an installable format or an executable format file, such as a CD-ROM, a flexible disk (FD), a CD-R, a DVD (Digital Versatile Disk), etc. And recorded on a computer-readable recording medium. The image processing program executed by the image processing apparatus 10 may be provided by being stored on a computer connected to a network such as the Internet and downloaded via the network. The image processing program executed by the image processing apparatus 10 may be provided or distributed via a network such as the Internet. Further, the image processing program may be provided by being incorporated in a ROM or the like.

  The image processing program executed by the image processing apparatus 10 has a module configuration including the above-described functional units (acquisition unit 123, synthesis unit 125, output unit 126), and a CPU (processor) is used as actual hardware. By reading out and executing the image processing program from the storage medium, each of the above functional units is loaded onto the main storage device, and the acquisition unit 123, the synthesis unit 125, and the output unit 126 are generated on the main storage device and stored as appropriate. Processing is performed using various information stored in the device or the external storage device.

  Information including processing procedures, control procedures, specific names, various data, parameters, and the like shown in the document and drawings can be arbitrarily changed unless otherwise specified. For example, various information stored in the three-dimensional model storage unit 111 is appropriately changed according to the area 50 to be imaged.

[Constitution]
The constituent elements of the illustrated image processing apparatus 10 are functionally conceptual, and need not be physically configured as illustrated. In other words, the specific form of distribution or integration of each device is not limited to the illustrated one, and all or a part thereof is functionally or physically distributed or arbitrarily distributed in arbitrary units according to various burdens or usage conditions. Can be integrated. For example, the synthesizing unit 125 determines a viewpoint of the three-dimensional model based on the camera position, and a two-dimensional projection image that is a two-dimensional image obtained by projecting the three-dimensional model onto a two-dimensional surface based on the determined viewpoint and imaging direction. And a “synthesis unit” that generates a synthesized image by synthesizing a captured image with a predetermined region of the generated two-dimensional projection image.

  In the embodiment described above, an example in which the image processing apparatus 10 is applied to the monitoring system 1 has been described. However, not only the monitoring system 1 but also various other systems such as a distribution system that distributes live video in real time. It can be applied to various uses.

DESCRIPTION OF SYMBOLS 10 Image processing apparatus 111 3D model memory | storage part 112 Viewpoint information memory | storage part 121 Operation part 122 Communication processing part 123 Acquisition part 124 Control part 125 Composition part 126 Output part

Claims (5)

Three-dimensional model data indicating a model of a three-dimensional region; a three-dimensional model storage unit that stores a position of a camera arranged in the three-dimensional region as a camera position in the three-dimensional model;
An acquisition unit that acquires a captured image captured by the camera and an imaging direction at the time of imaging;
An image storage unit that stores the captured image acquired by the acquisition unit in association with the imaging direction;
An operation unit for receiving an instruction to reproduce the captured image stored in the image storage unit;
When the operation unit receives a reproduction instruction, the viewpoint for the three-dimensional model is determined according to the camera position, and the three-dimensional model is determined according to the determined viewpoint and the imaging direction stored in the image storage unit. A two-dimensional projection image obtained by projecting the image onto a two-dimensional surface is generated, and a composite image is generated by combining the captured image stored in the image storage unit with a predetermined region on the generated two-dimensional projection image And
An output unit for outputting the composite image,
The three-dimensional model further includes another camera icon associated with another camera different from the camera in the three-dimensional region,
The acquisition unit includes selection information indicating whether or not the camera is in a selection state by a user when the captured image is acquired, a captured image acquired by the other camera, and the captured image when the captured image is acquired. Further acquiring the imaging direction of another camera and selection information indicating whether or not the other camera was in a selection state by the user when the captured image was acquired,
The image storage unit further stores selection information indicating whether or not the camera was selected by the user when the camera acquired the captured image, in association with the captured image, and the other cameras An acquired captured image, an imaging direction of the other camera when the captured image is acquired, and selection information indicating whether the other camera is in a selection state by the user when the captured image is acquired. Associate and remember more,
When the selection of the other camera icon is received by the operation unit, the combining unit newly determines a viewpoint position with respect to the three-dimensional model according to a position of the other camera icon in the three-dimensional model. And the two-dimensional projection of the three-dimensional model in accordance with the newly determined viewpoint position and the imaging direction associated with the captured image acquired by the other camera corresponding to the other camera icon. Generating a new composite image by generating an image and combining the captured image acquired by the other camera stored in the image storage unit with a predetermined region of the newly generated two-dimensional projection image An image processing apparatus. Three-dimensional model data indicating a model of a three-dimensional region; a three-dimensional model storage unit that stores a position of a camera arranged in the three-dimensional region as a camera position in the three-dimensional model;
An acquisition unit that acquires a captured image captured by the camera and an imaging direction at the time of imaging;
An image storage unit that stores the captured image acquired by the acquisition unit in association with the imaging direction;
An operation unit for accepting a reproduction instructions of the captured image by the image storing unit stores,
When the operation unit receives a reproduction instruction, the viewpoint for the three-dimensional model is determined according to the camera position, and the three-dimensional model is determined according to the determined viewpoint and the imaging direction stored in the image storage unit. A two-dimensional projection image obtained by projecting the image onto a two-dimensional surface is generated, and a composite image is generated by combining the captured image stored in the image storage unit with a predetermined region on the generated two-dimensional projection image And
An output unit for outputting the composite image,
The operation unit further accepts selection of another camera icon different from the camera in the two-dimensional projection image by the user,
The other camera icon is associated with another camera in the three-dimensional area,
The three-dimensional model further includes the other camera icon,
The acquisition unit further acquires a captured image acquired by the other camera and an imaging direction of the other camera when the captured image is acquired;
The image storage unit includes the captured image acquired by the other camera acquired by the acquisition unit;
Further storing the captured image in association with the imaging direction of the other camera when the captured image is acquired;
When the selection of the other camera icon is received by the operation unit, the combining unit newly determines a viewpoint position with respect to the three-dimensional model according to a position of the other camera icon in the three-dimensional model. The two-dimensional projection of the three-dimensional model according to the newly determined viewpoint position and the imaging direction associated with the captured image acquired by the other camera corresponding to the other camera icon Generating a new composite image by generating an image and combining the captured image acquired by the other camera stored in the image storage unit with a predetermined region of the newly generated two-dimensional projection image An image processing apparatus. Three-dimensional model data indicating a model of a three-dimensional region; a three-dimensional model storage unit that stores a position of a camera arranged in the three-dimensional region as a camera position in the three-dimensional model;
An acquisition unit that acquires a captured image captured by the camera and an imaging direction at the time of imaging;
An image storage unit that stores the captured image acquired by the acquisition unit in association with the imaging direction;
An operation unit for accepting a reproduction instructions of the captured image by the image storing unit stores,
When the operation unit receives a reproduction instruction, the viewpoint for the three-dimensional model is determined according to the camera position, and the three-dimensional model is determined according to the determined viewpoint and the imaging direction stored in the image storage unit. A two-dimensional projection image obtained by projecting the image onto a two-dimensional surface is generated, and a composite image is generated by combining the captured image stored in the image storage unit with a predetermined region on the generated two-dimensional projection image And
An output unit for outputting the composite image;
A sensor management table for managing a correspondence between a sensor icon associated with a sensor in the three-dimensional region and the camera icon;
The operation unit further accepts selection of the sensor icon in the two-dimensional projection image by a user,
When the selection of the sensor icon is received by the operation unit, the control unit identifies the camera icon associated with the sensor icon from the sensor management table, and the three-dimensional model of the identified camera icon The viewpoint position for the three-dimensional model is newly determined based on the position in the camera, and the newly determined viewpoint position is associated with the captured image acquired by the camera corresponding to the camera icon. The two-dimensional projection image of the three-dimensional model is generated according to the imaging direction, and the sensor icon selected in the predetermined region of the two-dimensional projection image generated using the newly determined viewpoint position By combining the captured images in the storage unit acquired by the camera corresponding to the camera icon associated with the Image processing apparatus and generates the formed image. Three-dimensional model data indicating a model of a three-dimensional area; and a three-dimensional model storage step of storing a position of a camera arranged in the three-dimensional area as a camera position in the three-dimensional model;
An acquisition step of acquiring a captured image captured by the camera and an imaging direction at the time of imaging;
An image storage step of storing the captured image acquired by the acquisition step in association with the imaging direction;
An operation step of receiving an instruction to reproduce the captured image stored in the image storage step;
When a reproduction instruction is received in the operation step, a viewpoint for the three-dimensional model is determined according to the camera position, and the 3 is determined according to the determined viewpoint and the imaging direction stored in the image storage step. A two-dimensional projection image obtained by projecting a two-dimensional model onto a two-dimensional plane is generated, and a composite image is generated by combining the captured image stored in the image storage step with a predetermined region on the generated two-dimensional projection image. A synthesis step to generate;
An output step of outputting the composite image,
The three-dimensional model further includes another camera icon associated with another camera different from the camera in the three-dimensional region,
The acquisition step includes selection information indicating whether or not the camera is selected by a user when the captured image is acquired, a captured image acquired by the other camera, and the captured image when the captured image is acquired. Further acquiring the imaging direction of another camera and selection information indicating whether or not the other camera was in a selection state by the user when the captured image was acquired,
The image storage step further stores selection information indicating whether or not the camera was selected by the user when the camera acquired the captured image, in association with the captured image, and the other cameras An acquired captured image, an imaging direction of the other camera when the captured image is acquired, and selection information indicating whether the other camera is in a selection state by the user when the captured image is acquired. Associate and remember more,
In the combining step, when selection of the other camera icon is accepted in the operation step, a viewpoint position for the three-dimensional model is newly determined according to a position of the other camera icon in the three-dimensional model. And the two-dimensional projection of the three-dimensional model in accordance with the newly determined viewpoint position and the imaging direction associated with the captured image acquired by the other camera corresponding to the other camera icon. An image is generated, and a new composite image is generated by combining the captured image acquired by the other camera stored in the image storage step with a predetermined region of the newly generated two-dimensional projection image. An image processing method. An image processing program for causing a computer to execute the method according to claim 4.