JP6586051B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing method.

2. Description of the Related Art Conventionally, a technique for capturing an object such as a vehicle passing through an intersection with a plurality of cameras and generating a composite image from the captured images is known.
For example, the abstract of the following Patent Document 1 states that “an image generation device that generates a bird's-eye view image capable of recognizing a three-dimensional object existing on a road surface with a good sense of distance from a photographed image of an in-vehicle camera, A peripheral bird's-eye view image generation unit that generates a peripheral bird's-eye view image from a photographed image using one projective transformation, and a three-dimensional object existing in the field of view of the in-vehicle camera are detected and three-dimensional object information including the position of the three-dimensional object is output. A three-dimensional object detection unit; a three-dimensional object extraction unit that extracts a three-dimensional object image that is an image region in which a three-dimensional object is captured from a captured image based on the three-dimensional object information; and a second that reduces distortion of the three-dimensional object in the surrounding overhead image. It includes a three-dimensional object overhead image generation unit that generates a three-dimensional object overhead image from a three-dimensional object image using projective transformation, and an image composition unit that synthesizes a three-dimensional object overhead image based on the three-dimensional object information. ing.

  In addition, the abstract of the following Patent Document 2 states that “an object is to provide a three-dimensional object confirmation device that can detect the presence or absence of a three-dimensional object when the vehicle is stopped.”, “A detection camera is provided on a door mirror. Each of the images taken in the door mirror retracted state and in the unfolded state is converted to a bird's eye view, and the images captured in the retracted state are rotated for alignment, and the captured images are compared ( 100 to 114), and if there is a portion that is shifted on the captured image, it is determined that the object is a three-dimensional object and an alarm is issued (116). "

  In addition, the abstract of the following Patent Document 3 states: “Vehicle periphery monitoring that eliminates the difficulty of viewing when combining a plurality of overhead images and presents an easy-to-see image to the driver so that an obstacle can be recognized reliably. "The vehicle periphery monitoring device 1 according to the present invention captures an image of the surroundings of the host vehicle, and is captured by the imaging unit 2 and a plurality of imaging units 2 in which imaging regions overlap." The image conversion unit 3 that converts an image to generate a plurality of overhead images, calculates a difference between the plurality of overhead images, and determines a mismatch region that does not match a matching region that matches the image in the overlap region And a display unit 6 for presenting the output image to the driver. The region determining unit 4 is configured to generate the output image by synthesizing the matching region and the non-matching region by different methods. . "

International Publication No. 2012/096058 JP2015-74318A JP 2007-027948

By the way, in each technique mentioned above, there existed a problem that it was difficult to grasp | ascertain the exact position of an object with the produced | generated synthesized image.
The present invention has been made in view of the above-described circumstances, and an object thereof is to provide an image processing apparatus and an image processing method capable of grasping an accurate position of a photographed object from an image.

In order to solve the above problems, an image processing apparatus according to the present invention normalizes a plurality of original images taken from different viewpoints, generates a plurality of normalized images, and a plurality of the normalized images. Each of object regions corresponding to an object satisfying a predetermined condition and generating a plurality of background images based on the plurality of normalized images; and the object region relating to the plurality of normalized images. A common area extracting unit that extracts a common area that is a part where the two overlap, and an image allocating unit that generates a general background image representing an image of a region other than the common region among the predetermined regions by combining a plurality of the background images and having a, the.

  According to the present invention, an accurate position of a photographed object can be grasped from an image.

1 is a block diagram of an image processing system according to a first embodiment of the present invention. It is a flowchart of the processing program in 1st Embodiment. It is a figure which shows one example of arrangement | positioning of each element in 1st Embodiment. FIG. 4 is a diagram illustrating (a) an example of a normalized image, (b) an example of a binarized image, and (c) an example of a common area display image in the arrangement example of FIG. 3. It is a figure which shows the other example of arrangement | positioning of each element in 1st Embodiment. FIG. 6 is a diagram illustrating (a) an example of a normalized image, (b) an example of a binarized image, and (c) an example of a common area display image in the arrangement example of FIG. 5. It is a block diagram of the image processing system by 2nd Embodiment. It is a flowchart of the processing program in 2nd Embodiment. It is a figure which shows the example of arrangement | positioning of each element in 2nd Embodiment. In the arrangement example of FIG. 9, (a) an example of a normalized image, (b) an example of a binarized image, (c) an example of a common area display image, (d) an example of a center of gravity position, (e) an example of a trajectory image FIG. It is a block diagram of the image processing system by 3rd Embodiment. It is a flowchart of the processing program in 3rd Embodiment. It is a figure which shows the example of arrangement | positioning of each element in 3rd Embodiment. FIG. 14 is a diagram illustrating (a) an example of a normalized image, (b) an example of a common area display image, (c) a display example of a composite image, and (d) a display example of a composite image according to a comparative example in the arrangement example of FIG. 13. . It is a block diagram of the image processing system by 4th Embodiment.

[First Embodiment]
<Configuration of First Embodiment>
FIG. 1 is a block diagram of an image processing system A1 according to the first embodiment of the present invention.
The image processing system A1 includes an image processing apparatus 201, N (N is a plurality) cameras 10-1 to 10-N, and a display unit 12. The cameras 10-1 to 10-N output camera images V1 to VN (original images) that are captured images. Hereinafter, the cameras 10-1 to 10-N may be collectively referred to as “camera 10”. The display unit 12 is a display such as an LCD (liquid crystal display).

  The image processing apparatus 201 includes hardware as a general computer such as a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), and an HDD (Hard Disk Drive). Stores an OS (Operating System), application programs, various data, and the like. The OS and application programs are expanded in the RAM and executed by the CPU. In FIG. 1, the inside of the image processing apparatus 201 shows functions realized by application programs developed in a RAM as blocks.

  The image processing apparatus 201 includes a normalization unit 22, an object region extraction unit 24, and a common region extraction unit 26. The normalizing unit 22 performs normalization processing on the camera images V1 to VN, and outputs the results as normalized images VS1 to VSN. The “normalization processing” is specifically processing for projective transformation of the camera images V1 to VN so that the viewpoints of the camera images V1 to VN are shared. The object area extraction unit 24 extracts an object area corresponding to an object satisfying a predetermined condition from the image areas of the normalized images VS1 to VSN, and outputs binarized images VB1 to VBN that specify the object area. To do.

  For example, if the normalized images VS1 to VSN are images of “roads”, the “object region” to be extracted can be, for example, “an area where a vehicle or a person is imaged”. As an object region extraction method, for example, a known technique such as obtaining a background difference may be employed. When the background difference method is employed, “having a color different from the background color measured in advance” is the above-mentioned “predetermined condition”. The binarized images VB <b> 1 to VBN can be expressed, for example, as a bitmap in which the pixel value of the pixels constituting the object region is “1” and the pixel value of the pixels constituting the other region is “0”.

  Here, a region where the object regions of the binarized images VB1 to VBN overlap is referred to as a “common region”, and the common region extraction unit 26 extracts the common region. That is, in the above example, when the logical product of the pixel values corresponding to each other of the binarized images VB1 to VBN is obtained, a range where the logical product is “1” becomes a common region. Then, the common area extraction unit 26 outputs a common area display image VC representing the common area. The output common area display image VC is displayed on the display unit 12.

<Operation of First Embodiment>
Next, the operation of this embodiment will be described with reference to FIG. FIG. 2 is a flowchart of a control program executed by the image processing apparatus 201.
In FIG. 2, when the process proceeds to step S10, the normalization unit 22 acquires frames of the camera images V1 to VN. Next, when the process proceeds to step S12 (normalization process), the normalization unit 22 converts the camera images V1 to VN into normalized images VS1 to VSN. That is, the frames of the camera images V1 to VN are converted into the frames of the normalized images VS1 to VSN. Next, when the process proceeds to step S14 (object region extraction process), the object region extraction unit 24 extracts an object region from each of the normalized images VS1 to VSN, and generates frames of the binarized images VB1 to VBN. .

  Next, when the process proceeds to step S16 (common area extraction process), the common area extraction unit 26 generates a common area display image VC from the binarized images VB1 to VBN. That is, a common area is extracted from the frames of the binarized images VB1 to VBN, and a frame of the common area display image VC is generated. Next, when the process proceeds to step S30, the normalization unit 22 determines whether or not the frame of the common area display image VC generated this time is the last frame. If the processing period is not particularly defined, it is always determined as “No” in step S30. If "No" is determined in step S30, the process returns to step S10, and the same process is repeated for the subsequent frames. Thereby, the common area display image VC representing the common area is continuously displayed on the display unit 12.

<Specific example of operation (1)>
Next, a specific example of the operation of the present embodiment will be described with reference to FIGS. 3 and 4A to 4C.
First, FIG. 3 is a diagram illustrating an arrangement example of each element in the present embodiment. In FIG. 3, the planar region 50 is a square region, and a prism member 52 is disposed at the approximate center thereof. The prism member 52 is a long member having a square cross-sectional shape. Therefore, the contact region 52a where the prism member 52 is in contact with the planar region 50 is also square. In the present embodiment, two cameras 10-1 and 10-2 are provided as the plurality of cameras 10.

  The cameras 10-1 and 10-2 are arranged at positions where the planar region 50 and the prism member 52 are looked down obliquely from above. The cameras 10-1 and 10-2 output the captured images as camera images V1 and V2. Although illustration of the contents of the camera images V1 and V2 is omitted, in these image areas, an area corresponding to the plane area 50 is substantially trapezoidal.

  Next, FIG. 4A is a diagram illustrating an example of normalized images VS1 and VS2 generated by the normalization unit 22 (see FIG. 1) based on the camera images V1 and V2 in FIG. In the camera images V1 and V2, the areas corresponding to the planar area 50 are substantially trapezoidal, but in the normalized images VS1 and VS2, these areas are square areas 70-1 and 70-2. Projective transformations have been made. As a result, the object images 72-1 and 72-2 corresponding to the prismatic member 52 are shaped such that the prismatic member 52 is expanded toward the top.

  Next, FIG. 4B shows an example of binarized images VB1 and VB2 generated by the object region extraction unit 24 (see FIG. 1) based on the normalized images VS1 and VS2 shown in FIG. FIG. In the binarized images VB1 and VB2, regions corresponding to the object images 72-1 and 72-2 of the normalized images VS1 and VS2 are object regions 82-1 and 82-2, and the pixel values of these regions are “ Set to 1 ". Further, the pixel values in the areas other than the object areas 82-1 and 82-2 are set to “0”.

  Next, FIG. 4C is a diagram illustrating an example of the common area display image VC generated by the common area extraction unit 26 (see FIG. 1) based on the binarized images VB1 and VB2. The common area display image VC can be obtained by a logical product of the binarized images VB1 and VB2. In FIG. 4C, regions corresponding to the object regions 82-1 and 82-2 are indicated by broken lines, and a portion where both overlap is a common region 90. The position, size, and shape of the common region 90 substantially match those of the contact region 52a of the prismatic member 52 shown in FIG. Therefore, the common area display image VC is an image that represents the position of the prism member 52 with high accuracy.

<Specific example of operation (2)>
Next, a specific example of another operation of the present embodiment will be described with reference to FIGS. 5 and 6A to 6C.
First, FIG. 5 is a diagram illustrating another arrangement example of each element in the present embodiment. In FIG. 5, the planar region 50 is a square region, and a prismatic member 52 and a cylindrical member 54 are disposed at the approximate center thereof. The prism member 52 is the same as that shown in FIG. Therefore, the contact regions 52a and 54a where the prismatic member 52 and the cylindrical member 54 are in contact with the planar region 50 are also square and circular. In the example of FIG. 5, three cameras 10-1 to 10-3 are provided as the plurality of cameras 10.

  The cameras 10-1 to 10-3 are arranged at positions where the planar region 50, the prismatic member 52, and the cylindrical member 54 are looked down obliquely from above, and output captured images as camera images V1 to V3. Although the contents of the camera images V <b> 1 to V <b> 3 are not shown, the area corresponding to the plane area 50 in these image areas is substantially trapezoidal.

  Next, FIG. 6A is a diagram illustrating an example of normalized images VS1 to VS3 generated by the normalization unit 22 (see FIG. 1) based on the camera images V1 to V3 in FIG. In the camera images V1 to V3, the regions corresponding to the planar region 50 are substantially trapezoidal, but in the normalized images VS1 to VS3, these regions become square regions 70-1 to 70-3. Projective transformations have been made. The object images 72-1 to 72-3 are images corresponding to the prismatic member 52 and the cylindrical member 54.

  Next, FIG. 6B shows examples of binarized images VB1 to VB3 generated by the object region extraction unit 24 (see FIG. 1) based on the normalized images VS1 to VS3 shown in FIG. FIG. In the binarized images VB1 to VB3, regions corresponding to the object images 72-1 to 72-3 of the normalized images VS1 to VS3 are object regions 82-1 to 82-3, and the pixel values of these regions are “ Set to 1 ". Further, the pixel values in the areas other than the object areas 82-1 to 82-3 are set to “0”.

  Next, FIG. 6C is a diagram illustrating an example of the common area display image VC generated by the common area extraction unit 26 (see FIG. 1) based on the binarized images VB1 to VB3. The common area display image VC can be obtained by a logical product of the binarized images VB1 to VB3. In FIG. 6C, regions corresponding to the object regions 82-1 to 82-3 are indicated by broken lines, and a portion where both overlap is a common region 90. The position, size, and shape of the common region 90 are similar to those of the contact regions 52a and 54a of the prismatic member 52 and the cylindrical member 54 shown in FIG. Therefore, the common area display image VC is an image that represents the positions of the prismatic member 52 and the cylindrical member 54 with high accuracy.

  As described above, according to the present embodiment, a normalization unit (22) that normalizes a plurality of original images (V1 to VN) photographed from different viewpoints and generates a plurality of normalized images (VS1 to VSN). ) And a plurality of normalized regions (VS1 to VSN), an object region extraction unit (24) that respectively extracts object regions (82-1 to 82-3) corresponding to objects that satisfy a predetermined condition, and a plurality of Since the common region extraction unit (26) that extracts the common region (90) that is a portion where the object regions (82-1 to 82-3) related to the normalized images (VS1 to VSN) overlap is provided, The exact position of the object can be grasped from the image. Further, as shown in FIGS. 5 and 6A to 6C, even when the number of objects (the prismatic member 52 and the cylindrical member 54) arranged in the planar region 50 is large, the camera 10 By increasing the number, the common area display image VC with high accuracy can be obtained.

[Second Embodiment]
<Configuration of Second Embodiment>
FIG. 7 is a block diagram of an image processing system A2 according to the second embodiment of the present invention. In FIG. 7, portions corresponding to the respective portions in FIG. 1 are denoted by the same reference numerals and description thereof may be omitted.
In the image processing system A2, an image processing apparatus 202 is provided instead of the image processing apparatus 201 of the image processing system A1 (see FIG. 1). Similar to the image processing apparatus 201, the image processing apparatus 202 includes hardware as a general computer such as a CPU, RAM, ROM, and HDD. In FIG. 7, the inside of the image processing apparatus 202 shows functions realized by an application program or the like developed in a RAM as a block.

  The image processing apparatus 202 includes a normalization unit 22, an object region extraction unit 24, a common region extraction unit 26, a centroid position calculation unit 30, and a trajectory image generation unit 32. Here, the functions of the normalization unit 22, the object region extraction unit 24, and the common region extraction unit 26 are the same as those in the first embodiment (see FIG. 1). That is, the normalizing unit 22 outputs the normalized images VS1 to VSN based on the camera images V1 to VN. The object region extraction unit 24 outputs binarized images VB1 to VBN that specify the object region based on the normalized images VS1 to VSN.

  Further, the common area extraction unit 26 outputs a common area display image VC representing the common area based on the binarized images VB1 to VBN. The centroid position calculation unit 30 calculates the centroid position (centroid position) of the common area. In addition, the trajectory image generation unit 32 generates a trajectory image VD that represents the common area and the trajectory of the center of gravity.

<Operation of Second Embodiment>
Next, the operation of this embodiment will be described with reference to FIG. FIG. 8 is a flowchart of a control program executed by the image processing apparatus 202.
In FIG. 8, the processing contents of steps S10 to S16 are the same as those in the first embodiment (see FIG. 2). That is, in steps S10 and S12, the normalization unit 22 acquires the frames of the camera images V1 to VN and outputs the frames of the normalized images VS1 to VSN. In step S14, the object region extraction unit 24 extracts the object region and generates frames of the binarized images VB1 to VBN. In step S16, the common area extraction unit 26 extracts a common area from the frames of the binarized images VB1 to VBN, and generates a frame of the common area display image VC.

  Next, when the process proceeds to step S18, the centroid position calculation unit 30 calculates the centroid position (centroid position) of the common area based on the common area display image VC. Next, when the process proceeds to step S <b> 20, the trajectory image generation unit 32 generates a trajectory image VD that represents the common area and the trajectory of the barycentric position. That is, if drawing is performed while superimposing the gravity center position of the past frame and the gravity center position of the current frame, the drawing result becomes a linear locus. The trajectory image generation unit 32 generates a trajectory image VD by superimposing the trajectory on the common area display image VC.

  Next, when the process proceeds to step S30, the normalizing unit 22 determines whether or not the frame of the trajectory image VD generated this time is the last frame. If “No” is determined here, the process returns to step S10, and the same process is repeated for the subsequent frames. Thereby, the trajectory image VD representing the common area and the trajectory of the center of gravity is displayed on the display unit 12.

<Example of operation>
Next, a specific example of the operation of the present embodiment will be described with reference to FIGS. 9 and 10A to 10E.
First, FIG. 9 is a diagram illustrating an arrangement example of each element in the present embodiment. In FIG. 9, a plane area 50 is a square area that is an intersection of roads. A bus 56 passes over the plane area 50. In the present embodiment, two cameras 10-1 and 10-2 are provided as the plurality of cameras 10.

  The cameras 10-1 and 10-2 are disposed at positions where the planar region 50 and the bus 56 are looked down obliquely from above. Although illustration of the contents of the camera images V1 and V2 output from the cameras 10-1 and 10-2 is omitted, in these image areas, an area corresponding to the planar area 50 is substantially trapezoidal.

  Next, FIG. 10A is a diagram illustrating an example of normalized images VS1 and VS2 generated by the normalization unit 22 (see FIG. 7) based on the camera images V1 and V2. As in the first embodiment (see FIG. 4A), in the normalized images VS1 and VS2, the areas corresponding to the planar area 50 are square areas 70-1 and 70-2. . Further, the object images 76-1 and 76-2 corresponding to the bus 56 have a shape obtained by extending the shape of the bus 56 in the vertical direction.

  Next, FIG. 10B is a diagram illustrating an example of the binarized images VB1 and VB2 generated by the object region extraction unit 24 (see FIG. 7) based on the normalized images VS1 and VS2. In the binarized images VB1 and VB2, regions corresponding to the object images 76-1 and 76-2 of the normalized images VS1 and VS2 are object regions 86-1 and 86-2, and pixel values of these regions are “ Set to 1 ". Further, the pixel values in the areas other than the object areas 86-1 and 86-2 are set to “0”.

  Next, FIG. 10C is a diagram illustrating an example of the common area display image VC generated by the common area extraction unit 26 (see FIG. 7) based on the binarized images VB1 and VB2. The common area display image VC can be obtained by a logical product of the binarized images VB1 and VB2. In FIG. 10C, the shape of the common region 90 substantially matches the shape of the floor surface of the bus 56.

  FIG. 10D shows the centroid position 94 of the common area 90 calculated by the centroid position calculation unit 30. FIG. 10E shows an example of the trajectory image VD generated by the trajectory image generation unit 32. According to the present embodiment, as in the first embodiment, the exact position of the photographed object (bus 56) can be grasped from the image. Furthermore, according to the present embodiment, the accurate center-of-gravity position 94 of the common area 90 can be obtained, and an accurate trajectory image VD can be obtained.

[Third Embodiment]
<Configuration of Third Embodiment>
FIG. 11 is a block diagram of an image processing system A3 according to the third embodiment of the present invention. In FIG. 11, parts corresponding to those in FIGS. 1 and 7 are denoted by the same reference numerals, and the description thereof may be omitted.
In the image processing system A3, an image processing apparatus 203 is provided instead of the image processing apparatus 201 of the image processing system A1 (see FIG. 1). Similar to the image processing apparatus 201, the image processing apparatus 203 includes hardware as a general computer such as a CPU, RAM, ROM, and HDD. In FIG. 11, the inside of the image processing apparatus 203 shows functions realized by application programs or the like developed in the RAM as blocks.

  The image processing apparatus 203 includes a normalization unit 22, an object region extraction unit 24, a common region extraction unit 26, a resizing unit 34, and an image allocation unit 36. Here, the functions of the normalization unit 22, the object region extraction unit 24, and the common region extraction unit 26 are the same as those in the first embodiment (see FIG. 1). That is, the normalizing unit 22 outputs the normalized images VS1 to VSN based on the camera images V1 to VN. The object region extraction unit 24 outputs binarized images VB1 to VBN that specify the object region based on the normalized images VS1 to VSN. Further, the common area extraction unit 26 outputs a common area display image VC representing the common area based on the binarized images VB1 to VBN.

  Furthermore, the object region extraction unit 24 of the present embodiment outputs background images VJ1 to VJN that are images obtained by removing the object regions of the normalized images VS1 to VSN, and the object in any of the normalized images VS1 to VSN. It has a function of outputting an image of a region as an object image VK. The resizing unit 34 resizes the size of the object image VK corresponding to the common area in the common area display image VC, and outputs the resized object image VL. For example, a contracted object image VL may be obtained by reducing the object image VK so as to fit in the common area.

  The image assignment unit 36 superimposes the background images VJ1 to VJN and generates a new background image. The generated background image is referred to as a “total background image”. The general background image is an image representing the planar area 50 in an area other than the common area. Furthermore, the image allocation unit 36 superimposes the general background image and the stretchable object image VL, and outputs the result as a composite image VE. The display unit 12 displays the composite image VE.

<Operation of Third Embodiment>
Next, the operation of this embodiment will be described with reference to FIG. FIG. 12 is a flowchart of a control program executed by the image processing apparatus 203.
In FIG. 12, the processing contents of steps S10 to S16 are the same as those in the first embodiment (see FIG. 2). That is, in steps S10 and S12, the normalization unit 22 acquires the frames of the camera images V1 to VN and outputs the frames of the normalized images VS1 to VSN. In step S14, the object region extraction unit 24 extracts the object region and generates frames of the binarized images VB1 to VBN. In step S16, the common area extraction unit 26 extracts a common area from the frames of the binarized images VB1 to VBN, and generates a frame of the common area display image VC.

  Next, when the process proceeds to step S22, the object region extraction unit 24 outputs the background images VJ1 to VJN of the normalized images VS1 to VSN and the object image VK. Next, when the process proceeds to step S24, the resizing unit 34 resizes the object image VK. That is, the object image VK is resized corresponding to the common area in the common area display image VC, and is output as a stretchable object image VL.

  Next, when the process proceeds to step S26, the image assignment unit 36 generates a composite image VE. That is, the image allocation unit 36 generates a combined background image by generating a combined background image based on the background images VJ1 to VJN and superimposing the combined background image and the expandable object image VL. Then, the generated composite image VE is displayed on the display unit 12.

  Next, when the process proceeds to step S30, the normalization unit 22 determines whether or not the frame of the composite image VE generated this time is the final frame. If “No” is determined here, the process returns to step S10, and the same process is repeated for the subsequent frames. As a result, the composite image VE continues to be displayed on the display unit 12.

<Example of operation>
Next, a specific example of the operation of the present embodiment will be described with reference to FIGS. 13 and 14A to 14D.
First, FIG. 13 is a diagram illustrating an arrangement example of each element in the present embodiment. In the arrangement example shown in FIG. 13, as in the arrangement example shown in FIG. 9, the bus 56 passes over the planar region 50, and two cameras 10-1 and 10-2 are used as the plurality of cameras 10. Is provided. Further, in FIG. 13, a pedestrian 58 exists near the bus 56. The cameras 10-1 and 10-2 output camera images V1 and V2. However, since the pedestrian 58 enters the blind spot of the bus 56 as viewed from the camera 10-2, the camera image V2 does not include the image of the pedestrian 58.

  Next, FIG. 14A is a diagram illustrating an example of normalized images VS1 and VS2 generated by the normalization unit 22 (see FIG. 11) based on the camera images V1 and V2. As in the second embodiment (see FIG. 10A), in the normalized images VS1 and VS2, the areas corresponding to the plane area 50 are square areas 70-1 and 70-2. . Further, the object images 76-1 and 76-2 corresponding to the bus 56 have a shape obtained by extending the shape of the bus 56 in the vertical direction. Here, the normalized image VS1 includes the object image 78 of the pedestrian 58, but the normalized image VS2 does not include the image of the pedestrian 58.

  Although illustration of the binarized images VB1 and VB2 output by the object region extraction unit 24 is omitted, the regions corresponding to the object images 76-1, 78, and 76-2 in FIG. The image is set to “0”. Next, FIG. 14B is a diagram illustrating an example of the common area display image VC generated by the common area extraction unit 26 (see FIG. 11). The content of the common area display image VC shown in the figure is the same as that of the second embodiment (see FIG. 10C). That is, the shape of the common area 90 in the common area display image VC substantially matches the shape of the floor surface of the bus 56.

  Next, FIG. 14C is a diagram illustrating an example of the composite image VE output from the image allocation unit 36. In the composite image VE, an area 100 corresponding to the planar area 50 (see FIG. 13) is generated based on the overall background image obtained by superimposing the background images VJ1 to VJN. Further, the object image 106 is obtained by resizing the object image 76-2 shown in FIG. 14A corresponding to the common area 90, and the object image 108 is obtained by using the object image 78 as it is.

  Next, FIG. 14D is a diagram illustrating an example of the composite image VF according to the comparative example. In this comparative example, the object image 76-2 which has not been resized is directly used as the object image 116 of the composite image VF in the plane region 50 by using the technique of Patent Document 3 (Japanese Patent Laid-Open No. 2007-027948) described above. The corresponding area 110 is superimposed. Even if the pedestrian 58 shown in FIG. 14A is captured by the camera 10-1, the position of the pedestrian 58 is occupied by the object image 116. Therefore, the user can select the pedestrian 58 based on the composite image VF. Cannot recognize existence. On the other hand, according to the composite image VE of the present embodiment (see FIG. 14C), the object image 106 of the bus 56 substantially matches the shape of the floor surface of the bus 56. The presence of the pedestrian 58 can be clearly recognized. More generally expressed, it is possible to display what exists on the planar region 50 (see FIG. 13) with high probability.

[Fourth Embodiment]
FIG. 15 is a block diagram of an image processing system A4 according to the fourth embodiment of the present invention. In FIG. 15, portions corresponding to the respective portions in FIGS. 1 to 14 are denoted by the same reference numerals, and description thereof may be omitted.
In the image processing system A4, an image processing apparatus 204 is provided instead of the image processing apparatus 203 of the image processing system A3 (see FIG. 11). Similar to the image processing apparatus 203, the image processing apparatus 204 includes hardware as a general computer such as a CPU, RAM, ROM, and HDD. In FIG. 15, the inside of the image processing apparatus 204 shows, as a block, functions realized by an application program or the like developed in the RAM.

  The image processing apparatus 204 includes a normalization unit 22, an object region extraction unit 24, and a common region extraction unit 26. These functions are the same as those of the third embodiment (see FIG. 11). Furthermore, the image processing apparatus 204 of the present embodiment has an image memory 33. The image memory 33 stores image data (hereinafter referred to as a symbol image) such as an animation image or an abstract image of a vehicle or a person.

  In addition, the resizing unit 34 in the present embodiment reads any symbol image from the image memory 33 as the object image VK, resizes the symbol image corresponding to the common area 90 of the common area display image VC, and forms the expandable object image VL. Output. Similar to the third embodiment, the image allocation unit 36 superimposes the background images VJ1 to VJN to generate a comprehensive background image. Furthermore, the image allocation unit 36 superimposes the general background image and the stretchable object image VL, and outputs the result as a composite image VE. The display unit 12 displays the composite image VE.

  According to the present embodiment, a composite image VE similar to that shown in FIG. 14C can be displayed on the display unit 12. However, in this embodiment, the object image 106 is obtained by resizing any symbol image stored in the image memory 33. According to the present embodiment, as in the third embodiment, it is possible to display what exists on the planar region 50 (see FIG. 13) with high probability. Furthermore, since the symbol image is applied to the composite image VE instead of the actually photographed object image, the composite image VE can be easily viewed by the user.

[Modification]
The present invention is not limited to the above-described embodiments, and various modifications can be made. The above-described embodiments are illustrated for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described. Further, a part of the configuration of an embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of an embodiment. Further, it is possible to delete a part of the configuration of each embodiment, or to add or replace another configuration. Further, the control lines and information lines are those that are considered necessary for the explanation, and do not necessarily indicate all the control lines and information lines that are necessary for the product. Actually, it may be considered that almost all the components are connected to each other. Examples of possible modifications to the above embodiment are as follows.

(1) In the third and fourth embodiments, a locus image generation unit 32 (see FIG. 7) similar to that in the second embodiment is added, and a locus 96 is added to these synthesized images VE (see FIG. 14C). (See FIG. 10E) may be displayed.

(2) In the second to fourth embodiments described above, an example in which a road intersection is photographed by a plurality of cameras 10 is described. However, the photographing target of the present invention is not limited to a road. The present invention can also be applied to a field in which a “person” is photographed by a plurality of cameras 10 in a station premises, a store, or the like and the flow line of the “person” is grasped.

(3) Since the hardware of the image processing apparatuses 201 to 204 in each of the above embodiments can be realized by a general computer, the programs according to the flowcharts shown in FIGS. 2, 8, and 12 are stored in a storage medium, Or you may distribute via a transmission line.

(4) In each of the above embodiments, the processing shown in FIGS. 2, 8, and 12 has been described as software processing using a program in each of the above embodiments. Application specific integrated circuit (IC for specific application), or hardware processing using FPGA (field-programmable gate array) or the like may be used.

22 Normalization unit 24 Object region extraction unit 26 Common region extraction unit 30 Center of gravity position calculation unit 32 Trajectory image generation unit 33 Image memory 34 Resize unit 36 Image allocation units 82-1 to 82-3, 86-1, 86-2 Object Area 90 Common area 94 Center of gravity position 201-204 Image processing device S12 Step (normalization process)
S14 step (object region extraction process)
S16 step (common area extraction process)
V1-VN camera image (original image)
VD locus image VE composite image VJ1 to VJN background image VK object image VL telescopic object image VS1 to VSN normalized image

Claims (6)

A normalization unit that normalizes each of a plurality of original images taken from different viewpoints and generates a plurality of normalized images;
An object region extraction unit that extracts object regions corresponding to objects satisfying a predetermined condition with respect to the plurality of normalized images, and generates a plurality of background images based on the plurality of normalized images ;
A common area extraction unit that extracts a common area that is a portion where the object areas related to the plurality of normalized images overlap;
An image allocating unit that generates a general background image representing an image of a region other than the common region in a predetermined region by combining a plurality of the background images;
An image processing apparatus comprising: A center-of-gravity position calculation unit for calculating the position of the center of gravity of the common area;
A trajectory image generation unit that generates a trajectory image representing the trajectory of the center of gravity position;
The image processing apparatus according to claim 1, further comprising: A resizing unit the test sheet has been the object image to generate an elastic object image resized to a size corresponding to the common region,
An image assigning unit that synthesizes the general background image and the stretchable object image and outputs a composite image;
The image processing apparatus according to claim 1, further comprising: The image processing apparatus according to claim 3, wherein the object image is an image of the object region in any one of the normalized images. An image memory for storing one or more symbol images in advance;
The image processing apparatus according to claim 3, wherein the object image is any one of the symbol images stored in the image memory. A normalization process of normalizing each of a plurality of original images taken from different viewpoints to generate a plurality of normalized images;
An object region extracting step of extracting object regions corresponding to objects satisfying a predetermined condition with respect to the plurality of normalized images, and generating a plurality of background images based on the plurality of normalized images ;
A common region extraction process for extracting a common region that is a portion where the object regions related to the plurality of normalized images overlap;
An image assignment process for generating a comprehensive background image representing an image of an area other than the common area in a predetermined area by combining a plurality of the background images;
An image processing method comprising:

Images