EP1262933A1 - Image processing apparatus, image processing method, and vehicle monitoring system - Google Patents

Image processing apparatus, image processing method, and vehicle monitoring system Download PDF

Info

Publication number
EP1262933A1
EP1262933A1 EP00987649A EP00987649A EP1262933A1 EP 1262933 A1 EP1262933 A1 EP 1262933A1 EP 00987649 A EP00987649 A EP 00987649A EP 00987649 A EP00987649 A EP 00987649A EP 1262933 A1 EP1262933 A1 EP 1262933A1
Authority
EP
European Patent Office
Prior art keywords
image
images
image processing
road
digital
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP00987649A
Other languages
German (de)
French (fr)
Other versions
EP1262933B1 (en
EP1262933A4 (en
Inventor
Takeshi Sumitomo Electric Ind. Ltd. IWAMOTO
Kunio Sakai
Kunihiko Suzuki
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sumitomo Electric Industries Ltd
Original Assignee
Sumitomo Electric Industries Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sumitomo Electric Industries Ltd filed Critical Sumitomo Electric Industries Ltd
Publication of EP1262933A1 publication Critical patent/EP1262933A1/en
Publication of EP1262933A4 publication Critical patent/EP1262933A4/en
Application granted granted Critical
Publication of EP1262933B1 publication Critical patent/EP1262933B1/en
Anticipated expiration legal-status Critical
Expired - Lifetime legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/01—Detecting movement of traffic to be counted or controlled
    • G08G1/04—Detecting movement of traffic to be counted or controlled using optical or ultrasonic detectors

Definitions

  • the present invention relates to an image processing apparatus, image processing method, and vehicle monitoring system used for, e.g., monitoring of vehicles.
  • CCTV Cell-Circuit TeleVision
  • a large number of cameras are installed to avoid monitoring dead angles.
  • Output images from these cameras are transmitted to a highway control center, displayed on monitors, and monitored by observers.
  • the above traffic monitoring method has the problem that the detection of a traffic abnormality is delayed. That is, since camera images are displayed on each monitor as they are switched, camera images which are not displayed exist. This may delay the detection of a traffic abnormality. Also, frequently switching camera images increases the fatigue of observers and makes them overlook traffic abnormalities.
  • the present invention has been made to solve the above problems, and has as its object to provide an image processing apparatus, image processing method, and vehicle monitoring system capable of improving the accuracy of traffic abnormality detection.
  • an image processing apparatus comprises converting means for receiving a plurality of video signals obtained by sensing images of a road, and A/D-converting the video signals into digital images, correcting means for correcting the digital images such that the road runs in the longitudinal or lateral direction of each digital image, and image compositing means for compositing a plurality of corrected images obtained by the correcting means, such that the roads shown in the corrected images continue.
  • the image processing apparatus is characterized by further comprising storage means for storing the composite image as image data.
  • An image processing method comprises the conversion step of receiving a plurality of video signals obtained by sensing images of a road, and A/D-converting the video signals into digital images, the correction step of correcting the digital images such that the road runs in the longitudinal direction or lateral direction of each digital image, and the image composition step of compositing a plurality of corrected images obtained in the correction step, such that the roads shown in the corrected images continue.
  • the image processing method according to the present invention is characterized by further comprising the storage step of storing the composite image as image data.
  • a vehicle monitoring system comprises a plurality of image sensing means arranged along a road to be monitored to sense images of vehicles traveling on the road, image processing means for receiving video signals obtained by the image sensing means, A/D-converting the video signals into digital images, correcting the digital images such that the road runs in the longitudinal direction or lateral direction of each digital image, and compositing the corrected digital images such that the roads shown in the digital images continue, and display means for displaying the image processed by the image processing means.
  • a vehicle monitoring system comprises a plurality of image sensing means arranged along a road to be monitored to sense images of vehicles traveling on the road, each of the image sensing means having a wide angle lens in an image sensing optical system thereof, image processing means for receiving video signals obtained by the image sensing means, A/D-converting the video signals into digital images, correcting distortion, caused by the wide angle lens, of each digital image, and compositing the corrected digital images such that the roads shown in the digital images continue, and display means for displaying the image processed by the image processing means.
  • the vehicle monitoring system according to the present invention is characterized in that the display means is a projector having a display area long from side to side.
  • the image processing system according to the present invention is characterized in that the image processing means stores the composite image as image data.
  • the whole of the road as an object of traffic monitoring can be displayed on a single screen. Accordingly, the entire road can be monitored without switching images to be displayed. Since there is no area which cannot be monitored owing to image switching, therefore, it is possible to prevent a delay of traffic abnormality detection and increase the speed of traffic abnormality detection.
  • traffic abnormalities can be detected on the basis of changes in an image. This facilitates visual traffic abnormality detection and reduces the monitoring load.
  • corrected composited image data is stored, after a traffic abnormality occurs it is possible to analyze the abnormality by checking this composite image.
  • Fig. 1 is a view for explaining a vehicle monitoring system. As shown in Fig. 1, this vehicle monitoring system monitors the state of traffic on a highway, e.g., detects traffic abnormalities in a tunnel 1.
  • the vehicle monitoring system includes a plurality of monitor cameras 2.
  • These monitor cameras 2 are image sensing means for sensing an image of a vehicle 11 traveling on a road in the tunnel 1, and are installed at predetermined intervals in the tunnel 1.
  • the installation interval of the monitor cameras 2 is, e.g., about 200 m.
  • the image sensing direction of the monitor camera 2 is a direction in which an image of the front or rear of the vehicle 11 traveling on a road 10 is sensed.
  • Each monitor camera 2 is connected to a distributor 3 which is installed in an institution 4 built near the tunnel 1. This distributor 3 receives sensed images from the monitor cameras 2 and outputs these images to an image processing apparatus 5.
  • the image processing apparatus 5 is an image processing means for receiving images from the monitor cameras 2, processing these images, and transmitting the processed images to a control center 6.
  • This image processing apparatus 5 comprises, e.g., an A/D converter, video frame memory, image processor, and signal output interface.
  • the A/D converter A/D-converts an input analog video signal into a digital signal.
  • a video A-D converter is used as this A/D converter.
  • the video frame memory stores converted digital video signals as image data.
  • This image data is an aggregate of pixels having tone values at individual coordinate runs on the x and y axes.
  • the pixel tone has, e.g., 8-bit 256 gray levels.
  • the image processor processes the image data stored in the video frame memory.
  • the signal output interface outputs the processed image to the outside in accordance with a command from the image processor.
  • the image transmitted from the image processing apparatus is input to a monitor 61, and this monitor 61 displays the image sensed in the tunnel 1. Through this image, a supervisor can visually monitor the traveling states of vehicles in the tunnel 1.
  • the monitor 61 is an image display means and desirably has a display area long from side to side. For example, a projector is used as this monitor 61.
  • the monitor cameras 2 sense images of the vehicle 11 traveling on the road 10 and input video signals to the distributor 3.
  • the distributor 3 suitably transfers these video signals from the monitor cameras 2 to the image processing apparatus 5.
  • the image processing apparatus 5 processes the video signals.
  • Fig. 2 is a flow chart of this image processing performed by the image processing apparatus 5.
  • an input video signal to the image processing apparatus 5 is A/D-converted.
  • This A/D conversion is the process of converting an analog video signal into a digital video signal.
  • the converted digital image is stored as image data into the video frame memory.
  • This image correction is the process of correcting the image data such that the road 10 shown in the A/D-converted digital image runs in the longitudinal or lateral direction of the image.
  • a digital image shown in Fig. 3 is corrected into a corrected image shown in Fig. 4.
  • the road 10 is so displayed in a digital image 20 as to go away from the lower right portion to the upper left portion.
  • this digital image 20 is corrected such that the road 10 runs in the lateral direction of the image.
  • This correcting process forms a corrected image 21.
  • the vehicle 11 traveling on the road 10 is also processed in accordance with the correction.
  • This image composition is the process of compositing corrected images subjected to the image correction.
  • corrected images 21 pertaining to video signals obtained by the monitor cameras 2 at the same time are composited into one image.
  • This compositing process is performed such that the roads 11 shown in the individual corrected images 21 continue.
  • the composition is so performed that more than one image of the same vehicle 11 will not be displayed.
  • Fig. 5 shows a composite image 22 subjected to the compositing process.
  • the composite image 22 is transmitted to the control center 6.
  • the whole of this composite image 22 is displayed on the monitor 61 in the control center 6.
  • the image processing apparatus, image processing method, and vehicle monitoring system can display, on a single screen, the whole of the road 10 as an object of traffic monitoring. Accordingly, the entire road can be monitored without switching images to be displayed.
  • the composite image 22 is transmitted to the control center 6.
  • the image processing apparatus, image processing method, and vehicle monitoring system according to the present invention are not limited to this embodiment.
  • the control center 6 can also perform all the image processing operations (A/D conversion, image correction, and image composition).
  • the image processing apparatus, image processing method, and vehicle monitoring system according to this embodiment relate to vehicle monitoring on a road in the tunnel 1.
  • the image processing apparatus, image processing method, and vehicle monitoring system according to the present invention are not restricted to this embodiment.
  • the image processing apparatus, image processing method, and vehicle monitoring system according to the present invention can also be used for vehicle monitoring on roads in other places.
  • the image processing apparatus, image processing method, and vehicle monitoring system according to this embodiment have the same configurations as the image processing apparatus, image processing method, and vehicle monitoring system according to the first embodiment, except that a monitor camera 2 as an image sensing means includes an image sensing optical system having a wide angle lens, and that an image processing apparatus 5 as an image processing means corrects image distortion caused by this wide angle lens.
  • Fig. 6 is a view for explaining the monitor cameras in the image processing apparatus, image processing method, and vehicle monitoring system according to this embodiment.
  • the monitor cameras 2 are so arranged as to sense images of vehicles 11 traveling on a road 10 from the side of the road 10.
  • Each monitor camera 2 has a wide angle lens in its image sensing optical system and can sense images over a wide view angle.
  • a fish-eye lens is preferably used as this wide angle lens.
  • a "wide angle lens" herein mentioned means a lens having a view angle wider than that of a standard lens, e.g., a view angle of 60° or more.
  • the monitor cameras 2 sense images of the vehicles 11 traveling on the road 10 and input video signals to a distributor 3.
  • This distributor 3 properly inputs these video signals from the monitor cameras 2 to the image processing apparatus 5.
  • the image processing apparatus 5 processes these video signals.
  • A/D conversion is performed as in the first embodiment.
  • correction for removing distortion of the image sensed through the wide angle lens is performed.
  • an A/D-converted digital image 30 has barrel-shaped form distortion because the image is sensed through the wide angle lens.
  • a corrected image 31 from which the distortion is removed is obtained.
  • an image compositing process composites the corrected images 31 pertaining to the video signals obtained at the same time by the monitor cameras 2 into a single composite image 32.
  • this image compositing process is performed such that the roads 10 shown in the individual corrected images 21 continue. Also, the composition is so performed that more than one image of the same vehicle 11 will not be displayed.
  • Fig. 9 shows a composite image 32 subjected to the compositing process.
  • the composite image 32 is transmitted to a control center 6.
  • the whole of this composite image 32 is displayed on a monitor 61 in the control center 6.
  • the image processing apparatus, image processing method, and vehicle monitoring system can display, on a single screen, the whole of the road 10 as an object of traffic monitoring, similar to the image processing apparatus, image processing method, and vehicle monitoring system according to the first embodiment. Accordingly, the entire road can be monitored without switching images to be displayed. Since there is no area which cannot be monitored owing to image switching, therefore, it is possible to prevent a delay of traffic abnormality detection and increase the speed of traffic abnormality detection. Also, since no image switching is performed, traffic abnormalities can be detected on the basis of changes in an image. This facilitates visual traffic abnormality detection and reduces the monitoring load.
  • a wide angle lens is used in the image sensing optical system of the monitor camera 2. Since this allows each monitor camera 2 to monitor a wide range, the number of the monitor cameras 2 to be installed can be reduced.
  • the image processing apparatus, image processing method, and vehicle monitoring system according to this embodiment have the same configurations as the image processing apparatuses, image processing methods, and vehicle monitoring systems according to the first and second embodiments, except that image storage is performed after image composition in an image processing means and in an image processing step.
  • Fig. 10 is a flow chart of image processing performed in the image processing apparatus, image processing method, and vehicle monitoring system according to this embodiment.
  • image storage is performed. This image storage is done by storing a composite image 22 or 32 subjected to image composition into a storage device such as a memory. Since the composite image 22 or 32 is thus stored, after a traffic abnormality occurs it is possible to analyze the abnormality by checking this composite image.
  • the whole of a road as an object of traffic monitoring can be displayed on a single screen. Accordingly, the entire road can be monitored without switching images to be displayed. Since there is no area which cannot be monitored owing to image switching, therefore, it is possible to prevent a delay of traffic abnormality detection and increase the speed of traffic abnormality detection.
  • traffic abnormalities can be detected on the basis of changes in an image. This facilitates visual traffic abnormality detection and reduces the monitoring load.
  • corrected composited image data is stored, after a traffic abnormality occurs it is possible to analyze the abnormality by checking this composite image.
  • the image processing apparatus, image processing method, and vehicle monitoring system of the present invention are useful in monitoring traffic of vehicles traveling on a road.

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Traffic Control Systems (AREA)
  • Closed-Circuit Television Systems (AREA)
  • Burglar Alarm Systems (AREA)
  • Image Processing (AREA)
  • Editing Of Facsimile Originals (AREA)
  • Controls And Circuits For Display Device (AREA)

Abstract

Images of vehicles traveling on a road are sensed by a plurality of monitor cameras arranged along the road. Video signals obtained by these monitor cameras are A/D-converted into digital images. These digital images are so corrected that the road shown in each digital image runs in the longitudinal or lateral direction of the image. The corrected digital images are composited such that the roads shown in these images continue. This allows the whole of the road as an object of traffic monitoring to be displayed on a single screen. Since the entire road can be monitored without switching images to be displayed, the speed of traffic abnormality detection increases.

Description

Technical Field
The present invention relates to an image processing apparatus, image processing method, and vehicle monitoring system used for, e.g., monitoring of vehicles.
Background Art
Generally, CCTV (Closed-Circuit TeleVision) cameras are installed on express highways and general highways to monitor traffic. Especially in long tunnels and the like, a large number of cameras are installed to avoid monitoring dead angles. Output images from these cameras are transmitted to a highway control center, displayed on monitors, and monitored by observers. It is usually difficult to install monitors equal in number to cameras owing to, e.g., the installation space. Therefore, the number of installed monitors is made smaller than that of installed cameras, and camera images from a plurality of cameras are displayed on one monitor. That is, a plurality of camera images are sequentially switched and displayed at intervals of a few seconds.
Disclosure of Invention
Unfortunately, the above traffic monitoring method has the problem that the detection of a traffic abnormality is delayed. That is, since camera images are displayed on each monitor as they are switched, camera images which are not displayed exist. This may delay the detection of a traffic abnormality. Also, frequently switching camera images increases the fatigue of observers and makes them overlook traffic abnormalities.
To solve these problems, the introduction of image processing technologies is being examined. However, it is technically difficult to detect traffic abnormalities with perfect accuracy.
The present invention has been made to solve the above problems, and has as its object to provide an image processing apparatus, image processing method, and vehicle monitoring system capable of improving the accuracy of traffic abnormality detection.
To achieve the above object, an image processing apparatus according to the present invention comprises converting means for receiving a plurality of video signals obtained by sensing images of a road, and A/D-converting the video signals into digital images, correcting means for correcting the digital images such that the road runs in the longitudinal or lateral direction of each digital image, and image compositing means for compositing a plurality of corrected images obtained by the correcting means, such that the roads shown in the corrected images continue.
The image processing apparatus according to the present invention is characterized by further comprising storage means for storing the composite image as image data.
An image processing method according to the present invention comprises the conversion step of receiving a plurality of video signals obtained by sensing images of a road, and A/D-converting the video signals into digital images, the correction step of correcting the digital images such that the road runs in the longitudinal direction or lateral direction of each digital image, and the image composition step of compositing a plurality of corrected images obtained in the correction step, such that the roads shown in the corrected images continue.
The image processing method according to the present invention is characterized by further comprising the storage step of storing the composite image as image data.
A vehicle monitoring system according to the present invention comprises a plurality of image sensing means arranged along a road to be monitored to sense images of vehicles traveling on the road, image processing means for receiving video signals obtained by the image sensing means, A/D-converting the video signals into digital images, correcting the digital images such that the road runs in the longitudinal direction or lateral direction of each digital image, and compositing the corrected digital images such that the roads shown in the digital images continue, and display means for displaying the image processed by the image processing means.
A vehicle monitoring system according to the present invention comprises a plurality of image sensing means arranged along a road to be monitored to sense images of vehicles traveling on the road, each of the image sensing means having a wide angle lens in an image sensing optical system thereof, image processing means for receiving video signals obtained by the image sensing means, A/D-converting the video signals into digital images, correcting distortion, caused by the wide angle lens, of each digital image, and compositing the corrected digital images such that the roads shown in the digital images continue, and display means for displaying the image processed by the image processing means.
The vehicle monitoring system according to the present invention is characterized in that the display means is a projector having a display area long from side to side.
The image processing system according to the present invention is characterized in that the image processing means stores the composite image as image data.
In these inventions described above, the whole of the road as an object of traffic monitoring can be displayed on a single screen. Accordingly, the entire road can be monitored without switching images to be displayed. Since there is no area which cannot be monitored owing to image switching, therefore, it is possible to prevent a delay of traffic abnormality detection and increase the speed of traffic abnormality detection.
Also, since no image switching is performed, traffic abnormalities can be detected on the basis of changes in an image. This facilitates visual traffic abnormality detection and reduces the monitoring load.
Additionally, since an image of a road to be monitored is sensed sideways by using a wide angle lens, image correction can be readily performed.
Furthermore, since corrected composited image data is stored, after a traffic abnormality occurs it is possible to analyze the abnormality by checking this composite image.
Brief Description of Drawings
  • Fig. 1 is a view for explaining a vehicle monitoring system according to the first embodiment of the present invention;
  • Fig. 2 is a flow chart showing the operation of an image processing apparatus in the vehicle monitoring system shown in Fig. 1;
  • Fig. 3 is a view for explaining an AID-converted digital image;
  • Fig. 4 is a view for explaining a corrected image subjected to a correcting process;
  • Fig. 5 is a view for explaining a composite image subjected to a compositing process;
  • Fig. 6 is a view for explaining a vehicle monitoring system and the like according to the second embodiment;
  • Fig. 7 is a view for explaining an A/D-converted digital image;
  • Fig. 8 is a view for explaining a corrected image subjected to a correcting process;
  • Fig. 9 is a view for explaining a composite image subjected to a compositing process; and
  • Fig. 10 is a view for explaining a vehicle monitoring system and the like according to the third embodiment.
    • Best Mode for Carrying Out the Invention
    Embodiments of the present invention will be described below with reference to the accompanying drawings. Note that the same reference numerals denote the same parts in these drawings, and a duplicate description thereof will be omitted. Note also that the dimensional ratio of each drawing is not necessarily consistent with that of the explanation.
    (First Embodiment)
    Fig. 1 is a view for explaining a vehicle monitoring system. As shown in Fig. 1, this vehicle monitoring system monitors the state of traffic on a highway, e.g., detects traffic abnormalities in a tunnel 1. The vehicle monitoring system includes a plurality of monitor cameras 2.
    These monitor cameras 2 are image sensing means for sensing an image of a vehicle 11 traveling on a road in the tunnel 1, and are installed at predetermined intervals in the tunnel 1. The installation interval of the monitor cameras 2 is, e.g., about 200 m. The image sensing direction of the monitor camera 2 is a direction in which an image of the front or rear of the vehicle 11 traveling on a road 10 is sensed. Each monitor camera 2 is connected to a distributor 3 which is installed in an institution 4 built near the tunnel 1. This distributor 3 receives sensed images from the monitor cameras 2 and outputs these images to an image processing apparatus 5.
    The image processing apparatus 5 is an image processing means for receiving images from the monitor cameras 2, processing these images, and transmitting the processed images to a control center 6. This image processing apparatus 5 comprises, e.g., an A/D converter, video frame memory, image processor, and signal output interface.
    The A/D converter A/D-converts an input analog video signal into a digital signal. For example, a video A-D converter is used as this A/D converter. The video frame memory stores converted digital video signals as image data. This image data is an aggregate of pixels having tone values at individual coordinate runs on the x and y axes. The pixel tone has, e.g., 8-bit 256 gray levels.
    The image processor processes the image data stored in the video frame memory. The signal output interface outputs the processed image to the outside in accordance with a command from the image processor.
    In the control center 6, the image transmitted from the image processing apparatus is input to a monitor 61, and this monitor 61 displays the image sensed in the tunnel 1. Through this image, a supervisor can visually monitor the traveling states of vehicles in the tunnel 1. The monitor 61 is an image display means and desirably has a display area long from side to side. For example, a projector is used as this monitor 61.
    The operation of the vehicle monitoring system, the operation of the image processing apparatus, and an image processing method will be described below.
    Referring to Fig. 1, the monitor cameras 2 sense images of the vehicle 11 traveling on the road 10 and input video signals to the distributor 3. The distributor 3 suitably transfers these video signals from the monitor cameras 2 to the image processing apparatus 5. The image processing apparatus 5 processes the video signals.
    Fig. 2 is a flow chart of this image processing performed by the image processing apparatus 5.
    As shown in S100 of Fig. 2, an input video signal to the image processing apparatus 5 is A/D-converted. This A/D conversion is the process of converting an analog video signal into a digital video signal. The converted digital image is stored as image data into the video frame memory.
    The flow advances to S200 to perform image correction. This image correction is the process of correcting the image data such that the road 10 shown in the A/D-converted digital image runs in the longitudinal or lateral direction of the image. By this image correction, a digital image shown in Fig. 3 is corrected into a corrected image shown in Fig. 4. Referring to Fig. 3, the road 10 is so displayed in a digital image 20 as to go away from the lower right portion to the upper left portion. As shown in Fig. 4, this digital image 20 is corrected such that the road 10 runs in the lateral direction of the image. This correcting process forms a corrected image 21. During the correction, the vehicle 11 traveling on the road 10 is also processed in accordance with the correction.
    The flow then advances to S300 to perform image composition. This image composition is the process of compositing corrected images subjected to the image correction. For example, corrected images 21 pertaining to video signals obtained by the monitor cameras 2 at the same time are composited into one image. This compositing process is performed such that the roads 11 shown in the individual corrected images 21 continue. Also, the composition is so performed that more than one image of the same vehicle 11 will not be displayed. Fig. 5 shows a composite image 22 subjected to the compositing process.
    After this image compositing process is completed, the composite image 22 is transmitted to the control center 6. The whole of this composite image 22 is displayed on the monitor 61 in the control center 6.
    As described above, the image processing apparatus, image processing method, and vehicle monitoring system according to this embodiment can display, on a single screen, the whole of the road 10 as an object of traffic monitoring. Accordingly, the entire road can be monitored without switching images to be displayed.
    Since there is no area which cannot be monitored owing to image switching, therefore, it is possible to prevent a delay of traffic abnormality detection and increase the speed of traffic abnormality detection. Also, since no image switching is performed, traffic abnormalities can be detected on the basis of changes in an image. This facilitates visual traffic abnormality detection and reduces the monitoring load.
    In the image processing apparatus, image processing method, and vehicle monitoring system according to this embodiment, the composite image 22 is transmitted to the control center 6. However, the image processing apparatus, image processing method, and vehicle monitoring system according to the present invention are not limited to this embodiment. For example, it is also possible to transmit a plurality of corrected images 21 to the control center 6 and composite these images in the control center 6. The control center 6 can also perform all the image processing operations (A/D conversion, image correction, and image composition).
    Furthermore, the image processing apparatus, image processing method, and vehicle monitoring system according to this embodiment relate to vehicle monitoring on a road in the tunnel 1. However, the image processing apparatus, image processing method, and vehicle monitoring system according to the present invention are not restricted to this embodiment. For example, the image processing apparatus, image processing method, and vehicle monitoring system according to the present invention can also be used for vehicle monitoring on roads in other places.
    (Second Embodiment)
    An image processing apparatus, image processing method, and vehicle monitoring system according to the second embodiment will be described below.
    The image processing apparatus, image processing method, and vehicle monitoring system according to this embodiment have the same configurations as the image processing apparatus, image processing method, and vehicle monitoring system according to the first embodiment, except that a monitor camera 2 as an image sensing means includes an image sensing optical system having a wide angle lens, and that an image processing apparatus 5 as an image processing means corrects image distortion caused by this wide angle lens.
    Fig. 6 is a view for explaining the monitor cameras in the image processing apparatus, image processing method, and vehicle monitoring system according to this embodiment. As shown in Fig. 6, the monitor cameras 2 are so arranged as to sense images of vehicles 11 traveling on a road 10 from the side of the road 10. Each monitor camera 2 has a wide angle lens in its image sensing optical system and can sense images over a wide view angle. As this wide angle lens, a fish-eye lens is preferably used. A "wide angle lens" herein mentioned means a lens having a view angle wider than that of a standard lens, e.g., a view angle of 60° or more.
    The operation of the vehicle monitoring system, the operation of the image processing apparatus, and the image processing method will be described below. Similar to the vehicle monitoring system according to the first embodiment, the monitor cameras 2 sense images of the vehicles 11 traveling on the road 10 and input video signals to a distributor 3. This distributor 3 properly inputs these video signals from the monitor cameras 2 to the image processing apparatus 5. The image processing apparatus 5 processes these video signals.
    As this image processing in the image processing apparatus 5, A/D conversion is performed as in the first embodiment. In subsequent image correction, correction for removing distortion of the image sensed through the wide angle lens is performed. As shown in Fig. 7, an A/D-converted digital image 30 has barrel-shaped form distortion because the image is sensed through the wide angle lens. By correcting this barrel form distortion of the digital image 30, as shown in Fig. 8, a corrected image 31 from which the distortion is removed is obtained.
    Subsequently, an image compositing process composites the corrected images 31 pertaining to the video signals obtained at the same time by the monitor cameras 2 into a single composite image 32. As in the first embodiment, this image compositing process is performed such that the roads 10 shown in the individual corrected images 21 continue. Also, the composition is so performed that more than one image of the same vehicle 11 will not be displayed. Fig. 9 shows a composite image 32 subjected to the compositing process.
    After the image compositing process is completed, the composite image 32 is transmitted to a control center 6. The whole of this composite image 32 is displayed on a monitor 61 in the control center 6.
    As described above, the image processing apparatus, image processing method, and vehicle monitoring system according to this embodiment can display, on a single screen, the whole of the road 10 as an object of traffic monitoring, similar to the image processing apparatus, image processing method, and vehicle monitoring system according to the first embodiment. Accordingly, the entire road can be monitored without switching images to be displayed. Since there is no area which cannot be monitored owing to image switching, therefore, it is possible to prevent a delay of traffic abnormality detection and increase the speed of traffic abnormality detection. Also, since no image switching is performed, traffic abnormalities can be detected on the basis of changes in an image. This facilitates visual traffic abnormality detection and reduces the monitoring load.
    Additionally, since an image of a road to be monitored is sensed sideways by using a wide angle lens, image correction can be readily performed.
    Furthermore, a wide angle lens is used in the image sensing optical system of the monitor camera 2. Since this allows each monitor camera 2 to monitor a wide range, the number of the monitor cameras 2 to be installed can be reduced.
    (Third Embodiment)
    An image processing apparatus, image processing method, and vehicle monitoring system according to the third embodiment will be described below.
    The image processing apparatus, image processing method, and vehicle monitoring system according to this embodiment have the same configurations as the image processing apparatuses, image processing methods, and vehicle monitoring systems according to the first and second embodiments, except that image storage is performed after image composition in an image processing means and in an image processing step.
    Fig. 10 is a flow chart of image processing performed in the image processing apparatus, image processing method, and vehicle monitoring system according to this embodiment.
    A/D conversion in S100, image correction in S200, and image composition in S300 of Fig. 10 are the same as explained in the first embodiment, so a detailed description thereof will be omitted. In S400, image storage is performed. This image storage is done by storing a composite image 22 or 32 subjected to image composition into a storage device such as a memory. Since the composite image 22 or 32 is thus stored, after a traffic abnormality occurs it is possible to analyze the abnormality by checking this composite image.
    In the present invention as has been described above, the whole of a road as an object of traffic monitoring can be displayed on a single screen. Accordingly, the entire road can be monitored without switching images to be displayed. Since there is no area which cannot be monitored owing to image switching, therefore, it is possible to prevent a delay of traffic abnormality detection and increase the speed of traffic abnormality detection.
    Also, since no image switching is performed, traffic abnormalities can be detected on the basis of changes in an image. This facilitates visual traffic abnormality detection and reduces the monitoring load.
    Additionally, since an image of a road to be monitored is sensed sideways by using a wide angle lens, image correction can be readily performed.
    Furthermore, since corrected composited image data is stored, after a traffic abnormality occurs it is possible to analyze the abnormality by checking this composite image.
    Industrial Applicability
    The image processing apparatus, image processing method, and vehicle monitoring system of the present invention are useful in monitoring traffic of vehicles traveling on a road.

    Claims (8)

    1. An image processing apparatus comprising:
      converting means for receiving a plurality of video signals obtained by sensing images of a road, and A/D-converting the video signals into digital images;
      correcting means for correcting the digital images such that the road runs in the longitudinal or lateral direction of each digital image; and
      image compositing means for compositing a plurality of corrected images obtained by said correcting means, such that the roads shown in the corrected images continue.
    2. An apparatus according to claim 1, characterized by further comprising storage means for storing the composite image as image data.
    3. An image processing method comprising:
      the conversion step of receiving a plurality of video signals obtained by sensing images of a road, and A/D-converting the video signals into digital images;
      the correction step of correcting the digital images such that the road runs in the longitudinal or lateral direction of each digital image; and
      the image composition step of compositing a plurality of corrected images obtained in the correction step, such that the roads shown in the corrected images continue.
    4. A method according to claim 3, characterized by further comprising the storage step of storing the composite image as image data.
    5. A vehicle monitoring system comprising:
      a plurality of image sensing means arranged along a road to be monitored to sense images of vehicles traveling on the road;
      image processing means for receiving video signals obtained by said image sensing means, A/D-converting the video signals into digital images, correcting the digital images such that the road runs in the longitudinal or lateral direction of each digital image, and compositing the corrected digital images such that the roads shown in the digital images continue; and
      display means for displaying the image processed by said image processing means.
    6. A vehicle monitoring system comprising:
      a plurality of image sensing means arranged along a road to be monitored to sense images of vehicles traveling on the road, each of said image sensing means having a wide angle lens in an image sensing optical system thereof;
      image processing means for receiving video signals obtained by said image sensing means, AID-converting the video signals into digital images, correcting distortion, caused by said wide angle lens, of each digital image, and compositing the corrected digital images such that the roads shown in the digital images continue; and
      display means for displaying the image processed by said image processing means.
    7. A system according to claim 5 or 6, characterized in that said display means is a projector having a display area long from side to side.
    8. A system according to any one of claims 5 to 7, characterized in that said image processing means stores the composite image as image data.
    EP00987649A 1999-12-27 2000-12-20 Image processing apparatus, image processing method, and vehicle monitoring system Expired - Lifetime EP1262933B1 (en)

    Applications Claiming Priority (3)

    Application Number Priority Date Filing Date Title
    JP37029899 1999-12-27
    JP37029899A JP3601392B2 (en) 1999-12-27 1999-12-27 Image processing apparatus, image processing method, and vehicle monitoring system
    PCT/JP2000/009025 WO2001048724A1 (en) 1999-12-27 2000-12-20 Image processing apparatus, image processing method, and vehicle monitoring system

    Publications (3)

    Publication Number Publication Date
    EP1262933A1 true EP1262933A1 (en) 2002-12-04
    EP1262933A4 EP1262933A4 (en) 2004-03-31
    EP1262933B1 EP1262933B1 (en) 2005-04-27

    Family

    ID=18496557

    Family Applications (1)

    Application Number Title Priority Date Filing Date
    EP00987649A Expired - Lifetime EP1262933B1 (en) 1999-12-27 2000-12-20 Image processing apparatus, image processing method, and vehicle monitoring system

    Country Status (7)

    Country Link
    EP (1) EP1262933B1 (en)
    JP (1) JP3601392B2 (en)
    AT (1) ATE294434T1 (en)
    DE (1) DE60019813T2 (en)
    ES (1) ES2237485T3 (en)
    HK (1) HK1047814B (en)
    WO (1) WO2001048724A1 (en)

    Cited By (3)

    * Cited by examiner, † Cited by third party
    Publication number Priority date Publication date Assignee Title
    EP2733677A1 (en) * 2012-11-19 2014-05-21 Kapsch TrafficCom AB Device for tolling or telematics systems
    EP2577958A4 (en) * 2010-06-03 2014-09-17 Microsoft Corp SIMULATED VIDEO WITH ADDITIONAL VIEWPOINTS AND ENHANCED RESOLUTION FOR TRAFFIC CAMERAS
    EP3561791A1 (en) * 2018-04-26 2019-10-30 Transdev Group System for monitoring road traffic with display of a virtual image of objects moving around a portion of road infrastructure

    Families Citing this family (5)

    * Cited by examiner, † Cited by third party
    Publication number Priority date Publication date Assignee Title
    DE10239026B4 (en) * 2002-08-21 2008-04-03 Deutsches Zentrum für Luft- und Raumfahrt e.V. Method and device for traffic monitoring
    CN101976341B (en) * 2010-08-27 2013-08-07 中国科学院自动化研究所 Method for detecting position, posture, and three-dimensional profile of vehicle from traffic images
    EP3923259B1 (en) 2020-06-12 2024-10-23 Ningbo Geely Automobile Research & Development Co. Ltd. A vehicle control system and method for providing information to support an overtaking maneuver
    CN111935451A (en) * 2020-07-16 2020-11-13 中国铁道科学研究院集团有限公司电子计算技术研究所 Railway safety monitoring device
    JP2023132159A (en) * 2022-03-10 2023-09-22 株式会社フジタ Image processing method and image processing system

    Family Cites Families (8)

    * Cited by examiner, † Cited by third party
    Publication number Priority date Publication date Assignee Title
    GB8913946D0 (en) * 1989-06-16 1989-08-02 Univ London Method and apparatus for traffic monitoring
    US5682198A (en) * 1993-06-28 1997-10-28 Canon Kabushiki Kaisha Double eye image pickup apparatus
    FR2714502A1 (en) * 1993-12-29 1995-06-30 Philips Laboratoire Electroniq An image processing method and apparatus for constructing from a source image a target image with perspective change.
    JP3569992B2 (en) * 1995-02-17 2004-09-29 株式会社日立製作所 Mobile object detection / extraction device, mobile object detection / extraction method, and mobile object monitoring system
    JP3577895B2 (en) * 1997-06-19 2004-10-20 松下電器産業株式会社 Wide area monitoring device and wide area monitoring system
    JPH1139589A (en) * 1997-07-18 1999-02-12 Fuji Electric Co Ltd Traffic monitoring device and traffic monitoring method
    JPH1196494A (en) * 1997-09-22 1999-04-09 Hitachi Ltd Traffic flow monitoring method and device
    JP2000331279A (en) * 1999-05-25 2000-11-30 Oki Electric Ind Co Ltd Wide area monitoring device

    Cited By (10)

    * Cited by examiner, † Cited by third party
    Publication number Priority date Publication date Assignee Title
    EP2577958A4 (en) * 2010-06-03 2014-09-17 Microsoft Corp SIMULATED VIDEO WITH ADDITIONAL VIEWPOINTS AND ENHANCED RESOLUTION FOR TRAFFIC CAMERAS
    EP2733677A1 (en) * 2012-11-19 2014-05-21 Kapsch TrafficCom AB Device for tolling or telematics systems
    WO2014076300A1 (en) * 2012-11-19 2014-05-22 Kapsch Trafficcom Ab Device for tolling or telematics systems
    US20160247398A1 (en) * 2012-11-19 2016-08-25 Kapsch Trafficcom Ab Device for tolling or telematics systems
    EP3182380A1 (en) * 2012-11-19 2017-06-21 Kapsch TrafficCom AB Device for tolling or telematics systems
    AU2017279793B2 (en) * 2012-11-19 2019-08-22 Kapsch Trafficcom Ag Device for tolling or telematics systems
    US10777075B2 (en) * 2012-11-19 2020-09-15 Kapsch Trafficcom Ab Device for tolling or telematics systems
    EP3561791A1 (en) * 2018-04-26 2019-10-30 Transdev Group System for monitoring road traffic with display of a virtual image of objects moving around a portion of road infrastructure
    FR3080701A1 (en) * 2018-04-26 2019-11-01 Transdev Group ROAD TRAFFIC MONITORING SYSTEM WITH DISPLAY OF A VIRTUAL IMAGE OF EVOLVING MOBILE OBJECTS IN A ROAD INFRASTRUCTURE PORTION
    US10885778B2 (en) 2018-04-26 2021-01-05 Transdev Group Traffic monitoring system with display of a virtual image of moving objects in a portion of roadway infrastructure

    Also Published As

    Publication number Publication date
    WO2001048724A1 (en) 2001-07-05
    ATE294434T1 (en) 2005-05-15
    EP1262933B1 (en) 2005-04-27
    DE60019813T2 (en) 2005-10-06
    HK1047814B (en) 2005-06-24
    DE60019813D1 (en) 2005-06-02
    JP3601392B2 (en) 2004-12-15
    JP2001184586A (en) 2001-07-06
    HK1047814A1 (en) 2003-03-07
    EP1262933A4 (en) 2004-03-31
    ES2237485T3 (en) 2005-08-01

    Similar Documents

    Publication Publication Date Title
    US7082209B2 (en) Object detecting method and object detecting apparatus and intruding object monitoring apparatus employing the object detecting method
    KR100939761B1 (en) Camera unit for a vehicle, method for display outside a vehicle, and system for generating driving corridor markers
    EP2410740B1 (en) Surrounding-area monitoring device for vehicles and surrounding-area image display method for vehicles
    JP2000090277A (en) Reference background image updating method, intruding object detecting method, and intruding object detecting device
    US20120033080A1 (en) Monitoring apparatus
    EP1262933A1 (en) Image processing apparatus, image processing method, and vehicle monitoring system
    JP3467774B2 (en) Image processing apparatus, image processing method, and vehicle monitoring system
    US20060008118A1 (en) Image processing apparatus and image monitoring system
    JP3788400B2 (en) Image processing apparatus, image processing method, and vehicle monitoring system
    JP2002158999A (en) Object detection method, object detection device, and intruding object monitoring device
    JP3354626B2 (en) Moving object detection method
    JPH10151991A (en) Vehicle rear monitoring device
    JP3945333B2 (en) Imaging display system
    JPH10322684A (en) Camera device
    JPH09116892A (en) Video surveillance equipment
    CN116828168A (en) Video signal processing apparatus
    JP2008059513A (en) Apparatus and method for monitoring bottom surface of vehicle
    JPH11339177A (en) Road monitoring device
    JP2001216598A (en) Traffic flow measurement device
    JP7708176B2 (en) Image processing device and image processing method
    EP4258653A1 (en) Image processing device that can measure distance to object, movable apparatus, image processing method, and storage medium
    JP5606774B2 (en) Video display device and video display method
    JP2006197614A (en) Image processing apparatus, image processing method, and vehicle monitoring system
    JP2001111899A (en) Image pickup device and its deteriorated pixel signal correction method
    JP7129270B2 (en) IMAGE PROCESSING APPARATUS, IMAGE PROCESSING METHOD AND IMAGE PROCESSING SYSTEM

    Legal Events

    Date Code Title Description
    PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

    Free format text: ORIGINAL CODE: 0009012

    17P Request for examination filed

    Effective date: 20020722

    AK Designated contracting states

    Kind code of ref document: A1

    Designated state(s): AT BE CH CY DE DK ES FI FR GB GR IE IT LI LU MC NL PT SE TR

    A4 Supplementary search report drawn up and despatched

    Effective date: 20040212

    RIC1 Information provided on ipc code assigned before grant

    Ipc: 7G 08G 1/01 B

    Ipc: 7G 08G 1/04 A

    17Q First examination report despatched

    Effective date: 20040301

    GRAP Despatch of communication of intention to grant a patent

    Free format text: ORIGINAL CODE: EPIDOSNIGR1

    GRAS Grant fee paid

    Free format text: ORIGINAL CODE: EPIDOSNIGR3

    GRAA (expected) grant

    Free format text: ORIGINAL CODE: 0009210

    AK Designated contracting states

    Kind code of ref document: B1

    Designated state(s): AT BE CH CY DE DK ES FI FR GB GR IE IT LI LU MC NL PT SE TR

    PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

    Ref country code: BE

    Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

    Effective date: 20050427

    Ref country code: TR

    Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

    Effective date: 20050427

    Ref country code: FI

    Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

    Effective date: 20050427

    REG Reference to a national code

    Ref country code: GB

    Ref legal event code: FG4D

    REG Reference to a national code

    Ref country code: CH

    Ref legal event code: EP

    REG Reference to a national code

    Ref country code: IE

    Ref legal event code: FG4D

    REF Corresponds to:

    Ref document number: 60019813

    Country of ref document: DE

    Date of ref document: 20050602

    Kind code of ref document: P

    REG Reference to a national code

    Ref country code: HK

    Ref legal event code: GR

    Ref document number: 1047814

    Country of ref document: HK

    REG Reference to a national code

    Ref country code: CH

    Ref legal event code: NV

    Representative=s name: BOVARD AG PATENTANWAELTE

    PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

    Ref country code: DK

    Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

    Effective date: 20050727

    Ref country code: GR

    Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

    Effective date: 20050727

    Ref country code: SE

    Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

    Effective date: 20050727

    REG Reference to a national code

    Ref country code: ES

    Ref legal event code: FG2A

    Ref document number: 2237485

    Country of ref document: ES

    Kind code of ref document: T3

    PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

    Ref country code: PT

    Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

    Effective date: 20051010

    PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

    Ref country code: CY

    Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

    Effective date: 20051220

    Ref country code: IE

    Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

    Effective date: 20051220

    PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

    Ref country code: MC

    Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

    Effective date: 20051231

    PLBE No opposition filed within time limit

    Free format text: ORIGINAL CODE: 0009261

    STAA Information on the status of an ep patent application or granted ep patent

    Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

    ET Fr: translation filed
    26N No opposition filed

    Effective date: 20060130

    REG Reference to a national code

    Ref country code: IE

    Ref legal event code: MM4A

    REG Reference to a national code

    Ref country code: GB

    Ref legal event code: 746

    Effective date: 20100330

    PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

    Ref country code: AT

    Payment date: 20101129

    Year of fee payment: 11

    PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

    Ref country code: LU

    Payment date: 20110118

    Year of fee payment: 11

    PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

    Ref country code: GB

    Payment date: 20101215

    Year of fee payment: 11

    Ref country code: IT

    Payment date: 20101224

    Year of fee payment: 11

    REG Reference to a national code

    Ref country code: CH

    Ref legal event code: PFA

    Owner name: SUMITOMO ELECTRIC INDUSTRIES, LTD.

    Free format text: SUMITOMO ELECTRIC INDUSTRIES, LTD.#5-33, KITAHAMA 4-CHOME CHUO-KU#OSAKA-SHI, OSAKA 541-0041 (JP) -TRANSFER TO- SUMITOMO ELECTRIC INDUSTRIES, LTD.#5-33, KITAHAMA 4-CHOME CHUO-KU#OSAKA-SHI, OSAKA 541-0041 (JP)

    PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

    Ref country code: CH

    Payment date: 20111213

    Year of fee payment: 12

    Ref country code: NL

    Payment date: 20111221

    Year of fee payment: 12

    Ref country code: ES

    Payment date: 20111228

    Year of fee payment: 12

    Ref country code: FR

    Payment date: 20111219

    Year of fee payment: 12

    REG Reference to a national code

    Ref country code: NL

    Ref legal event code: V1

    Effective date: 20130701

    REG Reference to a national code

    Ref country code: CH

    Ref legal event code: PL

    REG Reference to a national code

    Ref country code: AT

    Ref legal event code: MM01

    Ref document number: 294434

    Country of ref document: AT

    Kind code of ref document: T

    Effective date: 20121220

    GBPC Gb: european patent ceased through non-payment of renewal fee

    Effective date: 20121220

    REG Reference to a national code

    Ref country code: FR

    Ref legal event code: ST

    Effective date: 20130830

    PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

    Ref country code: NL

    Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

    Effective date: 20130701

    Ref country code: LI

    Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

    Effective date: 20121231

    Ref country code: AT

    Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

    Effective date: 20121220

    Ref country code: CH

    Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

    Effective date: 20121231

    PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

    Ref country code: FR

    Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

    Effective date: 20130102

    Ref country code: GB

    Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

    Effective date: 20121220

    PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

    Ref country code: IT

    Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

    Effective date: 20121220

    REG Reference to a national code

    Ref country code: ES

    Ref legal event code: FD2A

    Effective date: 20140306

    PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

    Ref country code: ES

    Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

    Effective date: 20121221

    Ref country code: LU

    Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

    Effective date: 20121220

    PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

    Ref country code: DE

    Payment date: 20151215

    Year of fee payment: 16

    REG Reference to a national code

    Ref country code: DE

    Ref legal event code: R119

    Ref document number: 60019813

    Country of ref document: DE

    PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

    Ref country code: DE

    Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

    Effective date: 20170701