JP2009217400A - Image processor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To stably extract an inter-frame difference regardless of an environment when an image is picked up. <P>SOLUTION: This image processor is provided with: an image management DB for storing a past image picked up in the past by a camera, and configured to: acquire environment information when an input image is picked up by the camera; determine, when the input image is input, an interval for executing an inter-frame difference based on the input image or the acquired environment information; acquire a past image input in the past corresponding to the determined difference interval from the image management DB; execute the inter-frame difference from the acquired past image and the inputted input image; and extract the difference. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image processing apparatus that extracts a difference by performing an inter-frame difference from an input image captured and input by an imaging unit that captures an image and a past image captured in the past.

  Conventionally, inter-frame differences are often used as a method for detecting a moving object from a moving image. The inter-frame difference is widely used because a moving area is extracted by taking a difference between the latest frame and the immediately preceding frame and can be realized by a lightweight process. In the inter-frame difference-based moving body tracking technology and the like, it is important to appropriately set the difference interval used in the inter-frame difference in order to increase the detection capability of detecting the moving body from the captured image. Various techniques for calculating appropriately are disclosed.

  For example, in Patent Document 1 (Japanese Patent Laid-Open No. 8-241414), speed information of a detection target is acquired from a predetermined area in an input image that is input, and the speed information is used for each area in the image. Set the interval to take the difference. To give a specific example, the image processing device (moving object detection device / tracking device) according to Patent Document 1 divides the input image into a plurality of regions, and acquires speed information from each of the divided regions. . Subsequently, the image processing apparatus determines an interval for performing the inter-frame difference from the acquired speed information for each region. Then, the image processing apparatus performs inter-frame differences in each of the plurality of regions using the intervals determined for each of the plurality of divided regions.

JP-A-8-241414

  However, the above-described conventional technique has a problem that the inter-frame difference interval cannot be determined accurately depending on the environment where the image is taken, and the extracted inter-frame difference is not accurate.

  Specifically, for example, the image changes greatly depending on the environment at the time of acquisition, for example, the position on the image and the amount of movement look different during the day and at night. In addition, for example, accurate speed measurement of a large number of moving objects under various environments is generally very difficult, and applicable scenes and effects are limited.

  Accordingly, the present invention has been made to solve the above-described problems of the prior art, and provides an image processing apparatus capable of stably extracting inter-frame differences regardless of the environment at the time of image capturing. The purpose is to do.

  In order to solve the above-described problems and achieve the object, the present apparatus performs an inter-frame difference from an input image captured and input by an imaging unit that captures an image and a past image captured in the past. An image processing apparatus for extracting a difference, an image storage unit for storing past images captured in the past by the imaging unit, and environment information for acquiring environment information when an input image is captured by the imaging unit A difference interval determination unit that determines an interval for performing the interframe difference based on the input image or the environment information acquired by the environment information acquisition unit when the input image is input; A past image input in the past corresponding to the difference interval determined by the difference interval determination unit is acquired from the image storage unit, and a frame is obtained from the acquired past image and the input image input. By carrying out the difference between the over-time, characterized by comprising a differential extraction means for extracting difference, the.

  According to this apparatus, it is possible to stably extract the interframe difference regardless of the environment.

  Embodiments of an image processing apparatus according to the present invention will be described below in detail with reference to the accompanying drawings. In the following, the main terms used in the present embodiment, the outline and features of the image processing apparatus according to the present embodiment, the configuration of the image processing apparatus and the flow of processing will be described in order, and finally various modifications to the present embodiment will be described. An example will be described.

[Explanation of terms]
First, main terms used in this embodiment will be described. An “image processing apparatus (corresponding to an“ image processing apparatus ”recited in the claims”) used in the present embodiment refers to an input image captured and input by a camera imaging unit that captures an image, This is a device that extracts a difference by performing an inter-frame difference from a past image captured in (1). In addition, the device that captures an image and inputs it to the “image processing device” is not limited to the above-described camera, and may be a video camera, a web camera, or the like, and further input from a storage medium such as a CD. May be.

  In addition, the “image processing apparatus” used in the present embodiment will be described for the case where it is installed on a traffic signal or a sign on a road on which an automobile or the like runs. However, the present invention is not limited to this, and It can also be installed in a moving body such as a vehicle, and can be fixedly installed such as traffic lights, signs, and runways. In this embodiment, the “image processing apparatus” is connected to a car navigation system or a car control system by wire or wirelessly. The car navigation system or the car control system receives information from the “image processing apparatus”. When (extracted difference) is received, navigation and brake control can be performed based on the information.

[Outline and Features of Image Processing Device]
Next, the outline and features of the image processing apparatus according to the first embodiment will be described. As described above, this image processing apparatus extracts a difference by performing an inter-frame difference from an input image captured and input by an imaging unit that captures an image and a past image captured in the past. In particular, the main feature is that the inter-frame difference can be stably extracted regardless of the environment.

  Specifically, the image processing apparatus according to the first embodiment acquires various information such as traffic volume, day / night information, traffic light status, weather, and speed information of a moving object (subject) as an environment in which an image is captured by a camera. To do. Then, when an input image captured by the camera is input, the image processing apparatus determines an interval for performing the inter-frame difference based on the input image or the acquired environment information. Thereafter, the image processing apparatus performs inter-frame difference using the determined interval, and extracts the difference (difference image).

  As described above, the image processing apparatus according to the first embodiment acquires not only the velocity information of the moving body but also various environment information to determine the interframe difference interval, and performs the interframe difference at the determined interval. As a result, as described above, it is possible to stably extract the inter-frame difference regardless of the environment at the time of image capturing as in the main feature.

[Configuration of image processing apparatus]
Next, the configuration of the image processing apparatus will be described with reference to FIG. FIG. 1 is a block diagram illustrating the configuration of the image processing apparatus according to the first embodiment. As shown in FIG. 1, the image processing apparatus 10 includes an input image receiving unit 11, an external sensor 12, an image management DB 20, a difference interval DB 21, an environment determination unit 30, and a difference processing unit 31. The

  The input image receiving unit 11 receives an input image that has been captured and input by an imaging unit that captures an image. To give a specific example, the input image receiving unit 11 receives an image captured by a shooting unit such as a camera, stores the image in the connected image management DB 20, and transmits the received input image to the environment determination unit 30. Output to.

  The external sensor 12 acquires environment information when an input image is captured by the imaging unit. For example, the external sensor 12 includes various sensors such as a timer, a temperature sensor, a visibility meter, a rain gauge, and a traffic meter such as an ultrasonic wave, and when the input image is captured by the imaging unit. Information such as time, date, temperature, weather, and traffic volume is acquired as environment information, and the acquired information is output to the connected environment determination unit 30. The external sensor 12 may be connected to an intersection traffic light or the like. In this case, the external sensor 12 acquires the lighting state of the traffic light and outputs it to the environment determination unit 30.

  For example, the external sensor 12 uses a sensor capable of measuring traffic flow to grasp the approximate number of vehicles on the road, or to detect the background difference of the input image received by the input image receiving unit 11. Process and predict the number of units. Accordingly, the external sensor 12 determines the traffic volume on the road and outputs the result to the environment determination unit 30 described later. Further, the external sensor 12 acquires a time when the input image is captured by the imaging unit from a time measuring device such as a timer, and environmental information such as “morning, noon, night” is described later based on the time. It can also be output to the environment determination unit 30.

  The image management DB 20 stores past images captured in the past by the imaging unit. For example, the image management DB 20 stores an input image (current processing target frame) received by the input image receiving unit 11 and subjected to inter-frame difference processing by a difference processing unit 31 described later. To do. Further, the image management DB 20 accumulates the input images processed by the difference processing unit 31, and outputs an image stored by an instruction operation of the difference processing unit 31.

  The difference interval DB 21 stores an interval for performing the inter-frame difference in association with the environment information acquired by the external sensor 12. Specifically, as shown in FIG. 2, the difference interval DB 21 includes “environment information indicating environment information acquired by the external sensor 12” and “interframe difference indicating an interval for performing the interframe difference. As “interval” ”,“ large traffic volume (congestion), 10 ”,“ low traffic volume, 1 ”, etc. are stored. In addition, information including various data and parameters can be arbitrarily changed unless otherwise specified. FIG. 2 is a diagram illustrating an example of information stored in the difference interval DB.

  When an input image is input, the environment determination unit 30 determines an interval for performing the inter-frame difference based on the input image or the environment information acquired by the external sensor 12. Specifically, in the above example, when the environmental information “traffic volume” is input from the external sensor 12, the environment determination unit 30 determines whether the traffic volume is “large (congested)” or “small” from the environmental information. (It is not a traffic jam). If the environment determination unit 30 determines that the traffic volume is “large (traffic jam)”, the environment determination unit 30 acquires “difference interval = 10” corresponding to “traffic volume (large)” from the difference interval DB 21 and acquires it. The difference interval “10” is output to the difference processor 31 described later.

  The environment determination unit 30 can also determine the environment information from the input image received from the input image reception unit 11 as well as the environment information received from the external sensor 12. For example, the environment determination unit 30 calculates the average luminance of the input image received from the input image reception unit 11, and determines “day” if the calculated value is equal to or greater than the threshold, “night” if the calculated value is less than the threshold. You can also.

  The difference processing unit 31 acquires a past image input in the past corresponding to the difference interval determined by the environment determination unit 30 from the image management DB 20, and calculates an inter-frame difference from the acquired past image and the input image that has been input. Implement and extract the differences. Specifically, in the above example, when the difference processing unit 31 receives the difference interval “1” determined by the environment determination unit 30, the current frame (time t image) received by the input image reception unit 11. The previous frame (time t-1 image) is acquired from the image management DB 20, and an inter-frame difference is performed from the acquired frame and the input image that has been input to extract the difference (difference image). Then, the difference processing unit 31 outputs the extracted difference (difference image) to a car navigation system, a vehicle control system, or the like connected to the image processing apparatus 10.

  For example, when the difference processing unit 31 receives the difference interval “10” determined by the environment determination unit 30, the difference processing unit 31 is ten frames before the current frame (time t image) received by the input image receiving unit 11. A frame (time t-10 image) is acquired from the image management DB 20, and an inter-frame difference is performed from the acquired frame and the input image that has been input, and the difference (difference image) is extracted. Then, the difference processing unit 31 outputs the extracted difference (difference image) to a car navigation system, a vehicle control system, or the like connected to the image processing apparatus 10.

[Processing by image processing device]
Next, difference extraction processing by the image processing apparatus will be described with reference to FIG. FIG. 3 is a flowchart illustrating the flow of the difference extraction process in the image processing apparatus according to the first embodiment.

(Difference extraction process flow)
As shown in FIG. 3, when an input image is input (Yes at Step S101), the image processing apparatus 10 performs past image management processing in order to store the input image in the image management DB 20 (Step S102). . The past image management process will be described in detail with reference to FIG.

  Then, the image processing apparatus 10 acquires environment information from the external sensor 12 (step S103) and performs environment determination (step S104). To give a specific example, the image processing apparatus 10 acquires environmental information “traffic volume” from the external sensor 12, and determines whether the traffic volume is “large” or “small”. Here, the image processing apparatus 10 determines that the traffic volume is “large”.

  Subsequently, the image processing apparatus 10 acquires a difference interval from the difference interval DB 21 based on the determined result (Step S105), and performs an inter-frame difference based on the acquired difference interval to extract the difference (Step S105). (S106), the extracted result is output to various devices connected (step S107). As described above, the image processing apparatus 10 performs the processes of steps S101 to S107 described above on the input image that has been input.

  As a specific example, the image processing apparatus 10 acquires the difference interval “10” from the difference interval DB 21 based on the determined result “high traffic volume”. Then, the image processing apparatus acquires from the image management DB 20 a frame (time t-10 image) that is ten frames before the current frame (time t image) received by the input image receiving unit 11, and acquires the acquired frame and input. The difference between the frames is performed from the input image thus obtained, and the difference (difference image) is extracted. Then, the image processing apparatus 10 outputs the extracted difference (difference image) to a connected car navigation system, vehicle control system, or the like.

(Past image management process flow)
Next, the past image management processing by the image processing apparatus will be described with reference to FIG. FIG. 4 is a flowchart illustrating the flow of past image management processing in the image processing apparatus according to the first embodiment. The past image management process described here is a process corresponding to step S102 in FIG.

  As shown in FIG. 4, when an input image is input, the image processing apparatus 10 acquires the list capacity of the image management DB 20 (step S201), and determines whether there is a free capacity (step S202).

  If there is free space in the image management DB 20 (Yes in step S202), the image processing apparatus registers the input image received by the input image receiving unit 11 in the image management DB 20 (step S203).

  On the other hand, when there is no free space in the image management DB 20 (No in step S202), the image processing apparatus deletes the most past image stored in the image management DB 20 to secure free space (step S204), and then the input image. The input image received by the receiving unit 11 is registered in the image management DB 20 (step S203).

[Effects of Example 1]
As described above, according to the first embodiment, the past image captured by the camera is stored, the environment information when the input image is captured by the camera is acquired, and when the input image is input, Based on the input image or the acquired environment information, the interval for performing the inter-frame difference is determined, the past image input in the past corresponding to the determined difference interval is acquired, and the acquired past image is input Since the difference between frames is extracted from the input image and the difference is extracted, not only the velocity information of the moving body but also various environment information is acquired to determine the difference interval between frames, and the difference between frames is determined at the determined interval. As a result, it is possible to stably extract the inter-frame difference regardless of the environment at the time of image capturing as in the main feature described above.

  In addition, according to the first embodiment, since the interval at which the difference is taken is determined according to the environment information including the detection target, it is possible to detect the image difference with high accuracy in any situation. Further, since only the frame interval for obtaining the difference is changed according to the environment information, the processing amount is lighter compared with the method of processing the image area separately.

  Further, according to the first embodiment, the traffic volume is acquired as environment information when the input image is captured by the camera, and the difference between frames when the acquired traffic volume is large (when traffic is congested) is set. Since it is determined as a longer interval than when the traffic volume is small, it can be easily applied to a vehicle such as an automobile such as a car navigation system.

  By the way, in Example 1, although the case where the difference space | interval was mainly determined using traffic as environmental information was demonstrated, this invention is not limited to this, A difference space | interval using other various information Can be determined.

  Accordingly, in the second embodiment, a case where the difference interval is determined using information other than the traffic volume will be described with reference to FIGS. In the second embodiment, three examples of an imaging time zone, a signal lighting state during imaging, an imaging time and a signal lighting state will be described.

(Determine difference interval using imaging time zone)
As shown in FIG. 5, the image processing apparatus 10 according to the second embodiment has “a time zone indicating environmental information acquired by the external sensor 12” and “interframe difference interval” indicating an interval for performing an interframe difference. As “daytime, 1”, “nighttime, 2” and the like in the difference interval DB 21. In addition, information including various data and parameters can be arbitrarily changed unless otherwise specified. FIG. 5 is a diagram illustrating an example of the imaging time period stored in the difference interval DB according to the second embodiment.

  In such a state, the image processing apparatus acquires environment information from the external sensor 12, performs environment determination, and determines a frame difference interval. Specifically, in the above example, the image processing apparatus 10 acquires the time when the input image is captured by the camera by the external sensor 12 using a time measuring device such as a timer. Then, the image processing apparatus 10 determines “daytime” if the acquired time is between a predetermined threshold value (for example, 9:00 to 16:00), and associates the determined “daytime” with the difference interval DB21. The inter-frame difference interval “1” stored in is acquired.

  The image processing apparatus 10 then, based on the acquired inter-frame difference interval “1”, the previous frame (time t−1 image) of the current frame (time t image) received by the input image receiving unit 11. ) Is acquired from the image management DB 20, and the difference between frames is extracted from the acquired frame and the input image that has been input, and the difference (difference image) is extracted. Then, the image processing apparatus 10 outputs the extracted difference (difference image) to a car navigation system or a vehicle control system connected to the image processing apparatus 10.

  Further, for example, the image processing apparatus 10 determines “night” if the acquired time is between a predetermined threshold (for example, 16:01 to 8:59), and the difference is associated with the determined “night”. The inter-frame difference interval “2” stored in the interval DB 21 is acquired.

  Then, the image processing apparatus 10, based on the acquired inter-frame difference interval “2”, the frame (time t−2 image) immediately before the current frame (time t image) received by the input image receiving unit 11. ) Is acquired from the image management DB 20, and the difference between frames is extracted from the acquired frame and the input image that has been input, and the difference (difference image) is extracted. Then, the image processing apparatus 10 outputs the extracted difference (difference image) to a car navigation system or a vehicle control system connected to the image processing apparatus 10.

  In this way, the image processing apparatus 10 acquires day / night information as environment information when an input image is captured by the camera, and the day / night information indicates the interval between frame differences when the acquired day / night information is night. Since it is determined as a longer interval than in the case of noon, that is, when it is difficult to discriminate the object on the image, the interval between frames is determined to be longer. Even when it is used, it is possible to stably extract the inter-frame difference.

(The difference interval is determined using the signal lighting state at the time of imaging.)
The image processing apparatus 10 according to the second embodiment, as shown in FIG. 6, “a signal indicating environmental information acquired by the external sensor 12 and an“ interframe difference interval ”indicating an interval for performing an interframe difference”. Are stored in the difference interval DB 21 as “blue, 1”, “yellow, 2”, “red, 10” and the like. In addition, information including various data and parameters can be arbitrarily changed unless otherwise specified. FIG. 6 is a diagram illustrating an example of a signal lighting state at the time of imaging stored in the difference interval DB according to the second embodiment.

  In such a state, the image processing apparatus acquires environment information from the external sensor 12 connected to the traffic light, performs environment determination, and determines a frame difference interval. Specifically, in the above example, the image processing apparatus 10 acquires the signal state when the input image is captured by the camera by the external sensor 12 that has received the signal state from the connected traffic signal. If the acquired signal state is “red”, the image processing apparatus 10 acquires the inter-frame difference interval “10” stored in the difference interval DB 21 in association with “red”.

  Then, the image processing apparatus 10, based on the acquired inter-frame difference interval “10”, the 10 frames before the current frame (time t image) received by the input image receiving unit 11 (time t-10 image). ) Is acquired from the image management DB 20, and the difference between frames is extracted from the acquired frame and the input image that has been input, and the difference (difference image) is extracted. Then, the image processing apparatus 10 outputs the extracted difference (difference image) to a car navigation system or a vehicle control system connected to the image processing apparatus 10.

  Further, for example, even when the signal state is “blue” or “yellow”, the image processing apparatus 10 can determine the inter-frame difference interval and extract the difference similarly to the above-described example. As described above, the image processing apparatus 10 acquires the traffic signal state as environment information when the input image is captured by the camera, and determines the interval of the inter-frame difference when the acquired traffic signal state is yellow or red. The interval is determined as a longer interval than when the state is blue, and the difference between frames when the acquired traffic light state is red is determined as a longer interval than when the traffic light state is yellow. In other words, when the moving speed of the object is slow on the image, it is determined to increase the difference between frames, so even if the traffic light state is used instead of the traffic volume, the frame can be stably displayed. It is possible to extract the difference between the two.

(Difference interval is determined using imaging time and signal lighting state)
The image processing apparatus 10 according to the second embodiment includes an “interframe difference interval” indicating an interval for performing an interframe difference in association with a “signal” indicating the environment information acquired by the external sensor 12 and a “time zone”. The difference interval DB 22 is stored. For example, as illustrated in FIG. 7, the image processing apparatus 10 stores “blue, daytime, 1”, “red, night, 15”, and the like in the difference interval DB 22. In addition, information including various data and parameters can be arbitrarily changed unless otherwise specified. FIG. 7 is a diagram illustrating an example of an imaging time and a signal lighting state stored in the difference interval DB according to the second embodiment.

  In such a state, the image processing apparatus includes a time counter such as a timer, acquires environment information from the external sensor 12 connected to the traffic light, performs environment determination, and determines a frame difference interval. Specifically, in the above example, the image processing apparatus 10 acquires the time when the input image is captured by the camera by the external sensor 12 using a time measuring device such as a timer. At the same time, the image processing apparatus 10 acquires the signal state when the input image is captured by the camera by the external sensor 12 that has received the signal state from the connected traffic signal.

  Then, the image processing apparatus 10 determines “night” if the signal state is “red” and the captured time zone is within a predetermined threshold (for example, 16:01 to 8:59), and “ The inter-frame difference interval “15” stored in the difference interval DB 22 in association with “red, night” is acquired.

  Then, the image processing apparatus 10 determines the 15 frames (time t-15 image) before the current frame (time t image) received by the input image receiving unit 11 based on the acquired difference interval “15”. An inter-frame difference is acquired from the acquired image management DB 20 and the acquired frame and the input image are input, and a difference (difference image) is extracted. Then, the image processing apparatus 10 outputs the extracted difference (difference image) to a car navigation system or a vehicle control system connected to the image processing apparatus 10.

  As described above, the image processing apparatus 10 can extract a more stable and more accurate inter-frame difference by using both the imaged time and the traffic light state.

  By the way, in the first and second embodiments, the example in which the inter-frame difference is performed on the entire past image and the entire input image corresponding to the determined difference interval and the difference is extracted has been described, but the present invention is limited to this. Instead, the input image may be divided into a plurality of regions, a difference interval may be determined for each divided region, and a difference between the frames may be extracted for each region.

  Therefore, in the third embodiment, an example in which an input image is divided into a plurality of regions, a difference interval is determined for each divided region, and a difference between frames is extracted for each region by using FIG. Will be described. FIG. 8 is a diagram for explaining processing of the image processing apparatus according to the third embodiment.

  The image processing apparatus according to the third embodiment includes a frame in association with a distance to an object captured in an input image and an environment state (for example, traffic light state, traffic volume, weather, time zone). A difference interval DB for storing the difference interval is provided. For example, the image processing apparatus includes a difference interval DB that stores “distance = 20 m, signal state = blue, traffic volume = small, interval = 3”, and the like.

  In such a state, as shown in FIG. 8, the image processing apparatus divides the input image into a plurality of regions ((1) to (n)), and is captured in each of the divided regions. A distance to an object (for example, a moving body such as an automobile or a signboard) is calculated. At the same time, the image processing apparatus acquires the same environment information as in the first and second embodiments. Here, as a method for calculating the distance to the object, for example, various known methods such as calculating from the pixel value of the object or calculating the distance from the size of the object can be used.

  Then, the image processing apparatus acquires the inter-frame difference interval from the difference interval DB based on the distance to the object and the environment information for each area of the divided input image. Then, the image processing apparatus acquires a past image for each area based on the difference interval acquired for each area of the divided input image, and generates a comparison image obtained by synthesizing them.

  For example, as shown in FIG. 8, the image processing apparatus has a difference interval “1” for the region (1), a difference interval “2” for the region (2), a difference interval “3” for the region (3), For the region (n), the difference interval “n” is acquired. Then, the image processing apparatus displays “time t-1 image” for area (1), “time t-2 image” for area (2), and “time t-3 image” for area (3). For (n), a “time t−n image” is acquired, and a comparison image obtained by combining these is generated. Here, the currently input image (frame) is described as “time t image”. That is, “time t−1 image” indicates an image immediately before the current input image.

  Thereafter, the image processing apparatus performs inter-frame difference processing between the generated comparison image and the currently input image (frame) between the “time t image” and extracts the difference.

  As described above, according to the third embodiment, the input image is divided into a plurality of areas by the camera, the environment information obtained when each of the areas is captured is acquired, and the acquired environment information for each of the plurality of areas is acquired. Based on the above, determine the interval for performing the inter-frame difference for each of the plurality of regions, obtain the past image input in the past corresponding to the determined difference interval for each of the plurality of regions, for each of the plurality of regions Since the difference is extracted by performing the inter-frame difference, the inter-frame difference interval can be determined more in consideration of the environment information than when determining the inter-frame difference interval in the entire image. Even in an environment, it is possible to obtain a difference with higher accuracy.

  In the third embodiment, an example is described in which an input image is divided into a plurality of regions, a difference interval is determined for each divided region, and an inter-frame difference is performed for each region to extract the difference. The invention is not limited to this, and a difference interval may be determined for each pixel of the input image, and the difference may be extracted by performing the inter-frame difference for each region.

  Therefore, in the fourth embodiment, a flow of processing when a difference interval is determined for each pixel of the input image and an inter-frame difference is performed for each region to extract the difference will be described with reference to FIG. FIG. 9 is a flowchart for explaining processing of the image processing apparatus according to the fourth embodiment.

  As illustrated in FIG. 9, when an input image is input, the image processing apparatus according to the fourth embodiment generates (reserves) an image buffer that is an area for generating a comparison image to be compared for inter-frame differences. (Step S301).

  Then, the image processing apparatus initializes the upper left pixel (X = 1, Y = 1) and the variables X and Y with the horizontal axis X coordinate = 0 and the vertical axis Y coordinate = 0 of the input image (step) S302). Note that the coordinates of the lower right pixel of the input image are “X = n, Y = n”.

  Thereafter, the image processing apparatus sets the distance of the target pixel to all the pixels from the upper left pixel “X = 1, Y = 1” of the input image to the lower right pixel “X = n, Y = n” of the input image. Calculate (step S303 and step S304), obtain the difference interval of the target pixel from the difference interval DB (step S305), obtain the past image corresponding to the obtained difference interval from the image management DB (step S306), and obtain A pixel value corresponding to the position of the processing target pixel (for example, “X = 1, Y = 1”) is acquired from the past image, and the pixel value is substituted into “X = 1, Y = 1” of the image buffer. (Step S307).

  The image processing apparatus then performs steps S304 to S304 described above for all the pixels from the upper left pixel “X = 1, Y = 1” of the input image to the lower right pixel “X = n, Y = n” of the input image. When the processing in step S307 is performed (Yes in step S308), the difference between the input image and the comparison image is extracted by using the generated image buffer as the comparison image, and the difference is extracted.

  As described above, according to the fourth embodiment, the environment information at the time of imaging is obtained for each pixel of the input image by the camera, and the frame is obtained for each pixel based on the obtained environment information for each pixel. Since the past image input in the past corresponding to the determined difference interval for each pixel is determined and the inter-frame difference is performed for each pixel, and the difference is extracted. Compared to the case where the interframe difference interval is determined for the entire image or the case where the interframe difference is determined for each divided area of the image, the interframe difference interval can be determined in consideration of environmental information. Even in a difficult environment, it is possible to obtain a difference with higher accuracy.

  By the way, although Example 1-4 demonstrated the example which acquires the difference space | interval memorize | stored beforehand based on the acquired environmental information, this invention is not limited to this, Based on the acquired environmental information The difference interval can also be calculated.

  Accordingly, in the fifth embodiment, various examples of calculating the difference interval based on the acquired environment information will be described with reference to FIGS.

(Example of calculating the difference interval from traffic volume)
First, an example of calculating the difference interval from the traffic volume will be described with reference to FIG. As shown in (1) of FIG. 10, the image processing apparatus uses “the length of the vehicle = 5 m, the distance between the vehicles = 2.5 m, and an observation range indicating a range for identifying the vehicle from the input image as parameters of the difference interval calculation formula. = 0 to 100 m "is preset. In addition, the above-mentioned numerical value is an example to the last, and is not limited to this. FIG. 10 is a diagram illustrating an example of a calculation formula for calculating the difference interval from the traffic volume.

  In such a state, for example, when five vehicles are captured in the input image, the image processing apparatus uses the calculation formula shown in (2) of FIG. 10 to calculate “frame interval = 100 / (5 / 13) = 4... ”And“ frame difference interval = 4 ”is determined.

  Further, for example, when 12 vehicles are captured in the input image, the image processing apparatus uses the calculation formula shown in (2) of FIG. 10 to calculate “frame interval = 100 / (12/13) = 9. ... ”And“ frame difference interval = 9 ”is determined. The subsequent process of acquiring the past image and performing the inter-frame difference is the same as that in the first embodiment, and is therefore omitted here.

(Example of calculating the difference interval from the time zone)
Next, an example of calculating the difference interval from the time zone will be described with reference to FIG. As shown in FIG. 11, in the image processing apparatus, a difference interval calculation formula is set in advance according to the time zone and the month when the image is taken. FIG. 11 is a diagram illustrating an example of a calculation formula for calculating the difference interval from the time zone.

  The calculation formula shown in FIG. 11 is “frame difference” when the imaged time zone is “7:00 to 17:00” and the imaged month is “November, December, January”. If the interval is determined to be “1”, the captured time zone is “6: 0 to 18:00”, and the captured month is “April or September”, “frame difference interval = 1”. If it is determined that the imaged time zone is between “5: 00-19: 00” and the imaged month is “May, June, July”, “frame difference interval = 1” is determined. The other combinations of time zone and month are determined as “frame difference interval = 2”.

  When receiving the environment information, the image processing apparatus determines an inter-frame difference depending on which pattern the environment information is included in. The subsequent process of acquiring the past image and performing the inter-frame difference is the same as that in the first embodiment, and is therefore omitted here. In addition, the pattern of the above-mentioned time slot | zone and the month is an example to the last, and is not limited to this.

(Example of calculating the difference interval from the weather)
Next, an example of calculating the difference interval from the weather will be described with reference to FIG. As shown in FIG. 12, in the image processing apparatus, a difference interval calculation formula is set in advance according to the captured weather. In addition, the numerical value shown in FIG. 12 is an example to the last, and is not limited to this. FIG. 12 is a diagram illustrating an example of a calculation formula for calculating the difference interval from the weather.

  Specifically, as shown in FIG. 12, the image processing apparatus includes “average luminance = total pixel value in frame (input image) / total number of pixels in frame (input image)” and “C1: threshold value. (Day / Night Brightness Determination), C2: Threshold (Daytime Cloudiness, Sunny Brightness Determination) "are preset as parameters.

  In such a state, for example, when “C1 = 30, C2 = 100” and “average brightness of the calculated input image = 120”, the image processing apparatus performs C2 based on (2) of FIG. Therefore, “weather = daytime, sunny” and “interframe difference interval = 5” are determined. Further, for example, when “the calculated average brightness of the input image = 20”, the image processing apparatus is smaller than C1 based on (2) of FIG. 12, and therefore “weather = night”, “inter-frame” When “difference interval = 1” is determined, and “average brightness of calculated input image = 70”, since it is larger than C1 and smaller than C2 based on (2) of FIG. 12, “weather = daytime, “Cloudy” and “interframe difference interval = 2”. The subsequent process of acquiring the past image and performing the inter-frame difference is the same as that in the first embodiment, and is therefore omitted here.

(Effects of Example 5)
As described above, according to the fifth embodiment, the calculation interval based on the input image or the environment information is used to determine the interval for performing the inter-frame difference. Specifically, the moving speed of the object is slow on the image. Alternatively, when it is difficult to identify the object, it is determined to increase the interval between frames, so that it is possible to flexibly respond to changes in the environment during image capture, so that the difference between frames can be extracted stably. Is possible.

  Although the embodiments of the present invention have been described so far, the present invention may be implemented in various different forms other than the embodiments described above. Therefore, as shown below, (1) application examples, (2) environmental information, (3) mobile object detection, (4) environmental information acquisition opportunity, (5) system configuration, etc. Different embodiments will be described.

(1) Application Examples For example, in the first to fifth embodiments, the present invention has been described with respect to a case where the present invention is installed on a traffic signal or a sign on a road on which an automobile or the like runs. However, the present invention is not limited to this, It can also be installed in a moving body such as a vehicle, and can be fixedly installed such as traffic lights, signs, and runways.

(2) Environmental information Moreover, in Examples 1-5, although the case where "traffic volume, traffic light state, weather, time" etc. were used as environmental information was described, this invention is not limited to this, By using an external sensor equipped with a visibility meter, a rain gauge, etc., it is also possible to use detailed information on weather such as fog and rain.

  In addition, the image processing apparatus acquires and acquires the distance to the object included in the input image from the camera as the environmental information when the input image is captured by the camera without using the environmental information that is external information. It is also possible to determine a shorter interval for performing the inter-frame difference as the distance is shorter or as the object is below the input image. As a result, the interval at which the difference is taken is determined according to the environment information including the detection target, so that it is possible to detect the image difference with high accuracy in any situation.

(3) Moving object detection Moreover, although Example 1-5 demonstrated the case where a difference (difference image) was obtained by implementing the inter-frame difference process based on the determined inter-frame difference interval, the present invention However, the present invention is not limited to this, and the moving object can be detected from the calculated difference. As a result, application to a vehicle-mounted control system mounted on a moving body such as a vehicle-mounted device is very effective. For example, it is possible to control the brake when the moving body is present at a position close to the extracted difference image (largely captured in the difference image). Note that if the inter-frame difference interval is too long, it may be difficult to detect a moving object in the image. Therefore, not only the method for obtaining the difference interval described in the above embodiment, but also depending on the actual detection target. The upper limit value and the lower limit value of the difference interval may be set, or the difference interval determination method may be changed.

(4) Environment Information Acquisition Trigger In the first to fifth embodiments, the case where the inter-frame difference interval is determined based on the acquired environment information every time an image is input has been described. The present invention is not limited, and the inter-frame difference interval can be determined based on the acquired environment information only when the acquired environment information has changed. As a result, it is possible to quickly respond to changes in the environment. In addition, the processing load can be reduced as compared with the case where the inter-frame difference interval is determined each time an image is input.

(5) System Configuration The components of the illustrated devices are functionally conceptual and do not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution / integration of each device is not limited to that shown in the figure, and all or a part thereof may be functionally or physically distributed or arbitrarily distributed in arbitrary units according to various loads or usage conditions. It can be configured by integrating (for example, integrating an external sensor and an environment determination unit). Further, all or any part of each processing function performed in each device may be realized by a CPU and a program analyzed and executed by the CPU, or may be realized as hardware by wired logic.

  In addition, among the processes described in the present embodiment, all or a part of the processes described as being automatically performed (for example, the imaging process) may be manually performed. In addition, the processing procedures, control procedures, specific names, and information including various data and parameters (for example, FIG. 2, FIG. 5 to FIG. 7, FIG. 10 to FIG. 12) It can be changed arbitrarily unless otherwise specified.

(6) Program By the way, the various processes described in the above embodiments can be realized by executing a program prepared in advance on a computer system such as a personal computer or a workstation. Therefore, hereinafter, a computer system that executes a program having the same function as that of the above embodiment will be described as another embodiment.

  FIG. 13 is a diagram illustrating an example of a computer system that executes an image processing program. As illustrated in FIG. 13, the computer system 100 includes a RAM 101, an HDD 102, a ROM 103, and a CPU 104. Here, the ROM 103 stores a program that exhibits the same function as the above-described embodiment, that is, as shown in FIG. 13, an input image reception program 103a, an environment information acquisition program 103b, and an environment determination program 103c. The processing program 103d is stored in advance.

  Then, the CPU 104 reads and executes these programs 103a to 103d, and as shown in FIG. 13, an input image reception process 104a, an environment information acquisition process 104b, an environment determination process 104c, and a difference processing process. 104d. The force image reception process 104a corresponds to the input image reception unit 11 shown in FIG. 1, and similarly, the environment information acquisition process 104b corresponds to the external sensor 12, and the environment determination process 104c corresponds to the environment determination unit. 30, the difference processing process 104 d corresponds to the difference processing unit 31.

  The HDD 102 also includes an image management table 102a that stores past images captured in the past by the imaging unit, and a difference interval table 102b that stores intervals for performing inter-frame differences in association with acquired environment information. Is provided. The image management table 102a corresponds to the image management DB 20 shown in FIG. 1, and the difference interval table 102b corresponds to the difference interval DB 21.

  By the way, the above-described programs 103a to 103d are not necessarily stored in the ROM 103. For example, a flexible disk (FD), a CD-ROM, an MO disk, a DVD disk, a magneto-optical disk inserted into the computer system 100, In addition to “portable physical media” such as IC cards, “fixed physical media” such as hard disk drives (HDDs) provided inside and outside the computer system 100, public lines, the Internet, LAN, WAN, etc. The program may be stored in “another computer system” connected to the computer system 100 via the computer system 100 so that the computer system 100 reads and executes the program.

  Regarding the embodiment including the above Examples 1 to 6, the following additional notes are disclosed.

(Supplementary Note 1) An image processing apparatus that extracts a difference by performing an inter-frame difference from an input image captured and input by an imaging unit that captures an image and a past image captured in the past,
Image storage means for storing past images captured in the past by the imaging unit;
Environmental information acquisition means for acquiring environmental information when an input image is captured by the imaging unit;
When the input image is input, based on the input image or the environment information acquired by the environment information acquisition unit, a difference interval determination unit that determines an interval for performing the inter-frame difference;
The past image input in the past corresponding to the difference interval determined by the difference interval determination unit is acquired from the image storage unit, and the inter-frame difference is performed from the acquired past image and the input image input, Difference extraction means for extracting the difference;
An image processing apparatus comprising:

(Supplementary note 2) The image processing apparatus according to supplementary note 1, further comprising a moving body detection unit that detects a moving body from a difference between the past input image and the input image extracted by the difference extraction unit.

(Additional remark 3) The said difference space | interval determination means determines the space | interval which implements the said inter-frame difference using the calculation formula based on the said input image or environmental information, The image processing apparatus of Additional remark 1 characterized by the above-mentioned.

(Additional remark 4) The said environmental information acquisition means acquires traffic as environmental information at the time of an input image being imaged by the said imaging part,
The difference interval determination unit determines the interval of the interframe difference when the traffic volume acquired by the environment information acquisition unit is large as a longer interval than when the traffic volume is low. An image processing apparatus according to 1.

(Additional remark 5) The said environment information acquisition means acquires day and night information as environment information at the time of an input image being imaged by the said imaging part,
The difference interval determining means determines an interval between frame differences when the day / night information acquired by the environment information acquiring means is night as a longer interval than when the day / night information is daytime. The image processing apparatus according to appendix 1.

(Additional remark 6) The said environment information acquisition means acquires a traffic light state as environment information at the time of an input image being imaged by the said imaging part,
The difference interval determination means determines an interval between frame differences when the traffic light state acquired by the environment information acquisition means is yellow or red as a longer interval than when the traffic light state is blue, Also, the interval of the inter-frame difference when the traffic light state acquired by the environmental information acquisition unit is red is determined as a longer interval than when the traffic light state is yellow. The image processing apparatus described.

(Appendix 7) The environment information acquisition unit acquires a distance to an object included in the input image from the imaging unit as environment information when the input image is captured by the imaging unit,
The difference interval determination means determines the interval for performing the inter-frame difference to be shorter as the distance acquired by the environment information acquisition means is shorter or as the object is located below the input image. The image processing apparatus according to appendix 1.

(Supplementary note 8) The environment information acquisition unit divides an input image into a plurality of areas by the imaging unit, acquires environment information when each area is captured,
The difference interval determining means determines an interval for performing an inter-frame difference for each of the plurality of areas based on the environment information for each of the plurality of areas acquired by the environment information acquiring means. A past image input in the past corresponding to the difference interval for each of the plurality of regions determined by the difference interval determining unit is acquired from the image storage unit, and an inter-frame difference is performed for each of the plurality of regions, and the difference is calculated. The image processing apparatus according to appendix 1, wherein the image processing apparatus is extracted.

(Supplementary Note 9) The environment information acquisition unit acquires environment information at the time of imaging for each pixel of the input image by the imaging unit,
The difference interval determining means determines an interval for performing an inter-frame difference for each pixel based on the environment information for each pixel acquired by the environment information acquiring means, and the difference extracting means determines the difference interval. Obtaining a past image input in the past corresponding to the difference interval for each pixel determined by the determination unit from the image storage unit, performing inter-frame difference for each pixel, and extracting the difference. The image processing apparatus according to appendix 1, which is characterized.

(Supplementary Note 10) When the environmental information acquired by the environmental information acquisition unit has changed from the previously acquired environmental information, the difference interval determination unit performs the inter-frame difference based on the environmental information. An image processing apparatus characterized by determining an interval to be performed.

(Supplementary Note 11) An image processing method suitable for extracting a difference by performing an inter-frame difference from an input image captured and input by an imaging unit that captures an image and a past image captured in the past. There,
Image storage means for storing past images captured in the past by the imaging unit;
An environment information acquisition step of acquiring environment information when an input image is captured by the imaging unit;
When the input image is input, based on the input image or the environment information acquired by the environment information acquisition step, a difference interval determination step for determining an interval for performing the inter-frame difference;
The past image input in the past corresponding to the difference interval determined by the difference interval determination step is acquired from the image storage means, and the inter-frame difference is performed from the acquired past image and the input image input, A difference extraction step for extracting the difference;
An image processing method comprising:

(Supplementary Note 12) Image processing that causes a computer to execute inter-frame difference from an input image captured and input by an imaging unit that captures an image and a past image captured in the past and extract the difference A program,
Image storage means for storing past images captured in the past by the imaging unit;
Environmental information acquisition procedure for acquiring environmental information when an input image is captured by the imaging unit;
A difference interval determination procedure for determining an interval for performing the inter-frame difference based on the input image or the environment information acquired by the environment information acquisition procedure when the input image is input;
The past image input in the past corresponding to the difference interval determined by the difference interval determination procedure is acquired from the image storage means, and the inter-frame difference is performed from the acquired past image and the input image input, A difference extraction procedure for extracting differences;
An image processing program for causing a computer to execute.

1 is a block diagram illustrating a configuration of an image processing apparatus according to a first embodiment. It is a figure which shows the example of the information memorize | stored in difference space | interval DB. 3 is a flowchart illustrating a flow of difference extraction processing in the image processing apparatus according to the first embodiment. 6 is a flowchart illustrating a flow of past image management processing in the image processing apparatus according to the first embodiment. It is a figure which shows the example of the imaging time slot memorize | stored in difference space | interval DB which concerns on Example 2. FIG. It is a figure which shows the example of the signal lighting state at the time of the imaging memorize | stored in difference space | interval DB which concerns on Example 2. FIG. It is a figure which shows the example of the imaging time memorize | stored in difference space | interval DB which concerns on Example 2, and a signal lighting state. FIG. 10 is a diagram for explaining processing of the image processing apparatus according to the third embodiment. 10 is a flowchart for explaining processing of an image processing apparatus according to a fourth embodiment. It is a figure which shows the example of the calculation formula for calculating a difference space | interval from traffic volume. It is a figure which shows the example of the calculation formula for calculating a difference space | interval from a time slot | zone. It is a figure which shows the example of the calculation formula for calculating a difference space | interval from the weather. It is a figure which shows the example of the computer system which performs an image processing program.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Image processing apparatus 11 Input image reception part 12 External sensor 20 Image management DB
21 Difference interval DB
30 Environment Determination Unit 31 Difference Processing Unit 100 Computer System 101 RAM
102 HDD
102a Image management table 102b Difference interval table 103 ROM
103a Input image reception program 103b Environment information acquisition program 103c Environment determination program 103d Difference processing program 104 CPU
104a Input image reception process 104b Environment information acquisition process 104c Environment determination process 104d Difference processing process

Claims (5)

An image processing apparatus that performs an inter-frame difference from an input image captured and input by an imaging unit that captures an image and a past image captured in the past, and extracts the difference,
Image storage means for storing past images captured in the past by the imaging unit;
Environmental information acquisition means for acquiring environmental information when an input image is captured by the imaging unit;
When the input image is input, based on the input image or the environment information acquired by the environment information acquisition unit, a difference interval determination unit that determines an interval for performing the inter-frame difference;
The past image input in the past corresponding to the difference interval determined by the difference interval determination unit is acquired from the image storage unit, and the inter-frame difference is performed from the acquired past image and the input image input, Difference extraction means for extracting the difference;
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the difference interval determination unit determines an interval for performing the interframe difference using a calculation formula based on the input image or environment information. The environmental information acquisition means acquires traffic volume as environmental information when an input image is captured by the imaging unit,
The difference interval determination unit determines an interval between frame differences when the traffic volume acquired by the environment information acquisition unit is large as a longer interval than when the traffic volume is low. The image processing apparatus according to 1. The environment information acquisition means acquires day and night information as environment information when an input image is captured by the imaging unit,
The difference interval determining means determines an interval between frame differences when the day / night information acquired by the environment information acquiring means is night as a longer interval than when the day / night information is daytime. The image processing apparatus according to claim 1. The environment information acquisition unit acquires a traffic light state as environment information when an input image is captured by the imaging unit,
The difference interval determination means determines an interval between frame differences when the traffic light state acquired by the environment information acquisition means is yellow or red as a longer interval than when the traffic light state is blue, The interval of the inter-frame difference when the traffic light state acquired by the environment information acquisition unit is red is determined as a longer interval than when the traffic light state is yellow. An image processing apparatus according to 1.

Images