JP2018147329A - Image processing device, image processing system, and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a background difference processing technique robust over a variation in illumination condition.SOLUTION: The image processing device for extracting, by using a background subtraction method, a foreground region from a moving image shot by a camera includes an acquisition unit, a background difference processing unit, and a threshold varying unit. The acquisition unit acquires a plurality of frame images included in the moving image. The background difference processing unit compares the background image which is the frame image with a processing target image which is the frame image and is acquired by the acquisition unit after the background image, and separates regions of the processing target image into the foreground region and the background region based on a threshold value. The threshold varying unit varies the threshold value according to at least one of a luminance value of the frame image acquired by the acquisition unit before the processing target image and an illuminance which influences the luminance value of the frame image acquired by the acquisition unit before the processing target image.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing technique for extracting a foreground region from a moving image captured by a camera using a background subtraction method.

  An image processing technique for extracting a foreground region from a moving image photographed by a camera using a background subtraction method is used, for example, for extracting a person (see Patent Document 1, for example).

JP 2008-250366 A

  However, the general background subtraction method has a drawback that it is vulnerable to variations in illumination conditions. For example, when a vehicle occupant is extracted from a moving image captured by a camera using a background difference method, the illumination conditions at the time of camera shooting vary depending on the external environment of the vehicle at the time of camera shooting. Background difference processing technology that is resistant to fluctuations in conditions is desired.

  In view of the above problems, an object of the present invention is to provide a background difference processing technique that is resistant to fluctuations in illumination conditions.

  An image processing apparatus of the present invention is an image processing apparatus that extracts a foreground region from a moving image captured by a camera using a background subtraction method, and an acquisition unit that acquires a plurality of frame images included in the moving image; The background image that is the frame image is compared with the processing target image that is the frame image and is acquired by the acquisition unit after the background image, and the region of the processing target image is determined based on a threshold value as the foreground region. And a background difference processing unit that distinguishes between a background region and a luminance value of the frame image acquired by the acquisition unit prior to the processing target image, and acquired by the acquisition unit prior to the processing target image A threshold variable unit configured to vary the threshold according to at least one of illuminances affecting the luminance value of the frame image (first configuration).

  In the image processing apparatus having the first configuration, the threshold variable unit is a time average luminance value obtained by time averaging luminance values of the plurality of frame images acquired by the acquisition unit prior to the processing target image. Depending on the configuration, the threshold may be varied (second configuration).

  In the image processing apparatus having the second configuration, the threshold variable unit divides the frame image into a plurality of divided areas, obtains the time average luminance value in units of the divided areas, and calculates the time in units of the divided areas. A configuration in which the threshold is variable (third configuration) may be used.

  In the image processing apparatus having the first configuration, the threshold variable unit is configured to respond to a luminance value of the frame image acquired by the acquisition unit prior to the processing target image and a luminance value of the processing target image. A configuration (fourth configuration) in which the threshold value is variable may be employed.

  In the image processing apparatus having the fourth configuration, the threshold variable unit divides each of the frame image and the processing target image acquired by the acquisition unit prior to the processing target image into a plurality of partitioned regions, A value obtained by averaging the luminance value of the frame image acquired by the acquisition unit prior to the processing target image in the unit of the segmented region, and the luminance value of the processing target image in the unit of the segmented region Depending on the value obtained in this way, the threshold value may be varied in units of the segmented regions (fifth configuration).

  In the image processing apparatus having the first configuration, the threshold variable unit receives illuminance information output from an illuminometer and corresponds to the plurality of frame images acquired by the acquisition unit prior to the processing target image. A configuration (sixth configuration) in which the threshold value is varied according to a time average illuminance obtained by time averaging the illuminance may be employed.

  In the image processing apparatus having the first configuration, the threshold variable unit receives illuminance information output from an illuminometer, and the illuminance corresponding to the frame image acquired by the acquisition unit prior to the processing target image And the structure (7th structure) which varies the said threshold value according to the said illumination intensity corresponding to the said process target image may be sufficient.

  In the image processing device having any one of the first to seventh configurations, when the threshold variable unit receives a communication signal notifying that a specific location has been reached from the navigation device, the threshold variable unit precedes the processing target image. The threshold value according to at least one of the luminance value of the frame image acquired by the acquisition unit and the illuminance that affects the luminance value of the frame image acquired by the acquisition unit prior to the processing target image. May be configured (eighth configuration).

  In the image processing apparatus having any one of the first to eighth configurations, the background difference processing unit may be configured to perform background difference processing on the entire region of the processing target image (ninth configuration). .

  In the image processing apparatus having any one of the first to eighth configurations, the background difference processing unit performs background difference processing only on a predetermined partial region in the processing target image (tenth configuration). It may be.

  In the image processing apparatus having any one of the first to tenth configurations, the background difference processing unit is configured to distinguish the region of the processing target image into the foreground region and the background region by a graph cut method (an eleventh configuration). Configuration).

  In the image processing apparatus having the eleventh configuration, the threshold value is a threshold value (twelfth configuration) for determining whether each node is strongly connected to the foreground region or the background region. Also good.

  In the image processing apparatus having any one of the first to twelfth configurations, an edge detection unit that detects an edge in a peripheral region of a pixel of interest in the frame image, and when the edge moves over time, the movement If the luminance value of the target pixel does not substantially change and the edge does not further move until a certain time elapses, a corrected background image is generated in which the luminance value of the target pixel is reflected in the background image. The background difference processing unit may be configured to use the corrected background image as the background image (a thirteenth configuration).

  An image processing system of the present invention is a configuration (fourteenth configuration) comprising a camera and the image processing apparatus having any one of the first to thirteenth configurations for processing a moving image captured by the camera.

  The image processing method of the present invention is an image processing method for extracting a foreground region from a moving image taken by a camera using a background subtraction method, and acquiring a plurality of frame images included in the moving image; The background image that is the frame image is compared with the processing target image that is the frame image and is acquired by the acquiring step after the background image, and the region of the processing target image is determined based on a threshold value. A background difference processing step for distinguishing between the image and the background region, the luminance value of the frame image acquired by the acquisition step prior to the processing target image, and the acquisition step prior to the processing target image. A configuration (fifteenth configuration) comprising: a threshold variable step that varies the threshold according to at least one of the illuminances affecting the luminance value of the frame image. That.

  According to the present invention, it is possible to provide a background difference processing technique that is resistant to fluctuations in illumination conditions.

1 is a block diagram showing the configuration of an image processing system according to a first embodiment. The figure which shows an example of the image image | photographed with a vehicle interior camera Flowchart showing an operation example of the image processing apparatus The figure which shows an example of the image image | photographed with a vehicle interior camera The figure which shows an example of the image image | photographed with a vehicle interior camera The block diagram which shows the structure of the image processing system which concerns on 3rd Embodiment. Block diagram showing the configuration of an image processing system according to the fifth embodiment A diagram conceptually showing the distinction between foreground and background areas by the graph cut method The block diagram which shows the structure of the image processing system which concerns on 7th Embodiment.

  Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings.

<1. First Embodiment>
FIG. 1 is a block diagram illustrating a configuration of an image processing system according to the present embodiment. The image processing system includes a vehicle interior photographing camera 10 and an image processing device 20. In the present embodiment, the image processing system is mounted on a vehicle.

  The vehicle interior photographing camera 10 is a monocular camera that photographs the interior of the vehicle from one direction. The vehicle interior camera 10 may be installed at the front center of the vehicle ceiling so that, for example, not only passengers sitting in the driver's seat and passenger seat but also passengers sitting in the rear seats can be viewed from above. For example, FIG. 2 shows an image captured by the vehicle interior camera 10 when the vehicle interior camera 10 is installed at the front center of the vehicle ceiling.

  The image processing apparatus 20 extracts a foreground region (occupant) from a moving image captured by the vehicle interior photographing camera 10 using a background difference processing method. It should be noted that a near-infrared camera is employed as the vehicle interior photographing camera 10 so that the image processing apparatus 20 can extract the foreground region (occupant) even at night, and the near infrared is applied to the photographing region of the vehicle interior photographing camera 10. It is preferable to install a near-infrared illumination device for irradiating light in the vehicle interior.

  The image processing apparatus 20 can be configured by hardware such as an application specific integrated circuit (ASIC) or a field-programmable gate array (FPGA), or a combination of hardware and software. When the image processing apparatus 20 is configured using software, a block diagram of a part realized by software represents a functional block diagram of the part. A function realized using software may be described as a program, and the function may be realized by executing the program on a program execution device. Examples of the program execution device include a computer including a CPU (Central Processing Unit), a RAM (Random Access Memory), and a ROM (Read Only Memory).

  The image processing device 20 includes an acquisition unit 21, a background difference processing unit 23, and a threshold variable unit 24.

  The acquisition unit 21 acquires analog or digital photographed images from the vehicle interior photographing camera 10 continuously in a predetermined cycle (for example, 1/30 second cycle). That is, a collection of captured images acquired by the acquisition unit 21 is a moving image captured by the vehicle interior camera 10. If the acquired captured image is analog, the acquiring unit 21 converts the analog captured image into a digital captured image (A / D conversion). The acquisition unit 21 outputs the acquired captured image or the acquired and converted captured image to the background difference processing unit 23. One captured image output from the acquisition unit 21 becomes one frame image.

  The acquisition unit 21 sends the frame image to the background difference processing unit 23 and the threshold variable unit 24.

  The background difference processing unit 23 compares the background image that is a frame image with the processing target image that is a frame image and is acquired by the acquisition unit 21 after the background image, and determines the region of the processing target image based on the threshold value. Distinguish between areas and background areas. In the present embodiment, the background difference processing unit 23 calculates a luminance difference between pixels at the same position in the background image and the processing target image, sets a position where the luminance difference is equal to or greater than a threshold as the foreground region of the processing target image, and determines the luminance difference. Is a background area of the processing target image.

  In the present embodiment, the threshold variable unit 24 varies the threshold used by the background difference processing unit 23 according to the luminance value of the frame image acquired by the acquisition unit 21 prior to the processing target image. More specifically, in the present embodiment, the threshold variable unit 24 divides the frame image into a plurality of divided regions, and sets the luminance values of the plurality of frame images acquired by the acquisition unit 21 before the processing target image as time. A time average luminance value obtained by averaging is obtained in units of segmented areas, and the threshold value is varied in units of segmented areas.

  FIG. 3 is a flowchart illustrating an operation example of the image processing apparatus 20. The image processing apparatus 20 starts the operation of the flowchart shown in FIG. 3 at the timing when the door lock of the vehicle is released. When the result of the background difference processing is that the foreground area (occupant) has not been extracted, the vehicle door is unloaded. The operation of the flowchart shown in FIG. 3 is forcibly terminated at the timing when the lock is switched to the lock. Therefore, the image processing apparatus 20 receives a signal for notifying whether the door of the vehicle is locked or unlocked from the outside of the image processing apparatus 20.

  First, the acquisition unit 21 acquires a background image (step S10). After acquiring the background image, the acquisition unit 21 continues to acquire the frame image at a predetermined cycle.

  Next, the background difference processing unit 23 determines a frame image to be a processing target image (step S20).

  Next, the threshold variable unit 24 updates the threshold (step S30). In the present embodiment, for the purpose of driver extraction, a predetermined partial region R1 of the frame image is set as shown in FIG. In addition, as shown in FIG. 5, the frame image is divided into a plurality of divided areas including areas R2 other than the predetermined partial area R1 and divided areas R3 obtained by equally dividing the predetermined partial area R1. . Then, the threshold variable unit 24 averages the luminance values of the frame images for each divided region R3 and updates the threshold for each divided region R3 according to the time-averaged luminance. For example, when the processing target image is the nth frame image, the luminance values of the (n−1) th frame image to the (n−k) th frame image may be time-averaged. In the above, n is a natural number of 3 or more, and k is a natural number smaller than n. Further, unlike the above example, the frame image (the (n−1) th frame image) acquired one ahead of the processing target image may not be included in the time averaging target.

  In step S30, for example, the threshold value may be decreased as the time-averaged luminance is separated from the reference value.

  Finally, the background difference processing unit 23 executes background difference processing using the threshold updated in step S30 (step S40). In the present embodiment, the background difference processing unit 23 performs background difference processing only on a predetermined partial region R1. This reduces the burden of background difference processing on the image processing apparatus 20. In addition, by removing vehicle movable parts such as a handle, seat belt retractor, and shift knob from the predetermined partial area R1, it is possible to prevent the vehicle movable parts from being mistakenly identified as the foreground area by moving. . In addition, the background difference processing unit 23 may perform background difference processing on the entire region of the processing target image including the predetermined partial region R1. Thereby, the background difference process can be executed on the processing target image without omission.

  Note that the background difference processing unit 23 may generate, for example, an image in which the boundary line between the foreground region and the background region is emphasized as a result of the background difference processing and output it to an external display device.

  When step S40 ends, the process returns to step S20, and the frame image immediately after the frame image that has been the processing target image is set as a new processing target image.

  According to the present embodiment, the threshold variable unit 24 varies the threshold used by the background difference processing unit 23 according to the luminance value of the frame image acquired by the acquisition unit 21 prior to the processing target image. The threshold value can be varied according to the illumination conditions before the image is taken. The lighting conditions do not fluctuate randomly but vary with some degree of continuity. Accordingly, it is possible to set a threshold value suitable for the illumination condition at the time of capturing the processing target image. As a result, it is possible to execute background difference processing that is resistant to variations in illumination conditions.

  Further, according to the present embodiment, the threshold variable unit 24 varies the threshold used in the background difference processing unit 23 in accordance with the time average luminance value obtained by time averaging the luminance values of a plurality of frame images. The threshold variable unit 24 can gradually change the threshold in accordance with the change of the illumination condition that changes with a certain degree of continuity. Therefore, a certain degree of continuity is maintained with respect to the foreground region extraction result, and sudden unextraction or erroneous extraction of the foreground region can be suppressed.

  Further, according to the present embodiment, since the threshold variable unit 24 updates the threshold for each divided region R3, the threshold variable unit 24 is configured to be divided even when the illumination conditions in the photographing region of the in-vehicle photographing camera 10 vary from place to place. The threshold value can be appropriately varied for each R3. Therefore, it becomes stronger against fluctuations in illumination conditions than when the divided region R3 is not set.

<2. Second Embodiment>
The image processing system according to the present embodiment differs from the image processing system according to the first embodiment described above in the operation content of the threshold variable unit 24. Therefore, FIGS. 1 to 5 used in the description of the first embodiment can be applied to the present embodiment as they are. Hereinafter, a different point from 1st Embodiment is demonstrated and description is abbreviate | omitted about the structure, operation | movement, and process similar to 1st Embodiment.

  In the present embodiment, the threshold variable unit 24 varies the threshold according to the luminance value of the frame image acquired by the acquisition unit 21 prior to the processing target image and the luminance value of the processing target image. For example, when the processing target image is the nth frame image, it is used in the previous background difference process based on the difference between the luminance value of the nth frame image and the luminance value of the (n−1) th frame image. The amount of change from the previous threshold value may be determined and updated from the threshold value used in the previous background difference process to the threshold value used in the current background difference process. For example, the larger the difference between the luminance value of the nth frame image and the luminance value of the (n−1) th frame image, the larger the amount of change from the threshold used in the previous background difference process.

  According to the present embodiment, the threshold variable unit 24 varies the threshold according to the luminance value of the frame image acquired by the acquisition unit 21 prior to the processing target image and the luminance value of the processing target image. Even when the illumination condition changes relatively abruptly from the time of capturing the frame image acquired by the acquiring unit 21 prior to the target image to the time of capturing the processing target image, the threshold variable unit 24 appropriately varies the threshold. Can do.

<3. Third Embodiment>
FIG. 6 is a block diagram showing the configuration of the image processing system according to the present embodiment. The point that the image processing system is configured by the vehicle interior photographing camera 10 and the image processing device 20 is the same as in the first embodiment and the second embodiment. Hereinafter, differences from the first embodiment and the second embodiment will be described.

  In the present embodiment, the background difference processing unit 23 receives the illuminance information output from the illuminometer 40. The illuminometer 40 is installed in a place where the illuminance detected by the illuminometer 40 becomes an illuminance that affects the luminance value of the frame image. The illuminometer 40 is installed, for example, at the front end (near the windshield) in the vehicle interior.

  Unlike the first embodiment and the second embodiment, in this embodiment, the threshold variable unit 24 does not need the luminance information of the frame image. The threshold variable unit 24 receives from the acquisition unit 21 the acquisition timing of each frame image that is correlated with the shooting timing of each frame image.

  The threshold variable unit 24 varies the threshold according to the time average illuminance obtained by time averaging the illuminance corresponding to the plurality of frame images acquired by the acquisition unit 21 prior to the processing target image. Unlike the first and second embodiments, in this embodiment, the frame image is not divided into a plurality of divided regions. However, the predetermined partial region R1 is set as in the first and second embodiments. The threshold variable unit 24 averages the illuminance corresponding to the frame image and updates the threshold according to the time-averaged illuminance. For example, when the processing target image is the nth frame image, the illuminance corresponding to the (n−1) th frame image to the (n−k) th frame image may be time-averaged. In the above, n is a natural number of 3 or more, and k is a natural number smaller than n. In addition, unlike the above example, the illuminance corresponding to the frame image (the (n−1) th frame image) acquired one ahead of the processing target image is not included in the time averaging target. Also good.

  According to this embodiment, the threshold variable unit 24 varies the threshold used by the background difference processing unit 23 according to the illuminance that affects the luminance value of the frame image acquired by the acquisition unit 21 prior to the processing target image. Therefore, the threshold value can be varied according to the illumination condition before the time when the processing target image is captured. The lighting conditions do not fluctuate randomly but vary with some degree of continuity. Accordingly, it is possible to set a threshold value suitable for the illumination condition at the time of capturing the processing target image. As a result, it is possible to execute background difference processing that is resistant to variations in illumination conditions.

  Further, according to the present embodiment, the threshold variable unit 24 varies the threshold without processing the luminance information of the frame image, so the burden of the threshold variable processing on the threshold variable unit 24 is reduced.

<4. Fourth Embodiment>
The image processing system according to the present embodiment differs from the image processing system according to the third embodiment described above in the operation content of the threshold variable unit 24. Hereinafter, a different point from 3rd Embodiment is demonstrated and description is abbreviate | omitted about the structure, operation | movement, and process similar to 3rd Embodiment.

  In the present embodiment, the threshold variable unit 24 varies the threshold according to the illuminance corresponding to the frame image acquired by the acquisition unit 21 prior to the processing target image and the illuminance corresponding to the processing target image. For example, when the processing target image is the nth frame image, the previous background difference process is performed based on the difference between the illuminance corresponding to the nth frame image and the illuminance corresponding to the (n−1) th frame image. The amount of change from the threshold value used in the above step may be determined and updated from the threshold value used in the previous background difference process to the threshold value used in the current background difference process. For example, the larger the difference between the illuminance corresponding to the nth frame image and the illuminance corresponding to the (n−1) th frame image, the greater the amount of change from the threshold used in the previous background difference process. .

  According to this embodiment, the threshold variable unit 24 varies the threshold according to the illuminance corresponding to the frame image acquired by the acquisition unit 21 prior to the processing target image and the illuminance corresponding to the processing target image. Even when the illumination condition changes relatively abruptly from when the frame image acquired by the acquisition unit 21 is captured before the processing target image to when the processing target image is captured, the threshold variable unit 24 appropriately varies the threshold. can do.

<5. Fifth Embodiment>
FIG. 7 is a block diagram showing the configuration of the image processing system according to this embodiment. The point by which an image processing system is comprised by the vehicle interior camera 10 and the image processing apparatus 20 is the same as that of 1st-4th embodiment. Hereinafter, differences from the first embodiment and the second embodiment will be described.

  When the threshold variable unit 24 receives a communication signal notifying that a specific place has been reached from the navigation device 50, the threshold variable unit 24, according to the luminance value of the frame image acquired by the acquisition unit 21 prior to the processing target image, Change the threshold. More specifically, the threshold variable unit 24 performs an operation of changing the threshold until a predetermined time elapses after receiving a communication signal notifying that a specific place has been reached from the navigation device 50, and otherwise. The threshold variable unit 24 stops updating the threshold. As the specific place, for example, a point immediately before a point where the lighting conditions greatly change, such as immediately before the entrance of the tunnel or immediately before the exit of the tunnel, is suitable.

  The operation of the threshold variable unit 24 may be the same as that of the first embodiment or may be the same as that of the second embodiment.

  According to the present embodiment, it is possible to appropriately cope with fluctuations in illumination conditions that may occur slightly ahead of a specific location, and the burden of threshold variable processing on the threshold variable unit 24 is reduced.

  Note that this embodiment may be combined with the third embodiment or the fourth embodiment, and the operation of the threshold variable unit 24 may be the same as that of the third embodiment or the fourth embodiment. In this case, the burden of the threshold variable process on the threshold variable unit 24 can be further reduced.

<6. Sixth Embodiment>
In the first to fifth embodiments described above, the background difference processing unit 23 calculates the luminance difference between the same-position pixels of the background image and the processing target image, and sets the position where the luminance difference is equal to or greater than the threshold as the processing target image. A foreground area is set, and a position where the luminance difference is less than the threshold is set as a background area of the processing target image.

  The present embodiment is a modification of any of the first to fifth embodiments described above, and in any of the first to fifth embodiments described above, the background difference processing unit 23 performs a graph cut (Graph Cut). ) Method so that the region of the processing target image is divided into a foreground region and a background region.

  FIG. 8 is a diagram conceptually showing the distinction between the foreground region and the background region by the graph cut method. In the graph cut method, a foreground area is obtained such that the total amount of energy required to divide the foreground area for the background image acquired by the acquisition unit 21 is minimized.

  In FIG. 8, s represents the foreground, t represents the background, and ◯ represents each adjacent node, and those connected by pipes are cut to divide the foreground and the background. The flow between each node and s or t is called a data term (broken line portion in FIG. 8), and the flow between adjacent nodes is called a smoothing term (solid line portion in FIG. 8). The data term represents how much the area is foreground (s) or background (t). The smoothing term represents how many connections there are to the four neighboring areas adjacent to the area. The stronger the connection between these nodes, the greater the energy required to divide.

  The threshold updated by the threshold variable unit 24 may be, for example, a threshold for determining whether each node is strongly connected to the foreground (s) or the background (t).

  By distinguishing between the foreground region and the background region by the graph cut method, it is possible to realize background difference processing that is more resistant to variations in illumination conditions.

<7. Seventh Embodiment>
FIG. 9 is a block diagram showing a configuration of the image processing system according to the present embodiment. In this embodiment, unlike the first embodiment and the second embodiment, the image processing apparatus 20 includes an edge detection unit 25 and a correction unit 26. Hereinafter, differences from the first embodiment and the second embodiment will be described.

  The edge detection unit 25 detects an edge in the peripheral region of the target pixel in the frame image. The target pixel is preferably all the pixels of the frame image, but the thinning process may be performed on all the pixels of the frame image, and only the pixel remaining after the thinning process may be set as the target pixel.

  When the edge detected by the edge detection unit 25 moves over time, the correction unit 26 does not substantially change the luminance value of the pixel of interest until the fixed time elapses from the movement, and the edge detection unit 25. If the detected edge does not move substantially further, a modified background image in which the luminance value of the target pixel is reflected in the background image is generated. When the corrected background image is generated by the correction unit 26, the background difference processing unit 23 uses the corrected background image generated by the correction unit 26 as a background image. In addition, when the edge detected by the edge detection unit 25 moves over time, the correction unit 26 does not change the luminance value of the pixel of interest at all until the fixed time elapses from the movement and detects the edge. If the edge detected by the unit 25 is not completely moved, a modified background image in which the luminance value of the target pixel is reflected in the background image may be generated. In addition, when the edge detected by the edge detection unit 25 moves over time, the correction unit 26 is within a predetermined range even if the luminance value of the pixel of interest fluctuates until a certain time elapses after the movement. Even if the edge within the range and the edge detected by the edge detection unit 25 further moves, a corrected background image in which the luminance value of the target pixel is reflected in the background image may be generated as long as it falls within a predetermined range. . As for the predetermined range relating to the change in the luminance value of the pixel of interest and the predetermined range relating to further movement of the edge, an appropriate range may be obtained by experiment or simulation.

  In the vehicle, after the background image is acquired, for example, the seat position and the seat reclining setting may be changed to move the seat. In this case, the sheet may be divided into foreground areas.

  Focusing on the fact that the movement of the seat as described above is not frequently performed and that the occupant does not move at all even if it takes the same posture, the edge detection unit 25 and the correction unit 26 is provided in the image processing apparatus 20.

  By providing the edge detection unit 25 and the correction unit 26, even if a foreground region other than the originally assumed foreground region (occupant in this embodiment) moves after the background image is once acquired, erroneous extraction of the foreground region is performed. Can be suppressed. That is, the foreground area extraction accuracy is improved.

  In addition, this embodiment can also be combined with the 3rd-6th embodiment.

<8. Modified example>
Various technical features disclosed in the present specification can be variously modified within the scope of the technical creation in addition to the above-described embodiment. In addition, a plurality of embodiments and modification examples shown in the present specification may be implemented in combination within a possible range.

  For example, the first embodiment or the second embodiment may be combined with the third embodiment or the fourth embodiment. In this case, since two threshold values are calculated for the same position of the processing target image, for example, a weighted average of the two threshold values may be used as the final threshold value.

DESCRIPTION OF SYMBOLS 10 Car interior camera 20 Image processing apparatus 21 Acquisition part 22 Interface 23 Background difference process part 24 Threshold variable part 25 Edge detection part 26 Correction part 40 Illuminance meter 50 Navigation apparatus

Claims (15)

An image processing apparatus that extracts a foreground region from a moving image captured by a camera using a background subtraction method,
An acquisition unit for acquiring a plurality of frame images included in the moving image;
The background image that is the frame image is compared with the processing target image that is the frame image and is acquired by the acquisition unit after the background image, and the region of the processing target image is determined as the foreground region based on a threshold value. A background difference processing unit for distinguishing the background region;
At least the luminance value of the frame image acquired by the acquisition unit prior to the processing target image and the illuminance that affects the luminance value of the frame image acquired by the acquisition unit prior to the processing target image An image processing apparatus comprising: a threshold value varying unit configured to vary the threshold value according to one.   The threshold variable unit varies the threshold according to a time average luminance value obtained by time averaging luminance values of the plurality of frame images acquired by the acquisition unit prior to the processing target image. Item 8. The image processing apparatus according to Item 1.   The threshold value changing unit divides the frame image into a plurality of divided regions, obtains the time average luminance value in units of the divided regions, and varies the threshold value in units of the divided regions. Image processing device.   The threshold value varying unit varies the threshold value according to a luminance value of the frame image acquired by the acquiring unit prior to the processing target image and a luminance value of the processing target image. The image processing apparatus described. The threshold variable unit includes:
Each of the frame image and the processing target image acquired by the acquisition unit prior to the processing target image is divided into a plurality of partitioned regions,
A value obtained by averaging the luminance value of the frame image acquired by the acquisition unit prior to the processing target image in the unit of the segmented area, and an average of the luminance value of the processing target image in the unit of the segmented area The image processing apparatus according to claim 4, wherein the threshold value is varied in units of the segmented areas in accordance with a value obtained by converting the threshold value.   The threshold variable unit receives illuminance information output from an illuminometer, and is a time average obtained by averaging the illuminance corresponding to the plurality of frame images acquired by the acquisition unit prior to the processing target image. The image processing apparatus according to claim 1, wherein the threshold value is varied according to illuminance.   The threshold variable unit receives illuminance information output from an illuminometer, and the illuminance corresponding to the frame image acquired by the acquisition unit prior to the processing target image and the illuminance corresponding to the processing target image The image processing apparatus according to claim 1, wherein the threshold value is varied in accordance with.   The threshold variable unit, when receiving a communication signal notifying that a specific place has been reached from the navigation device, the luminance value of the frame image acquired by the acquisition unit prior to the processing target image, and The threshold value is varied according to at least one of illuminance that affects the luminance value of the frame image acquired by the acquisition unit prior to the processing target image. Image processing apparatus.   The image processing apparatus according to claim 1, wherein the background difference processing unit performs background difference processing on the entire region of the processing target image.   The image processing apparatus according to claim 1, wherein the background difference processing unit performs background difference processing only on a predetermined partial region in the processing target image.   The image processing apparatus according to claim 1, wherein the background difference processing unit distinguishes the region of the processing target image into the foreground region and the background region by a graph cut method.   The image processing apparatus according to claim 11, wherein the threshold is a threshold for determining whether each node is strongly connected to the foreground area or the background area. An edge detection unit for detecting an edge in a peripheral region of the target pixel in the frame image;
When the edge moves over time, the luminance value of the pixel of interest does not change substantially and the edge does not move substantially until the fixed time elapses from the movement. A correction unit that generates a corrected background image in which a value is reflected in the background image, and
The image processing apparatus according to claim 1, wherein the background difference processing unit uses the corrected background image as the background image. A camera,
An image processing system comprising: the image processing apparatus according to any one of claims 1 to 13 that processes a moving image captured by the camera. An image processing method for extracting a foreground region from a moving image captured by a camera using a background subtraction method,
An acquisition step of acquiring a plurality of frame images included in the moving image;
The background image that is the frame image is compared with the processing target image that is the frame image and is acquired by the acquiring step after the background image, and the region of the processing target image is determined as the foreground region based on a threshold value. A background difference processing step for distinguishing from the background region;
The luminance value of the frame image acquired by the acquisition step prior to the processing target image, and the illuminance that affects the luminance value of the frame image acquired by the acquisition step prior to the processing target image An image processing method comprising: a threshold value changing step for changing the threshold value according to one of them.