JP2016115093A - Image processing system and image processing method used for the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing system which can distinguish an object from a background in an image even when not only the object changes but also the background changes at the time of distinguishing the object from the background in the image captured by a camera or the like, and provide an image processing method used for the same.SOLUTION: An image processing system includes processing of acquiring a present image at the time of processing and a previous image immediately prior to the present image from images captured consecutively at a fixed point, processing of comparing the present image and the previous image to define a region where a difference is detected as an operation region, processing of comparing the present image and a background image (initial value at the processing start, then updated by the present image) to define a region where the difference is detected as an object region, processing of setting a value to a timer corresponding to each of the operation region and the object region, processing of propagating the timer value of the object region to a peripheral region to set a proper value to the timer value of the operation region, processing of updating the background image of the region where the timer value of the operation region is in the time-out state to the present image, and processing of synthesizing an output image by using the background image.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an image processing system for distinguishing and displaying a stationary object and a moving object included in a captured image such as a camera and an image processing method used therefor. In particular, the present invention relates to an image processing system that distinguishes between an instructor and a board.

  In an image processing system that identifies moving objects in order to detect the region of the presenter in the video of a class, etc., the background should be registered and a person with a difference in color from the background should be a person. However, it is necessary to distinguish between changes in the background and changes in the person because the background changes depending on the board in class.

  There is Patent Document 1 as a technique for specifying a board book, and Patent Document 2 and Non-Patent Documents 1 and 2 are techniques for obtaining a picture of a board book that is not obstructed by a person. Since Patent Document 1 is a method specialized in characters and defines the order of places for writing on a board, it cannot be handled by using a board that does not follow this.

  Patent Document 2 is a method for distinguishing between a blackboard and a person based on the difference in color between the blackboard and the person. Therefore, a problem occurs in the case of a person wearing clothes of the same color as the letters used in the board. Also, it is not possible to paste a figure on a blackboard instead of a board.

  Non-Patent Document 1 is a method for distinguishing between a blackboard and a person using a difference between the pattern of the blackboard and the person as a starting point. In this method, a problem occurs in the case of a person wearing clothes printed with characters. In addition, it is not possible to explain when a photograph of a person is pasted on a blackboard.

  Non-Patent Document 2 is a method for discriminating between a board and a person based on the distance between the board and the person's camera. Although there is no restriction on the information of a person's clothes and the information written on or posted on a board, a device for measuring the distance is necessary, and this method cannot be used when such a device cannot be used.

  On the other hand, as a technique for specifying a person, Non-Patent Document 3 uses a temperature camera to detect the body temperature of a person and specifies a person based on images obtained from two cameras, a color camera and a temperature camera. Is disclosed. This method requires a specific device, a temperature camera, and cannot be distinguished in situations where such a device is not available.

JP 2007-166245 A JP 2010-092224 A

Masateru Onishi and two others, “Block Extraction of Board-written Areas in Lecture Video and Part 10 Automatic Shooting”, IEICE Technical Report, The Institute of Electronics, Information and Communication Engineers, November 1999, Vol.99, p.197 -204. Keikazu Yamane, Masayuki Okabe, Junji Umemura, "Implementation of a semi-transparent camera to support a board in lecture", Wiss2013, Software Science Society, pp203-204. Yoshiaki Nishigai: Research on video media using temperature information and color information in an integrated manner, Master's thesis, University of Tokyo, February 2006.

  As described above, an image processing system and image processing that distinguish and display a stationary object (hereinafter sometimes referred to as a stationary object) and a moving object (hereinafter also referred to as a moving object) included in a conventional captured image In the method, the object and the background can be distinguished when the background is stationary, but in the situation where a part of the background changes, there is a problem that the part of the background is recognized as an object and cannot be distinguished.

  In addition, if a part of the background changes, it follows the change in the background, so if there is no change for a certain time after the change, it is necessary to process it as a background. Even if the object moves, if most parts other than the part of the object are stationary, the object is regarded as a stationary object and is recognized as a background.

  The present invention has been made to solve the above problems.

The image processing system according to claim 1,
Image acquisition means for acquiring images continuously photographed at a fixed point;
A predetermined area dividing means for dividing a predetermined area into predetermined areas for the image acquired by the image acquisition means;
A still image creating means for detecting a stationary object from the image obtained by the image obtaining means and creating a still image by the stationary object;
An image recording means for recording a current image, which is the latest image at the time of processing, among the images acquired continuously, and an immediately preceding image acquired immediately before the current image;
An action area specifying means for specifying an action area by determining whether there is a change for each of the predetermined areas based on the current image and the immediately preceding image;
Object region specifying means for specifying an object region by determining a difference between the background image and the background image for each predetermined region based on the background image and the current image;
Region propagation means for reflecting a specific state of the motion region in the object region to a predetermined region belonging to the object region;
Output image creating means for creating an output image while synthesizing the entire image of the object region including the motion region with the still image.

The image processing system according to claim 2 is the image processing system according to claim 1,
Furthermore, a timer management means for managing a timer value given for each predetermined area is provided,
The region propagation means is
Timer processing means for setting the timer value for the motion area specified by the motion area specifying means, and for resetting the timer for all areas other than the object area specified by the object area specifying means;
Timer value propagation means for correcting the timer value of the object region based on the timer value of the motion region,
The whole image of the object area including the motion area is created by the timer management means based on a predetermined timer value.

The image processing system according to claim 3 is the image processing system according to claim 2,
The timer management means manages a timer value for a predetermined time set in advance,
The background image creating means identifies the predetermined area whose timer value during the predetermined time of the timer management means is less than a predetermined value as a background area, and creates a background image from the image in the background area. Features.

The image processing system according to claim 4 is the image processing system according to claim 2,
The background image creation means is characterized in that the background area is updated every predetermined time from the continuous images acquired by the image acquisition means.

The image processing method according to claim 5 comprises:
An image processing method for processing by a computer,
An image acquisition step of acquiring images continuously photographed at a fixed point;
A predetermined area dividing step for dividing the predetermined area as a predetermined area for the image acquired by the image acquiring step;
An initial still image creating step of detecting a stationary object from the image acquired by the image acquiring step and creating a temporary still image by the stationary object;
An image recording step of recording a current image that is the latest image at the time of processing and an immediately preceding image acquired immediately before the current image among images that are continuously acquired;
An operation region specifying step of specifying an operation region by determining whether there is a change for each of the predetermined regions based on the current image and the immediately preceding image;
An object region specifying step of specifying an object region by determining a difference between the background image and the background image for each of the predetermined regions based on the background image and the current image;
A timer processing step for setting a timer for the motion area specified by the motion area specifying means, and for resetting the timer for all areas other than the object area specified by the object area specifying means;
A timer value propagation step of correcting the timer value of the object region based on the timer value of the motion region;
Identifying a predetermined area where the value of the timer is less than a predetermined value, and certifying the predetermined area as a background area; and
A background image update step of reflecting the background region in the temporary background image and updating the background image;
And an output image creating step of creating an output image while synthesizing the object region including the motion region with the updated background image.

According to the present invention, even when both the background and the object change, it is possible to distinguish between both changes. Further, since a model for the background image and the object image is not constructed, the present invention can be applied without limiting the background and the object.

1 is a diagram illustrating a configuration of an image processing system according to the present invention. It is a figure which shows the detail of the image processing which concerns on this invention. It is a figure which shows the flow of the image processing which concerns on this invention. It is a figure which shows the initial state of the image processing which concerns on the Example of this invention. In the image processing which concerns on the Example of this invention, it is a figure which shows the result of having superimposed the background image and the present image immediately after lifting a box. In the image processing which concerns on the Example of this invention, it is a figure which shows the result of having overlap | superposed the background image and the present image after a fixed time passes immediately after lifting a box. It is a figure which shows the result when not performing the propagation process of the timer value until it starts update of a background image to the newest image in the image processing which concerns on the Example of this invention. In the image processing according to the embodiment of the present invention, it is a diagram showing the retention information (timer value) for each management section after a fixed time has passed from immediately after lifting the box, with symbols corresponding to each value.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  The configuration of the image processing system according to the present invention is shown in FIG. The configuration of the image processing system includes an image input device such as a camera, an image display device such as a display that displays images and videos, and a calculation process such as a CPU (Central Processing Unit) that performs calculations necessary for image processing according to the present invention. A device 101 and a storage device 102 that stores and holds a program and image information necessary for the image processing are connected by a bus. The image input device acquires images continuously captured at fixed points (functions as an image acquisition unit). The storage device 102 includes a ROM (Read Only Memory) and a RAM (Random Access Memory), or an external storage device such as a RAM and a hard disk, and stores and holds a program and image information necessary for image processing according to the present invention. (Functions as an image recording means), and can appropriately control the plurality of devices connected by the bus as necessary to obtain a predetermined processing result.

  Note that the configuration of the above-described image processing system is an example for describing the embodiment, and is not limited thereto.

  Next, details of the image processing according to the present invention are shown in FIGS.

  In FIG. 2, the storage device 102 of the image processing system stores the current image, the previous image, the background image, and the background image updated to the latest image for each management section (area) of the divided image. And an object area timer used for setting the update timer (can function as timer management means). Furthermore, the comparison result between the current image and the previous image is held.

  On the other hand, the arithmetic processing unit 101 compares two images and detects a difference portion, a propagation operation that sets the surrounding maximum value to the next value for each management partition, a bit operation, addition / subtraction, and a condition value Settings can be made. The arithmetic processing apparatus 101 can function as a predetermined area dividing unit, a still image creating unit, a region propagation unit, and an output image creating unit.

  In the image processing according to the present invention, first, the latest current image frame (hereinafter referred to as “currentFrame” including a figure) and the background image frame (hereinafter also referred to as “backgroundFrame” including a figure) whose background is substantially stationary. The image is held at the same resolution as the display resolution (image acquisition step, image recording step).

  In order to detect the most recent change, the image frame (hereinafter referred to as “previous frame” including the figure) at the previous time point is retained, and the resolution is reduced within the range where the change can be determined, or the color is changed. Or may be held by changing the resolution and color. The resolution is independent of the display resolution and may be the same as the display resolution, but is usually used at a low resolution.

  The previous frame holds information for area management such as an operation area and an object area (predetermined area dividing step). Since the unit of area management forms a section on the image, it is hereinafter referred to as a management section.

  In the image processing according to the present invention, for each of the management sections, a determination result as to whether or not the portion has changed most recently (hereinafter referred to as activeMap including the drawings), and a determination result as to whether the currentFrame and the backgroundFrame are the same. (Hereinafter referred to as objectMap, including the figure). Further, for each management section, an integer value of a timer (hereinafter referred to as “updateTimer” including the figure) indicating a remaining time until the background is determined and an integer value of the timer of the object area (hereinafter also including the figure). (Referred to as ObjectTimer).

  As a premise of the image processing according to the present invention, there are two cases (initial still image creation step).

  In the first case, there is no moving object in the background frame, and all the updateTimers are time-out values.

In the second case, the algorithm is continuously operated from the value of arbitrary backgroundFrame and updateTimer, and the image of the stationary object is obtained in backgroundFrame, and updateTimer has an appropriate value in the area covering the operation object. Is set. Information on whether or not the operation object exists is acquired by the objectTimer, and the value is transmitted to the surrounding area (covering area) of the operation object. An appropriate value is set for the area to be covered.
After obtaining the current image and the previous image from either of the two cases described above, the seven steps shown in FIG. 3 are repeated to generate the target image. Details of the seven steps shown in FIG. 3 will be described below.

Step 1. An operation area is obtained from the current image and the immediately preceding image (processing indicated by 201 in FIG. 2, operation area specifying step).
For each management partition, the current frame pixel and the previous frame pixel are compared to determine whether the current frame pixel has changed, and the determination result is stored in the activeMap. Then, the current frame information used for the determination is stored in the previous frame.

  Even if the memory for holding the value of the activeMap is not used, when the value of the activeMap is required, it may be obtained by referring to the pixels of the previous frame and the current frame. At this time, the previous Frame is updated after the value is used.

  There are many methods for determining whether or not there is a change in the currentFrame pixel, but the simplest method is to set a threshold value and change the average color in the management frame for the pixel of the previousFrame and the currentFrame pixel. It may be determined whether it is larger than the threshold value. Although it may be determined that there is a change in a case where the object is not occluded, it is effective to select a method that reduces the influence. Alternatively, even if the pixel does not change, the activeMap information may be set by regarding the feature of the pixel and the position of the pixel as equivalent to moving the part.

Step 2. An object area is obtained from the current image and the background image (processing indicated by 202 in FIG. 2, object area specifying step).
For each management partition, the currentFrame pixel and the backgroundFrame pixel are compared to determine whether there is a difference, and the result is stored in the objectMap. Alternatively, when the value of objectMap becomes necessary, the value may be obtained by referring to the currentFrame pixel and the backgroundFrame pixel.
There are many ways to determine the difference between the currentFrame pixel and the backgroundFrame pixel, but the simplest method is to create a threshold value and change the average color of the currentFrame pixel and the backgroundFrame pixel in the management section. It may be determined whether the change is larger than the threshold value. Although it may be determined that there is a change even though there is no actual change, if a method that reduces the influence is selected, the effect of the determination regarding the pixel difference increases. Thereafter, the object area is thickened. Specifically, when there is a change in neighboring pixels, it is assumed that an object exists there, and information on the presence or absence of the object is stored in the objectMap. As a neighborhood, a pixel immediately adjacent to the corresponding pixel may be taken.

Step 3. A timer is set for the operation region (the process indicated by 203 in FIG. 2, part of the timer processing step).
A predetermined time value is set in updateTimer corresponding to the management partition for which activeMap is determined to be true for each management partition. If it is false, the value is not changed. The fixed time may be any time, but it is effective to adjust for each application.

Step 4. The timer other than the object area is reset (the process indicated by 204 in FIG. 2, part of the timer processing step).
For each management partition, zero is set for the objectTimer corresponding to the partition whose objectMap is false. On the other hand, if objectMap is true, the value of updateTimer corresponding to objectTimer is set.

Step 5. The timer value is updated (the time is advanced after the value is propagated to the periphery) (the process indicated by 205 in FIG. 2, the timer value propagation step).
By propagating and updating the value of updateTimer based on the objectTimer, the value of the objectTimer is propagated all at once to surrounding values. Specifically, the next updateTimer value is obtained from the value of the objectTimer of the adjacent management partition and the value of the objectTimer of the corresponding management partition. For example, a maximum value of a total of nine values of the eight object timers adjacent to the corresponding management partition and the object timer of the corresponding management partition may be taken. Alternatively, the value obtained by subtracting the value of objectTimer around the corresponding management section may be calculated while taking the direction into account, and the maximum value may be taken. After obtaining the value of updateTimer, if the value of updateTimer is positive for each management partition, the value is decreased. Any value can be used to decrease the value. The simplest way is to subtract one. Further, the value to be decreased may be changed depending on characteristics such as the position of the management section.

Step 6. For the timeout area, the background image is updated to the current image (the process indicated by 206 in FIG. 2, the background area recognition step and the background image update step).
For each management partition, when the value of updateTime is equal to or less than the appropriate value, the background frame is brought closer to the current frame. In the simplest method, the backgroundFrame may be replaced with the currentFrame for the corresponding partition. Or you may update so that it may approach the said currentFrame gradually. The appropriate value may be a constant value. Further, it may be a value calculated according to the shape of the object.

Step 7. An image corresponding to the application is synthesized using the background image (processing indicated by 207 in FIG. 2, output image creation step).
Since the two images of backgroundFrame and currentFrame are high-resolution images, a composite image to be displayed is generated from the two images. For example, two images may be blended for the support of the board writing. Alternatively, only the background frame may be displayed. Alternatively, a precise contour of an object may be detected when a difference between two images is processed. Alternatively, only the object can be taken out and the background can all be a single color.

In the above image processing system, it is distinguished using the fact that there are some motions such as human beings in the explanation, but wearing special markers and artificially blinking instruments Only things can be selectively made into objects. For example, in a situation where a device that repeatedly blinks is worn, the value of the timer until it is determined whether or not it is stopped can be reduced, and the board can be updated quickly.

  FIG. 4 is a diagram illustrating a situation in which a monochrome box is placed on a desk. Here, when the box is lifted and held by hand, the box placed on the desk belongs to the background, and the held box belongs to the object covering the background. Therefore, in this embodiment, the change in the image of the box placed on the desk is immediately reflected in the background, and the image of the box held at that time is not reflected in the background. The flow until the above result is obtained is shown below.

  FIG. 5 is a diagram in which the background image immediately after the box on the desk is lifted by hand and the current image are superimposed. The box on the desk is translucent because the background image with the box on the desk is superimposed on the current image that is lifted by hand and is no longer on the desk. The reason why the arm and the box being lifted are translucent is the result of superimposing the background image without the box and the arm on the current image in which the box is lifted and held by hand.

  FIG. 6 is a diagram illustrating a state after a predetermined time has elapsed from a state where the box is lifted and held by hand. At this time, the box and arms were held as stationary as possible. The disappearance of the image of the box on the desk means that the image of the box on the desk has disappeared from the background image. On the other hand, a box held by hand remains translucent means that the background image does not have a held box image. For example, when used in a classroom, when a ruler is used to draw a line or a blackboard eraser erases characters, it means that both the ruler and the blackboard eraser can be treated as an object that hides the board. The feature of this method is that an object held by a person can be handled as a part of the person. With existing methods such as tracking an object area, it is difficult to distinguish between a box placed on a desk and a box held by hand. Alternatively, in the method using the characteristics of a person, when a person holds a ruler or the like, it is difficult to identify that it is an extension of the person.

  FIG. 7 shows that when the box is shaken from a state where the single color box is lifted by hand, the image processing system according to the present invention does not use the propagation process of the timer value until the update of the background image to the latest image is started. It is the figure which showed the result of having processed using the method which uses the division as a background, when there is no change even if the fixed time defined more than each management division passes. The reason why the outer part of the box is translucent is the result of being determined to have changed after a certain period of time. The reason why the center of the box is not translucent is that it has been determined that there has been no change after a certain period of time. In the above method, the outer part excluding the central part of the box is recognized as an object, but the center of the box is recognized as the background, and the entire box cannot be recognized as an object. In order to avoid this, it is conceivable to lengthen the detection time of the change. However, if the detection time becomes long, it takes a long time to recognize an object that has stopped moving as the background. Therefore, in this embodiment, a process is used in which the timer value of the management section that is determined to have changed after a certain time has passed is propagated to its surroundings.

  FIG. 8 shows the values of updateTimer for each management section (length 30 × width 40) after a certain time has elapsed since the box was lifted and held by hand. Similar to the case where the box is lifted and held by the hand (FIG. 6), the box and the arm are held as stationary as possible.

  In FIG. 8, the motion region (diamond symbol) is a portion of the motion region where it is determined that there is a difference between the current image and the previous image. In this embodiment, the operation area (diamond symbol) is a finger part. The finger portion has a large pixel spatial change and is easily moved slightly while holding an object and applying a force, so that a large color change is detected and determined as an operation region. The peripheral part of the operation area (diamond symbol) is an arm or a part of a holding box. In this embodiment, since it is held stationary, even if it moves strictly, it Is equal to or less than the detection threshold value, it is not determined as an operation region.

  In FIG. 8, the object region that is not timed out (star symbol) and the object region immediately before the time-out (diagonal cross symbol) are object regions that are determined to have a difference between the background image and the current image. There are two types of differences: the box portion on the desk that belonged to the background, and the raised box and arm portions in the latest image. Although the object (the box on the desk) does not exist in the current image in which the box on the desk disappears from the background, both are regarded as object areas in the sense that they existed in the background image.

There is a difference in the value of updateTimer, which is the time until update, between the object region that is not timed out (star symbol) and the object region immediately before the time-out (diagonal cross symbol). The non-timeout object area (star symbol) still has time to be captured in the background, and the object area just before the timeout (diagonal cross symbol) is about to expire.
The value of the updateTimer is propagated in the object region connected on the image as shown in Step 5 above. Then, since the value of the updateTimer is set to spread over the entire continuous region due to propagation, the region having the operation region (diamond symbol) somewhere, that is, the object region (star symbol) that is not timed out (here (Which corresponds to the area of the stored box) does not time out as a whole. On the other hand, the object area (diagonal cross symbol) immediately before the time-out corresponding to the place where the box on the desk disappears does not have the operation area (diamond symbol) inside or next to it. The whole will time out. Therefore, the object area (diagonal cross symbol) immediately before the timeout is a state immediately before the entire area is timed out due to the box being lifted and removed from the desk.

  In FIG. 8, a non-object area (cross symbol) that is not timed out is an area adjacent to the object area (star symbol) that is not timed out. By propagating the value of the updateTimer around the non-timeout object region (star symbol), the updateTimer of both the non-timeout non-object region (cross symbol) and the non-timeout object region (star symbol) The value is set. The fact that there is no non-object area (cross symbol) that is not time-out around the object area (diagonal cross symbol) immediately before the timeout indicates that the value of the updateTimer is not propagated because it is immediately before the timeout. .

In FIG. 8, a non-object area (dot symbol) that has timed out is an area that has timed out and is not determined to be an object area. The operation area (diamond symbol), the non-timeout object area (star symbol), and the non-timeout non-object area (cross symbol) are not timed out areas. In the time-out area, the background image is updated with the current image. When the update is completed, since the background image matches the current image, the area is not determined as the object area.

The present invention is applicable to image processing systems in various fields such as the robot industry, the medical field (surgical image), the entertainment industry and the like because the objects of the moving object and the stationary object to be distinguished are not limited.

101 arithmetic processing unit 102 storage device

Claims (5)

Image acquisition means for acquiring images continuously photographed at a fixed point;
A predetermined area dividing means for dividing a predetermined area into predetermined areas for the image acquired by the image acquisition means;
A still image creating means for detecting a stationary object from the image obtained by the image obtaining means and creating a still image by the stationary object;
An image recording means for recording a current image, which is the latest image at the time of processing, among the images acquired continuously, and an immediately preceding image acquired immediately before the current image;
An action area specifying means for specifying an action area by determining whether there is a change for each of the predetermined areas based on the current image and the immediately preceding image;
Object region specifying means for specifying an object region by determining a difference between the background image and the background image for each predetermined region based on the background image and the current image;
Region propagation means for reflecting a specific state of the motion region in the object region to a predetermined region belonging to the object region;
An image processing system comprising: an output image creating unit that creates an output image while synthesizing an entire image of the object region including the motion region with the still image. Furthermore, a timer management means for managing a timer value given for each predetermined area is provided,
The region propagation means is
Timer processing means for setting the timer value for the motion area specified by the motion area specifying means, and for resetting the timer for all areas other than the object area specified by the object area specifying means;
Timer value propagation means for correcting the timer value of the object region based on the timer value of the motion region,
The image processing system according to claim 1, wherein an entire image of the object area including the motion area is created based on a predetermined timer value by the timer management unit. The timer management means manages a timer value for a predetermined time set in advance,
The background image creating means identifies the predetermined area whose timer value during the predetermined time of the timer management means is less than a predetermined value as a background area, and creates a background image from the image in the background area. The image processing system according to claim 2, wherein: The image processing system according to claim 2, wherein the background image creating unit updates a background region every predetermined time from continuous images acquired by the image acquiring unit. An image processing method for processing by a computer,
An image acquisition step of acquiring images continuously photographed at a fixed point;
A predetermined area dividing step for dividing the predetermined area as a predetermined area for the image acquired by the image acquiring step;
An initial still image creating step of detecting a stationary object from the image acquired by the image acquiring step and creating a temporary still image by the stationary object;
An image recording step of recording a current image that is the latest image at the time of processing and an immediately preceding image acquired immediately before the current image among images that are continuously acquired;
An operation region specifying step of specifying an operation region by determining whether there is a change for each of the predetermined regions based on the current image and the immediately preceding image;
An object region specifying step of specifying an object region by determining a difference between the background image and the background image for each of the predetermined regions based on the background image and the current image;
A timer processing step for setting a timer for the motion area specified by the motion area specifying means, and for resetting the timer for all areas other than the object area specified by the object area specifying means;
A timer value propagation step of correcting the timer value of the object region based on the timer value of the motion region;
Identifying a predetermined area where the value of the timer is less than a predetermined value, and certifying the predetermined area as a background area; and
A background image update step of reflecting the background region in the temporary background image and updating the background image;
An image processing method comprising: an output image creation step of creating an output image while synthesizing the object region including the motion region with an updated background image.