JP2005109647A - Image processor and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processor and a program capable of performing the processing for removing an unnecessary object depending on a photographing object or purpose and photographing. <P>SOLUTION: An image photographed at a camera section 21 is recorded as image data 23 at an image recording section 22. A moving object removing section 24 judges an object to be removed in image data 23 based on previously specified conditions, and an image reproducing section 25 displays a composite image subjected to removing processing on a display 26. Consequently, an image from which an unnecessary image is removed is reproduced. Flexible removing processing can be carried out by specifying the conditions appropriately. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to an image processing apparatus that performs a processing process on a captured moving image, and more particularly to an image processing apparatus that removes an unnecessary object that is captured in a moving image.

When shooting landscapes with a video camera, you may see things you don't want to shoot. For example, in scenery and commemorative photography, it is desirable to avoid unnecessary objects such as cars and tourists coming and going on the road from appearing in photographs.
For this purpose, it is necessary to take measures such as timing the shooting, changing the shooting location, and cooperating so that the shooting time does not fall within the shooting range. Further, if the subject is close to the subject, it can be dealt with by these methods. However, it is difficult to prevent unnecessary objects from appearing when the subject is far away, such as in landscape photography.

In order to cope with this point, Patent Literature 1 captures a normal image and a three-dimensional image detected from a plurality of times by a three-dimensional image detection device capable of measuring a distance from a subject by emitting laser light. An image is generated by removing moving objects such as traffic objects from the data.
As a method for detecting a moving object, Patent Document 2 compares an image taken at time t with an image taken at the previous time (t-1) in time series, identifies the moving object, and moves the moving object. An image is generated using the image from which the image is removed as a background image.
JP 2001-119723 A (FIG. 3, [0021]) Japanese Patent Laid-Open No. 11-15882 (FIG. 2, [0018] to [0019])

  However, in the method described in Patent Document 1, it is necessary to provide a three-dimensional detection device in a photographing device, which makes the device large and effective. In the method disclosed in Patent Document 2, since the moving object and the background are determined only based on whether or not the object is moving, the method is applied as it is to a moving image obtained by shooting the moving object. It is not possible. Further, since only the determination criterion is whether or not the object is moving, it is difficult to determine the object that is really desired to be removed, and it is not possible to flexibly cope with the shooting situation.

  It is an object of the present invention to provide an image processing apparatus and a program that solve the above problems.

An image processing apparatus according to the present invention is premised on an image processing apparatus that performs processing on a moving image, and includes a determination unit, a removal unit, and a composite image generation unit in order to solve the above problems.
The determination means detects the moving object and its position by comparing the image data constituting the moving image with the image data in the moving image that is close to the image data in time series, and the detected moving object Whether or not to be removed is determined based on a designated condition.

The removing unit removes a region portion corresponding to the moving object from the image data including the moving object determined to be removed by the determining unit.
The composite image generation means performs interpolation on the area portion removed by the removal means, using image data that is close in time series to the image data from which the removal means removes the area portion corresponding to the moving object, A composite image without a moving object is generated.

With this configuration, the determination unit determines whether or not to remove the moving object based on the specified condition. Therefore, by setting an appropriate one for this condition depending on the shooting target and the shooting environment, the target moving object is determined. Can be removed.
By including, for example, a condition regarding the speed of the moving object in the designated condition, it is possible to remove a moving object such as a passing object that is immovable or imprinted in an image.

Further, by including, for example, a condition regarding the size of the moving object in the designated condition, an unnecessary moving object can be removed from an image obtained by photographing a fast moving object.
Further, the specified condition includes a condition regarding the number of frames in which the moving object is captured, so that it is possible to remove the captured image so as to cover the entire image.

The image processing apparatus further includes a condition determining unit that determines the specified condition based on an instruction from the operator, so that the operator uses the condition determining unit to set the specified condition in the shooting environment. It can be appropriately set based on the purpose and the like.
The image processing apparatus further includes a storage unit that stores image data of a moving image to be processed, and an image acquisition unit that acquires the image data input from the camera unit and stores the image data in the storage unit. The acquisition unit may store the image data in the storage unit and detect the moving object and the position of the image data stored in the storage unit. Thereby, the storage of the image data from the camera unit and the removal process of the moving object can be performed in parallel.

A recording area for recording data by further comprising means for storing the composite image generated by the composite image generation means in the storage means and deleting the original image data of the stored composite image from the storage means And the working storage area is not wasted.
Furthermore, as a second embodiment of the image processing apparatus of the present invention, image data constituting a moving image input from a plurality of camera units focused on different distances is acquired, and image data from each camera unit is obtained. The determination unit, the removal unit, and the composite image generation unit may generate a composite image, and the composite image generation unit may combine the composite image.

In the case of this configuration, even if there is a large difference in the distance to each object to be photographed, such as a landscape image, a composite image from which unnecessary moving objects are removed can be generated.
The present invention also includes within its scope a program that is executed in an apparatus that performs a processing process on a moving image input from the camera unit.

  According to the present invention, since the moving object is removed based on the designated condition, the removal process corresponding to the photographing object, the purpose, and the photographing can be performed by appropriately setting this condition.

An embodiment of the present invention will be described below with reference to the drawings.
FIG. 1 is a diagram illustrating a configuration example of an image processing apparatus according to the present embodiment.
FIG. 1 shows a configuration provided in a camera that captures an image. In FIG. 1, the image processing device includes a CPU 11, a RAM 12, a display device 13, an input device 14, a storage device 15, a communication device 16, a camera, and its camera. The control device 17 is included.

  The CPU 11 is a central processing unit, and when the CPU 11 executes a control program in the program storage area 151, each process of the present embodiment is realized. The RAM 12 is a memory area serving as a work area for the CPU 11, and is provided with a program execution area 121 where an execution program is executed by the CPU 11 and a work area 122 for executing the execution program. The display device 13 is a display, a viewfinder, or the like for displaying an image or showing various information to an operator. The input device 14 is an operation button or the like for an operator to input an operation instruction to the image processing apparatus. The storage device 15 includes a program storage area 151 that stores a control program in a nonvolatile memory and a data storage area that stores various data. The communication device 16 exchanges data with the outside, such as outputting a captured image to the outside of the device. The camera and its control device 17 are parts for taking images, and the data storage area 152 is stored for the taken images.

  FIG. 1 shows a case where the image processing apparatus is housed inside the housing of the camera apparatus. However, the image processing apparatus of the present embodiment is not limited to such a structure, and is configured as an apparatus independent of the camera apparatus. In addition, the captured image may be processed in real time by connecting to the camera device with a signal cable, or the captured image data may be received and processed from the camera device at a later date. In such a configuration, the display device 13 and the input device 14 are not used as camera devices such as a viewfinder and operation buttons, but are provided with dedicated devices such as a display, a pointing device, and a keyboard.

  FIG. 2 is a block diagram showing a flow of processing by the image processing apparatus. In the figure, the image recording processing unit 22, the moving object removal processing device 24, and the image reproduction processing unit 25 are realized by the CPU 11 executing the control program in the program storage area 151, but realized by dedicated hardware. You may do it.

  The camera unit 21 corresponds to the camera of FIG. 1 and its control device 17, and a captured image captured by the camera unit 21 is recorded as image data 23 (recorded image) by the image recording processing unit 22. Then, the image reproduction processing unit 25 displays a composite image in which the moving object removal processing unit 24 determines an object to be removed based on the conditions specified in advance for the recorded data 23 and performs processing. By displaying the image above, an image from which unnecessary moving objects are removed is reproduced. In addition, by operating the image recording processing unit 22, the moving object removal processing device 24, and the image reproduction processing unit 25 in parallel, it is possible to reproduce an image obtained by removing the moving object from the photographed image at the same time as photographing.

Next, a removal process performed by the moving object removal processing unit 24 to remove a moving object from a recorded image will be described.
FIG. 3 is a diagram for explaining the outline of the first embodiment of the moving object removal processing.
As shown in FIG. 5A, an example of processing when the moving object 32 crosses the shooting screen while shooting the landscape image 31 is shown.

In this case, when shooting is started from the image 31-1 as shown in FIG. 5B, the camera unit 21 records the shot images in the order of images 31-1, 31-2, 31-3,. Store and save as image 33.
Next, a composite image 35 is generated by removing the moving object from the recorded image 33. FIG. 3C shows an example of processing for the image 31-2.

  The image 31-1 which is an image at the start of recording is stored as the synthesized image 35 as it is because there is no previous recorded image. Since the next image 31-2 has an image 31-1 as an image recorded before that, it is compared with this image 31-1. As a result, it can be seen that the position of the object 32 has changed from the coordinates (x0, y0) to (x1, y1). From this, it is determined that this is an unnecessary moving object to be removed. The determination of whether or not the object is an unnecessary moving object is not performed only by moving the position, but is determined in consideration of conditions such as the size and moving speed of the moving object, as will be described later. Then, by appropriately setting these conditions, it is possible to perform a removal process suitable for the purpose, shooting target, shooting conditions, and the like.

  Since the coordinate position of the object 32 in the image 31-2 is (x1, y1), the partial extraction image 34 is partially extracted from the portion of the coordinate position (x1, y1) of the image 31-1. Then, the partially extracted image 34 is overwritten and combined with the coordinate position (x1, y1) of the moving object 32 in the image 31-2. Since the moving object 32 is removed from this image, it is stored as a composite image 35. Thereafter, the same processing is performed on the image after the image 31-3, so that the composite image becomes an image from which the moving object 32 is removed. After the moving object removal process is completed, the recorded image 33 that is the original before the moving object removal process is removed, so that the recording data recording area and the working storage area are not wasted. If necessary, the original recorded image 33 may be stored as it is as a live-action image.

  In the example of FIG. 3, a composite image 35 obtained by performing the removal process on the recorded image 33 is generated in advance, and the composite image 35 is reproduced to reproduce the image from which the moving object is removed. However, when the processing capability of the image processing apparatus is high, a synthesized image may not be generated in advance, but an image may be reproduced simultaneously while performing a recording process and a direct removal process on the recorded image 33. .

FIG. 4 is a diagram for explaining the outline of the second embodiment of the moving object removal process.
In the second embodiment, rain, snow, and the like reflected in the photographed image during snowfall are removed. The image processing apparatus used in the present embodiment is configured to require a high-sensitivity, high-image-quality, high-performance camera unit and image processing unit that can actually capture snow and rain particles, but in the following description, For the sake of simplification, description of each mounting technique is omitted, and only a basic concept is described.

In the following description, as shown in FIG. 4A, an example of processing when a landscape image is taken in a state where there is snow and rain is shown.
When shooting a landscape, a subject from a short distance to a long distance from the photographing point is a subject to be photographed, and there may be a large difference in distance to each subject to be photographed. The shooting screen 41-1 in FIG. 4A includes a mountain 42a and snow and rain particles 43a (shown by bold lines in the figure) at a short distance from the shooting point, a mountain 42b and snow and rain particles 43b at a medium distance. (Shown by a solid line in the figure), there is a mountain 42c at a long distance and snow and rain particles 43c (shown by a dotted line in the figure).

  In the second embodiment, a plurality of camera units 21 focused at different distances are installed at intervals within a range where snow and rain can be recognized, and images taken by the camera units 21 are combined. In this example, it is assumed that three camera units 21 having a focal length adjusted to a short distance, a middle distance, and a long distance are installed, and images captured by the cameras having the respective focal distances are shown in FIG. The short distance image 41a-2, the middle distance image 41b-2, and the long distance image 41c-2 are obtained.

  When the moving object removal processing shown in the first embodiment is performed on each of the images 41a-2, 41b-2, and 41c-2, the particles 43a, 43b, and 43c appearing on the image are removed. Images 41a-3, 41b-3, and 41c-3 shown in FIG. In these images 41a-3, 41b-3 and 41c-3, 43c of unnecessary snow and rain particles 43a and 43b are removed, and only the mountain 42a is shown.

By combining these three images 41a-3, 41b-3, and 41c-3 into one image 41-4, an image from which snow and rain has been removed is generated.
This composition is applied to images 41a-3, 41b-3, and 41c-3 obtained by performing moving object removal processing on images taken by a long-distance camera, a middle-distance camera, and a short-distance camera. It is determined whether the image corresponding to the camera is clearer, and the image corresponding to the camera determined to be clearer is preferentially combined. In addition, in a portion where it cannot be determined which image corresponding to which camera is clearer, the images of the respective cameras are synthesized at the same ratio to generate and record the synthesized image data for reproduction.

In the present embodiment, a condition as to whether or not a moving object in a captured image is to be removed is set in advance, and the moving object removal processing unit 24 performs a moving object removal process based on this set value.
FIG. 5 is a diagram illustrating a configuration example of the moving object removal condition registration table.
In the moving object removal condition registration table, conditions of moving objects to be removed are registered in a table configured in the data storage area 152 of the storage device 15, and conditions selected in the removal condition registration process described later from among these are registered. Based on the above, the moving object removal processing unit 24 performs a moving object removal process.

  In the table shown in the figure, as a moving object removal condition, a speed condition indicating a movement rate per second with respect to the entire image distance, a size condition indicating a ratio of the occupied area in the entire screen size, and the number of captured frames (photographed) Time) conditions are stored. For example, as the removal condition of condition number 1, the speed condition is 10% or less, the size condition is 0.5% or less, and the frame number condition is undefined (not used as a condition).

  As this removal condition, for example, by setting a speed condition, it is possible to remove a moving object from a motionless (slow) image such as a landscape. For example, when taking a commemorative photo at a tourist spot with a lot of people, by setting the speed of the moving object appropriately as a removal condition, only the target person and background are copied without copying the person passing in front of the camera A composite image can be generated.

  Alternatively, an unnecessary object can be removed from a moving (fast) image by setting, for example, a size condition as the removal condition. For example, by appropriately setting the size of the object to be removed as the removal condition, it is possible to cope with a case where the movement of the shooting target such as sports shooting is large and the deletion target cannot be defined by the speed.

Or, as this removal condition, for example, by setting the condition of the number of captured frames (captured time), the bag or passerby's back appears in front of the camera lens so as to cover the entire image. Even in this case, these can be removed as unnecessary objects.
FIG. 6 is a flowchart illustrating a process when the conditions for removal in the moving object removal process are set in the image processing apparatus. The processing in FIG. 7 is realized by the CPU 11 executing a control program in the storage device 15 using the RAM 12 as a work memory.

  When the removal condition registration process is started, first, a menu is displayed on the display device 13 as step S1, "new registration" for registering a new condition to the operator, and "register deletion" for deleting the registered condition. And “Exit” for ending the removal condition registration process.

  When “new registration” is selected in step S1, the operator is caused to input conditions for removing the moving object in step S2. This condition includes, for example, the speed / size of the moving object to be removed, and the lower limit / upper limit value of the number of frames in which the object is captured (the time during which the object is captured). Guidance is displayed on the display device 13. The operator inputs these values. In step S3, the removal condition input in step S2 is newly registered in the moving object removal condition registration table as shown in FIG. 5, and the process returns to step S1.

  When “register deletion” is selected in step S1, the removal condition currently registered with reference to the moving object removal condition registration table is displayed on the display device 13 and deleted to the operator in step S4. Specify a condition number. In step S5, the removal condition of the condition number specified in step S4 is deleted from the moving object removal condition registration table, and the process returns to step S1.

In step S1, when “end” is selected, the removal condition registration process is ended.
Next, an operation example during image recording processing, moving object removal processing, and reproduction processing by the image processing apparatus of the present embodiment will be described.
FIG. 7 is a diagram illustrating recording / playback condition setting data that is set and stored in the data storage area 152 of the image processing apparatus before the recording process, the moving object removal process, and the playback process are started.

The recording / playback condition setting data is data for setting various operations in the recording process, the moving object removal process, and the playback process, and is set in advance by the operator before starting these processes.
In the figure, in the recording / playback condition setting data, real-time playback designation, pre-removal data storage, post-removal data storage and removal conditions are set.

  The real-time playback designation is a designation of whether or not to perform the recording process and the real-time playback simultaneously with the playback process. The real-time playback designation is performed when ON, and is not performed when OFF. When the real-time playback designation is ON, priority is given to real-time recording and playback. Therefore, if the moving object removal process cannot keep up with the playback process, playback is performed using image data that has not been subjected to the moving object removal process. Therefore, real-time reproduction can be maintained, but removal of moving objects is not guaranteed. Also, when the real-time playback designation is OFF, priority is given to the moving object removal process. If the moving object removal process is not in time for the playback process, it waits for the moving object removal process to be completed, and the processed image is processed. Playback by. Therefore, in this case, the real-time reproduction is not guaranteed, but a reproduced image from which unnecessary moving objects are removed is output.

  Pre-removal data storage and post-removal data storage are specified by specifying whether to save the image before the moving object removal process and the image after the moving object removal process after the moving object removal process is performed. Saves and does not save when OFF. When these are ON, image data that is no longer needed at the same time as the processing is deleted, so that the recording data recording area and the work storage area are not wasted.

The removal condition is a designation of a condition used for determining whether or not to perform removal in the moving object removal process, and corresponds to the condition number shown in FIG.
In recording processing, moving object removal processing, and playback processing, which will be described later, processing is performed based on the set values of the recording / playback condition setting data.

  FIG. 8 is a flowchart showing the operation process in the recording process. The processing in FIG. 6 is performed by the image recording processing unit 22 and is realized by the CPU 11 executing a control program in the storage device 15 using the RAM 12 as a work memory. Further, in the following description, it is assumed that the processing in the figure and the moving object removal processing and reproduction processing described later are operating in parallel.

When the recording process is started, the recording current frame position is initialized to the first frame in step S11. This recording current frame position is a pointer that points to the currently recorded frame.
When the initialization process is completed, the recording process is repeated by repeating the processes of steps S12 to S15 until the recording process is completed (step S12, N).

  In step S13, image data for one frame is read from the camera unit 21, and recorded in the data storage area 152 as pre-removal image data at the recording current frame position. Subsequently, as step S14, the elapsed time (or current time) from the start of recording is recorded corresponding to the image data stored in step S13, and the removed flag is set to OFF.

Then, the current recording frame position is updated by one (step S15), and the process returns to step S12 to move to the process for the image data of the next frame.
Thereafter, the processes in steps S12 to S15 are repeated until the recording process is completed. When there is an instruction to end the recording by the operator or no image data is input from the camera unit 21 (step S12, Y), the process ends. .

FIG. 9 is a flowchart showing an operation process in the moving object removal process. The processing in FIG. 4 is performed by the moving object removal processing unit 24, and is realized by the CPU 11 executing a control program in the storage device 15 using the RAM 12 as a work memory.
When this processing is started, first, as an initialization process, the removal current frame position that points to the processing target frame is set to the first (first in time series of images to be processed) frame set (step S21). ). Thereafter, the processing of steps S22 to S35 is performed until the removal processing is completed (step S22, N).

  In step S23, it is determined whether or not the image data of the frame at the position before and after the frame pointed to by the removal current frame position is in the data storage area 152. If not (step S23, N), a waiting process is performed as step S24. And wait for the image data to be recorded by the recording process.

  If there is image data at the front and rear positions in step S23 (step S23, Y), it is determined in step S23 whether the moving object has been removed from the image data at the removal current frame position. This determination is made by a child that refers to the removed flag stored in association with the image data. If it is determined that it has been removed (step S25, Y), the process proceeds to step S36, and the process for the next frame is performed.

  If it is determined in step S25 that the image data at the removed current frame position has not been removed from the removed flag (step S25, N), the image data at the removed current frame position is replaced with temporally preceding and succeeding image data as step S26. By comparison, all the region portions where the image data is shifted by the same number of dots in the same direction are extracted.

  In step S27, it is determined whether or not there is an undetermined area among the areas extracted in step S26. If there is an undetermined area (step S27, Y), an undetermined area portion is set as 1 in step S28. The ratio of the size of the area portion to the entire area of the image data and the movement speed ratio to the distance of the entire image data are calculated. Then, referring to the moving object removal condition registration table, these are compared with each condition of the condition number set in the recording / playback condition setting data.

If there is a matching condition (step S29, Y), the selected area portion of the image data is erased as step S30. If there is no matching condition (step S29, N), the process is performed without erasing. Is returned to step S27.
If all the areas are processed and there is no undetermined area (step S27, N), then, as step S31, it is determined whether or not there is an uninterpolated area portion among those erased in step S30. to decide. If there is an uninterpolated area portion (step S31, Y), search for other image data that can interpolate the erased area portion in step S32, starting from the one closest in time to the removal current frame position. . If there is interpolable image data, the interpolable area portion is copied and combined with the image data at the removed current frame position (step S33), and the process returns to step S31.

  Interpolation processing is performed for all areas erased in step S30. If it is determined in step S31 that there is no uninterpolated area (step S31, N), the setting value of the recording / playback condition setting data is set as step S34. If there is no save designation before removal (step S35, N), after deleting the pre-removal image data at the removal current frame position as step S36, if there is a save designation, the process proceeds to step S36 without being erased. Transfer.

  In step S36, the post-update process for one removal current frame position is returned to step S22, and the process for the next frame is performed. If it is determined in step S22 that all the frames are processed and the removal process is finished (step S22, Y), this process is finished.

FIG. 10 is a flowchart illustrating an operation process in the moving object removal process when the moving object removal process according to the second embodiment described with reference to FIG. 4 is performed.
The processing example shown in the figure will be described by taking as an example a case of shooting using three camera units 21 for short-distance shooting, medium-distance shooting, and long-distance shooting.

  When this process is started, the moving object removal process shown in FIG. 9 is performed on the image data obtained by the short-distance shooting camera unit 21 as step S41, and the image obtained by the middle-distance shooting camera unit 21 is obtained as step S42. The moving object removal process shown in FIG. 9 is performed on the data, and the moving object removal process shown in FIG. 9 is performed on the image data from the long-distance shooting camera unit 21 in step S43. The processes in steps S41, S42, and S43 may be performed in parallel.

Then, as step S44, a portion that has been captured more clearly is extracted from the composite image of the images captured by the short-distance, medium-distance, and long-distance camera units 21 from which the moving object has been removed in steps S41, S42, and S43. After the images are combined, the processing is terminated.
FIG. 11 is a configuration example of the recording data 23 stored in the storage device 15 in the recording process and the moving object removal process.

In the figure, the image data before removal corresponds to the recorded image 33 in FIG. 3, and the image data after removal corresponds to the composite image 35.
In the drawing, the pre-removal image data is stored with frame numbers assigned in time series in frame units, and the temporal position of all the frames in the image is stored as the elapsed time.

  The pre-removal image data is associated with a removed flag that indicates whether or not the processing process has been performed by the moving object removal process. When the removed flag is ON, the image data of the frame is processed, and the image data from which the moving object after processing is removed is the image data before removal as post-removal image data. It is stored in association. For example, in relation to the frame number 58, the elapsed time “00: 00: 01: 56” from the beginning in the video in which the image was taken, the image data 57A before removal, the image data 57B after removal, and the removal completed A flag (OFF) is stored.

  The recording current frame position, the removal current frame position, and the reproduction current frame position to be described later point to the image data based on the frame number of the recording data 23, and the recording current frame position is the position of the frame where the current recording is performed, the removal The current frame position points to the position of the frame to be processed in the moving object removal process, and the reproduction current frame position points to the position of the frame read in the reproduction process.

FIG. 12 is a configuration example of the recording data 23 stored in the storage device 15 in the recording process and the moving object removal process when the moving object removal process according to the second embodiment described with reference to FIG. 4 is performed.
In the second embodiment, since images taken by a plurality of camera units 21 are used, the recorded data 23 is stored with images taken by the respective camera units 21 associated with each other by frame numbers. In the example of FIG. 12, image data from the long-distance shooting camera unit 21, image data from the middle-distance shooting camera unit 21, and image data from the short-distance shooting camera unit 21 at the same time are obtained by the moving object removal process. Along with image data and a removed flag (not shown), a frame number is added in time series and stored.

In the second embodiment, the images from these camera units 21 are further combined to generate image data for reproduction.
FIG. 13 is a flowchart showing an operation process in the image processing apparatus during the reproduction process. The processing in FIG. 4 is performed by the image reproduction processing unit 25, and is realized by the CPU 11 executing a control program in the storage device 15 using the RAM 12 as a work memory.

  When this process is started, the playback current frame position that points to the frame that is the target of the playback process as the initialization process is set to the first (first in the time series of images to be processed) frame set ( Step S51). Thereafter, the reproduction process of steps S52 to S62 is performed until the reproduction process is completed (step S52, N).

  In step S53 as a reproduction process, it is determined whether there is image data of a frame at the position of the reproduction current frame. If the recording process or moving object removal process for the image data of the frame is not completed and there is no image data (step S53, N), a post-process that performs a specific time waiting process to complete the process as step S54 Is returned to step S53.

  If there is image data of the frame at the position of the reproduction current frame in step S53 (step S53, Y), the removal flag is checked in step S55, and whether or not the moving object has been removed by the moving object removal processing is checked in the image data. to decide. If it has been removed (step S55, Y), the image data after removal of the reproduction current position is read in step S58. If it has not been removed (step S55, N), the recording / playback condition setting data is checked as step S56. If the real playback designation is not set to ON (step S56, N), playback by moving object removal processing is performed. A wait process is performed in step S59 so that the process for the current frame is completed, and then the process returns to step S55. If the real reproduction designation is set to ON in step S56 (step S56, Y), image data that has not been subjected to the reproduction current frame position removal process is read in step S56. In step S60, the image data read in step S57 or S58 is displayed on the display device to reproduce the image.

  Next, in step S61, the recording / playback condition setting data is checked to determine whether there is a data storage designation after removal, and if it is OFF (step S61, N), the storage current is not saved. Erase the image data after removing the position. If the recording / playback condition setting data is set to specify storage in step S61, the image data after removal is stored, so the process proceeds to step S63 without performing the deletion process.

  In step S63, the reproduction current frame position is updated by one, the process proceeds to step S52, and the process for the next frame is performed. When the processing for all the frames is completed or when there is an operation input for instructing the end of the reproduction processing from the operator (step S52, Y), this processing ends.

As described above, in the image processing apparatus according to the present embodiment, when recording a landscape or the like, even if an object such as a person or a car that is not to be recorded enters the frame of the camera, the object is reflected in the image and a rank image can be obtained. it can.
Also, moving objects are removed based on specified conditions. For example, large and slow objects can be removed from motionless (slow) images such as landscapes, small objects can be removed from moving images such as sports, and photography. It is possible to easily obtain an image from which unnecessary objects are removed according to the object and purpose.

In addition, if the image from this device is output as an image on the observation deck or the screen of the observation floor, or output to the observation mirror, visibility can be improved even during snowfall, so the number of visitors during snowfall is reduced. And, in turn, may reduce revenue decline.
In the above example, the captured image is handled and stored as image data in units of frames. However, a configuration may be used in which the captured image is handled and stored as compressed data. At this time, each image data constituting the video may be processed as difference data having only a changed portion as a difference.

It is a figure which shows the structural example of the image processing apparatus in this embodiment. It is a functional block diagram which shows the flow of a process by an image processing apparatus. It is a figure explaining the outline | summary of 1st Embodiment of the removal process of a moving object. It is a figure explaining the outline | summary of 2nd Embodiment of the removal process of a moving object. It is a figure which shows the structural example of a moving object removal condition registration table. It is a flowchart which shows the process at the time of setting the conditions for removing by a moving object removal process to an image processing apparatus. It is a figure which shows recording / playback condition setting data. It is a flowchart which shows the operation | movement process in a video recording process. It is a flowchart which shows the operation | movement process in a moving object removal process. It is a flowchart which shows the operation | movement process in the moving object removal process in 2nd Embodiment. It is a figure which shows the structural example of video recording data. It is a figure which shows the structural example of the video recording data at the time of performing the removal process of the moving object of 2nd Embodiment. It is a flowchart which shows the operation | movement process in the image processing apparatus at the time of reproduction | regeneration processing.

Explanation of symbols

11 CPU
12 RAM
DESCRIPTION OF SYMBOLS 13 Display apparatus 14 Input apparatus 15 Storage apparatus 16 Communication apparatus 17 Camera & control part 121 Program execution area 122 Work area 151 Program storage area 152 Data storage area 21 Camera part 22 Image recording process part 23 Image data 24 Moving object removal process part 25 Image data reproduction processing unit 26 Display processing unit

Claims (14)

In an image processing apparatus that performs processing on a moving image, the moving object and its position are compared by comparing the image data constituting the moving image with the image data in the moving image that is close in time series to the image data. Determining means for determining whether or not to remove the detected moving object based on a designated condition;
Removing means for removing a region portion corresponding to the moving object from the image data of the moving object determined to be removed by the determining means;
The removal means performs interpolation on the area portion removed by the removal means using image data that is close in time series to the image data from which the area corresponding to the moving object is removed, and the moving object is not captured. An image processing apparatus comprising: composite image generation means for generating a composite image.   The storage means for storing image data of a moving image to be processed is further provided, and the determination means detects the moving object and the position of the image data stored in the storage means. The image processing apparatus according to 1.   The image processing apparatus further includes a storage unit that stores image data of a moving image to be processed, and an image acquisition unit that acquires the image data input from the camera unit and stores the image data in the storage unit. 2. The image processing apparatus according to claim 1, wherein means stores image data in the storage means and detects the moving object and the position of the image data stored in the storage means.   4. The apparatus according to claim 2, further comprising means for deleting the original image data of the stored composite image from the storage means simultaneously with storing the composite image generated by the composite image generation means in the storage means. An image processing apparatus according to 1.   5. The reproduction apparatus according to claim 2, further comprising a reproduction unit configured to reproduce a moving image based on the composite image generated by the composite image generation unit in parallel with the acquisition of the image data by the image acquisition unit. The image processing apparatus according to item 1.   The image processing apparatus according to claim 5, wherein the determination unit, the removal unit, and the composite image generation unit operate in parallel with the image acquisition unit and the reproduction unit.   The image processing apparatus according to claim 1, further comprising a condition determining unit that determines the specified condition based on an instruction from an operator.   The image processing apparatus according to claim 1, wherein the specified condition includes a condition regarding a speed of the moving object.   The image processing apparatus according to claim 1, wherein the specified condition includes a condition regarding a size of the moving object.   The image processing apparatus according to claim 1, wherein the specified condition includes a condition regarding the number of frames in which the moving object is captured. Image acquisition means for acquiring image data constituting moving images input from a plurality of camera units focused on different distances;
The image data acquired from each of the plurality of camera units is compared with time-sequential image data to detect a moving object and its position, and whether or not the detected moving object should be removed is specified. Determining means for determining based on conditions;
Removing means for removing a region portion corresponding to the moving object from the image data of the moving object determined to be removed by the determining means;
Using the image data that is close in time series to the image data, interpolating the area removed by the removing means, and generating a composite image in which the moving object is not shown;
An image processing apparatus comprising: a synthesis unit configured to synthesize the synthesized image obtained from the synthesized image generation unit corresponding to each of the plurality of camera units.   12. The image processing apparatus according to claim 11, wherein the synthesizing unit preferentially synthesizes a clearer synthesized image among the synthesized images obtained corresponding to each of the plurality of camera units. . A program that is executed in a device that performs processing on a moving image,
Whether the moving object and its position should be detected by comparing the image data constituting the moving image with the image data in the moving image that is close to the image data in time series, and whether the detected moving object should be removed A function of determining whether or not based on a specified condition;
A function of removing a region corresponding to a moving object from image data of the moving object determined to be removed;
Using image data that is time-sequentially similar to image data from which the region corresponding to the moving object is removed, interpolation is performed on the region that has been removed by the removing means, and image data that does not include the moving object is obtained. A program that realizes the functions to be generated. A program that is executed in a device that performs processing on a moving image,
A function of acquiring image data constituting moving images input from a plurality of camera units focused on different distances;
The image data acquired from each of the plurality of camera units is compared with time-sequential image data to detect a moving object and its position, and whether or not the detected moving object should be removed is specified. A function to determine based on conditions;
A function of removing a region portion corresponding to the moving object from the image data of the moving object determined to be removed;
A function of performing interpolation on the removed region portion using image data close in time series to the image data, and generating a composite image in which the moving object is not captured;
A program for realizing a function of combining the composite images obtained corresponding to each of the plurality of camera units.

Images