JP2020154732A - Image processor, image processing method and program - Google Patents

Abstract

To create a natural interpolation frame image in a frame interpolation when a part of a frame image moves between continuous two frames.SOLUTION: When creating an interpolation frame for interpolating a clearance between fore-and-aft frames being continuous two frames, a region including an object having movement between the fore-and-aft frames is extracted from each of the fore-and-aft frames, a moving object image is acquired, a moving direction of the region is determined on the basis of respective regions of the fore-and-aft frames, permeability being a permeation degree of a background in the moving object image which is extracted from each of the fore-and-aft frames is calculated, creation means for creating an interpolation object is decided on the basis of the moving object image which is extracted from each of the fore-and-aft frames, the permeability and a prescribed threshold, the interpolation object is created by the decided creation means, a background image being a background of the interpolation object is created from either of the fore-and-aft frames, and the interpolation frame is created from the interpolation object and the background image.SELECTED DRAWING: Figure 6

Description

The present invention relates to an image processing apparatus, an image processing method and a program.

Conventionally, a so-called frame interpolation technique in which a motion vector between two consecutive frames in a moving image is detected for each block, an interpolation frame is created using the motion vector, and the frame rate is increased by inserting the motion vector between the original frames. It has been known.

As the above technique, for example, there is a technique disclosed in Patent Document 1. In the document, when creating an interpolation frame between two frames, the motion vector between the two frames is calculated, and an image when the motion vector is halved for each of the front and rear frames is created and averaged. A device for creating an interpolation frame is described.

Further, in Patent Document 2, a moving body image is cut out into a rectangle from a frame image at a certain time point, a background image is created from the cut out frame image, and the movement of the moving body at another time point is predicted on the created background image. Then, an image processing device for obtaining an interpolated frame image that supplements between two frames by overwriting the clipped moving body image is described.

However, in the case of frame interpolation by the conventional technique as disclosed in the above-mentioned documents, for example, the screen of a PC (Personal Computer) is shared at a remote place, and the mouse pointer on the receiving side while suppressing the amount of transmitted / received data. A challenge remains when trying to make the movement of the mouse look smooth. To smooth the movement of the mouse pointer, cut out the part including the mouse pointer from the frame image used as the material of the interpolation frame image to make the mouse pointer image, use the cut rest as the background image, and use the mouse pointer image as the background image. The image that is moved and superimposed is made into an interpolated frame image. In this case, if the mouse pointer image contains characters or parts of the image or can be seen through, they will appear to move momentarily with the mouse pointer and then disappear, resulting in unnatural frame interpolation. turn into.

The present invention has been made in view of the above, and an object of the present invention is to generate a natural interpolated frame image in frame interpolation when a part of a frame image moves between two consecutive frames.

In order to solve the above-mentioned problems and achieve the object, the present invention is an image processing device that generates an interpolation frame that complements the space between the front and rear frames, which are two consecutive frames of the input moving object. The movement direction of the area is determined based on the object extraction unit that extracts the area including the object that moves between the frames from each of the front and rear frames to obtain the motion object image and the area of each of the front and rear frames. Object motion determination unit, a transparency calculation unit that calculates transparency, which is the degree of transparency of the background in the motion object image extracted from each of the front and rear frames, and the motion object extracted from each of the front and rear frames. The interpolation object is generated by an interpolation object generation means determination unit that determines a generation means for generating an interpolation object based on an image, the transparency, and a predetermined threshold, and a generation means determined by the interpolation object generation means determination unit. An interpolating object generation unit, an object peripheral image extraction unit that generates a background image as a background of the interpolation object from any of the preceding and following frames, and an image composition that generates the interpolation frame from the interpolation object and the background image. It has a part and.

According to the present invention, it is possible to generate a natural interpolated frame image in frame interpolation when a part of the frame image moves between two consecutive frames.

FIG. 1 is a configuration diagram of the image processing system 1000 according to the first embodiment. FIG. 2-1 is a diagram showing an example of the front frame. FIG. 2-2 is a diagram showing an example of an interpolation frame. FIG. 2-3 is a diagram showing an example of an interpolation frame. FIG. 3 is a hardware configuration diagram of the user device 100 according to the first embodiment. FIG. 4 is a functional block diagram of the user device 100 according to the first embodiment. FIG. 5 is a diagram illustrating the function of the frame interpolation unit 12. FIG. 6 is a diagram illustrating a difference in a method of generating an interpolation frame. FIG. 7 is a flowchart showing an example of the flow of processing in which the frame interpolation unit 12 generates an interpolation frame. FIG. 8 is a flowchart showing an example of a processing flow in which the frame interpolation unit 12 determines the interpolation object generation means. FIG. 9 is a diagram illustrating a specific example when calculating the standard deviation S1 (or S1') from the motion object image (pointer cut-out image).

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In each drawing for explaining the embodiment of the present invention, components such as members and components having the same function or shape are once described by giving the same reference numerals as much as possible. The description will be omitted.

FIG. 1 is a diagram showing a configuration example of the image processing system 1000 according to the first embodiment. As shown in FIG. 1, the image processing system 1000 includes user devices 100a to 100e, which are image processing devices, a server 200, a projection device 201, and a network 300. The user device is connected to another user device 100 and the server 200 via the network 300.

The user devices 100a to 100e can receive moving image data from other user devices 100a to 100e via the network 300, perform frame interpolation processing by an image processing method as described later, and display the moving image after the interpolation processing. It is a device. Further, the user devices 100a to 100e are devices capable of performing frame interpolation processing on the moving image data received from the server 200 as described later and displaying the moving image after the interpolation processing.

The user devices 100a and 100b are, for example, a PC (Personal Computer), a tablet PC, a smartphone, or the like. A projection device 201 is connected to the user device 100b.

The projection device 201 is, for example, a projector, which receives a signal output from the user device 100b and projects an image. The projection device 201 may be directly connected to the network 300 to receive moving image data and project an image.

The user device 100c is a display capable of displaying moving image data received via the network 300.

The user device 100d is a network conference device including a display and an audio device, capable of reproducing moving image data and audio data received via the network 300.

The user device 100e is an electronic blackboard provided with a display capable of displaying moving image data received via the network 300.

Hereinafter, when the user devices 100a to 100e are not distinguished from each other, they may be referred to as the user device 100.

The server 200 has a function of transmitting moving image data to the user device 100 via the network 300. For example, it is a multipoint control device for video conferencing, and receives video data taken by a camera from a certain user device 100 and video data on a PC screen on which office documents and the like are operated, and the video data is transmitted via a network 300. Can be transmitted to another user device 100.

As described above, the user devices 100a to 100e can display and play back at least the video data received via the network 300, but the user devices 100a to 100e themselves distribute the video data and the audio data via the network 300 or via the network 300 and the server 200. It may have a function to perform.

Here, in the image processing system 1000, since the user devices 100a and 100b have a PC function, the screen operated by the PC using the server 200 and the network 300 can be used as moving image data for another user device. Can be sent. That is, the screens operated by the PC on the user devices 100a and 100b can be shared with other devices as moving image data. This state may be particularly referred to as a PC screen sharing mode.

In FIG. 1, the user devices 100a and 100b can operate as a side that provides a screen to be shared in the PC screen sharing mode, and can also operate as a side that receives the screen. On the other hand, the user devices 100c, 100d, 100e in FIG. 1 operate as a receiving side. However, the present invention is not limited to this, and even user devices 100c, 100d, and 100e can operate as both a providing side and a receiving side by connecting to a PC using their respective I / Fs (Interfaces). It becomes.

This embodiment is an example of image processing in the case of performing frame interpolation so that an image quality abnormality does not occur when transmitting at a low frame rate from the transmitting side and increasing the frame rate on the receiving side in the PC screen sharing mode. is there. In the present embodiment, an example of an object that moves with respect to a stationary background is used as a mouse pointer, and an image obtained by extracting an area separated by a rectangle surrounding the mouse pointer is used as an example of a moving object image.

In the present embodiment, the movement of the mouse pointer is detected between two consecutive frames in the moving image, and an image including the mouse pointer extracted from each frame image (an example of a pointer cut image and a motion object image) is used for interpolation. Generate a frame.

FIG. 2-1 is a diagram showing an example of the front frame. 2-2 and 2-3 are diagrams showing an example of an interpolation frame. Conventionally, when generating an interpolation frame, if the background (characters, etc.) is transparent or included in the pointer cut image (see Fig. 2-1), they are displayed at different positions in the interpolation frame. (See FIGS. 2-2 and 2-3), resulting in unnatural frame interpolation (image quality abnormality), which is inconvenient. In the present embodiment, in order to reduce the above-mentioned inconveniences and perform natural frame interpolation, the acquisition source of the pointer cut image can be selected from a plurality of sources, and the more suitable one is adopted for the interpolation frame.

FIG. 3 is a hardware configuration diagram of the user device 100 according to the first embodiment. The user device 100 includes a CPU (Central Processing Unit) 101, a display device 102, a ROM (Read Only Memory) 103, a RAM (Random Access Memory) 104, a communication I / F 105, and a storage device 106, which are connected to each other. It has an output I / F 107 and the like.

The program that realizes the processing in the user device 100 is stored in the ROM 103 or the storage device 106. The storage device 106 is, for example, a storage medium such as an HDD (Hard Disc Drive), which stores installed programs and stores necessary data. As another example, the storage device 106 may have an external storage medium attached to the user device 100.

The RAM 104 reads and stores the program from the ROM 103 or the storage device 106 when the program is instructed to start. The CPU 101 realizes the function related to the user device 100 according to the program stored in the RAM 104.

The display device 102 is a display device included in the user device 100, and may be, for example, a liquid crystal display, a touch panel display, an organic EL (Electro Luminescence), or a projection device such as a projector.

The communication I / F 105 is a wired or wireless interface for communicating with another user device 100, a server 200, or the like via the network 300.

The input / output I / F 107 is an interface for connecting to various input / output devices such as a USB (Universal Serial Bus) device, a hardware key, a status notification LED, and a liquid crystal display.

Then, in addition to the above-mentioned hardware configuration, the user device 100 includes hardware configurations specific to each device such as a camera, a scanner, a printer, a keyboard, a mouse, a microphone, a speaker, and an electronic pen.

The server 200 may also have the same hardware configuration as in FIG.

FIG. 4 is a functional block diagram of the user device 100 according to the first embodiment. The user device 100 includes a storage unit 11, a frame interpolation unit 12, and an image display unit 13. Each of these parts is realized by a process of causing the CPU 101 to execute one or more programs installed in the user device 100.

The storage unit 11 stores moving images sequentially input from the network 300 in chronological order for a predetermined number of frames by tracing back from the latest frame to a continuous past frame. The frame is, for example, color image data having a predetermined horizontal resolution and vertical resolution, one dot having three colors of RGB, and a color depth of each color being 8 bits. The number of bits of the color depth is not limited to 8 bits and may be further larger such as 16 bits. Further, the frame may be recorded in a data format of a luminance color difference signal such as YUV (YCbCr).

The frame interpolation unit 12 generates an interpolation frame to be inserted between the latest frame and the previous frame based on a plurality of input frames read from the storage unit 11. Here, the number of frames to be interpolated may be one or a plurality of frames. In addition, continuous frames may be referred to as N-1 frame, N frame, N + 1 frame, etc., where N is an integer.

Further, two consecutive input frames, for example, an N-1 frame and an N frame may be collectively referred to as a front-rear frame. The latest frame of the two consecutive input frames, that is, the N frame may be referred to as a rear frame, and the previous frame, that is, N-1 may be referred to as a front frame. At this time, if an interpolation frame is inserted between two consecutive input frames, the input frame immediately before the interpolation frame is the N-1 frame which is the front frame, and the input frame immediately after the interpolation frame is the rear frame. It is an N frame.

The image display unit 13 sequentially outputs or displays the moving image in which the interpolation frame is inserted between the input frames on the display device 102 or the projection device 201 included in the user device 100 after the interpolation processing.

FIG. 5 is a diagram illustrating the function of the frame interpolation unit 12. The frame interpolation unit 12 includes an object extraction unit 120, an object motion determination unit 121, an object transparency calculation unit 122, an interpolation object generation means determination unit 123, an interpolation object generation unit 124, an object peripheral image extraction unit 125, and an image composition unit 126. Including.

The object extraction unit 120 extracts the position information of the movement object and the image data of the movement object (movement object image, more specifically, the pointer cut image) from the plurality of frames input from the storage unit 11. In the present embodiment, two consecutive frames, which are the insertion targets of the interpolation frames, are used as the acquisition source of the motion object image. That is, in the present embodiment, the image including the mouse pointer is cut out not only from the previous frame but also from the subsequent frame in chronological order of two consecutive frames.

The object movement determination unit 121 determines whether or not there is movement of the movement object before and after the frame from the position information of the movement object obtained from the object extraction unit 120, and also determines the direction thereof.

The object transparency calculation unit 122 analyzes the motion object image obtained from the object extraction unit 120 and calculates the background transparency degree. The background see-through degree is an image feature amount such as a standard deviation.

The interpolation object generation means determination unit 123 determines the interpolation object generation means from the background transparency degree of each motion object obtained from the object transparency calculation unit 122 and a predetermined threshold value. Here, the interpolation object is a motion object used to generate an interpolation frame. For example, the interpolating object generation means determination unit 123 compares the degree of transparency of the background of a moving object with a predetermined threshold value, and if the threshold value is lower than the threshold value, the background is not transparent.

The interpolation object generation unit 124 interpolates using the movement direction determined by the object movement determination unit 121, the interpolation object generation means determined by the interpolation object generation means determination unit 123, and the movement object image obtained from the object extraction unit 120. Generate an object image.

More specifically, when the background of either the front frame or the rear frame is not transparent, the interpolation object generation unit 124 uses the motion object image whose background is not transparent as the interpolation object. Further, when the background is transparent in both the front frame and the rear frame, the interpolation object generation unit 124 corrects the background so as not to be conspicuous with respect to any of the motion object images, and uses it as the interpolation object.

Here, FIG. 6 is a diagram for explaining the difference in the method of generating the interpolation frame. In the present embodiment, the combination of the appearance of the motion object image (pointer cut image) of the front frame and the rear frame is considered by dividing into the following four types A to D.
A: The background is not transparent in either the front frame or the rear frame B: The background is transparent in the front frame C: The background is transparent in the rear frame D: The background is transparent in both the front frame and the rear frame An example of generating an interpolation frame in the above four cases A to D is as follows.

In the case of A (the background is not transparent in either the front frame or the rear frame), the image quality does not become abnormal regardless of which of the motion object images of the front frame and the rear frame is used.

In the case of B (the background can be seen through in the front frame), the background transparency of the front frame and the rear frame is compared, and the rear frame having a low transparency is used as the motion object (interpolation object) of the interpolation frame. Therefore, the background is not transparent in the interpolated object, and the image quality does not become abnormal. In this case, since the interpolation object is always generated from the previous frame in the past, the background is transparent in the interpolation object, resulting in abnormal image quality.

In the case of C (the background can be seen through in the rear frame), the background transparency of the front frame and the rear frame is compared, and the front frame having a low transparency is used as the interpolation object. Therefore, the background is not transparent in the interpolated object, and the image quality does not become abnormal. In this case, since the interpolation object is always generated from the previous frame in the conventional case, the background is not transparent in the interpolation object and the image quality does not become abnormal.

In the case of D (the background can be seen through in both the front frame and the rear frame), both the front frame and the rear frame exceed a predetermined threshold value. In this case, the area of the movement object is narrowed. This reduces the part where the background is transparent. In this case, since the interpolation object is always generated from the previous frame in the past, the background can be seen through in the interpolation object, resulting in abnormal image quality. That is, according to the present embodiment, the image quality abnormality can be reduced.

Returning to FIG. 5, the object peripheral image extraction unit 125 extracts an image (background) around the motion object in the frame (rear frame) immediately after the interpolation frame obtained from the storage unit 11.

The image synthesizing unit 126 generates an interpolation frame by synthesizing a frame (previous frame) immediately before the interpolation frame obtained from the storage unit 11 and an interpolation object obtained from the interpolation object generation unit 124. Then, the image synthesis unit 126 transmits the frame obtained from the storage unit 11 and the interpolation frame to the image display unit 13.

FIG. 7 is a flowchart showing an example of the flow of processing in which the frame interpolation unit 12 generates an interpolation frame. First, the frame interpolation unit 12 extracts the position information of the motion object and the motion object image from each of the plurality of frames input from the storage unit 11 as the object extraction unit 120 (object extraction step) (step S1). Further, the frame interpolation unit 12 extracts the object peripheral image in the frame (rear frame) immediately after the interpolation frame obtained from the storage unit 11 as the object peripheral image extraction unit 125 (object peripheral image extraction step) (step S2). ).

When the object extraction step (step S1) is completed, the frame interpolation unit 12 moves between the front frame and the rear frame as the object movement determination unit 121 based on the position information of the movement object obtained from the object extraction unit 120. The movement direction of the object is determined (object movement determination step) (step S3). Further, the frame interpolation unit 12 analyzes the motion object image obtained from the object extraction unit 120 as the object transparency calculation unit 122, and calculates the background transparency degree of the motion object (object background transparency calculation step) (step S4). ).

When the object background transparency calculation step (step S4) is completed, the frame interpolation unit 12 serves as the interpolation object generation means determination unit 123 from the background transparency of the motion object obtained from the object transparency calculation unit 122 and a predetermined threshold value. , The interpolating object generating means is determined (interpolating object generating means determining step) (step S5).

After completing the object movement determination step (step S3) and the interpolation object generation means determination step (step S5), the frame interpolation unit 12 interpolates with the movement direction determined by the object movement determination unit 121 as the interpolation object generation unit 124. Based on the object generation means determined by the object generation means determination unit 123, an interpolation object composed of the position information of the motion object in the interpolation frame and the motion object image in the interpolation frame is generated (interpolation object generation step) (step S6). ).

After completing the interpolation object generation step (step S6) and the object peripheral image extraction step (step S2), the frame interpolation unit 12 acts as the image composition unit 126 on the object peripheral image obtained from the object peripheral image extraction unit 125. The interpolation objects obtained from the generation unit 124 are combined to generate an interpolation frame (image composition step) (step S7).

Then, the image synthesizing unit 126 transmits the frame obtained from the accumulating unit 11 and the interpolating frame to the image display unit 13. In the object peripheral region extraction step, an example of using the rear frame has been described, but the front frame may be used, and other methods may be used without being limited to these.

FIG. 8 is a flowchart showing an example of a processing flow in which the frame interpolation unit 12 determines the interpolation object generation means.

First, the frame interpolation unit 12 calculates the standard deviation S1 of the previous frame object as the object transparency calculation unit 122, and passes the standard deviation S1 to the interpolation object generation means determination unit 123 (step S101).

Next, the frame interpolation unit 12 compares the predetermined threshold value Sth with the standard deviation S1 as the interpolation object generation means determination unit 123 (step S102). When the standard deviation S1 does not exceed the predetermined threshold value Sth (No in step S102), the frame interpolation unit 12 determines that the motion object of the previous frame is used as the interpolation object, and ends this process (step S103).

In step S102, when the standard deviation S1 exceeds a predetermined threshold value Sth (Yes in step S102), the frame interpolation unit 12 calculates the standard deviation S2 of the rear frame object as the object transparency calculation unit 122, and the standard deviation S2 is passed to the interpolation object generation means determination unit 123 (step S104).

Next, the frame interpolation unit 12 compares the predetermined threshold value Sth with the standard deviation S2 as the interpolation object generation means determination unit 123 (step S105). When the standard deviation S2 does not exceed the predetermined threshold value Sth (No in step S105), the frame interpolation unit 12 determines that the motion object of the rear frame is used as the interpolation object, and ends this process (step S106).

In step S105, when the standard deviation S2 exceeds a predetermined threshold value Sth (Yes in step S105), the frame interpolation unit 12 corrects the movement object image of the previous frame by cutting a certain width from the entire outer frame (step S107). .. The cutting width a (the number of pixels to be cut) at this time is arbitrarily determined.

Next, the frame interpolation unit 12 calculates the standard deviation S1'of the motion object image corrected in step S107 as the object transparency calculation unit 122, and passes the standard deviation S1'to the interpolation object generation means determination unit 123 (step). S108).

Next, the frame interpolation unit 12 compares the predetermined threshold value Sth with the standard deviation S1'as the interpolation object generation means determination unit 123 (step S109). When the standard deviation S1'does not exceed the predetermined threshold value Sth (No in step S109), the frame interpolation unit 12 determines that the motion object image of the previous frame corrected in step S107 is used as the interpolation object. The process ends (S110).

In step S109, when the standard deviation S1'exceeds a predetermined threshold value Sth (Yes in step S109), the frame interpolation unit 12 increases the cutting width a (the number of pixels to be cut) (S111) and corrects the previous frame object. (Step S107). The frame interpolation unit 12 repeats the processing loops of steps S107 to S109 and S111 until the standard deviation S1'is equal to or less than a predetermined threshold value Sth.

FIG. 9 is a diagram illustrating a specific example when calculating the standard deviation S1 (or S1') from the motion object image (pointer cut-out image). This figure shows a state in which steps S107, S108, S109, and S111 of FIG. 8 are repeatedly executed, and the pointer cut image is corrected until the standard deviation S1'of the motion object image falls below a predetermined threshold value Sth. ..

The broken line indicates the image area when calculating the standard deviation S1 (or S1'). When the initial standard deviation S1 exceeds the threshold value Sth, the frame interpolation unit 12 decides not to use the current motion object image as the interpolation object image, and corrects the outer peripheral portion of the motion object image.

Next, when the standard deviation S1'after cutting the object with the cutting width a exceeds the threshold value Sth, the frame interpolation unit 12 uses the current motion object image (after cutting with the cutting width a from the initial state) as an interpolation object. It will not be used as an image, and the outer circumference of the motion object image will be further trimmed. The frame interpolation unit 12 repeats the above modification until the standard deviation S1'is equal to or less than the threshold value Sth.

Then, when the standard deviation S1'after N corrections becomes equal to or less than the threshold value Sth, the frame interpolation unit 12 determines that the motion object image at that time is used as the interpolation object.

As described above, according to the present embodiment, the degree to which the background is transparent is calculated for the mouse pointer image (pointer cut image) cut from each of the front frame and the rear frame, and each background is transparent. Since the pointer cropped image to be used for the interpolation frame is determined from the degree and a predetermined threshold, it is possible to create an interpolation frame in which the background that can be seen through the translucent part of the mouse pointer is less noticeable. Can be reduced. Therefore, a natural interpolated frame image can be generated in frame interpolation when a part of the frame image moves between two consecutive frames.

The image processing device of the present embodiment includes a control device such as a CPU, a storage device such as a ROM (Read Only Memory) or RAM, an external storage device such as an HDD or a CD drive device, and a display device such as a display device. , Equipped with input devices such as a keyboard and mouse, and has a hardware configuration using a normal computer.

The program executed by the image processing apparatus of the present embodiment is a file in an installable format or an executable format on a computer such as a CD-ROM, a flexible disk (FD), a CD-R, or a DVD (Digital Versaille Disk). It is recorded and provided on a readable recording medium.

Further, the program executed by the image processing device of the present embodiment may be stored on a computer connected to a network such as the Internet and provided by downloading via the network. Further, the program executed by the image processing device of the present embodiment may be configured to be provided or distributed via a network such as the Internet.

Further, the program for performing the image processing of the present embodiment may be configured to be provided by incorporating it in a ROM or the like in advance.

The program executed by the image processing apparatus of the present embodiment includes the above-mentioned parts (object extraction unit 120, object motion determination unit 121, object transparency calculation unit 122, interpolation object generation means determination unit 123, and interpolation object generation unit 124. , Object peripheral image extraction unit 125, image composition unit 126), and as actual hardware, the CPU (processor) reads the ~ program from the storage medium and executes it, so that each part is mainly Loaded on the storage device, object extraction unit 120, object movement determination unit 121, object transparency calculation unit 122, interpolation object generation means determination unit 123, interpolation object generation unit 124, object peripheral image extraction unit 125, image composition unit 126. Is to be generated on the main memory.

11 ... Accumulation part,
12 ... Frame interpolation unit 13 ... Image display unit 100, 100a to 100e ... User device 101 ... CPU, 102 ... Display device, 103 ... ROM, 104 ... RAM,
105 ... Communication I / F, 106 ... Storage device, 107 ... Input / output I / F
120 ... Object extraction unit 121 ... Object movement determination unit 122 ... Object transparency calculation unit 123 ... Interpolation object generation means determination unit 124 ... Interpolation object generation unit 125 ... Object peripheral image extraction unit 126 ... Image composition unit 200 ... Server 201 ... Projection Device 300 ... Network 1000 ... Image processing system

Japanese Unexamined Patent Publication No. 3-263789 Japanese Unexamined Patent Publication No. 10-336599

Claims (6)

An image processing device that generates an interpolation frame that complements the space between the previous and next frames, which are two consecutive frames of the input video.
An object extraction unit that extracts an area including an object that moves between the front and rear frames from each of the front and rear frames to obtain a movement object image, and an object extraction unit.
An object movement determination unit that determines the movement direction of the area based on each of the areas of the front and rear frames,
A transparency calculation unit that calculates the transparency, which is the transparency of the background in the motion object image extracted from each of the front and rear frames, and
An interpolation object generation means determination unit that determines a generation means for generating an interpolation object based on the motion object image extracted from each of the front and rear frames, the transparency, and a predetermined threshold value.
An interpolation object generation unit that generates the interpolation object by the generation means determined by the interpolation object generation means determination unit,
An object peripheral image extraction unit that generates a background image as a background of the interpolating object from any of the preceding and following frames, and
An image compositing unit that generates the interpolation frame from the interpolation object and the background image, and
An image processing device comprising. The image processing apparatus according to claim 1, wherein the transmittance is a standard deviation indicating the degree of variation of the motion object image. When the transparency of the motion object image of each of the front and rear frames is higher than the predetermined threshold value, the interpolating object generation means determining unit determines the motion object image of the front frame and the motion object image of the rear frame. The image processing apparatus according to claim 1 or 2, wherein it is determined that the motion object image having the lower transparency is used for generating the interpolation object in the interpolation object generation unit. When the transparency of the motion object image of each of the front and rear frames is lower than the predetermined threshold value, the interpolating object generation means determining unit determines the motion object image of the front frame and the motion object image of the rear frame. The image processing apparatus according to any one of claims 1 to 3, wherein any one of the above is determined to be used for generating the interpolating object in the interpolating object generation unit. It is an image processing method that generates an interpolation frame that complements the space between the previous and next frames, which are two consecutive frames of the input video.
An object extraction step of extracting a region including an object that moves between the front and rear frames from each of the front and rear frames to obtain a movement object image, and
An object movement determination step of determining the movement direction of the region based on each of the regions of the front and rear frames,
A transparency calculation step for calculating the transparency, which is the transparency of the background in the motion object image extracted from each of the front and rear frames, and
An interpolation object generation means determination step of determining a generation means for generating an interpolation object based on the motion object image extracted from each of the front and rear frames, the transparency, and a predetermined threshold value.
An interpolation object generation step of generating the interpolation object by the generation means determined in the interpolation object generation means determination step, and
An object peripheral image extraction step of generating a background image as a background of the interpolated object from any of the preceding and following frames, and
An image composition step of generating the interpolation frame from the interpolation object and the background image, and
Image processing method including. It is a program for performing image processing that generates an interpolation frame that complements the space between the previous and next frames, which are two consecutive frames of the input video.
An object extraction means for obtaining a motion object image by extracting an area including an object moving between the front and rear frames from each of the front and rear frames.
An object movement determining means for determining the movement direction of the region based on the respective regions of the front and rear frames, and
A transparency calculating means for calculating the transparency, which is the transparency of the background in the motion object image extracted from each of the front and rear frames, and
An interpolation object generation means determining means for determining a generation means for generating an interpolation object based on the motion object image extracted from each of the front and rear frames, the transparency, and a predetermined threshold value.
An interpolation object generation means that generates the interpolation object by the generation means determined in the interpolation object generation means determination means, and an interpolation object generation means.
An object peripheral image extraction means for generating a background image as a background of the interpolating object from any of the preceding and following frames, and
An image compositing means for generating the interpolation frame from the interpolation object and the background image, and
Programs that include.

Images