JP6866707B2 - Image processing equipment, image processing method and image processing system - Google Patents

Description

The present invention relates to an image processing apparatus, an image processing method, and an image processing system.

A frame interpolation technology that detects a motion vector between two consecutive frames in a moving image for each block, generates an interpolation frame using the motion vector, and inserts it between the original frames to increase the frame rate is already known. ing. In cases where frame interpolation is used, for example, in a system in which an image displayed on a PC display is read at a predetermined frame rate, transmitted to a remote location in real time via a network, and a conference is held while sharing the same image, transmission / reception in real time is performed. In order to make it possible, there are cases where a small amount of data with a low frame rate is transmitted and the frame rate is increased by frame interpolation on the receiving side. Further, for example, in a projector that is connected to a PC and projects a moving image sent from the PC onto a screen, in order to make the movement of the moving image appear smoother, the projector performs frame interpolation to display a high frame rate image. There is a case.

The motion vector detection tends to be erroneously detected in the case where a plurality of motions exist in the image or the case where the plurality of motions intersect, and the interpolated image may be broken. Therefore, as one method of preventing the collapse of the interpolated image, it is determined from the detected motion vector whether or not it is a scroll scene, and in the case of a scroll scene, a scroll vector indicating the motion of the entire frame from the motion vector of each block. A frame interpolation technique for detecting an object and generating an interpolation frame using a scroll vector is already known (for example, Patent Document 1).

However, frame interpolation using a scroll vector for a conventional scroll scene mainly targets natural images taken by a camera, and targets office documents such as those displayed on a PC (Personal Computer) display. Therefore, there is a problem that an image quality abnormality occurs such that the mouse cursor or the outer frame of the application window screen, which should be stationary on the display display, moves in the interpolation frame and appears to be vibrating.

The present invention has been made in view of the above points, and an object of the present invention is to prevent the occurrence of image quality abnormality due to frame interpolation when an office document containing characters and figures and tables is scrolled on a display screen.

Therefore, in order to solve the above problem, the image processing device includes a detection unit that detects a motion vector from a plurality of frames included in the input moving image, a determination unit that determines the scene type based on the plurality of frames, and the determination. Based on the scene type determined by the unit, it is determined whether or not to refer to the motion vector for generating the interpolation frame, and the generation unit has a generation unit for generating the interpolation frame of the moving image. When the determination unit determines that the scene type is a scroll scene in which a part or all of the screen moves, one of the plurality of frames is generated as an interpolation frame without referring to the motion vector, and the generation is performed. When the determination unit determines that the scene type is neither the scroll scene nor the pointer scene in which only the mouse cursor is moving in a non-scrolling state, the unit generates an interpolation frame by weighting and adding the plurality of frames. To do.

Prevents image quality abnormalities due to frame interpolation when an office document containing characters and charts is scrolled on the display screen.

It is a figure which shows the configuration example of the image processing system 1000 in 1st Embodiment. It is a figure which shows the hardware configuration example of the user device 100 in 1st Embodiment. It is a figure which shows the functional configuration example of the user device 100 in 1st Embodiment. It is a flowchart for demonstrating an example of the procedure for determining a scene in 1st Embodiment. It is a figure for demonstrating the function of the scene determination part 13 in 1st Embodiment. It is a flowchart for demonstrating an example of the procedure for generating the interpolation frame in 1st Embodiment. It is a figure for demonstrating the operation example of the interpolation frame generation part 15 in 1st Embodiment. It is a flowchart for demonstrating an example of the procedure for determining a scene in 2nd Embodiment. It is a figure for demonstrating the function of the scene determination part 13 in the 2nd Embodiment. It is a flowchart for demonstrating an example of the procedure for generating the interpolation frame in 2nd Embodiment. It is a flowchart for demonstrating an example of the procedure for determining a scene in 3rd Embodiment. It is a figure for demonstrating the function of the motion vector detection unit 12 in 3rd Embodiment. It is a figure which shows an example of scrolling in an office document. It is a figure for demonstrating the function of the scene determination part 13 in 3rd Embodiment. It is a flowchart for demonstrating an example of the procedure for determining a scene in 4th Embodiment. It is a figure for demonstrating the function of the scene determination part 13 in 4th Embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a diagram showing a configuration example of the image processing system 1000 according to the first embodiment. As shown in FIG. 1, the image processing system 1000 includes a user device 100a, a user device 100b, a user device 100c, a display device 200, a projection device 201, a server 300, and a network.

The user device 100a, the user device 100b, and the user device 100c (hereinafter, referred to as "user device 100" when they are not distinguished from each other) are, for example, a PC (Personal Computer), a tablet, a smartphone, or the like. The user device 100 is connected to another user device 100 and the server 300 via a network. The user device 100 receives moving image data from another user device 100 and the server 300 via a network, performs interpolation processing (details will be described later), and is included in the display device 200, the projection device 201, or the user device 100. The display device displays the moving image after the interpolation processing.

The display device 200 is, for example, a large-screen liquid crystal display device, which receives a signal output from the user device 100 and displays an image. The display device 200 may be directly connected to the network to receive moving image data from the user device 100 or the server 300.

The projection device 201 is, for example, a projector, which receives a signal output from the user device 100 and displays an image. The projection device 201 may be directly connected to the network and receive moving image data from the user device 100 or the server 300.

The server 300 is, for example, a multipoint control device for video conferencing. The server 300 transmits the moving image data of the PC screen on which the camera image, the office document, etc. are displayed to the user device 100 via the network. That is, the server 300 has a transmission unit that realizes a function of transmitting moving image data.

FIG. 2 is a diagram showing a hardware configuration example of the user device 100 according to the first embodiment. As shown in FIG. 2, the user device 100 has a CPU (Central Processing Unit) 101, a display device 102, a ROM (Read Only Memory) 103, a RAM (Random Access Memory) 104, and a communication I, which are connected to each other. It has / F105, a storage device 106, an input / output I / F107, and the like.

The program that realizes the processing in the user device 100 is stored in the ROM 103 or the storage device 106. The ROM 103 or the storage device 106 stores the installed program and also stores the necessary data. The storage device 106 may be, for example, one in which an external storage medium is attached to the user device 100.

The RAM 104 reads and stores the program from the ROM 103 or the storage device 106 when the program is instructed to start. The CPU 101 realizes the function related to the user device 100 according to the program stored in the RAM 104.

The display device 102 is a display device included in the user device, and is, for example, a liquid crystal display.

The communication I / F 105 is a wired or wireless interface for communicating with another PC user device 100, a server 300, or the like via a network.

The input / output I / F 107 is an interface for connecting to various input / output devices such as a USB (Universal Serial Bus) device, a hardware key, a status notification LED, and a liquid crystal display.

The display device 200, the projection device 201, and the server 300 may also have the same hardware configuration as in FIG.

FIG. 3 is a diagram showing a functional configuration example of the user device 100 according to the first embodiment. As shown in FIG. 3, the user device 100 includes a storage unit 11, a motion vector detection unit 12, a scene determination unit 13, a processing mode selection unit 14, an interpolation frame generation unit 15, and an image display unit 16. Each of these parts is realized by a process of causing the CPU 101 to execute one or more programs installed in the user device 100. Further, the storage unit 11 can be further realized by using the communication I / F 105, and the processing mode selection unit 14 and the image display unit 16 can be realized by using the input / output I / F 107.

The storage unit 11 stores a predetermined number of frames retroactively from the latest frame of the moving image sequentially input from the network in chronological order to the continuous past frames. The frame is, for example, color image data having a predetermined horizontal resolution and vertical resolution, one dot having three colors of RGB, and a color depth of each color being 8 bits. The number of bits of the color depth is not limited to 8 bits and may be further larger such as 16 bits. Further, the frame may be recorded in a data format of a luminance color difference signal such as YUV (YCbCr).

The motion vector detection unit 12 detects a motion vector based on a plurality of frames read from the storage unit 11. In the present embodiment, motion vectors are detected from two frames, the latest frame and the previous frame, but three or more frames are used so as to deal with non-linear movements and complicated cases where multiple movements intersect. There is also an existing method for detecting a motion vector, and the existing method may be used in the present embodiment.

The scene determination unit 13 determines the scene type based on the plurality of frames read from the storage unit 11 and the motion vector detected by the motion vector detection unit 12. In the present embodiment, the scene type is a scroll scene corresponding to a state in which a part or all of the screen is moved, a pointer scene in which only the mouse cursor is moving in a non-scrolling state, a scroll scene, or a scene other than the pointer scene. Is included.

The processing mode selection unit 14 selects one of a plurality of processing modes including a first processing mode for office documents including characters and figures and a second processing mode for natural images. The processing mode may be selected by the user.

The interpolation frame generation unit 15 generates an interpolation frame to be inserted between the latest frame and the previous frame by a method according to the selected processing mode and the scene determination result.

The image display unit 16 sequentially outputs or displays the moving image in which the interpolation frame is inserted between the input frames to the display device included in the display device 200, the projection device 201, or the user device 100, after the interpolation processing. ..

FIG. 4 is a flowchart for explaining an example of a procedure for determining a scene in the first embodiment. In FIG. 4, the pixels of the frame of the input moving image are composed of 8 bits each of RGB, and the processing mode selection unit 14 assumes that the first processing mode for the office document including characters and figures and tables is selected. To do. Regarding the blocks (Xi, Yi), Xi indicates the horizontal coordinates of the blocks constituting the frame, and Yi indicates the vertical coordinates of the blocks constituting the frame. The block is composed of, for example, 32 pixels in the horizontal direction and 32 pixels in the vertical direction. It is assumed that the horizontal direction of the frame is composed of Xn blocks and the vertical direction of the frame is composed of Yn blocks, and Xi takes an integer value of 0 to Xn-1, and Yi takes an integer value of 0 to Yn-1.

In step S101, the scene determination unit 13 executes initialization. The scene determination unit 13 sets Xi = 0 and Yi = 0. Further, the scene determination unit 13 sets the number of blocks with movement S = 0 and the number of blocks with density difference D = 0. The number of blocks with motion is the number of blocks in which a predetermined change is detected by comparing the pixels of the same coordinates in the blocks of the same coordinates (Xi, Yi) in the N-1th frame and the Nth frame. The Nth frame is the Nth frame in the time series of the moving image data, and the N-1th frame is a frame immediately before the Nth frame. The number of blocks S with a density difference is the number of blocks in which a predetermined change is detected by comparing the pixels in the blocks (Xi, Yi) in the N-1th frame.

Subsequently, the scene determination unit 13 reads the blocks (Xi, Yi) of the Nth frame and the N-1th frame from the storage unit 11 (S102).

Subsequently, the scene determination unit 13 performs representative difference value calculation (S103a) and block extraction with density difference (S103b). Steps S103a and S104a and steps S103b and S104b are independent processes related to blocks (Xi, Yi), and may be performed in parallel.

FIG. 5 is a diagram for explaining the function of the scene determination unit 13 in the first embodiment. As shown in FIG. 5, the scene determination unit 13 includes an inter-frame difference calculation unit 130, a density difference block extraction unit 131, and a determination unit 132.

The inter-frame difference calculation unit 130 calculates the absolute difference value between the N-1th frame and the Nth frame for each pixel for the blocks (Xi, Yi) of the N-1th frame and the Nth frame, and the block (Xi). , Yi), the maximum value among the maximum value of the absolute difference value of R, the maximum value of the absolute value of the difference of G, and the maximum value of the absolute value of the difference of B is calculated as the representative difference value.

The block extraction unit 131 with a density difference indicates that the difference between the maximum value of R and the minimum value of R with respect to the pixels in the block (Xi, Yi) of the N-1th frame is equal to or more than a predetermined threshold th1 or the maximum of G. If either the difference between the value and the minimum value of G is th1 or more, or the difference between the maximum value of B and the minimum value of B is th1 or more, the block (Xi, Yi) has a density difference. Extract as a block with.

The determination unit 132 compares the representative difference value output from the inter-frame difference calculation unit 130 with the predetermined threshold value th2, and if the representative difference value is th2 or more, blocks the block (Xi, Yi) with motion. Is determined, and the number of moving blocks S is increased by 1. Further, the determination unit 132 increases the number of blocks D with a density difference by 1 based on the block extraction result with a density difference output from the block extraction unit 131 with a density difference.

Return to FIG. In step S104a, the determination unit 132 counts the number of moving blocks S. In step S104b, the determination unit 132 counts the number of blocks D with a density difference.

Subsequently, the scene determination unit 13 determines whether or not Xi = Xn-1, and if Xi is equal to Xn-1 (YES in S105), that is, all the horizontal blocks of the in-frame line are already present. If it has been read, the process proceeds to step S107, and if Xi is not equal to Xn-1 (NO in S105), that is, if all the horizontal blocks of the in-frame line have not been read yet, the process proceeds to step S106.

In step S106, the scene determination unit 13 proceeds to step S102 with Xi = Xi + 1. That is, in the next step S102, the scene determination unit 13 reads the next block in the horizontal direction.

In step S107, the scene determination unit 13 determines whether or not Yi = Yn-1, and if Yi is equal to Yi-1 (YES in S107), that is, all the blocks in the frame have already been read. If so, the process proceeds to step S109, and if Yi is not equal to Yi-1 (NO in S107), that is, if there is an in-frame line to be read next, the process proceeds to step S108.

In step S108, the scene determination unit 13 proceeds to step S102 with Xi = 0 and Yi = Yi + 1. That is, in the next step S102, the scene determination unit 13 reads the next block in the vertical direction and the first block in the horizontal direction.

In step S109, the determination unit 132 determines whether or not the S / D is equal to or greater than the predetermined threshold value th3 with respect to the counted number of blocks with movement S and the number of counted blocks with difference in density D. That is, the determination unit 132 makes a determination based on the ratio obtained from the area of the moving block with respect to the area of the block having a density difference. When the S / D is equal to or higher than the predetermined threshold value th3 (YES in S109), the scene determination unit 13 determines that the scene is a scroll scene and completes the flow. When the S / D is less than the predetermined threshold value th3 (NO in S109), the scene determination unit 13 determines that the scene is a non-scrolling scene and completes the flow.

The predetermined threshold values th1, th2, and th3 are set in advance. Although th1 and th2 can take values from 0 to 255, both may be relatively small values of around 30. Since the input frame image is often a compressed or decompressed image, a value with a margin of 0 may be used so as not to make a false determination in response to noise generated by compression or decompression. th3 may be set to a relatively large value of around 0.7 between 0 and 1.

FIG. 6 is a flowchart for explaining an example of a procedure for generating an interpolated frame according to the first embodiment. In FIG. 6, it is assumed that the processing mode selection unit 14 has selected the first processing mode for office documents including characters and charts.

In step S121, when the interpolation frame generation unit 15 is determined to be a scroll scene as a result of the scene determination in step S109 shown in FIG. 4 (YES in S121), proceeds to step S122 and is determined to be a non-scroll scene. (NO in S121), the process proceeds to step S123.

In step S122, the interpolation frame generation unit 15 generates a copy of the previous frame as an interpolation frame, and completes the flow.

In step S123, the interpolation frame generation unit 15 generates an interpolation frame using the motion vector and completes the flow. A conventional frame complementing technique may be used to generate an interpolated frame using a motion vector.

FIG. 7 is a diagram for explaining an operation example of the interpolation frame generation unit 15 in the first embodiment. FIG. 7 shows a specific operation example in which the interpolation frame generation unit 15 inserts the interpolation frames generated one by one between the input frames of the office document.

For (a) input images that are sequentially input, (b) the moving image after conventional frame interpolation is displayed by moving the mouse cursor upward by the scroll vector in the same way as scrolling the background in the interpolated frame. Therefore (interpolation frames of the first frame and the second frame, interpolation frames of the second frame and the third frame), the mouse cursor vibrates up and down, resulting in abnormal image quality. In (b), the mouse cursor is moved up in the second frame, which is an interpolation frame, with respect to the input first frame. In the input second frame, the mouse cursor returns to the bottom, and in the fourth frame, which is the interpolation frame, the mouse cursor moves up again. In the input third frame, it returns to the bottom again. Similarly, although not shown in FIG. 7, the outer frame of the application window screen also vibrates.

On the other hand, in the moving image after frame interpolation of the first embodiment (c), the interpolation frame generation unit 15 generates an interpolation frame for the first frame, the second frame, and the third frame determined to be scroll scenes. In this case, a copy of the previous frame is used as an interpolation frame (interpolation frame of the first frame and the second frame, interpolation frame of the second frame and the third frame). If it is determined to be a non-scrolling scene, an interpolated frame is generated from the previous and next frames using motion vectors (interpolated frames of the third frame and the fourth frame) as in the conventional technique. Therefore, in the moving image after frame interpolation of the first embodiment (c), the mouse cursor does not move in the scroll scene. Therefore, in the moving image of (c), the image quality abnormality that occurred in (b) does not occur. The pointer scene in the figure will be described in the second embodiment.

As described above, according to the first embodiment, the image processing system 1000 is a method of generating an interpolation frame based on the result of scene determination based on the number of difference blocks between frames and the number of blocks having a density difference. By changing the above, it is possible to prevent an image quality abnormality that has occurred in the conventional method of generating an interpolated frame. That is, when an office document containing characters and figures and tables is scrolled on the display screen, it is possible to prevent the occurrence of image quality abnormality due to frame interpolation.

FIG. 8 is a flowchart for explaining an example of a procedure for determining a scene in the second embodiment. The second embodiment will be described which is different from the first embodiment. Therefore, the same as in the first embodiment may be used without particular mention.

In step S201, the scene determination unit 13 executes initialization. In addition to the initialization of step S101 shown in FIG. 4, the number of groups of moving blocks is set to cnt = 0. The number of groups of moving blocks is the number counted as 1 for a predetermined range of adjacent blocks (for example, 2 × 2 blocks) in the frame even when a plurality of moving blocks are detected. Is.

FIG. 9 is a diagram for explaining the function of the scene determination unit 13 in the second embodiment. In the second embodiment, the determination unit 132 of the scene determination unit 13 is any of the above three scenes: 1) scroll scene, 2) pointer scene in which the mouse cursor is moved, 3) other scenes. Or make a scene judgment. 1) The determination of the scroll scene is the same as that of the first embodiment. From the cases determined to be non-scrolling scenes in the first embodiment, 2) the pointer scene is determined, and the cases other than the pointer scene are determined to be 3) other scenes.

The method of determining 2) the pointer scene by the determination unit 132 will be described below. Assuming that the shaded blocks shown in FIG. 9 are moving blocks, a group of moving blocks composed of a predetermined number of blocks surrounded by broken lines (for example, within 2 × 2 blocks) is 2 in the frame. If there is one, it is determined that it is a pointer scene. When the mouse cursor that was in the position related to the lower left 2 blocks in the N-1 frame moves to the position that hangs on the upper right 2 blocks in the Nth frame, or hangs on the upper right 2 blocks in the N-1 frame. When the mouse cursor at the position moves to the position where it hangs on the lower left two blocks in the Nth frame, the moving blocks as shown in FIG. 9 are detected, and the number of groups of moving blocks is counted as 2. ..

A specific method for the determination unit 132 to determine whether or not there are two groups of moving blocks in a frame, which are composed of a predetermined range of adjacent blocks, will be described below. The scene determination unit 13 sets a variable that counts a group of moving blocks as cnt, and initializes it to cnt = 0. The scene determination unit 13 scans one line in the X-axis direction from the block (X, Y) = (0,0), adds 1 to Y when finished, and scans the next line. When the determination unit 132 detects a moving block, it adds 1 to ct. Assuming that the coordinates of the moving block are (X, Y) = (x1, y1), the determination unit 132 moves with respect to the three blocks (x1 + 1, y1), (x1, y1 + 1), and (x1 + 1, y1 + 1). Even if a block is detected, the process of adding 1 to ct is not performed. If cnt = 2 when scanning is completed to the end of the frame, it can be determined that there are two groups of moving blocks in the frame.

Return to FIG. Since step S204c and step S210 are different from the first embodiment, the steps will be described below.

In step S204c, the determination unit 132 counts the number of groups of moving blocks cnt. Since the number of moving blocks S is counted in step S204a, the CNTs can be counted based on S. Further, step S204c may be performed in parallel with step S203b and step S204b.

In step S210, the determination unit 132 determines whether or not ct = 2. When ct is 2 (YES in S210), the scene determination unit 13 determines that the scene is a pointer scene and completes the flow. When ct is not 2, the scene determination unit 13 determines that the scene is another scene and completes the flow.

FIG. 10 is a flowchart for explaining an example of a procedure for generating an interpolated frame in the second embodiment. Since steps S223 to S225 are different from the first embodiment, the steps will be described below.

If the scene determination unit 13 determines in step S223 that the result of the scene determination is a pointer scene (YES in S223), the process proceeds to step S224. When the scene determination unit 13 determines that the result of the scene determination is not a pointer scene (NO in S223), the process proceeds to step S225.

In step S224, the interpolation frame generation unit 15 generates an interpolation frame using the motion vector and completes the flow. A conventional frame complementing technique may be used to generate an interpolated frame using a motion vector.

In step S225, the interpolation frame generation unit 15 generates an interpolation frame by weighting and adding the preceding and following frames, and completes the flow. A conventional frame interpolation technique may be used to generate the weighted and added interpolation frame. For example, as shown in FIG. 7, when one frame is inserted between the input frames, the preceding and following frames are added with a weight of 1/2, that is, the average image is the interpolation frame. When two frames are inserted between the input frames, the first interpolation frame is an image obtained by adding a weight of 2/3 to the previous frame and a weight of 1/3 to the rear frame. The first interpolated frame is an image obtained by adding a weight of 1/3 to the previous frame and a weight of 2/3 to the rear frame. A case determined as another scene is, for example, a mouse drag operation scene. It is known that the mouse drag operation scene becomes a moving image in which the movement looks relatively natural and there is little discomfort by generating an interpolated frame by weighting addition.

As described above, according to the second embodiment, the image processing system 1000 determines the result of the scene determination based on the number of difference blocks between frames, the number of groups of difference blocks between frames, and the number of blocks having a density difference. By changing the method of generating the interpolation frame based on the above, it is possible to display a screen suitable for displaying the mouse cursor.

FIG. 11 is a flowchart for explaining an example of a procedure for determining a scene in the third embodiment. The third embodiment will be described which is different from the first embodiment. Therefore, the same as in the first embodiment may be used without particular mention.

In step S301, the scene determination unit 13 executes initialization. Similar to the initialization in step S101 shown in FIG. 4, Xi = 0, Yi = 0, and D = 0. Further, the number of blocks with upward movement M1 = 0 and the number of blocks with downward movement M2 = 0. The number of blocks with motion is the number of blocks in which motion in the upward or downward direction is detected based on the motion vector.

FIG. 12 is a diagram for explaining the motion vector detection unit 12 according to the third embodiment. Although some existing methods are known for motion vector detection, in the third embodiment, the method by block matching is used. The N-1th frame is the frame immediately before the latest frame, and the Nth frame is the latest frame. In the block matching method, the frame is divided into blocks composed of a predetermined number of pixels (for example, 32x32 pixels), and a block of the Nth frame having a high correlation with the block of the N-1th frame is searched for. Assuming that the blocks filled with diagonal lines shown in FIG. 12 have a high correlation, the motion vector is indicated by an arrow extending from the position of the diagonal line block of the N-1th frame to the position of the diagonal line block of the Nth frame. When the motion vector (Vx, Vy) is expressed by the coordinates in block units, it becomes (0, -2). That is, the movement of -2 blocks in the Y direction is detected. In this way, motion vector detection is performed for all blocks of the Nth frame.

FIG. 13 is a diagram showing an example of scrolling in an office document. As shown in FIG. 13, the motion vector is illustrated by superimposing it on the display display image of the PC. It is assumed that characters or charts are drawn in the area where the motion vector is displayed.

Example 1 of the scroll screen shown in FIG. 13 is an example in which the office document is displayed in full screen on the display of the PC. Images other than the application window frame and mouse cursor are scrolled and moving downward. Further, the scroll screen example 2 shown in FIG. 13 is an example in the case where the scroll screen is not displayed in full screen. Only the inside of the window frame is scrolling downward, and the outside of the window frame is displayed with the wallpaper or icon image still. When scrolling an office document, it is assumed that the office document is scrolled downward in order to read the office document. In addition, scrolling upward, which is the opposite direction, is also assumed in order to return the display of the office document to the previous state.

FIG. 14 is a diagram for explaining the scene determination unit 13 in the third embodiment. In the third embodiment, the determination unit 135 of the scene determination unit 13 enhances the determination accuracy by determining the scene using the motion vector. The N-1th frame is the frame immediately before the latest frame, and the Nth frame is the latest frame.

The block extraction unit 133 with upward movement refers to the motion vector of each block in the N-1th frame, and if the motion vector (Vx, Vy) is Vx = 0 and Vy <0, the block extraction unit 133 moves upward. Extract as a moving block. The block extraction unit 134 with downward movement refers to the motion vector of each block in the N-1th frame, and if the motion vector (Vx, Vy) is Vx = 0 and Vy> 0, the block extraction unit 134 moves downward. Extract as a moving block. Therefore, the number of blocks with upward movement and the number of motion vectors indicating upward movement, the number of blocks with downward movement and the number of motion vectors indicating downward movement are the same, respectively.

The block extraction unit 131 with a density difference is the same as that of the first embodiment of the scene determination unit 13. The determination unit 135 is based on the number of blocks with upward movement M1 and the blocks extracted by the block extraction unit 134 with downward movement, based on the blocks extracted by the block extraction unit 133 with upward movement. Then, the number of blocks with downward movement M2 is counted. The number of blocks D with a density difference is counted based on the blocks extracted by the block extraction unit 131 with a density difference as in the first embodiment. The determination unit 135 of the scene determination unit 13 determines that the scene is a scroll scene when M1 / D or M2 / D is equal to or higher than the predetermined threshold value th4. When M1 / D and M2 / D are less than the predetermined threshold value th4, it is determined that the scene is a non-scrolling scene.

Similar to the example shown in FIG. 13, the scrolling operation on the PC display screen is mainly an upward or downward movement, and for example, scrolling in an oblique direction is small. Therefore, the determination accuracy of the scroll scene is determined by determining the ratio of the number of blocks with upward movement M1 to the number of blocks with difference in density D or the ratio of the number of blocks with downward movement M2 to the number of blocks with density difference D. Is increasing. The threshold value th4 is a predetermined threshold value and is set in advance. Similar to the threshold value th3, the threshold value th4 is preferably set to a relatively large value.

Return to FIG. Since step S302a, step S303a, step S303c, step S304a, step S304c and step S309 are different from the first embodiment, the steps will be described below.

In step S302a, the scene determination unit 13 reads the motion vector of the (Xi, Yi) block of the N-1th frame from the motion vector detection unit 12. Subsequently, the upwardly moving block extraction unit 133 extracts the upwardly moving blocks (S303a), and the determination unit 135 counts the number of upwardly moving blocks M1 (S304a). At the same time, the downwardly moving block extraction unit 134 extracts the downwardly moving blocks (S303c), and the determination unit 135 counts the number of downwardly moving blocks M2 (S304c). ..

In step S309, when the determination unit 135 determines that M1 / D or M2 / D is th4 or more (YES in S309), the scene determination unit 13 determines that the scene is a scroll scene and performs a flow. Complete. When the determination unit 135 determines that M1 / D and M2 / D are less than th4, the scene determination unit 13 determines that the scene is a non-scrolling scene and completes the flow.

As described above, according to the third embodiment, the image processing system 1000 has the number of blocks with upward movement due to the motion vector, the number of blocks with downward movement due to the motion vector, and the blocks having a density difference. By judging the scene based on the number, it is possible to improve the accuracy of the scene judgment for scrolling in the vertical direction.

FIG. 15 is a flowchart for explaining an example of a procedure for determining a scene in the fourth embodiment. A fourth embodiment aims to accurately determine that the scroll scene is a scroll scene in either of the two examples of the scroll screen shown in FIG. The fourth embodiment will be described which is different from the third embodiment. Therefore, the same as in the third embodiment may be applied without particular mention.

In step S401, similarly to the initialization in step S301 shown in FIG. 11, Xi = 0, Yi = 0, M1 = 0, M2 = 0, D = 0, and the number of blocks with motion based on the motion vector. Let M3 = 0. The number of blocks with motion M3 is the number of blocks in which motion in any direction is detected based on the motion vector.

FIG. 16 is a diagram for explaining the scene determination unit 13 in the fourth embodiment. The block extraction unit 133 with an upward movement, the block extraction unit 134 with a downward movement, and the block extraction unit 131 with a concentration difference are the same as those in the third embodiment shown in FIG.

The motion block extraction unit 136 moves the motion vector (Vx, Vy) of each block in the N-1th frame if the absolute value in the X direction or the Y direction is | Vx |> 0 or | Vy |> 0. Extract as a block with. That is, if the motion vector of a certain block is non-zero, the motion block extraction unit 136 extracts the block as a motion block.

The determination unit 137 counts the number of blocks with movement M1 in the upward direction, the number of blocks M2 with movement in the downward direction, and the number of blocks with movement M3, and further counts the number of blocks D with a density difference. The determination unit 137 of the scene determination unit 13 determines that the scene is a scroll scene when M1 / D or M2 / D is th4 or more and M1 / M3 or M2 / M3 is a predetermined threshold value th5 or more. The determination unit 137 of the scene determination unit 13 determines that the scene is a non-scrolling scene when M1 / D and M2 / D are less than th4, or M1 / M3 and M2 / M3 are less than th5.

The threshold value th5 is a predetermined threshold value and is set in advance. th5 is a value between 0 and 1, and may be set to a value close to 1. It may have a margin or a large value of about 0.95. That is, assuming Example 2 of the scroll screen shown in FIG. 13, the area of the scroll portion occupies a certain area in the frame, and the moving portion and the scroll portion are substantially the same area, except for the scroll portion. When there is almost no moving area, it is determined that the scroll scene is a scroll scene, thereby improving the determination accuracy of the scroll scene.

The block extraction unit 133 with movement in the upward direction, the block extraction unit 134 with movement in the downward direction, and the block extraction unit 136 with movement all determine 0 as a threshold value, but an error occurs due to the motion vector detection accuracy. In consideration of the possibility, a margin may be provided, for example, 1 may be determined as a threshold value.

Return to FIG. Since step S403d, step S404d, step S409 and step S410 are different from the third embodiment, the steps will be described below.

In step S403d, the moving block extraction unit 136 extracts the moving blocks (S403d), and the determination unit 137 counts the number of moving blocks M1 (S404d).

If the determination unit 137 determines in step S409 that M1 / D or M2 / D is th4 or more (YES in S409), the process proceeds to step S410, and M1 / D and M2 / D are less than th4. If it is determined (NO in S409), the scene determination unit 13 determines that the scene is a non-scrolling scene and completes the flow.

In step S410, when the determination unit 137 determines that M1 / M3 or M2 / M3 is th5 or more (YES in S410), the scene determination unit 13 determines that the scene is a scroll scene and performs a flow. Complete. When the determination unit 137 determines that M1 / M3 and M2 / M3 are less than th5 (NO in S410), the scene determination unit 13 determines that the scene is a non-scrolling scene and completes the flow.

As described above, according to the fourth embodiment, the image processing system 1000 includes the number of blocks with upward movement due to the motion vector, the number of blocks with downward movement due to the motion vector, the number of blocks with movement, and the number of blocks with movement. By making a scene judgment based on the number of blocks having a density difference, it is possible to improve the accuracy of the scene judgment especially for scrolling only in the window on the PC display.

Regarding video processing, if the video is accompanied by audio, there will be a time lag between the audio and the video unless the video is processed immediately and the video is output. It is necessary to output a video that has been processed and frame-interpolated. In each of the first to fourth embodiments, the scene determination unit 13 has a configuration in which the corresponding block is extracted, the number of the blocks is counted, and the threshold value is determined. The configuration is simpler than the conventional method of creating and analyzing a histogram and determining a scene, and is excellent in immediacy.

In the embodiment of the present invention, the motion vector detection unit 12 is an example of the detection unit. The scene determination unit 13 is an example of the determination unit. The interpolation frame generation unit 15 is an example of a generation unit. The user device 100 is an example of an image processing device. The moving block is an example of a moving area. The block with a concentration difference is an example of a region where the concentration changes.

Although the examples of the present invention have been described in detail above, the present invention is not limited to such specific embodiments, and various modifications are made within the scope of the gist of the present invention described in the claims.・ Can be changed.

1000 Image processing system 100 User device 200 Display device 201 Projection device 300 Server 101 CPU
102 Display device 103 ROM
104 RAM
105 Communication I / F
106 Storage device 107 I / O I / F
11 Accumulation unit 12 Motion vector detection unit 13 Scene judgment unit 14 Processing mode selection unit 15 Interpolation frame generation unit 16 Image display unit 130 Interframe difference calculation unit 131 Density difference block extraction unit 132 Judgment unit 133 Upward movement block Extraction unit 134 Downward movement Block extraction unit 135 Judgment unit 136 Movement Block extraction unit 137 Judgment unit

Japanese Unexamined Patent Publication No. 2011-130409

Claims (9)

A detector that detects motion vectors from multiple frames included in the input video,
A determination unit that determines the scene type based on the plurality of frames,
Based on the scene type determined by the determination unit, it has a generation unit that determines whether or not to refer to the motion vector for generation of the interpolation frame and generates the interpolation frame of the moving image.
When the determination unit determines that the scene type is a scroll scene in which a part or all of the screen moves, the generation unit uses one of the plurality of frames as an interpolation frame without referring to the motion vector. Generate and
When the determination unit determines that the scene type is neither the scroll scene nor the pointer scene in which only the mouse cursor is moving in a non-scrolling state, the generation unit weights and adds the plurality of frames to the interpolated frame. An image processing device that produces. The determination unit counts the number of moving regions from the difference between the plurality of frames, and determines whether or not the scene type is the scroll scene based on the area calculated from the number of moving regions. The image processing apparatus according to claim 1. The determination unit counts the number of regions having a density change in one of the plurality of frames, and is calculated from the number of regions having the movement with respect to the area calculated from the number of regions having the density change. The image processing apparatus according to claim 2, wherein it is determined whether or not the scene type is the scroll scene based on the area ratio. The determination unit counts the number of groups in a moving region from the difference between the plurality of frames, and when the number of the groups is 2, it determines that the scene type is a pointer scene in which the mouse cursor moves. Item 2. The image processing apparatus according to item 2 or 3. The first aspect of claim 1, wherein the determination unit determines whether or not the scroll scene is based on a motion vector indicating an upward movement or a motion vector indicating a downward movement among the motion vectors. Image processing device. The determination unit counts the number of regions having a density change in one of the plurality of frames, and is the ratio of the number of motion vectors indicating upward movement to the number of regions having a density change, or the said. The image processing apparatus according to claim 5 , wherein it is determined whether or not the scene is a scroll scene based on the ratio of the number of motion vectors indicating the downward motion to the number of regions having a density change. The determination unit counts the number of motion regions from the motion vector, and the ratio of the number of motion vectors indicating upward movement to the number of motion regions, or the number of motion regions. The image processing apparatus according to claim 5 , wherein it is determined whether or not the scene is a scroll scene based on the ratio of the number of motion vectors indicating the downward movement. A detection procedure that detects motion vectors from multiple frames included in the input video,
Judgment procedure for determining the scene type based on the plurality of frames and
Based on the scene type determined by the determination procedure, it is determined whether or not to refer to the motion vector for generating the interpolation frame, and the generation procedure for generating the interpolation frame of the moving image and the generation procedure.
If a scene type is determined to some or all of the screen is scrolled scene moved by prior Symbol judging procedure, a procedure for generating one of said plurality of frames without reference to the motion vector as the interpolation frame When,
When it is determined by the determination procedure that the scene type is neither the scroll scene nor the pointer scene in which only the mouse cursor is moving in a non-scrolling state, the plurality of frames are weighted and added to generate an interpolation frame. Image processing method to execute and. An image processing system including a server and an image processing device connected to the server via a network.
The server
It has a transmitter that transmits a moving image to the image processing device via the network.
The image processing device is
A detector that detects motion vectors from a plurality of frames included in the received moving image,
A determination unit that determines the scene type based on the plurality of frames,
Based on the scene type determined by the determination unit, it has a generation unit that determines whether or not to refer to the motion vector for generation of the interpolation frame and generates the interpolation frame of the moving image.
When the determination unit determines that the scene type is a scroll scene in which a part or all of the screen moves, the generation unit uses one of the plurality of frames as an interpolation frame without referring to the motion vector. Generate and
When the determination unit determines that the scene type is neither the scroll scene nor the pointer scene in which only the mouse cursor is moving in a non-scrolling state, the generation unit weights and adds the plurality of frames to the interpolated frame. An image processing system that produces.

Images