JP2011053587A - Image processing device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To enable to reopen automatic following of partial expansion display and an expansion region without re-operating by a user even when an attention object deviated temporarily from a display image is displayed again. <P>SOLUTION: An image processing device 1 includes: an attention object detecting part 3 detecting feature point information and feature point position which indicates an image of the attention object in a user specification region; a memory 8 for the feature point information; a frame memory 10 for afront frame; a movement quantity calculating part 6 calculating movement quantity of the feature point information between the frame stored in the memory 10 and the present frame; a memory 9 storing the feature point position in the present frame; a re-display detecting part 7 in which when movement quantity cannot be calculated, it is detected whether the feature point is included in an afterward frame or not, and when it is included, the feature point position is detected; an expansion area deciding part 4 in which the expansion area including the attention object is decided based on the feature point position detected by the attention object detecting part 3, a movement quantity of the feature points calculated by the movement quantity calculating part 6, or the feature point position detected by the re-display detecting part 7. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image processing apparatus that enlarges a part of an image and displays the enlarged image on a display device.

  In the image display device, a function of enlarging and displaying a part of an image according to a user operation is known. For example, in Patent Document 1, when a user specifies a position that the user wants to pay attention to, a circular area centered on the specified position is enlarged and displayed on the original image as if the user had viewed the image using a loupe. An information processing apparatus is shown.

  FIG. 7 is a block diagram illustrating an example of a functional configuration of a conventional image display device described in Patent Document 1. In FIG. In the image display apparatus 100 in the figure, the user input center coordinate acquisition unit 101 acquires the center coordinates of the area to be enlarged by the input from the user, and outputs it to the partial enlargement calculation unit 102. The partial enlargement calculation unit 102 determines a circular area centered on the center coordinates, performs image enlargement processing only within the determined area of the input image, and generates a partial enlarged image in which the enlarged image is superimposed on the entire original input image. obtain. The partial enlarged image output from the partial enlargement calculation unit 102 is presented to the user on the display unit 103.

However, when a moving image is displayed, the target of attention of the user in the image moves. In this case, in the technique disclosed in Patent Document 1, in order to continue to display the target object in an enlarged manner, it is necessary for the user to manually specify the position of the enlarged region in accordance with the movement of the image. Such manual tracking is troublesome for the user.
On the other hand, Patent Document 2 discloses a technique for moving a display position of an enlarged image of an object designated by a user according to a motion vector of the object when performing partial enlarged display.

JP-A-4-142661 JP 2009-10016 A

  When displaying a moving image, not only the target that the user pays attention to moves within the image, but also the target of interest may deviate from the frame or may not be displayed on the video due to a scene change. After that, it may appear again in the frame, or may be redisplayed on the video due to another scene change. In the above-described case, it is preferable that after the redisplay, the partially enlarged display of the target of interest and the automatic tracking of the enlarged region are resumed without being operated again.

  The present invention has been made in view of the above-described situation, and when performing a partial enlarged display for displaying a moving image in which a target object in an area designated by a user is enlarged so as to be superimposed on the entire moving image. In addition to automatically following the enlarged area, it is possible to resume the partially enlarged display and the above-described automatic tracking of the enlarged area without the user's operation again even if the target of interest once deviated from the display image is redisplayed. It is an object of the present invention to provide an image processing apparatus that can be used.

  In order to solve the above problems, a first technical means of the present invention is an image processing apparatus, in a peripheral area at a position specified by a user with respect to a display image on a display screen, or an area specified by a user Feature point information indicating an image of the target object included in the target object, a target object detection unit that detects a display position of the feature point, a feature point information storage unit that stores the detected feature point information, and a current display frame Detecting the display position of the feature point information stored in the feature point information storage unit in the frame memory storing the image of the previous frame, the image stored in the frame memory and the image of the current display frame, and / or A movement amount calculation unit for calculating a movement amount of a display position of feature point information; a current frame feature point position information storage unit for storing a display position of feature point information in a current display frame; and the movement amount calculation unit. If the amount of movement could not be calculated, whether or not the feature point information is included in the image of the subsequent frame, and if included, a re-display detection unit that detects the display position of the feature point information; The display position of the feature point information detected by the target object detection unit, the movement amount of the feature point information calculated by the movement amount calculation unit, or the display position of the feature point information detected in the current frame, or the re-display detection unit A partial enlarged display unit that determines an enlarged region including the target of interest based on the detected display position of the feature point information, and displays an enlarged image of the enlarged region superimposed on the display location of the target of interest; The enlarged image of the target of interest is superimposed on the display location of the target of interest and displayed following the movement of the target image as specified by the user, and the image of the target of interest has moved out of the display screen , In which is characterized in that also display the enlarged image follows to the case where it is re-displayed.

  According to a second technical means, in the first technical means, an acquisition unit is provided for acquiring at least one of a size and an enlargement ratio of the enlarged image from a user.

  According to a third technical means, in the first or second technical means, the attention object detection unit detects a face image and / or a character image as the attention object.

  According to a fourth technical means, in the first or second technical means, when the target object detection unit includes depth information attached to the input image, an object existing in front of the depth information is detected. In this case, the object is detected as the target of interest.

  The fifth technical means highlights the target of interest detected by the target of interest detection unit in any one of the first to fourth technical means, and determines whether or not the target of interest is in accordance with the user's intention. It is characterized by comprising confirmation means for allowing the user to confirm.

  In a sixth technical means, in any one of the first to fifth technical means, when the re-display detection unit detects the feature point information that is equal to or greater than a predetermined threshold value, It is characterized by determining that the feature point information is included.

  According to the present invention, an enlarged region can be automatically followed when performing partial enlarged display. Furthermore, even when a target of interest that once deviates from the display image is redisplayed, the partially enlarged display and the follow-up of the enlarged region can be resumed without re-operation by the user.

1 is a block diagram showing a schematic configuration of an image processing apparatus according to a first embodiment of the present invention. 3 is a flowchart for explaining an example of a processing flow in the image processing apparatus of FIG. 1. 3 is a flowchart for explaining an example of a processing flow in the image processing apparatus of FIG. 1. 3 is a flowchart for explaining an example of a processing flow in the image processing apparatus of FIG. 1. It is a figure explaining the example of the partial expansion display which the image processing apparatus of this invention performs. It is a block diagram which shows schematic structure of the image processing apparatus which concerns on the 2nd Embodiment of this invention. It is a figure explaining the conventional partial expansion display technique.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, portions having the same function are denoted by the same reference numerals, and repeated description is omitted.

(First embodiment)
FIG. 1 is a block diagram showing a schematic configuration of the image processing apparatus according to the first embodiment of the present invention. The image processing apparatus 20 according to the present embodiment performs partial enlargement display with automatic tracking of a target of interest. As shown in FIG. 1, a tracking execution signal acquisition unit 1, a user-specified region acquisition unit 2, and a target Object detection unit 3, enlargement area determination unit 4, partial enlargement calculation unit 5, feature point movement amount calculation unit 6, attention object redisplay detection unit 7, first memory 8, and second memory 9 And the frame memory 10 and outputs the generated image to the display unit 11. The image processing apparatus 20 can perform not only partial enlarged display with automatic tracking of the target of interest but also partial enlarged display without automatic tracking, but here, the former partial expansion according to the present invention is performed. The display will be mainly described.

  In the follow-up execution signal acquisition unit 1, a signal instructing the start and end of the follow-up function input from the user is acquired and output to the enlarged region determination unit 4. The follow-up execution signal that determines the execution of the follow-up function takes either a value of “ON” representing “execution of follow-up” or “OFF” representing “not execute follow-up”.

Similar to the conventional technique, the partial enlargement calculation unit 5 performs image enlargement processing on an area specified by partial enlargement area information Cout described later on the input frame image when performing partial enlargement display of the target of interest. Do. For example, the image enlargement method is performed using a bilinear method with a preset fixed value enlargement ratio, but is not limited thereto.
The frame memory 10 stores the previous frame image for processing in a feature point movement amount calculation unit 6 described later.
The display unit 11 is composed of a liquid crystal display, for example, and displays and outputs a partially enlarged image.

  For example, when the display unit 11 has a touch panel function, the user designated area acquisition unit 2 encloses the object of interest with a circle in the display image on the screen. When the center of the object is touched or a part of the target object is touched, the position of the area including the target object of the user is acquired for the first frame, and the enlarged area determination unit 4 and the user specified area information Cin are obtained. This is output to the target object detection unit 3. In addition, the user may instruct using a mouse or a remote controller instead of the touch panel, or the area may be calculated based on the image position calculated from the user's line-of-sight direction detected by a line-of-sight detection device (not shown). .

  The target object detection unit 3 detects the target object in the area specified by the user for the first frame in the partial enlarged display, and calculates the feature point information for identifying the target object and the position information of the feature point / The feature point information is acquired, stored in the first memory (feature point information storage unit) 8, and the feature point position information is output to the enlarged region determination unit 4. As a method of detecting a target of interest, a method realized by an automatic subject tracking function in a camera may be used. For example, when a face image exists in an area to be enlarged, the face image is likely to be a target of attention, and therefore the face image may be set as a target of attention. As a method of detecting a face image, a method of detecting a face image from the shape and arrangement of eyes, nose, and mouth, or a general face image region detection method that detects from the degree of gradation of a skin color region, etc. Good. Further, when a character is displayed in an area to be enlarged, it is highly likely that the character is a target of attention, and therefore the character image may be set as a target of attention. In general, since the target of interest has a motion, it is possible to correlate between several frames, and to select the subject of motion in the region to be enlarged.

As a method for detecting feature point information from a target of interest, there is a method using a corner defined as a location where main edges intersect. As a corner detection method, a known Harris operator or SUSAN operator can be used. Although corners are used here as feature points, the present invention is not limited to corners. Edges may be used as feature points, and other feature quantities such as color information may be used instead of luminance information. You may combine a feature-value.
In the image processing apparatus 20, the feature point information acquired in this way is stored in the first memory 8 even when the target of interest disappears from the screen during partial enlargement display. The attention object redisplayed by the attention object redisplay detection unit 7 can be detected.

The enlargement area determination unit 4 determines an area to be enlarged in the input frame image, and outputs the area to the partial enlargement calculation unit 5 as partial enlargement area information Cout.
For the first frame, the enlargement area determination unit 4 sets an area in which the corresponding feature point is stored as an enlargement area based on the feature point position information acquired from the target object detection unit 3. For example, a circle in which all the corresponding feature points are drawn is drawn, and as the partial enlarged region information Cout, the shape of the enlarged region is a circle, its center coordinates, and a value obtained by adding a fixed margin to the radius of the drawn circle Are output to the partial enlargement calculation unit 5. Further, the enlarged region determination unit 4 stores the feature point position information C1 of the first frame in the second memory (current frame feature point position information storage unit) 9 and stores the first frame in the frame memory 10. To do.

  The operation of the enlarged region determination unit 4 regarding the second and subsequent frames in the automatic follow-up partial enlarged display will be described later.

  The feature point movement amount calculation unit 6 holds, for example, the input frame image, the previous frame image held in the frame memory 10, and the first memory 8 for the second and subsequent frames in the partial enlarged display. Based on the feature point information, the coordinates of each feature point are detected in both the front and back frame images, and the movement amount V of each feature point between the input frame image and the previous frame image is detected from these coordinates, The result is output to the enlarged area determination unit 4. Further, the feature point movement amount calculation unit 6 stores the input frame image in the frame memory 10 after the detection.

The enlarged region determination unit 4 has the feature point position information of the first frame stored in the second memory 9 and the features calculated by the feature point movement amount calculation unit 6 for the second frame in the partial enlarged display. Based on the movement amount V of the point, the feature point position information C2 of the frame is obtained, an enlarged region in which all the corresponding feature points are contained is determined, and partial enlarged region information Cout is output. At this time, the memory of the second memory 9 is updated and stored for the feature point position information C2 of the second frame.
For the third and subsequent frames, the enlarged region determination unit 4 operates in the same manner as described above, so that the partial enlarged display can be performed.

Also, the feature point movement amount calculation unit 6 cannot calculate the feature point movement amount V when the target of interest disappears from the display image due to frame-out or scene change. Output to part 4.
Upon receiving this information, the enlargement area determination unit 4 outputs information for invalidating the partial enlargement display as the partial enlargement area information Cout to the partial enlargement calculation part 5 and stops the partial enlargement display. In response to this, the partial enlargement calculation unit 5 outputs the input frame image to the display unit 11 as it is.

The attention object re-display detection unit 7 detects whether or not the attention object disappears from the video due to a frame-out or a scene change, and after the partial enlarged display is stopped, the attention object is displayed again on the screen. Here, since the feature point information used for detecting the movement amount of the feature point is held in the first memory 8, it may be determined whether or not the feature point appears again in the screen. In addition, when redisplayed, some of the feature points may have changed, so not only when the feature points are completely matched, but also when the matching rate of the feature points is greater than a certain threshold, You may determine with the attention object displayed again.
The attention object re-display detection unit 7 further calculates / acquires feature point position information and outputs it to the enlarged region determination unit 4 when the attention object is displayed again. After the calculation / acquisition, the input frame image is stored in the frame memory 10.

  For the frame in which the redisplay of the target of interest is detected, the enlarged region determination unit 4 determines, as the expansion region, the region in which the corresponding feature point is contained based on the feature point position information acquired from the target redisplay detection unit 7 Then, the partial enlargement area information Cout is output to the partial enlargement calculation unit 5, and the feature point position information Cn of the frame is stored in the second memory 9. The subsequent frames are processed in the same manner as the above-described normal partial enlarged display, so that the partial enlarged display can be resumed without the user performing another operation after the redisplay of the target of interest is detected.

FIG. 2 is a flowchart for explaining an example of the flow of processing performed by the image processing apparatus 20 of FIG. 1 for the first frame for performing partial enlarged display.
When the user designates an area using the input device and instructs partial enlargement display, the user designated area acquisition unit 2 obtains the position of the area as user designated area position information Cin, as shown in FIG. The attention object detection unit 3 detects the attention object in the first frame from the user-specified region position information Cin, calculates the feature point information and the feature point position information C1 of the attention object (step S1), and the feature points The information is stored in the first memory 8, and the feature point position information C 1 is output to the enlarged region determination unit 4.

  Then, the enlarged region determination unit 4 sets a region in which each feature point falls based on the feature point position information C1, and outputs it to the partial enlargement calculation unit 5 as the enlarged region position information Cout (step S2). Is stored in the second memory 9 (step S3), and the first input frame image is stored in the frame memory 10 (step S4). In step S5, the partial enlargement calculation unit 5 performs image enlargement processing on the pixels in the enlargement area specified by the partial enlargement area information Cout, and further, for the enlarged image and the area outside the enlarged image. An image using the pixel value of the input image as it is is synthesized, and the obtained partial enlarged image is output to the display unit 11.

FIG. 3 is a flowchart for explaining an example of a flow of processing performed by the image processing apparatus 20 in FIG. 1 for the second and subsequent frames in the partial enlarged display.
For the second and subsequent frames in the partial enlarged display, as shown in FIG. 3, the feature point movement amount calculation unit 6 uses the feature point information stored in the first memory 8 and the previous frame in the frame memory 10. A moving amount V of each feature point is calculated from the image and the input frame image (step S11). Next, in step S12, it is determined whether or not the calculation of the movement amount V of each feature point in step S11 has failed. When the movement amount V can be calculated (in the case of NO), the process proceeds to step S13. In addition, when it fails (in the case of YES), it is a case where an attention object remove | deviates from the inside of a display image, and is a case where partial enlarged display is interrupted. The processing in this case will be described later with reference to FIG.

  In step S13, the enlarged region determination unit 4 reads the previous frame feature point position information Cn-1 from the second memory 9, and adds the movement amount V of each feature point to the current frame feature point position information Cn. Then, an area in which each feature point falls is set from the information, and is output as enlarged area position information Cout, and the process proceeds to step S11. In this way, not only the target object is automatically tracked, but if the target object is enlarged / reduced in the input frame image, the shape of the partially enlarged display area can be enlarged / reduced accordingly.

In step S14, the content of the feature point position information in the second memory 9 is updated, and in step S15, the frame memory 10 is updated with the input frame image. Then, the partial enlargement calculation unit 5 performs image enlargement processing on the pixels in the enlargement region specified by the partial enlargement region information Cout, acquires a partial enlargement image, and outputs it to the display unit 11 (step S16).
Thereafter, in step S17, it is determined whether or not the user has instructed to end the enlargement process. If instructed (in the case of YES), the process is terminated. If not instructed (in the case of NO), the process proceeds to step S11. Return.

FIG. 4 illustrates an example of the flow of processing performed by the image processing apparatus 20 in FIG. 1 when the calculation of the amount of movement of the feature points in step S12 in FIG. It is a flowchart for doing.
If the calculation of the amount of movement of the feature point fails (when the target object is out of the display image), information indicating that fact is output to the enlarged region determining unit 4, and the determining unit 4 displays the enlarged region position information Cout. As a result, information indicating invalidity is output to the partial enlargement calculation unit 5 (step S21). At this time, the partial enlargement calculation unit 5 does not perform the partial enlargement process, and outputs the input frame image as it is to the display unit 11 (step S22).
Thereafter, in step S23, it is determined whether or not the user has instructed to end the enlargement process. If instructed (in the case of YES), the process is terminated. If not instructed (in the case of NO), the process proceeds to step S24. move on.

In step S <b> 24, it is determined whether or not the attention object re-display detection unit 7 detects that the attention object is displayed again in the next input frame image. The re-display is detected based on the input frame image and the feature point information held in the first memory 8.
If not detected (NO in step S24), the process proceeds to step S22 with the information indicating that the enlarged area information is invalid as before, so that the partial enlarged display is not performed. On the other hand, when the attention object is detected (in the case of YES), the feature point position information Cn is calculated by the attention object re-display detection unit 7 (step S25), and the enlarged region determination unit 4 performs the feature point position information. Based on Cn, an area in which each feature point falls is set, and is output as the enlarged area position information Cout to the partial enlargement calculation unit 5 (step S26). Then, the process proceeds to step S14 in FIG. It moves to the same processing as the case where it had. As a result, the partial enlarged display is restored.

  As described above, according to the image processing apparatus of the present embodiment, when performing a partial enlargement process on a moving image, it is possible to automatically follow the enlargement area in accordance with the movement of the image. Furthermore, even when the target of interest that has once deviated from the display image is displayed again, it is possible to resume the partially enlarged display and the tracking of the enlarged region without the user again specifying the target of interest.

FIG. 5 is a diagram for explaining an example of a captured image of an athletic meet by the user, which is an example of the case where the target of interest disappears from the screen and is displayed again as described above.
5A to 5C are diagrams illustrating an example of a moving image input to the image processing apparatus. FIG. 5 (A) is an image of the start points of the three Athlete participants. While shooting this video, tilt the camera to the left and focus on the background building. When projected, the image shown in FIG. From this point, the video shown in FIG. 5C is obtained by returning to the shooting of the video centering on the starting point of the race.

  Here, FIGS. 5D to 5F show an enlarged display method in the case where the face image of the person at the center of the three student race participants is the target of attention. When the user surrounds the periphery of the face of the person at the center of the student race participant, it is recognized that the target object is present in the enclosed area. Next, by recognizing that a face image exists in the region, it is recognized that the target object is a face image. Therefore, when the partial enlargement function process is performed on the image in FIG. 5A, the face portion of the central person is partially enlarged as shown in FIG. Here, the area on which the partial enlargement display is performed is indicated by a magnifying glass shape, but the display shape is not limited to this shape. Here, when the video in FIG. 5B is projected centering on the building in the background, it is recognized that the target of interest is no longer displayed on the screen, and the movement amount detection and partial enlargement display functions are interrupted. ) Is displayed without partial enlargement. Then, when returning to the video at the start point, the video shown in FIG. 5C is obtained, and it is detected that the target of interest has appeared again in the screen, the partial enlarged display is resumed again, and the video shown in FIG. 5F is obtained.

(Second Embodiment)
FIG. 6 is a block diagram showing a schematic configuration of an image processing apparatus according to the second embodiment of the present invention. The difference from the first embodiment of FIG. 1 is that a parameter acquisition unit 12 is added and the processing performed by the partial enlargement calculation unit 13 is different.
The parameter acquisition unit 12 acquires the enlargement ratio of the partial enlargement process and the size of the image after enlargement according to the input from the user, and outputs them to the partial enlargement calculation unit 13.

  For example, the user inputs “higher enlargement ratio” or “lower enlargement ratio” by operating the remote controller. When these inputs are made, the parameter acquisition unit 12 sets a magnification factor correction coefficient designated in advance. For example, a value of +0.5 is designated in advance for “higher enlargement ratio” and −0.5 in the case of “lower enlargement ratio”. Then, a value obtained by adding an enlargement factor correction coefficient to the enlargement factor used in the previous frame is output as an enlargement factor. As a result, the enlargement ratio can be adjusted according to the user's intention.

  Further, for example, the user inputs “enlarged image after enlargement” or “decrease image after enlargement” by operating the remote controller. When these inputs are made, the parameter acquisition unit 12 sets a pre-enlarged image size correction coefficient designated in advance. For example, a value of 1.1 is designated in advance for “larger image after enlargement”, and 0.9 is designated for “smaller image after enlargement”. Then, a value obtained by adding the post-enlargement image size correction coefficient to the post-enlargement image size magnification used in the previous frame is output as the post-enlargement image size magnification. Thereby, the size of the enlarged image can be adjusted according to the user's intention.

  In the above example, correction coefficient candidates are set in advance. However, the present invention is not limited to this, and a means for directly inputting a correction coefficient value may be provided. Further, a means for directly inputting the enlargement ratio and the image size magnification after enlargement may be provided without using the correction coefficient. Further, the input means is not limited to the remote controller, and other input devices such as a touch pad and a keyboard may be used. In the above example, both the enlargement ratio and the image size magnification after enlargement are used as parameters, but only one of them may be used, and the other may be a fixed value.

Similarly to the first embodiment, the partial enlargement calculation unit 13 performs image enlargement processing on the inside of the region specified by the enlargement region position information Cout on the input frame image. However, in this example, the values acquired by the parameter acquisition unit 12 are used for the enlargement ratio and the image size magnification after enlargement.
As described above, according to the image processing apparatus of the present embodiment, when the partial enlargement process is performed while following the enlargement area, the enlargement ratio and the enlargement area size magnification can be adjusted according to the user's intention.

(Variations in each embodiment)
The follow-up signal is set based on, for example, an input of a signal representing “start” or “end” using a remote controller. In the case of “start”, the follow execution signal is set to “ON”, and in the case of “end”, the follow execution signal is set to “OFF”. While there is no input, the follow-up execution signal is not changed as it was when there was an input before. The input device is not limited to a remote controller. For example, when the display unit 11 has a touch panel function, a menu screen may be displayed on the display unit, and a user may input from the touch panel, or a keyboard or the like is not illustrated. An input device may be used.

  Further, the previous frame image held in the frame memory 10 may hold the entire image or may be only a partial area used in the feature point movement amount calculation unit 6.

In the above, a method for setting a face image, a character image, and an image with motion as a target of interest has been described. However, the present invention is not limited to this. In addition, prioritization when there are a plurality of these may be set in the menu in advance, and any of face image detection mode, character image detection mode, and image detection mode with motion Alternatively, the user may select an area to be enlarged and then select an area to be enlarged.
Further, in the case of a video with depth information attached thereto, a method of discriminating an object existing in front from the depth information and recognizing the object as a target of attention may be used.
Further, when recognizing that there are a plurality of attention targets, a screen prompting to highlight the attention targets one by one and prompting the user to enter YES / NO is displayed. The highlighted attention target is the user's intention. The user may select (confirm) whether or not it is appropriate.

  In addition, the calculation method of the movement amount of each feature point is not limited to the above example, and the feature point detected in the previous frame image is calculated to which point in the input frame image is moved using the matching method. It is also possible to do it. Alternatively, block matching can be used. First, in the previous frame image, an area surrounded by squares of R pixels in the vertical and horizontal directions is set as a calculation area so that the target of interest fits, and this area is divided into vertical and horizontal Q blocks. However, R is set in advance so that the block size is P pixels vertically and horizontally, and R = P × Q. Then, block matching is performed between each block and the input frame image, and this is output as the amount of movement of each feature point. In the above example, the calculation area and the block have a square shape. However, the shape is not limited to a square, and other shapes may be used.

  The method described here detects the amount of movement for each feature point or small block, so it can detect the position of the target object not only for simple movements but also for changes such as rotation, deformation, and enlargement / reduction. Thus, the enlarged display of the target object can be followed. Note that the enlargement / reduction here can be applied not only when the entire image is enlarged / reduced, such as zoom-in / zoom-out, but also when only the target of interest approaches or moves away from the camera.

  In the above example, the shape of the enlarged region is a circle, but the shape is not limited to a circle, and a square or any other shape may be used. When the shape of the enlarged region is a circle, the shape is a circle, the center coordinates of the circle, and the radius are used as the enlarged region position information. In the case of an ellipse, the coordinate position is the enlarged region position information, and in the case of an ellipse, the shape is an ellipse, and the major axis, minor axis, ellipticity, and slope are the enlarged region position information. What is necessary is just to make it an arbitrary shape and the coordinate of an outer periphery position into enlarged area position information.

  In addition, the set target of interest may be always valid, and when a new target of interest is set, the target of interest may be updated, or multiple target targets can be set. The oldest attention object or the least frequently used attention object may be deleted. In addition, when performing partial enlargement during broadcast viewing, the target of attention may be cleared each time the program is switched, or a plurality of target of interest can be set for each channel, and the channel is switched, When returning to the original channel, the target of interest set in that channel last time may be enabled. Also, at the time of playback of content that has recorded a broadcast program or content that the user has taken with a camera, the target of attention may be set for each content, or the target of interest is set every time another content is played back. You may make it clear.

  So far, the method of calculating the movement amount of the feature point and the calculation method of whether or not the target object, that is, the feature point has been redisplayed, have been described in different ways, but instead of calculating the movement amount, for each frame Both may be the same method by detecting the position of the feature point. Depending on the environment where the machine power is scarce, it is possible to calculate the amount of movement of the feature points only around the area before the movement by using the previous frame feature point position information stored in the second memory 9. Since it is necessary to perform the entire region to determine whether or not the target (feature point) has been redisplayed, it may not be possible to perform every frame. In this case, the determination as to whether or not the target of interest (feature point) has been redisplayed may detect one frame over a period of a plurality of frames. In addition, when multiple attention targets are set, if not all attention targets are detected in the same frame, but one target is detected at intervals of time and cannot be detected. A method may be used in which the next target of interest is detected from that point.

The embodiments of the image processing apparatus according to the present invention have been described above. From these descriptions, the image processing method according to the present invention and a processing program for executing the image processing method as a program by a computer are also described. It will be easy to understand.
The image processing apparatus of the present invention may be configured as an image display apparatus integrated with an image display unit. Furthermore, it goes without saying that it may be provided in a video output device such as various recording media playback devices.

DESCRIPTION OF SYMBOLS 1 ... Following execution signal acquisition part, 2 ... User designation area | region acquisition part, 3 ... Attention object detection part, 4 ... Enlargement area determination part, 5, 13 ... Partial enlargement calculation part, 6 ... Feature point movement amount calculation part, 7 ... Attention target redisplay detection unit, 8... First memory, 9... Second memory, 10... Frame memory, 11.

Claims (6)

Feature point information indicating a target image included in a peripheral region at a position designated by the user with respect to a display image on the display screen or a region designated by the user, and a target of interest for detecting the display position of the feature point A detection unit;
A feature point information storage unit for storing the detected feature point information;
A frame memory for storing an image of the previous frame of the current display frame;
Detecting the display position of the feature point information stored in the feature point information storage unit in the image stored in the frame memory and the image of the current display frame, and / or calculating the movement amount of the display position of the feature point information A movement amount calculation unit to perform,
A current frame feature point position information storage unit for storing a display position of feature point information in the current display frame;
When the movement amount calculation unit cannot calculate the movement amount, whether or not the feature point information is included in the image of the subsequent frame, and if it is included, the display position of the feature point information is detected. A re-display detection unit to
The display position of the feature point information detected by the target object detection unit, the movement amount of the feature point information calculated by the movement amount calculation unit, or the display position of the feature point information detected in the current frame, or the re-display detection unit A partial enlarged display unit that determines an enlarged region including the target of interest based on the detected display position of the feature point information, and displays an enlarged image of the enlarged region superimposed on the display location of the target of interest; Prepared,
The enlarged image of the target of interest is superimposed on the display location of the target of interest according to the user's designation and displayed following the movement of the target of interest, and the image of the target of interest has been re-displayed after moving outside the display screen An image processing apparatus that displays an enlarged image following a case.   The image processing apparatus according to claim 1, further comprising an acquisition unit configured to acquire at least one of a size and an enlargement ratio of the enlarged image from a user.   The image processing apparatus according to claim 1, wherein the attention object detection unit detects a face image and / or a character image as the attention object.   When the depth information is attached to the input image, the attention target detection unit discriminates an object existing in front from the depth information, and detects the object as the attention target. The image processing apparatus according to claim 1 or 2.   5. The apparatus according to claim 1, further comprising: a confirmation unit that highlights the target of interest detected by the target of interest detection unit and causes the user to confirm whether or not the target of interest matches the user's intention. The image processing apparatus according to item 1.   The target object re-display detection unit determines that the feature point information is included in an image of a subsequent frame when the feature point information equal to or greater than a predetermined threshold is detected. The image processing apparatus according to any one of the above.

Images