JP2014182481A - Image processing program and image processing apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing program for facilitating a user specifying a parameter for deforming or rotating an image when a plurality of images are superimposed.SOLUTION: An image processing program, for superimposing images so that object images captured in the images overlap each other, causes a computer to execute the steps of: displaying an object image included in the images so as to superimpose an object image included in the images on a reference image included in the images; detecting user operation direction for drawing one linear trajectory in the object image; determining deformation direction or rotation direction of the object image according to the operation direction, to generate a processed image formed by deforming or rotating the object image in the deformation direction or the rotation direction; and displaying the processed image so as to be superimposed on the reference image.

Description

  The present invention relates to an image processing program and an image processing apparatus.

  Conventionally, many image processing techniques for superimposing a plurality of images so that images of a specific object overlap each other have been proposed and used in various applications. These applications are, for example, comparison and sharpening of images. In Patent Document 1, a plurality of images taken from different directions are superimposed and compared. In Patent Document 1, these images are converted into images viewed from a predetermined reference angle in order to cancel the influence of the difference in shooting direction. Specifically, distortion correction for converting a trapezoidal image into a rectangular image and correction by rotational movement are executed.

JP 2012-33734 A

  By the way, the series of processes described in Patent Document 1 are automatically executed by a computer. However, automatic superposition has a limit in terms of accuracy, and it may be desired to use it together with manual superposition. Therefore, the applicant of the present application has applied for a patent on a method for allowing a user to specify a deformation parameter such as distortion, a rotational movement parameter, or the like in a state where a differential image of a plurality of images is displayed (January 12, 2012). Application Japanese Patent Application No. 2012-4440). Further, the specification of the application discloses an example in which designation is performed numerically as a parameter designation mode.

  However, depending on the user, even such an operation of specifying a numerical value may be annoying. That is, the inventor of the present application has thought that there is room for further improvement in terms of user operability, although the method of Japanese Patent Application No. 2012-4440 is sufficiently expected to improve accuracy.

  An object of the present invention is to provide an image processing program and an image processing apparatus that allow a user to easily specify parameters for deformation or rotation of an image when a plurality of images are superimposed.

  An image processing program according to a first aspect of the present invention is an image processing program for superimposing the plurality of images so that images of an object captured by the plurality of images overlap. A step of displaying a target image included on a reference image included in the plurality of images, a step of detecting an operation direction of a user operation for drawing a single-line trajectory in the target image, and the operation direction And determining a deformation direction or a rotation direction of the target image, generating a processed image by deforming or rotating the target image in the deformation direction or the rotation direction, and processing the processed image on the reference image. And causing the computer to execute a step of displaying the image on the computer.

  Here, the deformation direction or the rotation direction of the target image is determined in accordance with the operation direction of the user's operation in the target image displayed to be superimposed on the reference image. The user's operation here is an operation that draws a single-line trajectory in the target image. Accordingly, the user can intuitively specify the deformation direction or the rotation direction of the target image so as to match the images of the specific target captured in both images while comparing the target image with the reference image. Therefore, the user can easily specify parameters for deformation or rotation of an image when a plurality of images are superimposed.

  The “user's operation to draw a single line trajectory” is a single trajectory such as a mouse operation for moving a single cursor or a touch panel operation for moving a single finger. The operation that draws. On the other hand, a touch panel operation called pinch-in / pinch-out, in which two fingers are brought closer / away from each other, is an operation that draws two trajectories and is not included in the user's operation here. . The “single line” referred to here may be, for example, a straight line, a curved line, or a bent line.

  An image processing program according to a second aspect of the present invention is the image processing program according to the first aspect, further causing the computer to execute a step of detecting an operation amount of the user operation in the operation direction. The step of generating the processed image further determines a deformation amount or a rotation amount of the target image according to the operation amount, and determines the target image by the deformation amount or the rotation amount in the deformation direction or the rotation direction. It is a step of generating the processed image deformed or rotated.

  Here, the deformation amount or the rotation amount of the target image is determined in accordance with the operation amount of the user's operation in the target image displayed to be superimposed on the reference image. The user's operation here is an operation that draws a single-line trajectory in the target image. Therefore, the user can intuitively specify the deformation amount or the rotation amount of the target image so as to match the images of the specific target object captured in both images while comparing the target image with the reference image. Therefore, the user can easily specify parameters for deformation or rotation of an image when a plurality of images are superimposed.

  An image processing program according to a third aspect of the present invention is the image processing program according to the first aspect or the second aspect, and in the step of detecting the operation direction, the operation direction is predetermined on the target image. Determining whether the direction is toward a given reference point or away from the reference point. Note that the reference point can be set at an arbitrary point in the target image, and is, for example, the center of the target image.

  Here, the determination is made based on whether the operation direction of the user's operation is a direction toward a predetermined reference point or a direction away from the predetermined reference point. Further, as described above, the operation direction determined here determines the deformation direction or the rotation direction of the target image. Therefore, for example, when the user performs an operation in a direction toward the reference point, the target image is reduced, and when an operation in a direction away from the reference point is performed, an intuitive operation interface that is enlarged can be realized. .

  An image processing program according to a fourth aspect of the present invention is the image processing program according to any one of the first aspect to the third aspect, wherein the operation direction is determined in advance in the step of detecting the operation direction. Determining whether the current direction is a predetermined direction or a direction opposite to the predetermined direction.

  Here, determination is made based on whether the operation direction of the user's operation is a predetermined direction or the opposite direction. Further, as described above, the operation direction determined here determines the deformation direction or the rotation direction of the target image. Therefore, for example, when the user performs an operation in the upward direction, the target image is distorted into a parallelogram so that the lower right point of the target image moves upward, and when the operation in the downward direction is performed, It is possible to realize an intuitive operation interface that distorts into a parallelogram so that the same point moves downward. As another example, when the user performs an upward operation, around a predetermined axis in a plane parallel to the target image (for example, an axis extending in the left-right direction through the center of the target image). When the target image is rotated in a predetermined direction and an operation in a downward direction is performed, an intuitive operation interface that reversely rotates around the same axis can be realized.

  An image processing program according to a fifth aspect of the present invention is the image processing program according to any one of the first to fourth aspects, wherein the step of displaying the target image superimposed on the reference image includes the target It is a step of displaying a difference image between an image and the reference image, or an image in which at least one of the target image and the reference image is translucent.

  Here, a difference image between the target image and the reference image or an image in which at least one of the target image and the reference image is made translucent and the two images are superimposed is displayed. The latter image is, for example, an average image or a weighted average image. When the former image is displayed, the user may deform or rotate the target image so that the target object image disappears from the difference image. Further, when the latter image is displayed, the target image may be deformed or rotated so that the target object in the latter image overlaps well. Therefore, the user can more intuitively specify the deformation or rotation parameters.

  An image processing program according to a sixth aspect of the present invention is an image processing program according to any one of the first to fifth aspects, wherein the target image and the image are selected from the plurality of images included in the same moving image. The computer is further caused to perform a step of selecting a reference image.

  The same moving image is likely to contain similar images with respect to the shape of the target object. Here, since a plurality of images to be superimposed are selected from the same moving image, it is possible to match the image of the target object of interest more accurately.

  An image processing program according to a seventh aspect of the present invention is the image processing program according to any one of the first to sixth aspects, wherein the deformation mode includes enlargement / reduction, distortion, and the object. At least one rotation about an axis extending in a plane parallel to the image is included.

  Note that “rotation” includes rotation as one aspect of deformation, that is, rotation around an axis extending in a plane parallel to the target image (see FIGS. 11A and 11B), and parallel to the target image. There is a rotation around a predetermined point in the plane (see FIG. 11C). In this specification, unless specifically stated otherwise, simply “rotating” means rotating movement without deformation of the target image.

  An image processing apparatus according to an eighth aspect of the present invention is an image processing apparatus for superimposing the plurality of images so that images of an object captured by the plurality of images overlap, A display control unit that displays a target image included on a reference image included in the plurality of images, and an operation detection unit that detects an operation direction of a user's operation that draws a single-line trajectory in the target image; An image processing unit that determines a deformation direction or a rotation direction of the target image according to the operation direction, and generates a processed image obtained by deforming or rotating the target image in the deformation direction or the rotation direction. The display control unit displays the processed image so as to overlap the reference image. Here, the same effect as the first aspect can be achieved.

  According to the present invention, the deformation direction or the rotation direction of the target image is determined in accordance with the operation direction of the user's operation in the target image displayed to be superimposed on the reference image. The user's operation here is an operation that draws a single-line trajectory in the target image. Accordingly, the user can intuitively specify the deformation direction or the rotation direction of the target image so as to match the images of the specific target captured in both images while comparing the target image with the reference image. Therefore, the user can easily specify parameters for deformation or rotation of an image when a plurality of images are superimposed.

1 is a block diagram of an image processing apparatus according to an embodiment of the present invention. It is a figure of the basic screen before image data is taken in. It is a figure of the basic screen after image data was taken in. The figure which shows the still image group which belongs to 1 timeline. The figure which shows a superimposition window. The figure which shows a candidate selection window. Another figure which shows a candidate selection window. Another figure which shows a candidate selection window. Another figure which shows a superposition window. Another figure which shows a superimposition window. The figure which shows the aspect of rotation deformation | transformation and rotation movement of an image. The figure which shows the aspect of the distortion deformation | transformation of an image. The figure which shows the example of a target image. The figure which shows the example of a reference | standard image. The figure which shows the image which overlap | superposed the image of FIG.13 and FIG.14. FIG. 15 is a diagram illustrating an image obtained by moving and transforming the image of FIG. 13 and overlaying the images of FIGS. 13 and 14. The figure which shows the difference image corresponding to the image of FIG. The figure which shows the difference image corresponding to the image of FIG. The figure which shows a feature point selection window. Another figure which shows a feature point selection window.

Hereinafter, an image processing program and an image processing apparatus according to an embodiment of the present invention will be described with reference to the drawings.
<1. Overview of Image Processing Device>
An image processing apparatus 1 shown in FIG. 1 is an embodiment of an image processing apparatus according to the present invention. The image processing apparatus 1 is a general-purpose personal computer. The image processing apparatus 1 includes a computer-readable recording medium 60 such as a CD-ROM, a DVD-ROM, or a USB memory that stores an image processing program 2 that is an embodiment of an image processing program according to the present invention. Etc. are provided and installed by etc. The image processing program 2 is application software for supporting image processing for moving images and still images. The image processing program 2 causes the image processing apparatus 1 to execute steps included in the operations described later.

  The image processing apparatus 1 includes a display 10, an input unit 20, a storage unit 30, and a control unit 40. These units 10 to 40 are connected to each other via a bus line, a cable 5 or the like and can communicate appropriately. The display 10 is composed of a liquid crystal display or the like, and displays a screen or the like described later to the user. The input unit 20 includes a mouse 21, a keyboard 22, and the like, and accepts an operation from the user for the image processing apparatus 1. The storage unit 30 is a non-volatile storage area configured from a hard disk or the like. The control unit 40 includes a CPU, a ROM, a RAM, and the like.

  The image processing program 2 is stored in the storage unit 30. A software management area 50 is secured in the storage unit 30. The software management area 50 is an area used by the image processing program 2. In the software management area 50, an original image area 51 and a processed file area 52 are secured. The role of each of the areas 51 and 52 will be described later.

  The control unit 40 virtually operates as a display control unit 41, an operation detection unit 42, and an image processing unit 43 by reading and executing the image processing program 2 stored in the storage unit 30. The display control unit 41 controls display of all elements such as screens, windows, buttons, and the like displayed on the display 10. The operation detection unit 42 detects all operations input from the user via the input unit 20. The image processing unit 43 executes various types of image processing.

<2. Details of Configuration and Operation of Image Processing Apparatus>
When the control unit 40 detects that the user has performed a predetermined operation via the input unit 20, the control unit 40 activates the image processing program 2. When the image processing program 2 is activated, a basic screen W1 (see FIG. 2) is displayed on the display 10.

<2-1. Importing image data>
The basic screen W1 accepts an instruction for taking image data into the original image area 51 from the user. The image data taken into the original image area 51 is subject to later-described reproduction processing and image processing. The control unit 40 captures image data from the still image file or the moving image file into the original image area 51. In this specification, a still image file is a data file in a still image format, and a moving image file is a data file in a moving image format.

  When capturing image data from a still image file, the user operates the input unit 20 to specify one still image file or one folder. In the former case, the control unit 40 causes the user to input the address path and file name in the storage unit 30 of the still image file. In the latter case, the control unit 40 allows the user to input the address path and folder name in the storage unit 30 of the folder. Thereafter, the control unit 40 stores the specified still image file or all the still image files in the specified folder as a still image file group in the original image area 51. In the present specification, when referring to a “group”, the number of elements is not limited to a plurality, and may be one.

  On the other hand, when capturing image data from a moving image file, the user operates the input unit 20 to input the address path and file name in the storage unit 30 of one moving image file. When the control unit 40 detects that the user has specified a moving image file, the control unit 40 displays a moving image capturing window (not shown) in an overlapping manner on the basic screen W1. The moving image capture window accepts selection of an arbitrary section from the user among all the sections of the timeline of the specified moving image file. When the control unit 40 detects that the user has selected a specific section via the input unit 20, the control unit 40 generates a still image file group corresponding to the frame group included in the section of the specified moving image file on a one-to-one basis. To do. Thereafter, the control unit 40 stores the still image file group in the original image area 51. Therefore, in the present embodiment, the image data to be subjected to playback processing and image processing to be described later is not a moving image file but a still image file.

  Note that the control unit 40 determines that the still image file group is taken along the timeline even if the still image file group captured in the original image area 51 is not derived from the moving image file but derived from the still image file. Are recognized as being arranged. The array is automatically determined from file attributes (file name, creation date, update date, etc.).

<2-2. Playback processing of still image files>
When the still image file group is taken into the original image area 51, the display control unit 41 displays the display window W2 (see FIG. 3) on the basic screen W1. The display windows W2 are created by the number of timelines of the still image file group taken into the original image area 51.

  First, one still image file (for example, a still image file corresponding to the first frame on the timeline) included in the still image file group captured in the original image area 51 is displayed in the display window W2. . Thereafter, as will be described later, the frame displayed in the display window W2 is switched in response to a user operation.

  As shown in FIG. 3, a window selection pull-down menu T1, a playback button T2, a frame advance button T3, a frame return button T4, and a timeline bar T5 are arranged on the basic screen W1.

  Even when there are a plurality of display windows W2, there is only one active display window W2. The window selection pull-down menu T1 accepts selection of which display window W2 is active from the user. Since the display window W2 corresponds to the timeline on a one-to-one basis, the window selection pull-down menu T1 is also a timeline selection pull-down menu. Hereinafter, a timeline corresponding to the active display window W2 is referred to as an active timeline, and a frame group belonging to the active timeline is referred to as an active frame group. A frame currently displayed in the active display window W2 is referred to as an active frame.

  The display control unit 41 can reproduce the active frame group as a moving image in the active display window W2. The playback button T2 receives a playback command from the user as a moving image of the active frame group. When it is detected that the user has pressed the play button T2 via the input unit 20, the display control unit 41 sequentially displays the frames included in the active frame group in the active display window W2 along the timeline. Display in frame-by-frame format. Note that the reproduction starts from the active frame at the time when the reproduction button T2 is pressed. The playback button T2 accepts a playback stop command from the user. When it is detected that the user has pressed the playback button T2 via the input unit 20 during playback, the display control unit 41 fixes the display in the active display window W2 to the active frame at that time.

  Each of the frame advance button T3 and the frame return button T4 accepts an instruction from the user to switch the active frame to the previous frame one by one along the active timeline.

  The timeline bar T5 is an object that schematically shows the active timeline. The timeline bar T5 is equally divided by the number of frames of the active frame group in the direction in which the bar extends. The nth divided region from the left on the timeline bar T5 corresponds to the nth frame on the active timeline (n is a natural number).

  As illustrated in FIG. 3, the display control unit 41 displays the divided area A1 corresponding to the selected frame group and the divided area A2 corresponding to the non-selected frame group in different display formats on the timeline bar T5. . The selected frame group is a frame group that belongs to the currently selected section on the active timeline. The non-selected frame group is a frame group that belongs to a section that is not currently selected on the active timeline.

  The timeline bar T5 accepts selection of an arbitrary section on the active timeline from the user. The section selected at this time may be a continuous section or a discontinuous section as shown in FIG. Specifically, the user can select any number of arbitrary frames from the active frame group by manipulating the divided areas on the timeline bar T5 via the input unit 20. A plurality of divided areas can be selected at the same time. The display control unit 41 immediately switches the active frame to the frame corresponding to the most recently selected divided region every time the divided region on the timeline bar T5 is selected by the user. The image processing unit 43 recognizes the selected frame group or the active frame as an image processing target to be described later.

<2-3. Image processing>
The image processing unit 43 can execute a plurality of image processing modules such as noise removal, sharpness, brightness / contrast / saturation adjustment, image resolution, addition of characters / arrows / mosaic, and superposition. The image processing module is incorporated in the image processing program 2.

  The user can select any one of the image processing modules in any order and any number of times by operating the basic screen W1 via the input unit 20. Each time the image processing unit 43 detects that the user has selected an image processing module, the image processing unit 43 executes the image processing module on the selected frame group or active frame at that time. Note that whether the execution target of the image processing module is the selected frame group or the active frame depends on the type of image processing. Also, executing the image processing module for the selected frame group means executing the image processing module for each frame included in the selected frame group.

  As the image processing module is sequentially executed once, twice, three times,... With respect to the frame, the frame is sequentially sorted into the first order, the second order, the third order,. It will be processed. The 0th frame corresponds to the still image file stored in the original image area 51. The (m + 1) th frame corresponds to the still image file after the image processing module is executed once for the mth frame still image file (m is an integer of 0 or more). The image processing unit 43 sequentially generates still image files corresponding to the first and subsequent frames, and separately stores these still image files in the processed file area 52.

  FIG. 4 is a conceptual diagram showing how an image group belonging to one timeline is managed by the image processing program 2. In FIG. 4, the N axis on the horizontal axis indicates the order of frames on the timeline, and the M axis on the vertical axis indicates the order of processing. A square corresponding to the coordinates (n, m) in the NM space in FIG. 4 represents the image I (n, m). The image I (n, m) is an mth-order image of the nth frame on the timeline (n is a natural number, and m is an integer of 0 or more).

The control unit 40 manages the value of the currently selected coordinate m as the parameter m _s for each frame. Immediately after the still image file group is taken into the original image area 51, the coordinate m _s has an initial value of 0. Thereafter, each time the image processing module is executed once, the coordinate m _s of the frame is incremented by one. Further, the user can freely change the coordinate m _s of an arbitrary frame by performing a predetermined operation via the input unit 20. Note that to execute the image processing module to the frame, it is to perform an image processing module to the m _s next image of that frame. Therefore, changing the coordinate m _s means changing the execution target of the image processing module. In addition, displaying a frame means displaying an image of the coordinate m _s of the frame. Therefore, changing the coordinate m _s also means changing the object displayed in the active display window W2.

<2-3-1. Overlay processing>
Hereinafter, superposition processing, which is one of image processing implemented in the image processing program 2, will be described. The superimposing process is a process of superimposing a plurality of images so that the images of the object captured by the plurality of images overlap for the purpose of clarifying and comparing the images. More specifically, the superimposition process of the present embodiment is an arbitrary timeline on the m _sth image (hereinafter referred to as the reference image) of the active frame (hereinafter referred to as the reference frame) at the start of the overlay process. The 0th-order or m _sth- order image of an arbitrary frame belonging to is superimposed.

  First, when it is detected that the user has instructed execution of the superimposition process, the display control unit 41 displays the superimposition window W3 shown in FIG. 5 on the basic screen W1. Furthermore, when it is detected that the user presses the additional button B1 on the overlapping window W3, the display control unit 41 displays the candidate selection window W4 shown in FIG. 6 on the basic screen W1. The candidate selection window W4 includes one or a plurality of images (hereinafter referred to as target) to be superimposed on the reference image from among all frame groups belonging to all timelines including the timeline to which the reference frame belongs (hereinafter referred to as reference timeline). This is a screen for allowing a user to select a frame (hereinafter referred to as a candidate frame) to which an image (hereinafter referred to as a candidate image) belongs.

  As shown in FIG. 6, a timeline selection pull-down menu C1, a playback button C2, a frame advance button C3, a frame return button C4, and a timeline bar C5 are arranged on the candidate selection window W4. These objects C1 to C5 have the same functions as the above-described objects T1 to T5 on the basic screen W1, respectively. A frame list D1 is arranged on the candidate selection window W4.

  In the frame list D1, the IDs of all the frames belonging to the time line currently selected in the candidate selection window W4 (hereinafter, selected time line) are arranged vertically according to the arrangement on the selected time line. . The selected timeline immediately after the candidate selection window W4 is displayed is the reference timeline, but the timeline selection pull-down menu C1 accepts an operation for changing the selected timeline to an arbitrary timeline from the user. . In the frame list D1, a check box is arranged on the left side of the ID of each frame. As a result, the user can select an arbitrary number of arbitrary frames from the frame group whose ID is displayed in the frame list D1 via this check box. The image processing unit 43 recognizes the frame selected by the user via this check box as the above candidate frame.

  Alternatively, the user can select a candidate frame via the timeline bar C5. The timeline bar C5 has the same function as the timeline bar T5 described above, but accepts selection of an arbitrary section on the selected timeline, not the active timeline, from the user. Specifically, when the user presses the add button E1 on the candidate selection window W4, one or a plurality of frames corresponding to one or a plurality of divided regions selected at that time on the timeline bar C5 are displayed. Recognized as a candidate frame by the image processing unit 43. Conversely, when the user presses the delete button E2 on the candidate selection window W4, one or more frames corresponding to the one or more divided areas selected at that time on the timeline bar C5 are candidate frames. Is released from.

  Further, as shown in FIGS. 7 and 8, a bar E3 is displayed below the divided area corresponding to the reference frame on the timeline bar C5, and the divided area corresponding to the candidate frame on the timeline bar C5 is displayed. A bar E4 having a display format different from that of the bar E3 is displayed below. Therefore, the user can easily grasp the position n on the selection timeline between the reference frame and the candidate frame. FIG. 7 clearly shows that the bar E3 indicating the position of the reference frame exists, but shows a state where the timeline to which the reference frame belongs is the selected timeline. On the other hand, FIG. 8 in which the bar E3 is not displayed shows a state where the timeline other than the timeline to which the reference frame belongs is the selected timeline.

  A reference image panel P1 on which a reference image is displayed is arranged on the upper left side of the candidate selection window W4, and a comparison image panel P2 is arranged on the right side thereof. The display control unit 41 can reproduce a frame group on the selected timeline in the comparative image panel P2 in response to a user operation on the play button C2, the frame advance button C3, and the frame return button C4. The reproduction process at this time is executed in the same manner as the reproduction process described above in the active display window W2. Therefore, the user can easily find an appropriate candidate image as a target image to be superimposed on the reference image displayed on the reference image panel P1, while reproducing various images in the comparison image panel P2. it can. For example, if it is desired to sharpen an image of a specific object captured in the reference image, an image in which the same object is captured from a similar angle may be found and used as a candidate image. At this time, if there is an image in which the target object is captured in almost the same shape as that in the reference image, the candidate can be used regardless of slight deformation of the target image or misalignment in the screen. It can be an image. This is because the target image selected from the candidate images to be superimposed on the reference image is appropriately subjected to parallel movement, rotational movement, enlargement / reduction, rotational deformation, and distortion deformation at the time of superposition as described later. Because it can.

The image displayed in the comparative image panel P2 is a reference image immediately after the candidate selection window W4 is displayed. Display in Comparative image panel P2 is also switched by the reproduction process described above, even when the divided area on the timeline bar C5 is selected by the user, immediately the m _s next frame corresponding to the divided region The image can be switched to. Also, other, when the frame regardless of the check boxes in the frame list D1 is selected by the user is also switched immediately to the m _s next image of the frame. Here, selecting a frame regardless of a check box in the frame list D1 means selecting a frame by clicking an area in which the frame ID is displayed instead of the check box portion. In the frame list D1, the background area of the ID of the most recently clicked frame, the background area of the ID of the reference image frame, and the background area of the ID of the other frame are represented by different colors and patterns. . Therefore, the user can easily identify the meaning of these frames.

  When it is detected that the user has pressed the OK button E5 on the candidate selection window W4, the display control unit 41 closes the candidate selection window W4 and updates the display of the overlapping window W3. Specifically, as shown in FIG. 9, the display control unit 41 displays the frame list D2 near the center of the overlapping window W3. In the frame list D2, the IDs of candidate frames selected via the candidate selection window W4 are arranged in the vertical direction, and a check box is arranged on the left side of the ID of each candidate frame. As a result, the user can select an arbitrary number of arbitrary candidate frames from the candidate frame group selected via the candidate selection window W4 via this check box. The image processing unit 43 recognizes the candidate frame selected by the user via the check box as a frame to which the target image belongs (hereinafter, target frame).

  Also, the frame list D2 accepts an operation for selecting one candidate frame from the user by clicking an area in which the candidate frame ID is displayed instead of the check box portion. Hereinafter, a candidate frame selected by this method is referred to as a selected frame. In the frame list D2, the background area of the ID of the selected frame and the background area of the ID of the other frame are represented by different tones (see FIG. 10).

Then, as shown in FIG. 10, in the state where the selection frame is set, the display control unit 41 displays the candidate image area B2 on the upper part of the overlapping window W3. Candidate image area B2 is an area that displays the zeroth-order or the m _s next image of the selected frame. Here, or so that the zeroth-order or the m _s One of the following images are displayed, by setting on the overlay window W3 using image selection pull-down menu is displayed on B3. The use image selection pull-down menu B3 has two choices of “latest image” or “original image”, and can select any value for the selected frame. As a result, when the former is selected by the user, the m _sth image of the selected frame is displayed in the candidate image area B2, and when the latter is selected, the 0th image of the selected frame is displayed. Is displayed. The setting value of the “latest image” or “original image” for each candidate frame is the default value for the former. The user operates the use image selection pull-down menu B3 via the input unit 20 to switch the 0th or m _sth image of the selected frame in the candidate image area B2 and display the selected frame. In the case of the target frame, it can be determined which image is appropriate as the target image to be superimposed on the reference image.

  Also, a preview selection area B4 and a method designation area B5 are arranged on the overlapping window W3. Here, when the user does not check the preview selection area B4, the reference image is displayed in the active display window W2, but when the preview selection area B4 is checked, the reference image is active. Various images other than the reference image, such as a target image or an image obtained by processing the target image, can be displayed in the display window W2. The method designation area B5 is an area that is validated only while the preview selection area B4 is checked. The method designation area B5 is used to designate how to preview an image generated in the active display window W2. Accept from. Specifically, as shown in FIG. 10, the method designation area B5 is an image that is previewed in the active display window W2 from four options “all”, “reference”, “immediately”, and “selection”. The designation of the generation method (hereinafter, preview image) is alternatively accepted from the user.

More specifically, in a state where “all” is selected in the method designation area B5, the display control unit 41 is currently selected by the user via the frame list D2 in the active display window W2. all of the target frame 0th or the m _s the following images are, that is, displayed over all of the target image on the reference image. Here, the image of the order m currently set by the user is selected as the target image of each target frame via the above-described use image selection pull-down menu B3. If no target image is selected, the reference image is displayed in the active display window W2.

  Here, the user can easily move and / or move an image that is the target image and is currently selected in the frame list D2 regardless of the check box (hereinafter, the selection target image) in the active display window W2. Can be deformed. In this embodiment, as movement and / or deformation modes, parallel movement in the X-axis and Y-axis directions, enlargement / reduction in the X-axis and Y-axis directions, rotational deformation in the X-axis and Y-axis directions, and around the Z-axis Rotational movement and distortion deformation in the X-axis and Y-axis directions. Here, the X-axis and the Y-axis are axes that pass through the center of the screen and extend in the horizontal direction and the vertical direction, respectively, and the Z-axis is a direction that passes through the center of the screen and is orthogonal to the screen.

  A specific movement and / or deformation operation method will be described. First, in the case of parallel movement in the X-axis and Y-axis directions, the user operates the mouse 21 to move the cursor in a clicked state in the selection target image displayed in the active display window W2 (hereinafter referred to as “click”). , Click move). At this time, the operation direction and the operation amount of the click movement operation are detected by the operation detection unit 42. The image processing unit 43 determines the parallel movement direction of the selection target image according to the operation direction detected by the operation detection unit 42, and determines the selection target image according to the operation amount detected by the operation detection unit 42. Determine the amount of translation. More specifically, the image processing unit 43 translates the selection target image by a1 pixels in the X direction when the cursor moves by a1 pixels in the X direction and a2 pixels in the Y direction by the click movement operation. And translate by a2 pixels in the Y direction. A1 and a2 may be positive values or negative values. Further, in the present embodiment, the operation amount of the user's cursor matches the translation amount, but in other actual forms, the two may have a correlation such as a proportional relationship and may be mismatched.

  Next, in the case of enlargement / reduction in the X-axis and Y-axis directions, the user operates the mouse 21 while pressing a specific key (for example, the F1 key) on the keyboard 22 to move within the active display window W2. Click and move the cursor in the selection target image displayed in. At this time, when a specific key on the keyboard 22 is pressed, the image processing unit 43 recognizes that the click movement operation means enlargement / reduction. Also at this time, the operation detection unit 42 detects the operation direction and the operation amount of the click movement operation, as in the case of the parallel movement described above. Then, the image processing unit 43 determines the enlargement / reduction direction of the selection target image according to the operation direction detected by the operation detection unit 42, and selects according to the operation amount detected by the operation detection unit 42. The amount of enlargement / reduction of the target image is determined. More specifically, for each of the X-axis and Y-axis directions, the image processing unit 43 moves the cursor to a predetermined reference point on the selection target image (in this embodiment, the center of the selection target image). ) Or in a direction away from the reference point, and a quantitative movement amount is also determined. Then, when it is determined that the cursor is separated by b pixels from the reference point in the X direction, the image processing unit 43 enlarges the selection target image in the X direction at a magnification of (1 + c1 · b) times, and in the X direction. If it is determined that the reference point has approached b pixels, the image is reduced in the X direction at a magnification of (1-c1 · b) (c1 is a positive constant). Note that b is a positive value. The same applies to the Y direction.

  Subsequently, in the case of rotational deformation in the X-axis and Y-axis directions, the user operates the mouse 21 while pressing a specific key (for example, F2 key) on the keyboard 22 to enter the active display window W2. Click and move the cursor in the displayed selection target image. At this time, when a specific key on the keyboard 22 is pressed, the image processing unit 43 recognizes that the click movement operation means rotational deformation. Also at this time, as in the case of the above-described parallel movement or the like, the operation detection unit 42 detects the operation direction and the operation amount of the click movement operation. The image processing unit 43 determines the rotational deformation direction of the selection target image according to the operation direction detected by the operation detection unit 42, and selects the selection target according to the operation amount detected by the operation detection unit 42. Determine the amount of rotational deformation of the image. More specifically, the image processing unit 43 determines whether the cursor is moved in the positive Y-axis direction by the click movement operation or in the negative Y-axis direction, which is the opposite direction, and quantitative movement is performed. Also judge the amount. When it is determined that the cursor has moved by d1 pixels in the positive Y-axis direction as shown in FIG. 11A, the image processing unit 43 moves the selection target image around the X axis by c2 · d1 °. When it is determined that the lens has moved by d1 pixels in the negative Y-axis direction, it is deformed to rotate in the positive direction around the X-axis by c2 · d1 ° (c2 is , Positive constant). Similarly, the image processing unit 43 determines whether the cursor is moved in the positive X-axis direction by the click movement operation or in the negative X-axis direction, which is the opposite direction, and also determines the quantitative movement amount. To do. Then, as shown in FIG. 11B, when it is determined that the cursor has moved by d2 pixels in the positive direction of the X axis, the image processing unit 43 moves the selection target image around the Y axis by c2 · d2 °. If it is determined that the pixel has moved by d2 pixels in the negative direction of the X axis, it is deformed to rotate in the positive direction around the Y axis by c2 · d2 °. D1 and d2 are positive values.

  Subsequently, in the case of rotational movement around the Z axis, the user operates the mouse 21 while pressing a specific key (for example, the F3 key) on the keyboard 22 and is displayed in the active display window W2. Click and move the cursor in the selected image. At this time, when a specific key on the keyboard 22 is pressed, the image processing unit 43 recognizes that the click movement operation means a rotational movement. Also at this time, as in the case of the above-described parallel movement or the like, the operation detection unit 42 detects the operation direction and the operation amount of the click movement operation. Then, the image processing unit 43 determines the rotation direction of the selection target image according to the operation direction detected by the operation detection unit 42, and selects the selection target image according to the operation amount detected by the operation detection unit 42. Determine the amount of rotation. More specifically, the image processing unit 43 determines whether the cursor has been rotated in the positive direction around the Z-axis by a click movement operation or in the negative direction, which is the opposite direction, and quantitative rotation is performed. Also judge the amount. Then, as shown in FIG. 11C, when it is determined that the cursor has rotated by e ° around the Z axis in the positive direction, the image processing unit 43 sets the selection target image by c3 · e ° in the Z axis. When it is determined that the lens has been rotated so as to rotate in the positive direction and is rotated by e ° in the negative direction, it is deformed so as to be rotated in the negative direction by c3 · e ° (c3 is a positive constant). Note that e is a positive value. In addition, the click movement operation that can be recognized by the image processing unit 43 here does not have to be an operation in which the user moves the cursor strictly in an arc shape around the Z axis. For example, the value of e can be detected even with a linear click movement operation. For example, the value of e is a line segment connecting a cursor movement start point by a click movement operation and a predetermined reference point on the selection target image (in this embodiment, the center of the selection target image), and the movement of the cursor. It can be detected as an angle formed by a line segment connecting the end point and the reference point.

  Subsequently, in the case of distortion deformation in the X-axis and Y-axis directions, the user operates the mouse 21 while pressing a specific key (for example, the F4 key) on the keyboard 22, and enters the active display window W2. Click and move the cursor in the displayed selection target image. At this time, when a specific key on the keyboard 22 is pressed, the image processing unit 43 recognizes that the click movement operation means distortion deformation. Also at this time, as in the case of the above-described parallel movement or the like, the operation detection unit 42 detects the operation direction and the operation amount of the click movement operation. Then, the image processing unit 43 determines the distortion deformation direction of the selection target image according to the operation direction detected by the operation detection unit 42, and selects the selection target according to the operation amount detected by the operation detection unit 42. Determine the amount of distortion of the image. More specifically, the image processing unit 43 determines whether the cursor is moved in the positive X-axis direction by the click movement operation or in the negative X-axis direction, which is the opposite direction, and quantitative movement is performed. Also judge the amount. When it is determined that the cursor has moved by f1 pixels in the positive direction of the X axis, as shown in FIG. 12A, the image processing unit 43 sets the selection target image to the lower right point of the selection target image. Is distorted into a parallelogram so that it moves in the positive direction of the X axis by c4 · f1 pixels, and if it is determined that it has moved by f1 pixels in the negative direction of the X axis, It is distorted into a parallelogram so as to move (c4 is a positive constant). Similarly, the image processing unit 43 determines whether the cursor has been moved in the positive Y-axis direction by the click movement operation or in the negative Y-axis direction, which is the opposite direction, and also determines the quantitative movement amount. To do. When it is determined that the cursor has moved by f2 pixels in the positive Y-axis direction as shown in FIG. 12B, the image processing unit 43 sets the selection target image to the lower right point of the selection target image. Is distorted into a parallelogram so that it moves by c4 · f2 pixels in the positive direction of the Y axis, and if it is determined that it has moved by f2 pixels in the negative direction of the Y axis, Distort into a parallelogram to move. Note that f1 and f2 are positive values.

  Each time the click movement operation as described above is performed, the movement and / or deformation is immediately reflected in the display in the active display window W2. In other words, the display control unit 41, in the active display window W2, replaces the original selection target image with a preview image obtained by superimposing the moved and / or deformed selection target image on the reference image and other target images. Is displayed.

  As a result, the user can intuitively move and / or deform the selection target image by a click movement operation that draws a single-line trajectory in the selection target image. As described above, in the state where “all” is selected in the method designation area B5, the selection target image is displayed on the reference image in the active display window W2. Therefore, the user can easily specify parameters for movement and / or deformation so as to match the images of the specific object captured in both images while comparing the selection target image with the reference image. In addition, the user can check the selection target image after movement and / or deformation in real time in the active display window W2 every time a click movement operation is performed. Therefore, the user repeats the execution of the above-described click movement operation as necessary, and moves the selection target image such that the target image of interest is appropriately overlapped between the selection target image and the reference image. / Or deformation parameters can be specified.

  Here, as shown in FIG. 10, a parameter designation area B6 that displays the above-described movement and deformation parameters of the selection target image and receives an operation for changing these parameters from the user is displayed on the overlapping window W3. Has been placed. In this area B6, parameters are displayed and changed quantitatively using numerical values. The parameters displayed in this area B6 are linked in real time to the above-described click movement operation for movement and / or deformation, and each time the selection target image is moved and / or deformed in the above-described manner. And immediately updated to a value reflecting the movement and / or deformation. Further, the display control unit 41 displays the selection target image displayed in the active display window W2 every time it is detected that the user has performed an operation of changing these parameters on the parameter designation area B6. Move and / or deform immediately according to the amount of change. That is, the movement and / or deformation parameter designation is realized by two operations having different properties, that is, an intuitive operation by clicking movement and a quantitative designation operation using numerical values. Therefore, the user can make a fine adjustment finely by the latter operation after the image of the target object is almost easily matched between the selection target image and the reference image by the former operation.

  When the user wants to superimpose a plurality of target images on the reference image, the above-described operation of superimposing each target image on the reference image while sequentially selecting the plurality of target images as the selection target image. Can be done. Then, at the stage where the movement and / or deformation of the plurality of target images is completed, the plurality of target images and the reference image are displayed on the preview image in a state where “all” is selected in the method designation area B5. The images of all the objects shown in the above are almost the same.

  Here, returning to the description of the method for generating the preview image, in a state where “reference” is selected in the method designation area B5, the display control unit 41 uses only the selection target image as a reference. Display overlaid on the image. In addition, in a state where “selection” is selected in the method designation area B5, the display control unit 41 displays the selection target image alone as a preview image. Further, in a state where “immediately before” is selected in the method designation area B5, the display control unit 41 displays the selection target image as a preview image so as to be superimposed on the target image one level higher in the frame list D2. . However, when such a target image does not exist, only the selection target image is displayed superimposed on the reference image. Even when such a preview image is displayed, the selection target image displayed in the active display window W2 is the same as the case where “all” is selected in the method designation area B5. In a manner, it can be moved and / or deformed. Also in this case, when the selection target image is moved and / or deformed, the movement and / or deformation is immediately reflected in the display in the active display window W2 and the parameter designation area B6.

  FIG. 15 is an image obtained by superimposing the target image shown in FIG. 13 on the reference image shown in FIG. FIG. 16 is an example in which the target image is translated and enlarged so that the images of the car that is the target object of interest overlap appropriately.

  Further, a difference selection area B7 is arranged in the overlapping window W3. The difference selection area B <b> 7 is an area that accepts designation of an aspect of superposition of a plurality of images that can be previewed in the active display window W <b> 2 from the user. Specifically, the display control unit 41 displays a difference image when “difference” is checked in the difference selection area B7, and displays a plurality of images semi-transparently when the check is not checked. The superimposed image (hereinafter, translucent image) is displayed.

  Here, the semi-transparent image is an average image in principle in this embodiment, but may be a weighted average image. Specifically, a transmittance designation area B8 for adjusting the transmittance of the selection target image is arranged in the overlapping window W3. The display control unit 41 generates a superimposed image after setting the transparency (weight at the time of average) of the selection target image in accordance with the transmittance designated by the user via the area B8. On the other hand, a difference image is an image that displays only a difference between a plurality of images to be superimposed, for example, a selection target image and a reference image. Therefore, the user can intuitively understand the positional shift between the two images regardless of the superposition mode. Note that FIGS. 15 and 16 described above are examples of the semi-transparent image, but FIGS. 17 and 18 show cases where FIGS. 15 and 16 are changed to the mode of the difference image, respectively. As can be seen from these figures, when the semi-transparent image is displayed, the user can move and / or deform the selection target image so that the target objects in the semi-transparent image overlap well. That's fine. Further, when the difference image is displayed, the selection target image may be moved and / or deformed so that the image of the target object on the difference image disappears.

  When the image processing unit 43 detects that the user has pressed the OK button B9 on the overlapping window W3, the image processing unit 43 performs all the movement and / or deformation after that point according to the transmittance at that point. An image in which the image is superimposed on the reference image is generated. At the same time, the superimposed image is stored in the processed file area 52 as a still image of the next order m of the reference frame. In the present embodiment, the order m of the target frame does not advance.

<3. Application>
The image processing program 2 can handle image processing for a wide variety of moving images. For example, the image processing program 2 can be used in a scene where an organization such as the police analyzes a surveillance video of a security camera for investigation of an incident. For example, the suspect's car license plate may be captured in the surveillance video of the security camera. In such a case, if the above-described superimposition process is executed with the license plate as an object, the resulting superimposed image will read a number that was previously unclear and could not be read. Is also possible. Further, the above-described superimposing process can also be used to confirm the size and position of the target object of interest. For example, to estimate the size of an object by superimposing another image of an object with a known size on an image that shows the object of interest. Can do.

<4. Modification>
As mentioned above, although one Embodiment of this invention was described, this invention is not limited to the said embodiment, A various change is possible unless it deviates from the meaning. For example, the following changes can be made. Moreover, the gist of the following modifications can be combined as appropriate.

<4-1>
The mode of operation of the user who draws a single linear locus for movement and / or deformation of the target image is not limited to that described above. For example, the click movement operation described above can be an operation of simply moving the cursor within the selection target image with the mouse 21 while designating the start and end of the movement and / or deformation operation with the keyboard 22.

  Further, instead of or in addition to the mouse 21 and the keyboard 22, a touch panel is provided on the display 10, and the above-described click movement operation is equivalent to an operation using the touch panel, for example, a drag operation such as dragging a finger on the screen, a screen It can be substituted by flicking such as paying a finger above.

<4-2>
In the above embodiment, the movement amount and / or deformation amount of the selection target image is determined by the image processing unit 43 in accordance with the operation amount of the click movement operation. However, such control may be omitted, and only the movement direction and / or the deformation direction may be automatically determined from the click movement operation. In this case, the amount of movement and / or the amount of deformation by one click movement operation can be a constant amount.

<4-3>
The above superimposition processing can be implemented even in an image processing program that manages a plurality of still images without having a management unit called a timeline. However, there is a high possibility that similar images are included in the same moving image with respect to the shape of the target object. Therefore, as in the above-described embodiment, it is desirable to have a device that makes it easy to select images included in the same moving image. In the above embodiment, the target image and the reference image can be selected from an arbitrary timeline. However, from the above viewpoint, the target image can be selected only from the same timeline as the reference image. Also good.

<4-4>
In the above-described embodiment, the target image is moved and / or deformed during the superimposition process. However, the reference image may be moved and / or deformed, and both may be moved and / or deformed.

<4-5>
The example of a translucent image is not restricted to what was mentioned above. For example, in the above-described embodiment, all of the plurality of superimposed images are made translucent, but only a part of the plurality of superimposed images may be made translucent (for example, only the reference image, the target Only images, only images specified by the user, etc.). Further, instead of the translucent image or the difference image, a plurality of images may be superimposed in another manner.

<4-6>
In the above embodiment, the types of translation and enlargement / reduction, rotational deformation, rotational movement and distortion deformation and / or deformation are distinguished by pressing a specific key on the keyboard 22. However, the mode may be switched by pressing a virtual button on the screen.

<4-7>
The deformation | transformation aspect of the target image in the said embodiment is an illustration, and it is also possible to deform | transform a target image by the other method. For example, the image can be deformed so as to be distorted into a trapezoid.

<4-8>
In the above embodiment, the movement and / or deformation parameter designation is realized by an intuitive operation by clicking movement and a quantitative designation operation using numerical values. Matching can also be performed automatically. For example, when the user orders execution of automatic superposition by performing a predetermined operation, the display control unit 41 displays a feature point selection window W5 for displaying a reference image as shown in FIG. In this image, when the user surrounds an object to be superimposed, for example, a car with a rectangular region R by operating the mouse 21 or the keyboard 22, the image processing unit 43 displays the feature points in the region R. Is automatically extracted. In response to this, the display control unit 41 displays the feature points with marks S on the feature point selection window W5 as shown in FIG. When the OK button on this window W5 is pressed, these feature points are stored. Thereafter, when the target image is selected on the candidate selection window W4, the image processing unit 43 extracts the corresponding feature points from the target image, and sets the target image and the reference image so that the corresponding feature points match each other. Overlapping. This superimposed image is displayed in the active display window W2 in response to an appropriate operation by the user, and then the user performs the fine adjustment in the active display window W2 as in the above embodiment. An intuitive operation by clicking movement and a quantitative designation operation using numerical values can be performed on the superimposed image.

<4-9>
The gist of the above modification can be combined as appropriate.

1 Image processing device (computer)
2 Image processing program 41 Display control unit 42 Operation detection unit 43 Image processing unit

Claims (8)

An image processing program for superimposing the plurality of images so that images of an object captured by the plurality of images overlap.
Displaying the target images included in the plurality of images superimposed on a reference image included in the plurality of images;
Detecting an operation direction of a user's operation for drawing a single-line trajectory in the target image;
Determining a deformation direction or a rotation direction of the target image according to the operation direction, and generating a processed image obtained by deforming or rotating the target image in the deformation direction or the rotation direction;
Causing the computer to execute a step of displaying the processed image superimposed on the reference image.
Image processing program. Further causing the computer to execute a step of detecting an operation amount of the user operation in the operation direction,
The step of generating the processed image further determines a deformation amount or a rotation amount of the target image according to the operation amount, and determines the target image by the deformation amount or the rotation amount in the deformation direction or the rotation direction. Generating the transformed or rotated processed image;
The image processing program according to claim 1. The step of detecting the operation direction includes a step of determining whether the operation direction is a direction toward a predetermined reference point on the target image or a direction away from the reference point.
The image processing program according to claim 1 or 2. The step of detecting the operation direction includes a step of determining whether the operation direction is the predetermined direction or a direction opposite to the predetermined direction.
The image processing program according to claim 1. The step of displaying the target image superimposed on the reference image is a step of displaying a difference image between the target image and the reference image, or an image in which at least one of the target image and the reference image is translucent. is there,
The image processing program according to claim 1. Causing the computer to further execute a step of selecting the target image and the reference image from the plurality of images included in the same moving image.
The image processing program according to claim 1. The deformation mode includes at least one of enlargement / reduction, distortion, and rotation around an axis extending in a plane parallel to the target image.
The image processing program according to claim 1. An image processing apparatus for superimposing the plurality of images so that images of an object captured by the plurality of images overlap.
A display control unit that displays the target images included in the plurality of images superimposed on a reference image included in the plurality of images;
An operation detection unit that detects an operation direction of a user's operation that draws a single-line trajectory in the target image;
An image processing unit that determines a deformation direction or a rotation direction of the target image according to the operation direction, and generates a processed image obtained by deforming or rotating the target image in the deformation direction or the rotation direction;
The display control unit displays the processed image so as to overlap the reference image.
Image processing device.