JPWO2020170604A1 - Information processing equipment, information processing methods, programs - Google Patents

Abstract

To improve the degree of freedom in drawing for subject tracking. On the first screen for displaying a predetermined image in the display area, the designation of the arrangement position in the display area to be tracked is accepted. For example, the user can specify the position (position in the display area) where the tracking target subject is placed while the image content can be recognized on the display screen.

Description

The present technology relates to an information processing apparatus, an information processing method, and a program, and particularly to a technical field of a user interface for accepting various specifications related to image processing.

There are known techniques for performing various image processing on moving images captured by an image pickup device.
Further, in the field of image processing, a tracking technique for tracking a target subject among the subjects projected in an image is widely known (see, for example, Patent Document 1 below).

Japanese Unexamined Patent Publication No. 2015-111746

By the way, in recent years, users can easily perform image imaging and image editing using mobile terminals such as smartphones and tablets, cameras themselves, personal computers, and the like, and video posting is also popular.
In such an environment, the user may not output the captured image as it is, but may execute an image process of cutting out a part of the captured image (original image) in consideration of composition and the like. It is conceivable to cut out such an image so that the portion of the original image in which the target subject is projected is included. At this time, the device that performs the image processing uses the subject tracking technology to perform the image processing. It is conceivable to set the cropping range so that the target subject specified by the user is included in the image frame, and crop the image in the set range.

Therefore, the purpose of this technique is to improve the degree of freedom in drawing with respect to the tracking of the subject.

The information processing apparatus according to the present technology includes a reception unit that accepts designation of an arrangement position in the display area to be tracked on a first screen for displaying a predetermined image in the display area.
For example, the user can specify the position (position in the display area) where the tracking target subject is placed while the image content can be recognized on the display screen.

In the information processing apparatus according to the present technology described above, it is conceivable that the predetermined image is an image cut out from the original image.
That is, the image displayed in the display area is a cutout image from the original image.

In the information processing apparatus according to the present technology described above, it is conceivable that the reception unit receives the designation of the tracking target.
That is, from among the subjects in the image, the designation of the subject as the tracking target is accepted.

The information processing apparatus according to the present technology described above includes a display control unit that controls the display of the predetermined image so that the tracking target is arranged at a position in the display area based on the designated arrangement position. Can be considered.
For example, an image that reflects the placement position of the tracking target subject specified by the user is displayed.

In the information processing apparatus according to the present technology described above, it is conceivable that the reception unit receives the designation of the range to be cut out from the original image.
For example, on the display screen, the user can specify the range included in the image frame of the cutout image among the original images.

In the information processing apparatus according to the present technology described above, it is conceivable that the display control unit displays an operation image used for the operation of designating the arrangement position of the tracking target on the first screen.
For example, an operation image that serves as a aim for designating an arrangement position is superimposed and displayed on the display area so that the user can specify a position in the image frame by the operation image.

In the information processing apparatus according to the present technology described above, it is conceivable that the reception unit receives the designation of the tracking target and the designation of the arrangement position of the tracking target based on the operation on the operation image.
For example, by an operation of designating a certain place with an operation image on the display area, the subject of that part is set as the tracking target, and the position of the operation image is accepted as the placement position of the tracking target.

In the information processing apparatus according to the present technology described above, the predetermined image is an image cut out from the original image, and the reception unit specifies a range to be cut out from the original image and the operation image is used as an operation for the screen. It is conceivable to accept operations related to the tracking target.
For example, the range cut out from the original image is displayed on the display screen, and the operation image is displayed on the image. On the screen, the change of the cutout range and the movement of the operation image are accepted.

In the information processing apparatus according to the present technology described above, the reception unit recognizes the operation on the screen as a state of recognizing the operation on the screen as a designated operation of a range cut out from the original image and an operation related to the tracking target by the operation image. It is conceivable to switch between states.
For example, the recognition mode of the user operation performed on the screen is switched by the operation prepared by the icon on the display screen.

In the information processing apparatus according to the present technology described above, the reception unit recognizes the operation on the screen as the operation for designating the image range cut out from the original image on the second screen, and on the first screen, the operation is designated. It is conceivable to recognize the operation on the screen as an operation related to the tracking target by the operation image.
The recognition of the user operation performed on the screen is switched depending on whether the display screen is in the state of the first screen or the state of the second screen.

In the information processing apparatus according to the present technology described above, the display control unit performs a display for changing the range of the image to be the cutout range in response to the operation recognized by the reception unit on the second screen. It is conceivable to display the operation image to move the designated position in response to the operation recognized by the reception unit on the first screen.
When the display screen is the second screen, for example, the image of the cutout range is displayed in the entire screen or in the display area, but it is changed according to the operation. When it is the first screen, the designated position by the operation image is moved according to the operation.

In the information processing apparatus according to the present technology described above, in the second screen, the original image changes in the display area according to the operation, and in the first screen, the display area changes according to the operation. It is conceivable that the arrangement position of the operation image is changed in the above.
That is, on the second screen, image changes such as movement, enlargement, reduction, and rotation of the original image occur in the display area according to the operation, and on the first screen, for example, in the display area according to the operation. The original image does not change, but the position of the operation image changes.

In the information processing apparatus according to the present technology described above, it is conceivable that the original image is one image of an image group composed of a plurality of images.
This makes it possible for the user to specify the arrangement position of the target subject in the image frame of the cutout image at an arbitrary position for each image constituting the image group.

In the above-mentioned information processing apparatus according to the present technology, the designated image is specified for at least one image after the order of the images for which the designation of the arrangement position is accepted among the plurality of images consecutive in a predetermined order. It is conceivable to perform image cutting processing so that the tracking target is arranged at a position in the display area based on the arrangement position.
As a result, the placement position of the tracking target is specified for one image out of a plurality of consecutive images in a predetermined order, and the frame images located after that in order are also tracked to the position based on the specified position. Image cropping processing to position the target will be performed.

In the information processing apparatus according to the present technology described above, it is considered that the image group is composed of a series of frame images constituting a moving image.
This makes it possible for the user to specify the arrangement position of the target subject in the image frame of the cut-out image at an arbitrary position when the image is cut out for each frame image constituting the moving image.

In the information processing apparatus according to the present technology described above, the reception unit cuts out from the original image so that the tracking target is arranged at a position in the display area based on the designated arrangement position. It is conceivable to accept the designation of the tracking section.
This allows the user to arbitrarily specify the tracking section as well.

In the information processing apparatus according to the present technology described above, it is conceivable that the display control unit controls to display an image within the cutout range from the original image and an image outside the cutout range in different display modes. Be done.
For example, the entire original image is displayed on the display screen, but the display mode is such that the inside of the cutout range and the outside of the cutout range can be distinguished.

In the above-mentioned information processing apparatus according to the present technology, a shake correction unit that performs a sticking process to a virtual celestial sphere for each frame of image data constituting a moving image and performs shake correction using the attitude information corresponding to the frame. The display is provided with a cutout range setting unit for performing a protrusion prevention process for correcting the cutout range according to the designation of the cutout range received by the reception unit so that the cutout range does not protrude from the area where the shaking correction is effectively maintained. It is conceivable that the control unit controls the display of the cut-out image reflecting the protrusion prevention processing performed by the cut-out range setting unit.
For example, the shake caused by camera shake in the image captured by the image pickup device is corrected by coordinate transformation on the virtual celestial sphere, and the cutout range of the image projected in a plane with this shake correction is set based on the user's operation. do. In this case, the protrusion prevention process is performed so that the cutout range does not exceed the range in which the shake correction is effectively maintained. The display control unit performs display control so that the user can recognize the cutout range of the range limited by this protrusion prevention process.

In the above-mentioned information processing apparatus according to the present technology, the display control unit controls so that display is performed based on the allowable movable range that allows fluctuation of the image cutout range set in the protrusion prevention process. Can be considered.
The protrusion prevention process is a process of keeping the cutout range set based on the operation information within the range in which the shake correction is effective, and at that time, the allowable movable range is set as the range in which the shake correction is effectively maintained. That is, it is a range in which the cutting range may be moved as long as it is within the allowable movable range. On the display, the display is made so that the user can recognize this allowable movable range.

In the information processing apparatus according to the present technology described above, it is conceivable that the display control unit controls the display so as to clearly indicate that the operation for designating the cutout range exceeding the allowable movable range is restricted.
For example, when the allowable movable range is exceeded, the display is executed so that the cutout range cannot be changed or the cutout range returns to the allowable movable range.

In the information processing method according to the present technology, the information processing apparatus performs a process of accepting the designation of the arrangement position in the display area to be tracked on the first screen for displaying a predetermined image in the display area.
This enables the user to specify the position (position in the display area) where the tracking target subject is placed, for example, in a state where the image content can be recognized on the display screen.
The program according to the present technology is a program that causes an information processing apparatus to execute a process corresponding to such an information processing method.
This makes it possible for various information processing devices to execute the processing related to this technology.

It is explanatory drawing of the apparatus used in embodiment of this technique. It is a block diagram of the image pickup apparatus of embodiment. It is explanatory drawing of the image correction processing in the image pickup apparatus of embodiment. It is a block diagram of the information processing apparatus of embodiment. It is explanatory drawing of the functional structure as an image processing apparatus of embodiment. It is explanatory drawing of the example of the process flow of embodiment. It is explanatory drawing of the example of the process flow of embodiment. It is explanatory drawing of the example of the process flow of embodiment. It is explanatory drawing of the contents of the image file and metadata of an embodiment. It is explanatory drawing of the metadata about the lens distortion correction. It is explanatory drawing of the screen transition concerning the image processing of an embodiment. It is explanatory drawing of the cutout area designation of embodiment. It is explanatory drawing of the tracking designation of embodiment. It is explanatory drawing of the image processing of embodiment. It is explanatory drawing of attachment to the celestial sphere model of embodiment. It is explanatory drawing of the sample timing of the IMU data of embodiment. It is explanatory drawing of the correspondence between the output image of an embodiment and a celestial sphere model. It is explanatory drawing of rotation and perspective projection of the output coordinate plane of embodiment. It is explanatory drawing of the outline of the protrusion prevention processing of embodiment. It is explanatory drawing of the calculation of the available area in the protrusion prevention processing of embodiment. It is explanatory drawing of how to make an available area of an embodiment. It is explanatory drawing of how to make an available area of an embodiment. It is explanatory drawing of the expansion of the available area of embodiment. It is explanatory drawing of the correction of the display range in the protrusion prevention processing of an embodiment. It is explanatory drawing of the procedure of the protrusion prevention processing of embodiment. It is explanatory drawing of the difference of the capture area by the lens of embodiment. It is the figure which showed the example of the import screen. It is a figure which showed the example of the import screen at the time of overlay reproduction. It is a figure for demonstrating the screen transition at the time of image import in an embodiment. It is a figure which showed the example of the preview screen. It is a figure which showed the example of the preview screen at the time of calling a video menu panel. It is a figure which showed the example of the preview screen which was displayed which showed the section which the reproduction speed was changed and the tracking process was performed. It is a figure which showed the example of the preview screen which made the preview area zoom out display state. It is a figure which showed the example of the aspect ratio setting screen. It is the figure which showed the example of the aspect ratio setting screen in which the preview area was zoomed out display state. It is a figure which showed the example of the trimming screen. It is a figure for demonstrating the operation of specifying a trimming section. It is a figure which showed the example of the trimming screen which the preview area was zoomed out display state. It is a figure which showed the example of the speed screen. It is a figure for demonstrating the designation operation of the reproduction speed change section. It is a figure which showed the example of the speed screen which the preview area was zoomed out display state. It is a figure which showed the example of the framing screen. It is a figure which showed the example of the framing screen which the preview area was zoomed out display state. It is a figure for demonstrating an example of designation of a cut-out range. It is a figure which showed the example of the framing screen in the tracking mode. It is a figure for demonstrating the example of the position designation operation using the target designation aiming. It is a figure which showed the example of the framing screen when the mode which can start tracking is set. It is a figure which showed the example of the framing screen after the start of tracking. It is explanatory drawing of the appearance that the bar display which shows the tracking section is extended. It is a figure exemplifying the relationship between the capture area, the available area, and the display frame about the original image to be preview-displayed. It is a figure which showed the display example of the image which expresses an available area. It is a figure which showed the other display example of the image which expresses an available area. Is. It is a figure which showed the example of the export screen. It is a figure which showed the example of the export screen during rendering. It is a figure which showed the example of the export screen after rendering. It is a flowchart exemplifying the processing of the UI processing part of embodiment. It is a flowchart which exemplifies the process which concerns on the cut-out range designation and tracking. It is a flowchart which exemplifies the tracking correspondence processing. It is a figure which showed the example of the framing screen as a modification. It is a figure which showed the transition example of the framing screen in the tracking mode in the modification. It is a figure which illustrated the shape of the target designation aiming as a modification. It is a figure which showed the transition example of the framing screen in the enlargement / viewpoint change mode in the modification. It is a figure which showed the transition example of the framing screen in the tilt correction mode in the modification. It is explanatory drawing about the modification of the aspect ratio designation acceptance. Similarly, it is explanatory drawing about the modification of the aspect ratio designation acceptance. It is explanatory drawing of the modification of the operation image which concerns on the designation acceptance of an aspect ratio.

Hereinafter, embodiments will be described in the following order.
<1. Configuration of equipment that can be applied as an image processing device>
<2. Image files and metadata>
<3. Screen transition overview>
<4. Image processing>
<5. Summary and modification examples of image processing>
<6. About UI for image processing>
<7. Processing related to UI>
<8. Modification example about UI>
<9. Summary about UI>
<10. Program>
<11. This technology>

<1. Configuration of equipment that can be applied as an image processing device>

In the following embodiments, an example in which an image processing device according to the present technology is realized mainly by a mobile terminal such as a smartphone will be described, but the image processing device can be realized in various devices. First, the equipment to which this technology can be applied will be explained.

FIG. 1A shows an image source and an example as an image processing device that acquires an image file MF from the image source.
As the image source, an image pickup device 1, a server 4, a recording medium 5, and the like are assumed.
As the image processing device, a mobile terminal 2 such as a smartphone, a personal computer 3, or the like is assumed.

The image pickup device 1 as an image source is a digital camera or the like capable of performing moving image imaging, and transfers an image file MF obtained by moving image pickup to a mobile terminal 2 or a personal computer 3 via wired communication or wireless communication.
The server 4 may be a local server, a network server, a cloud server, or the like, but refers to a device capable of providing an image file MF captured by the image pickup device 1. It is conceivable that the server 4 transfers the image file MF to the mobile terminal 2 or the personal computer 3 via some transmission path.

The recording medium 5 may be a solid-state memory such as a memory card, a disk-shaped recording medium such as an optical disk, a tape-shaped recording medium such as a magnetic tape, or the like, but removable recording in which the moving image file MF captured by the image pickup apparatus 1 is recorded. Pointing to the medium. It is conceivable that the moving image file MF read from the recording medium 5 is read by the mobile terminal 2 or the personal computer 3.

The mobile terminal 2 and the personal computer 3 as the image processing device can perform image processing on the image file MF acquired from the above image sources. The image processing referred to here includes, for example, shake correction processing, cropping area setting processing, and effective cropping area image generation processing.
The shake correction process is a shake correction process performed by using the posture information corresponding to the frame after the pasting process is performed on the celestial sphere model for each frame of the image data constituting the moving image.
The cutout area setting process is a process of setting the cutout area of the image based on the user's operation information and then performing the protrusion prevention process of correcting the cutout area so that it does not protrude from the area where the shake correction is effectively maintained. ..
The effective cut-out area image generation process produces an effective cut-out area image by cutting out an image projected on a plane in a state where shake correction processing has been performed in a cut-out area (effective cut-out area CL) that has undergone protrusion prevention processing. It is a process to generate. The cut out image can be an output image.

A certain mobile terminal 2 or a personal computer 3 may be an image source for another mobile terminal 2 or a personal computer 3 that functions as an image processing device.

FIG. 1B shows an image pickup device 1 and a mobile terminal 2 in a case where one device functioning as an image processing device also serves as an image source.
For example, a microcomputer or the like inside the image pickup apparatus 1 performs the above image processing. That is, the image pickup apparatus 1 can output the image as the image processing result by performing the above-mentioned image processing as it is on the image file MF generated by the image pickup.

The same applies to the mobile terminal 2, and since it can be an image source by having an image pickup function, it is possible to output an image as an image processing result by performing the above image processing on the image file MF generated by the image pickup.
Of course, not limited to the image pickup device 1 and the mobile terminal 2, various other devices that can serve as an image source and an image processing device can be considered.

As described above, there are various devices and image sources that function as the image processing device of the embodiment, but in the following, an example will be described in which the image pickup device 1 is an image source and the mobile terminal 2 is an image processing device. There is. That is, it is an example in which the image file MF formed by the image pickup by the image pickup apparatus 1 is transferred to the mobile terminal 2, and the image processing on the acquired image file MF is performed in the mobile terminal 2.

First, a configuration example of the image pickup apparatus 1 serving as an image source will be described with reference to FIG.
As described in FIG. 1B, when it is assumed that the image file MF captured by the mobile terminal 2 is subjected to image processing by the mobile terminal 2, the mobile terminal 2 has the same configuration as the following image pickup device 1 in terms of the image pickup function. You just have to prepare.

As shown in FIG. 2, the image pickup apparatus 1 includes a lens system 11, an image pickup element 12, a camera signal processing unit 13, a recording control unit 14, a display unit 15, an output unit 16, an operation unit 17, a camera control unit 18, and a memory unit. It has 19, a driver unit 22, and a sensor unit 23.

The lens system 11 includes a lens such as a cover lens, a zoom lens, and a focus lens, an aperture mechanism, and the like. The lens system 11 guides the light (incident light) from the subject and condenses it on the image sensor unit 12.
Although not shown, when the lens system 11 is provided with an optical image stabilization mechanism that corrects image shake (interframe shake) and blur due to camera shake or the like. There is.

The image sensor unit 12 includes, for example, an image sensor 12a (image sensor) such as a CMOS (Complementary Metal Oxide Semiconductor) type or a CCD (Charge Coupled Device) type.
The image sensor unit 12 executes, for example, CDS (Correlated Double Sampling) processing, AGC (Automatic Gain Control) processing, and the like on the electric signal obtained by photoelectric conversion of the light received by the image sensor 12a, and further performs A / D. (Analog / Digital) Perform conversion processing. Then, the image pickup signal as digital data is output to the camera signal processing unit 13 and the camera control unit 18 in the subsequent stage.
The optical image stabilization mechanism (not shown) is a mechanism that corrects image shake by moving the image sensor 12a side instead of the lens system 11 side, or spatial optical image stabilization using a gimbal. In some cases, it may be a balanced optical image stabilization mechanism, and any method may be used.
In the optical image stabilization mechanism, in addition to the interframe shake, the blur in the frame is also corrected, which will be described later.

The camera signal processing unit 13 is configured as an image processing processor by, for example, a DSP (Digital Signal Processor) or the like. The camera signal processing unit 13 performs various signal processing on the digital signal (image pickup image signal) from the image pickup element unit 12. For example, as a camera process, the camera signal processing unit 13 performs preprocessing, simultaneous processing, YC generation processing, resolution conversion processing, codec processing, and the like.
The camera signal processing unit 13 also performs various correction processes. However, it is assumed that the image stabilization may or may not be performed in the image pickup apparatus 1.

In the preprocessing, a clamping process for clamping the black level of R, G, B to a predetermined level, a correction process between the color channels of R, G, B, etc. are performed on the captured image signal from the image sensor unit 12. conduct.
In the simultaneous processing, a color separation processing is performed so that the image data for each pixel has all the color components of R, G, and B. For example, in the case of an image sensor using a Bayer array color filter, demosaic processing is performed as color separation processing.
In the YC generation process, a luminance (Y) signal and a color (C) signal are generated (separated) from the image data of R, G, and B.
In the resolution conversion process, the resolution conversion process is executed for the image data to which various signal processes have been performed.

FIG. 3 shows an example of various correction processes (internal correction of the image pickup apparatus 1) performed by the camera signal processing unit 13. FIG. 3 exemplifies the correction processing performed by the camera signal processing unit 13 together with the optical image stabilization performed by the lens system 11 in the order of execution.
In the optical image stabilization as processing F1, the in-lens image stabilization by shifting the yaw direction and pitch direction of the lens system 11 and the in-body image stabilization by shifting the yaw direction and pitch direction of the image sensor 12a are performed to perform image stabilization. The image of the subject is formed on the image sensor 12a in a state where the influence of the above is physically canceled. In-lens image stabilization and in-body image stabilization may be one or both. When both in-lens image stabilization and in-body image stabilization are used, it is conceivable that the in-body image stabilization does not shift in the yaw direction or pitch direction.
In addition, neither in-lens image stabilization nor in-body image stabilization is adopted, and in some cases, only electronic image stabilization or only optical image stabilization is performed for image stabilization.

In the camera signal processing unit 13, processing from processing F2 to processing F7 is performed by spatial coordinate conversion for each pixel.
In the process F2, lens distortion correction is performed.
In the process F3, focal plane distortion correction is performed as one element of electronic image stabilization. It should be noted that this corrects the distortion when the rolling shutter method is read out by, for example, the CMOS type image sensor 12a.

Roll correction is performed in the process F4. That is, the roll component is corrected as one element of the electronic image stabilization.
In the process F5, trapezoidal distortion correction is performed for the trapezoidal distortion amount generated by the electronic image stabilization. The trapezoidal distortion caused by electronic image stabilization is perspective distortion caused by cutting out a place away from the center of the image.
In the process F6, shift and cutout in the pitch direction and the yaw direction are performed as one element of the electronic image stabilization.
For example, camera shake correction, lens distortion correction, and trapezoidal distortion correction are performed by the above procedure.
It is not essential to carry out all of the processes listed here, and the order of the processes may be changed as appropriate.

In the codec processing in the camera signal processing unit 13, for example, coding processing for recording or communication and file generation are performed on the image data subjected to the above various processing. For example, an image file MF as an MP4 format used for recording MPEG-4 compliant video / audio is generated. It is also conceivable to generate files in formats such as JPEG (Joint Photographic Experts Group), TIFF (Tagged Image File Format), and GIF (Graphics Interchange Format) as still image files.
The camera signal processing unit 13 also generates metadata to be added to the image file MF by using the information from the camera control unit 18 and the like.
Although the audio processing system is not shown in FIG. 2, it actually has an audio recording system and an audio processing system, and the image file MF includes audio data as well as image data as a moving image. May be good.
Note that this image data has a preset aspect ratio, and the entire frame of the image data corresponds to an imaging area (captured area) described later.

The recording control unit 14 records and reproduces, for example, a recording medium using a non-volatile memory. The recording control unit 14 performs a process of recording an image file MF such as moving image data or still image data, a thumbnail image, or the like on a recording medium, for example.
The actual form of the recording control unit 14 can be considered in various ways. For example, the recording control unit 14 may be configured as a flash memory built in the image pickup apparatus 1 and a write / read circuit thereof, or a recording medium that can be attached to and detached from the image pickup apparatus 1, such as a memory card (portable flash memory, etc.). ) May be in the form of a card recording / playback unit that performs recording / playback access. Further, it may be realized as an HDD (Hard Disk Drive) or the like as a form built in the image pickup apparatus 1.

The display unit 15 is a display unit that displays various displays to the imager, and is, for example, a display such as a liquid crystal panel (LCD: Liquid Crystal Display) or an organic EL (Electro-Luminescence) display arranged in the housing of the image pickup device 1. It is said to be a display panel or viewfinder depending on the device.
The display unit 15 causes various displays to be executed on the display screen based on the instructions of the camera control unit 18.
For example, the display unit 15 displays a reproduced image of image data read from a recording medium by the recording control unit 14.
Further, the display unit 15 is supplied with image data of the captured image whose resolution has been converted for display by the camera signal processing unit 13, and the display unit 15 is based on the image data of the captured image according to the instruction of the camera control unit 18. May be displayed. As a result, a so-called through image (subject monitoring image), which is an captured image during composition confirmation, is displayed.
Further, the display unit 15 causes various operation menus, icons, messages, etc., that is, display as a GUI (Graphical User Interface) to be executed on the screen based on the instruction of the camera control unit 18.

The output unit 16 performs data communication and network communication with an external device by wire or wirelessly.
For example, the image data (still image file or moving image file) is transmitted and output to an external display device, recording device, playback device, or the like.
Further, assuming that the output unit 16 is a network communication unit, it communicates with various networks such as the Internet, a home network, and a LAN (Local Area Network), and transmits / receives various data to / from a server, a terminal, etc. on the network. You may do so.

The operation unit 17 collectively shows input devices for the user to perform various operation inputs. Specifically, the operation unit 17 shows various controls (keys, dials, touch panels, touch pads, etc.) provided in the housing of the image pickup apparatus 1.
The operation unit 17 detects the user's operation, and the signal corresponding to the input operation is sent to the camera control unit 18.

The camera control unit 18 is composed of a microcomputer (arithmetic processing unit) provided with a CPU (Central Processing Unit).
The memory unit 19 stores information and the like used for processing by the camera control unit 18. As the illustrated memory unit 19, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a flash memory, and the like are comprehensively shown.
The memory unit 19 may be a memory area built in the microcomputer chip as the camera control unit 18, or may be configured by a separate memory chip.
The camera control unit 18 controls the entire image pickup device 1 by executing a program stored in the ROM of the memory unit 19, the flash memory, or the like.
For example, the camera control unit 18 controls the shutter speed of the image sensor unit 12, gives instructions for various signal processing in the camera signal processing unit 13, captures and records operations according to user operations, reproduces recorded image files, and lenses. It controls the operation of each necessary part regarding the operation of the lens system 11 such as zoom, focus, and aperture adjustment in the lens barrel, and the operation of the user interface.

The RAM in the memory unit 19 is used for temporarily storing data, programs, and the like as a work area for various data processing of the CPU of the camera control unit 18.
The ROM and flash memory (nonvolatile memory) in the memory unit 19 include an OS (Operating System) for the CPU to control each unit, content files such as image files, application programs for various operations, and firmware. It is used for memory of etc.

The driver unit 22 is provided with, for example, a motor driver for a zoom lens drive motor, a motor driver for a focus lens drive motor, a motor driver for a motor of an aperture mechanism, and the like.
These motor drivers apply a drive current to the corresponding driver in response to an instruction from the camera control unit 18, to move the focus lens and the zoom lens, open and close the diaphragm blades of the diaphragm mechanism, and the like.

The sensor unit 23 comprehensively shows various sensors mounted on the image pickup apparatus.
The sensor unit 23 is equipped with, for example, an IMU (inertial measurement unit). For example, an angular velocity is detected by a three-axis angular velocity (gyro) sensor of pitch-, yaw, and roll, and acceleration is detected by an acceleration sensor. can do.
Further, the sensor unit 23 may be equipped with a position information sensor, an illuminance sensor, or the like.

For example, the image file MF as a moving image captured and generated by the above image pickup device 1 can be transferred to an image processing device such as a mobile terminal 2 for image processing.
The mobile terminal 2 can be realized as an information processing device having the configuration shown in FIG. 4, for example. The personal computer 3 and the server 4 can also be realized by the information processing device having the configuration of FIG. 4 in the same manner.

In FIG. 4, the CPU 71 of the information processing apparatus 70 executes various processes according to a program stored in the ROM 72 or a program loaded from the storage unit 79 into the RAM 73. The RAM 73 also appropriately stores data and the like necessary for the CPU 71 to execute various processes.
The CPU 71, ROM 72, and RAM 73 are connected to each other via a bus 74. An input / output interface 75 is also connected to the bus 74.

An input unit 76 including an operator and an operation device is connected to the input / output interface 75.
For example, as the input unit 76, various controls and operation devices such as a keyboard, a mouse, a key, a dial, a touch panel, a touch pad, and a remote controller are assumed.
The user's operation is detected by the input unit 76, and the signal corresponding to the input operation is interpreted by the CPU 71.

Further, a display unit 77 made of an LCD or an organic EL panel and an audio output unit 78 made of a speaker or the like are connected to the input / output interface 75 as one or as a separate body.
The display unit 77 is a display unit that performs various displays, and is composed of, for example, a display device provided in the housing of the information processing device 70, a separate display device connected to the information processing device 70, or the like.
The display unit 77 executes the display of various images for image processing, moving images to be processed, and the like on the display screen based on the instruction of the CPU 71. Further, the display unit 77 displays various operation menus, icons, messages, etc., that is, as a GUI (Graphical User Interface) based on the instruction of the CPU 71.

A storage unit 79 composed of a hard disk, a solid-state memory, or the like, or a communication unit 80 composed of a modem or the like may be connected to the input / output interface 75.
The communication unit 80 performs communication processing via a transmission line such as the Internet, and performs communication with various devices by wire / wireless communication, bus communication, or the like.

A drive 82 is also connected to the input / output interface 75, if necessary, and a removable recording medium 81 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is appropriately mounted.
The drive 82 can read data files such as an image file MF and various computer programs from the removable recording medium 81. The read data file is stored in the storage unit 79, and the image and sound included in the data file are output by the display unit 77 and the sound output unit 78. Further, the computer program or the like read from the removable recording medium 81 is installed in the storage unit 79 as needed.

In the information processing apparatus 70, for example, software for image processing as the image processing apparatus of the present disclosure can be installed via network communication by the communication unit 80 or a removable recording medium 81. Alternatively, the software may be stored in the ROM 72, the storage unit 79, or the like in advance.

For example, by such software (application program), the functional configuration as shown in FIG. 5 is constructed in the CPU 71 of the information processing apparatus 70.
That is, the information processing device 70 (CPU 71) has functions as a pre-processing unit 31, an image processing unit 32, and a UI processing unit 36.

The preprocessing unit 31 is, for example, a function of importing an image file MF constituting a moving image generated by the image pickup apparatus 1 and performing preprocessing at the time of import.
In the present specification, "import" refers to targeting an image file MF or the like that can be accessed by the information processing apparatus 70 being captured in, for example, a storage unit 79, and is subjected to preprocessing. It means developing so that image processing is possible. For example, it does not mean transferring from the image pickup device 1 to the mobile terminal 2.
The pre-processing unit 31 imports the image file MF designated by the user operation or the like so as to be the image processing target, and also performs processing related to the metadata added to the image file MF as pre-processing. For example, a process of extracting and storing metadata corresponding to each frame of a moving image is performed.

The image processing unit 32 shows a function of performing image processing on the imported image file MF. The function as the image processing unit 32 has functions as a shake correction unit 33, an effective cutting area setting unit 34, and an effective cutting area image generation processing unit 35 as shown.

The shake correction unit 33 is a function of performing shake correction using the posture information corresponding to each frame as image data constituting a moving image as an image file MF. For example, the shake correction unit 33 performs shake correction in a state where the plane model is attached to the celestial sphere model for each frame. That is, when the image is captured as the image file MF, the subject image is projected on a plane, but the shaking is corrected while this is projected on the celestial sphere model.
Here, the “shaking” to be corrected refers to the vibration of the image due to the movement of the image pickup device 1 that has taken the image of the image file MF, and in particular, the vibration component generated between the frames (the fluctuation of the image between the frames). To say. And "(inter-frame) vibration correction" refers to correcting the vibration that appears as such vibration between frames. The above-mentioned shift and cutout method of electronic image stabilization is included in "shake correction". Further, in the above-mentioned optical image stabilization, "interframe shake correction" and "blurring correction" are performed at the same time.
When the image sensor unit 12 is equipped with a rolling shutter type CMOS image sensor 12a, the amount of shaking and blurring differs for each line.

The effective cutout area setting unit 34 sets the effective cutout area of the image based on the user's operation information, tracking target information, and the like. The effective cutout area setting unit 34 includes a function of performing a protrusion prevention process for correcting the cutout area so that it does not protrude from the area where the shaking correction is effectively maintained.
The setting of the effective cutout area is not limited to setting a fixed cutout area according to the user's operation, and the effective cutout area is included so as to include the tracking target according to the user's specification of the tracking target (tracking target). The area may be set.
In this case, the effective cutout area setting unit 34 also performs a process of specifying the position of the tracking target in each frame of the moving image by image analysis, frame-to-frame comparison, or the like.

The effective cut-out area image generation processing unit 35 determines the effective cut-out area designated by the effective cut-out area setting unit 34 from the image projected onto the plane from the virtual spherical surface of the celestial sphere model in the state where the shake correction unit 33 performs the shake correction. It is a function to cut out the output image which is an image.

The UI processing unit 36 is a function that accepts user operations for import and image processing, and controls display output according to the processing.
As shown in the figure, the UI processing unit 36 has a reception unit 36a and a display control unit 36b. The reception unit 36a mainly accepts user operations for image processing, and the display control unit 36b mainly performs display control according to the processing executed based on the user operations for image processing.
The UI processing unit 36 can be operated by the user by the reception unit 36a and the display control unit 26b on each screen such as an import screen, a preview screen, a framing screen, a trimming screen, a speed screen, an aspect ratio setting screen, and an export screen, which will be described later. Control reception and display.
The details of the processing executed by the CPU 71 as the reception unit 36a and the display control unit 36b will be described later.

Details of the processing by the above-mentioned function of FIG. 5 will be described later.
For example, the image pickup apparatus 1 and the mobile terminal 2 corresponding to the information processing apparatus 70 having the function of FIG. 5 perform processing according to the flow as shown in FIG. 6, for example.

Image pickup (step S91), camera process (step S92), metadata generation process (step S93), and image file generation process (step S94) are performed in the image pickup apparatus 1.
The image pickup in step S91 indicates the output (RAW data output) of the image pickup image signal by the image pickup element unit 12.
The camera process in step S92 is a so-called development process, which is the above-mentioned various signal processes performed by the camera signal processing unit 13 for the captured image signal. It includes various correction processes described with reference to FIG.
Further, the metadata generation process in step S93 is a process of generating metadata associated with an image according to, for example, IMU data obtained by the sensor unit 23 or a camera control signal, and the camera signal processing unit 13 or the camera control unit 18 generates metadata. It will be done.
In the camera signal processing unit 13, the image file generation process of step S94 is performed including the image data obtained by the camera process of step S92 and the metadata generated by the metadata generation process of step S93, and the image file MF is generated. Generated.

The image file MF is acquired by the mobile terminal 2 via some route as described above.
In the mobile terminal 2, as a pre-processing (step S95) at the time of importing the image file MF, the image file MF is targeted for image processing, and the storage management of the metadata associated with each frame is performed.
Further, for the image file MF that has been imported and preprocessed, the image processing in step S96 can be performed at any time. In this image processing, shaking correction by the shaking correction unit 33, effective cutting area setting by the effective cutting area setting unit 34, plane projection and cutting processing by the effective cutting area image generation processing unit 35, and the like are performed.
Then, the process of saving / displaying is performed as step S97 in order to display or save the progress and result of the image processing.
With such a flow, the user can perform image processing on the mobile terminal 2 on the moving image captured by the image pickup device 1, and reproduce, store, upload, and the like.

When the image pickup apparatus 1 has an image processing function as the image processing apparatus of the present disclosure as shown in FIG. 1B, the camera control unit 18 and the camera signal processing unit 13 may be provided with the functions as shown in FIG. good.
The flow of processing in that case is shown in FIG. That is, the case where the preprocessing of step S95, the image processing of step S96, and the storage / display processing of step S97 are also performed in the image pickup apparatus 1 is shown.
Further, in FIG. 1B, the case where the mobile terminal 2 performs imaging is also mentioned, but in that case, it is conceivable that the processing shown in FIG. 7 is performed in the mobile terminal 2.

As another example, FIG. 8 shows a case where RAW data is targeted for image processing.
For example, when image pickup and image processing are performed by the image pickup device 1 and the mobile terminal 2 as shown by being surrounded by a broken line, the image pickup device 1 has the image pickup image signal (RAW data) obtained by the image pickup in step S91 and the metadata of step S93. Using the metadata generated in the generation process, the image file generation process in step S94 is performed to generate the image file MF. In the case of RAW data, the camera process in step S92 of FIGS. 6 and 7 is not performed.
The mobile terminal 2 performs pre-processing (step S95), image processing (step S96), and storage / display processing (step S97) on such an image file MF.
When RAW data is used as an image processing target as shown in FIG. 8, it is assumed that electronic image stabilization and optical image stabilization are performed or not performed in the image pickup apparatus 1.
In addition, in the cases of FIGS. 6 and 7 in which the camera process is performed, it is assumed that electronic image stabilization and optical image stabilization are performed in the image pickup apparatus 1, and it is assumed that they are not performed.

Further, the image pickup device 1 (or the mobile terminal 2) surrounded by the alternate long and short dash line in FIG. 8 shows a case where image pickup and image processing for RAW data are performed in the image pickup device 1 (or in the mobile terminal 2). It is a thing.
That is, there may be a case where the preprocessing of step S95, the image processing of step S96, and the storage / display processing of step S97 are performed in the image pickup apparatus 1.
Further, in the mobile terminal 2, the image pickup in step S91, the metadata generation process in step S93, and the image file generation process in step S94 may be performed.

<2. Image files and metadata>

The contents of the image file MF and the contents of the metadata will be described.
FIG. 9A shows the data included in the image file MF. As shown in the figure, the image file MF includes various data as "header", "sound", "movie", and "metadata".

In the "header", information such as a file name and a file size as well as information indicating the presence or absence of metadata are described.
"Sound" is audio data recorded with a moving image. For example, 2-channel stereo audio data is stored.
The "movie" is moving image data, and is composed of image data as each frame (# 1, # 2, # 3 ...) constituting the moving image.
As the "metadata", additional information associated with each frame (# 1, # 2, # 3, ...) Constituting the moving image is described.

An example of the content of the metadata is shown in FIG. 9B. For example, IMU data, coordinate conversion parameter HP, timing information TM, and camera parameter CP are described for one frame. It should be noted that these are a part of the metadata contents, and here, only the information related to the image processing described later is shown.

As the IMU data, a gyro (angular velocity data), an accelerator (acceleration data), and a sampling rate are described.
The IMU mounted on the image pickup apparatus 1 as the sensor unit 23 outputs the angular velocity data and the acceleration data at a predetermined sampling rate. Generally, this sampling rate is higher than the frame rate of the captured image, so that many IMU data samples can be obtained in one frame period.

Therefore, as the angular velocity data, n samples are associated with each frame, such as gyro sample # 1, gyro sample # 2, ... Gyro sample # n shown in FIG. 9C.
Also, as acceleration data, m samples are associated with each frame, such as accelerator sample # 1, accelerator sample # 2, ... accelerator sample # m.
In some cases, n = m, and in some cases, n ≠ m.
Although the metadata is described here in the example of being associated with each frame, for example, the IMU data may not be completely synchronized with the frame. In such a case, for example, the time information associated with the time information of each frame is provided as the IMU sample timing offset in the timing information TM.

The coordinate conversion parameter HP is a general term for parameters used for correction accompanied by coordinate conversion of each pixel in an image. It also includes non-linear coordinate transformations such as lens distortion.
The coordinate conversion parameter HP is a term that can include at least a lens distortion correction parameter, a trapezoidal distortion correction parameter, a focal plane distortion correction parameter, an electronic image stabilization parameter, and an optical image stabilization parameter.

The lens distortion correction parameter is information for directly or indirectly grasping how the distortion such as barrel aberration and pincushion aberration is corrected and returning to the image before the lens distortion correction. The metadata regarding the lens distortion correction parameter as one of the metadata will be briefly described.
FIG. 10A shows the image height Y, the angle α, the entrance pupil position d1 and the exit pupil position d2 in the schematic diagram of the lens system 11 and the image sensor 12a.
The lens distortion correction parameter is used in image processing to know the incident angle of each pixel of the image sensor 12a. Therefore, it is sufficient to know the relationship between the image height Y and the angle α.

FIG. 10B shows the image 110 before the lens distortion correction and the image 111 after the lens distortion correction. The maximum image height H0 is the maximum image height before distortion correction, and is the distance from the center of the optical axis to the farthest point. The maximum image height H1 is the maximum image height after distortion correction.
To understand the relationship between the image height Y and the angle α, the metadata required is the maximum image height H0 before distortion correction and the incident angle data d0, d1, ... d (N−) for each of the N image heights. It becomes 1). It is assumed that "N" is about 10 as an example.

Returning to FIG. 9B, the trapezoidal distortion correction parameter is a correction amount for correcting trapezoidal distortion caused by shifting the cutout region from the center by electronic image stabilization, and is also a value corresponding to the correction amount for electronic image stabilization.

The focal plane distortion correction parameter is a value indicating the correction amount for each line with respect to the focal plane distortion.

With regard to electronic image stabilization and optical image stabilization, it is a parameter indicating the amount of correction in each axial direction of yaw, pitch, and roll.

The parameters of lens distortion correction, trapezoidal distortion correction, focal plane distortion correction, and electronic image stabilization are collectively referred to as coordinate conversion parameters, but these correction processes are performed by each of the image sensors 12a of the image sensor unit 12. This is because it is a correction process for the image formed on the pixel and is a parameter of the correction process accompanied by the coordinate conversion of each pixel. Optical image stabilization is also one of the coordinate conversion parameters, but this is because the correction of the fluctuation of the inter-frame component in the optical image stabilization is a process that involves the coordinate conversion of each pixel.
That is, if the reverse correction is performed using these parameters, the image data to which the lens distortion correction, the trapezoidal distortion correction, the focal plane distortion correction, the electronic image stabilization, and the optical image stabilization are performed can be captured before each correction processing, that is, by imaging. It is possible to return to the state when the image was formed on the image sensor 12a of the element unit 12.

In addition, each parameter of lens distortion correction, trapezoidal distortion correction, and focal plane distortion correction is distortion correction processing for the case where the optical image itself from the subject is an image captured in an optically distorted state, and each is optical distortion. Since it is intended for correction, it is collectively referred to as an optical distortion correction parameter.
That is, if the reverse correction is performed using these parameters, the image data to which the lens distortion correction, the trapezoidal distortion correction, and the focal plane distortion correction have been performed can be returned to the state before the optical distortion correction.

The timing information TM in the metadata includes each information of exposure time (shutter speed), exposure start timing, readout time (curtain speed), number of exposure frames (long exposure information), IMU sample offset, and frame rate.
In the image processing of the present embodiment, these are mainly used to associate the line of each frame with the IMU data.
However, even if the image sensor 12a is a CCD or a global shutter type CMOS, if the exposure center of gravity shifts using an electronic shutter or mechanical shutter, the exposure start timing and curtain speed are also used to match the exposure center of gravity. Correction is possible.

The angle of view (focal length), zoom position, and lens distortion information are described as the camera parameter CP in the metadata.

<3. Screen transition overview>

An example of screen transition during image processing in the mobile terminal 2 will be described.
FIG. 11 shows the transition of various screens appearing on the display screen of the mobile terminal 2. Each screen will be briefly described.
Of the screens described below, the images output on the preview screen 102, the framing screen 103, the trimming screen 104, the speed screen 105, the aspect ratio setting screen 106, and the export screen 108 are shown in step ST19 of FIG. It is an output image oPD that has undergone processing.
However, the image output on these screens does not necessarily have to be the output image oPD that has undergone the processing of FIG. 14, but in that case, the area set or visually recognized by the user and the final image of FIG. 14 The area after processing and the degree of distortion may be slightly different.

[Import screen 101]
When the application program for video processing is started, the import screen 101 appears.
The import screen 101 is a screen before the start of moving image processing. On the import screen 101, thumbnails and the like of the image file MF are displayed so that the user can select the image file MF to be processed.
The selected image file MF is imported and preprocessed as the image file MF to be processed.

[Preview screen 102]
When the user completes the operation of selecting one or a plurality of image file MFs on the import screen 101, the display transitions to the preview screen 102 through the import process. On the preview screen 102, a list of imported moving images (image file MF) that can be processed with images is displayed, and preview playback of the moving images specified in the list is possible.

With respect to the image file MF shown on the preview screen 102, the user can perform various image processing operations, such as changing the brightness and changing the color, from the preview screen as a starting point.
On the preview screen 102, the user can perform operations such as playing the selected moving image (image file MF), adding or deleting the moving image, and resetting the image processing.
The moving image is played back on the preview screen 102.
When a moving image to be image-processed is added or a new project (new image processing action) is operated, the screen returns to the import screen 101 and selection is possible. The moving image can be deleted (excluded from the image processing target) on the preview screen 102.

[Flaming screen 103]
By a predetermined operation from the preview screen 102, the display transitions to the framing screen 103. The framing screen 103 is a screen for individually framing the moving images designated as the processing target on the preview screen 102 among the imported moving images.
On the framing screen 103, for example, in a state where a still image of a certain frame of the moving image to be processed is displayed, the user can specify an effective cropping area of the image, enlarge / reduce the image, and so on on the image of the frame. Operations such as rotating the image, specifying the tracking target (tracking target), and specifying the placement position of the tracking target on the screen are possible.
After that, the display returns to the preview screen 102 by a predetermined operation from the framing screen 103.

The image displayed on the framing screen 103 is the entire image of a certain frame (the first frame of the moving image, the paused frame, etc.), or a part cut out from the entire image. The user can specify the effective cropping area of the image by moving the displayed range up / down / left / right, and performing enlargement / reduction / rotation operations.
Take an example. FIG. 12A shows the entire image 114 of a frame and the display frame 115. It is assumed that the range of the display frame 115 is the range displayed on the framing screen 103. That is, on the framing screen 103, a part of the entire image 114 is cut out and displayed. The range of the display frame 115 corresponds to the effective cutting area.

When the user performs, for example, a drag operation or a pinch-out / pinch-in operation on the framing screen 103, the range included in the display frame 115 can be moved, enlarged / reduced / rotated.
For example, by a drag operation on the screen, the range included in the display frame 115 is moved from FIG. 12A to FIG. 12B. That is, the displayed range changes by moving the image up, down, left, and right with respect to the display frame 115.
Further, when the user instructs the enlargement / reduction of the image by, for example, a pinch-out / pinch-in operation, the range included in the display frame 115 changes. For example, when the reduction operation is performed from FIG. 12B, the result is as shown in FIG. 12C. That is, the entire image 114 is reduced with respect to the display frame 115 so that the image included in the display frame 115 (viewed by the user) is reduced, and the range of the subject included in the display frame 115 is widened. That is, by reducing the displayed image, a wider range is displayed.
Although not shown, conversely, when the user performs an enlargement operation, the entire image 114 is enlarged with respect to the display frame 115, and the range of the subject included in the display frame 115 is narrowed. That is, a narrower range is enlarged and displayed.
The operation for the range displayed by the display frame 115 is an operation for designating the cutout area by the user.

In addition, the user can specify the effective cutout area and the tracking target and its placement position as described above.
For example, when the user performs an operation for specifying a tracking target on the framing screen 103, the target designation aiming 116 is displayed as shown in FIG. 13A. The user can arbitrarily specify the position of the target specification aim by dragging or tapping. For example, as shown in FIG. 13B, a human face portion can be specified.
In the state of FIG. 13B, the face of the person is designated as the tracking target, and further, the placement position in the tracking target screen, which is to be placed in the upper right of the tracking target screen, is also specified. Become.

That is, on the framing screen 103, the effective cutout area can be specified by selecting the range included in the display frame 115, and the tracking target can be specified by combining the range included in the display frame 115 and the operation of specifying the tracking target. And the operation of designating the arrangement position is possible.

The above operation is performed as an operation on the displayed still image of one frame, but the designation is also reflected in other frames in the same moving image.
If the tracking target is not specified only by moving, enlarging, reducing, and rotating the image range in the display frame 115, in each frame, the range specified by the operation of moving, enlarging / reducing / rotating the image range is specified. It is a fixed effective cutting area.
On the other hand, when the tracking target and its arrangement position are specified together with the image range to be included in the display frame as described above, the effective cutout area is calculated for the other frames in the moving image according to the specification. That is, in each frame constituting the moving image, the position in the screen of the tracking target also changes, so that the effective cutout area is calculated and changed for each frame.
Specifically, in each frame, the effective cutting area is set so that the specified tracking target is located at the specified placement position in the size of the effective cutting area specified by the enlargement / reduction operation of the display frame 115. It is calculated.
The size of the effective cutting area may be appropriately changed according to the change in the size of the tracking target.

In addition, on the framing screen 103, it is possible to move, enlarge, reduce, and rotate the range included in the display frame 115 while playing back the moving image, and to specify the tracking target and its arrangement position. May be good.

[Trimming screen 104]
By a predetermined operation from the preview screen 102, the display transitions to the trimming screen 104. The trimming screen 104 is a screen for individually trimming (cutting in the time axis direction) a moving image designated as a processing target on the preview screen 102 among the imported moving images.
On the trimming screen 104, the range to be trimmed in the moving image can be selected. For example, it is possible to display a bar in the time axis direction of the moving image and specify the start point and end point of the range to be cut.
The display returns to the preview screen 102 by a predetermined operation from the trimming screen 104.
In this specification, trimming does not include a process of cutting out a part of a frame.

[Speed screen 105]
By a predetermined operation from the preview screen 102, the display transitions to the speed screen 105. The speed screen 105 is a screen for individually speed-processing a moving image designated as a processing target on the preview screen 102 among the imported moving images.
For example, on the speed screen 105, it is possible to select a range for setting a speed (reproduction speed) in a moving image or to specify a reproduction speed in the range.
The display returns to the preview screen 102 by a predetermined operation from the speed screen 105.
More specifically, it is a reproduction speed for performing slow motion or high-speed reproduction by setting a frame rate different from the frame rate set in the timing information TM included in the metadata.

[Aspect ratio setting screen 106]
By a predetermined operation from the preview screen 102, the display transitions to the aspect ratio setting screen 106. The aspect ratio setting screen 106 is a screen for setting the aspect ratio of the output moving image for each file of the imported moving image.
If the aspect ratio of the imported video and the set output image (output video) are different, it may have the same meaning as the setting of the effective cutout area performed on the framing screen.
That is, even if the effective cutout area is not set on the framing screen, the effective cutout area of the output images having different aspect ratios is set from the image pickup area.
For example, on the aspect ratio setting screen 106, aspect ratios such as "1: 1", "4: 5", "5: 4", "16: 9", and "9:16" can be selected.
The display returns to the preview screen 102 by a predetermined operation from the aspect ratio setting screen 106.

It should be noted that all or part of the functions of the aspect ratio setting screen 106, for example, the batch selection operation of the aspect ratio may be operated by, for example, a dialog displayed on the preview screen 102.
Further, the aspect ratio setting screen 106 may be used as a screen for setting the overall aspect ratio of a plurality of frames constituting the moving image designated as the processing target on the preview screen 102 among the imported moving images.

[Setting screen 107]
By a predetermined operation from the preview screen 102, the display transitions to the setting screen 107. On the setting screen 107, setting operations for the operation of the application program and the like are possible.
The display returns to the preview screen 102 by a predetermined operation from the setting screen 107.

[Export screen 108]
By a predetermined operation from the preview screen 102, the display transitions to the export screen 108. The export screen 108 is a screen when a process of writing a moving image reflecting the image processing contents performed on each of the above screens is being performed. That is, after performing various image processing operations, the user returns to the preview screen 102 and instructs export. As a result, the screen transitions to the export screen 108, and the video writing process processed according to the image processing content is executed.
The exported video can be arbitrarily recorded by the user, or uploaded to a website, SNS (Social Networking Service), or the like via the network communication of the mobile terminal 2 and shared.
If the user performs a cancel operation during the export process, or if the export process is completed, the display returns to the preview screen 102.

<4. Image processing>

A specific example of the processing related to the moving image executed while performing the user interface on each of the above screens will be described.
FIG. 14 shows a procedure as preprocessing performed at the time of importing a moving image (image file MF) by the preprocessing unit 31 (CPU71) of FIG. 5 and a procedure as image processing performed by the image processing unit 32 (CPU71). At the same time, the relationship of the information used in each process is shown.

First, the preprocessing will be described. The preprocessing is performed on the image file MF selected as the image processing target on the import screen 101. In this preprocessing, metadata extraction (step ST1), all IMU data concatenation (step ST2), metadata retention (step ST3), conversion to quotation (posture information of image pickup apparatus 1), and retention (step ST4) are performed. Will be done.

As the metadata extraction in step ST1, the preprocessing unit 31 reads the target image file MF and extracts the metadata included in the image file MF as described with reference to FIG.
Note that part or all of steps ST1, ST2, ST3, and ST4 may be performed on the image pickup apparatus 1 side. In that case, in the pre-processing, the contents after those processing described below are acquired as metadata.

Of the extracted metadata, the preprocessing unit 31 performs the concatenation processing of the IMU data (angular velocity data (gyro sample) and acceleration data (accelerator sample)) in step ST2.
This is a process of constructing IMU data corresponding to the entire sequence of moving images by arranging and concatenating all the IMU data associated with all frames in chronological order.
Then, integration processing is performed on the connected IMU data to calculate a quaternion QD representing the posture of the image pickup apparatus 1 at each time point on the sequence of moving images, and this is stored and stored. It is an example that the quaternion QD is calculated.
It is also possible to calculate the quaternion QD using only the angular velocity data.

Among the extracted metadata, the pre-processing unit 31 performs a process of holding the metadata other than the IMU data, that is, the coordinate conversion parameter HP, the timing information TM, and the camera parameter CP in step ST3. That is, the coordinate conversion parameter HP, the timing information TM, and the camera parameter CP are stored in a state corresponding to each frame.

By performing the above preprocessing, various image processings are performed via the interface of each screen starting from the preview screen 102 described above.
In the image processing of FIG. 14, each screen as the preview screen 102, the framing screen 103, the trimming screen, the speed screen 105, and the aspect ratio setting screen 106 is reflected while reflecting the information of the effective cutout area designated by the user on the framing screen 103. Shows the processing that is constantly performed when the moving image of the image file MF or the still image that is paused is played back.

The image processing unit 32 (CPU71) takes out one frame of moving image (step ST11), cancels the internal correction of the image pickup device (step ST12), pastes it on the celestial sphere model (step ST13), synchronizes processing (step ST14), and shakes correction (ST15). ) Is performed. The processing of these steps ST11 to ST15 is the processing by the function of the shake correction unit 33 of FIG.
Further, the image processing unit 32 (CPU71) performs each processing as tracking (step ST16), display area designation (step ST17), and protrusion prevention (step ST18). The processing of these steps ST16, ST17, and ST18 is the processing by the function of the effective cutting area setting unit 34 of FIG.
Further, the image processing unit 32 (CPU71) performs plane projection and cutting (step ST19) processing. The process of this step ST19 is a process by the function of the effective cutout area image generation processing unit 35 of FIG.

The image processing unit 32 performs each of the above steps ST11 to ST19 for each frame when the image of the image file MF is reproduced. For example, when displaying an image on each screen such as the framing screen 103 or the trimming screen 104 described above, each process of steps ST11 to ST19 is performed for each frame.

In step ST11, the image processing unit 32 decodes one frame of the moving image (image file MF) along the frame number FN. Then, one frame of image data PD (#FN) is output. It should be noted that "(#FN)" indicates a frame number and indicates that the information corresponds to the frame.
If the moving image is not compressed or otherwise encoded, the decoding process in step ST11 is not necessary.

In step ST12, the image processing unit 32 performs a process of canceling the internal correction performed by the image pickup apparatus 1 for the image data PD (#FN) of one frame. For this purpose, the coordinate conversion parameter HP (#FN) stored corresponding to the frame number (#FN) at the time of preprocessing is referred to, and the correction opposite to the correction performed by the image pickup apparatus 1 is performed. As a result, image data iPD (#FN) in a state in which lens distortion correction, trapezoidal distortion correction, focal plane distortion correction, electronic image stabilization, and optical image stabilization in the image pickup apparatus 1 are canceled is obtained.

In step ST13, the image processing unit 32 attaches one frame of image data iPD (#FN) in a state where various corrections are canceled to the celestial sphere model. At this time, the camera parameter CP (#FN) stored corresponding to the frame number (#FN), that is, the angle of view, the zoom position, and the lens distortion information are referred to.

FIG. 15 shows an outline of attachment to the celestial sphere model.
FIG. 15A shows the image data iPD. The image height h is the distance from the center of the image. Each circle in the figure indicates a position where the image heights h are equal.
From the angle of view, zoom position, and lens distortion information of this image data iPD frame, the "relationship between the image sensor surface and the incident angle φ" in that frame is calculated, and "data0" at each position on the image sensor surface ... Let's say "dataN-1". Then, from "data0" ... "dataN-1", it is expressed as a one-dimensional graph of the relationship between the image height h and the incident angle φ as shown in FIG. 15B. The incident angle φ is the angle of the light ray (angle seen from the optical axis).
This one-dimensional graph is rotated once around the center of the captured image, and the relationship between each pixel and the incident angle is obtained.
Accordingly, mapping from the pixel G1 in FIG. 15C to the celestial sphere model MT is performed for each pixel of the image data iPD as in the pixel G2 on the celestial sphere coordinates.

As described above, an image (data) of the celestial sphere model MT in which the captured image is attached to the ideal celestial sphere with the lens distortion removed can be obtained. In this celestial sphere model MT, the parameters and distortions peculiar to the image pickup apparatus 1 that originally captured the image data iPD are removed, and the range that can be seen by an ideal pinhole camera is pasted on the celestial sphere.
Therefore, by rotating the image of the celestial sphere model MT in a predetermined direction in this state, the shaking correction can be realized.

Here, the attitude information (quaternion QD) of the image pickup apparatus 1 is used for the shaking correction. For this reason, the image processing unit 32 performs synchronization processing in step S14.
In the synchronization process, a process of identifying and acquiring a quaternion QD (#LN) suitable for each line corresponding to the frame number FN is performed. Note that "(#LN)" indicates a line number in the frame, and indicates that the information corresponds to that line.

The reason why the quaternion QD (#LN) for each line is used is that when the image sensor 12a is a CMOS type and the rolling shutter type image pickup is performed, the amount of shaking differs for each line.
For example, when the image sensor 12a is a CCD type and the imaging is performed by the global shutter method, a quaternion QD (#FN) for each frame may be used.
Even when using a CCD or CMOS global shutter as the image sensor 12a, the center of gravity shifts when an electronic shutter (similar to a mechanical shutter) is used, so the center of the exposure period of the frame (shifts according to the shutter speed of the electronic shutter). ) Timing quotation should be used.

Now consider the blur that appears in the image.
Blur is image bleeding due to relative movement between the image pickup device and the subject within the same frame. That is, image bleeding due to shaking within the exposure time. The longer the exposure time, the stronger the effect of blurring.
Electronic image stabilization can reduce / eliminate "shake" that occurs between frames when a method that controls the image range to be cut out for each frame is used, but relative shake within the exposure time is such an electron. It cannot be reduced by image stabilization.
When changing the cutout area by image stabilization, the posture information of each frame is used, but if the posture information is deviated from the center of the exposure period such as the start or end timing of the exposure period, the posture. The direction of shaking within the exposure time based on the above is biased, and bleeding is easily noticeable. Further, in the CMOS rolling shutter, the exposure period is different for each line.

Therefore, in the synchronization process of step ST14, the quaternion QD is acquired for each frame of the image data based on the timing of the exposure center of gravity for each line.
FIG. 16 shows the synchronization signal cV of the vertical period of the image pickup apparatus 1, the synchronization signal sV of the image sensor 12a generated from the synchronization signal cV, and the sample timing of the IMU data, and also shows the exposure timing range 120. There is.
The exposure timing range is schematically shown by a parallelogram showing the exposure period of each line of one frame when the exposure time is t4 by the rolling shutter method. Further, the temporal offset t0 of the synchronization signal cV and the synchronization signal sV, the IMU sample timing offset t1, the read start timing t2, the read time (shutter speed) t3, and the exposure time t4 are shown. The read start timing t2 is the timing after a predetermined time t2of has passed from the synchronization signal sV.
Each IMU data obtained at each IMU sample timing is associated with a frame. For example, the IMU data in the period FH1 is the metadata associated with the current frame indicating the exposure period in a parallelogram, and the IMU data in the period FH1 is the metadata associated with the next frame. However, by concatenating all the IMU data in step ST2 of FIG. 14, the association between each frame and the IMU data is released, and the IMU data can be managed in chronological order.
In this case, the IMU data corresponding to the exposure center of gravity (timing of the broken line W) of each line of the current frame is specified. This can be calculated if the temporal relationship between the IMU data and the effective pixel area of the image sensor 12a is known.

Therefore, using the information that can be acquired as the timing information TM corresponding to the frame (#FN), the IMU data corresponding to the exposure center of gravity (timing of the broken line W) of each line is specified.
That is, it is information on the exposure time, the exposure start timing, the readout time, the number of exposure frames, the IMU sample offset, and the frame rate.
Then, the quaternion QD calculated from the IMU data of the exposure center of gravity is specified, and the quaternion QD (#LN) which is the attitude information for each line is used.

This quaternion QD (#LN) is provided for the shake correction in step S15.
In the shaking correction in step S15, the image processing unit 32 cancels the posture change (shaking) using the quaternion QD (#LN) for each line of the image of the celestial sphere model MT to which the frame image is pasted in step S14. Shake correction is performed by rotating it so as to make it. The image of the shake-corrected celestial sphere model hMT is sent to the process of step ST19.

Then, in step ST19, the image processing unit 32 projects the image of the celestial sphere model hMT that has been shake-corrected onto a plane and cuts it out to obtain an image (output image data oPD) that has been shake-corrected.

In this case, the shaking correction is realized by the rotation of the celestial sphere model MT, and by using the celestial sphere model MT, the trapezoidal shape is not formed no matter where it is cut out, and as a result, the trapezoidal distortion is also eliminated. Further, as described above, the celestial sphere model MT has no lens distortion because the range that can be seen by an ideal pinhole camera is attached to the celestial sphere. By rotating the celestial sphere model MT according to the quaternion QD (#LN) for each line, the focal plane distortion correction is also eliminated.
Further, since the quaternion QD (#LN) corresponds to the exposure center of gravity of each line, the blur is inconspicuous in the image.

The correspondence between the image after the plane projection in step S19 and the celestial sphere model MT is as follows.
FIG. 17A shows an example of a rectangular coordinate plane 131 that is projected in a plane. Let each coordinate of the image projected on the plane be (x, y).
As shown in FIG. 17B, the coordinate plane 131 is arranged (normalized) in the three-dimensional space so as to be in contact with the celestial sphere model MT in the center. That is, the center of the coordinate plane 131 coincides with the center of the celestial sphere model MT and is arranged at a position in contact with the celestial sphere model MT.

In this case, the coordinates are normalized based on the zoom magnification and the size of the effective cutout area CL. For example, when the horizontal coordinates of the coordinate plane 131 are 0 to outh and the vertical coordinates are 0 to outv as shown in FIG. 17A, outh and outv are the image sizes. Then, for example, the coordinates are normalized by the following equation.

In the above (Equation 1), min (A, B) is a function that returns the smaller value of A and B. Further, "zoom" is a parameter for controlling enlargement / reduction.
Further, xnorm, ynorm, and znorm are normalized x, y, and z coordinates.
By each of the above equations (Equation 1), the coordinates of the coordinate plane 131 are normalized to the coordinates on the spherical surface of the hemisphere having a radius of 1.0.

As for the rotation for obtaining the direction of the effective cutout region CL, the coordinate plane 131 is rotated by the rotation matrix operation as shown in FIG. 18A. That is, the following rotation matrix (Equation 2) is used to rotate the pan angle, tilt angle, and roll angle. Here, the pan angle is a rotation angle that rotates the coordinates around the z-axis. The tilt angle is a rotation angle for rotating the coordinates around the x-axis, and the roll angle is a rotation angle for rotating the coordinates around the y-axis.

In the above (Equation 2), "Rt" is a tilt angle, "Rr" is a roll angle, and "Rp" is a pan angle. Further, (xrot, yrot, zrot) are the coordinates after rotation.

These coordinates (xrot, yrot, zrot) are used to calculate the celestial sphere corresponding points in perspective projection.
As shown in FIG. 18B, the coordinate plane 131 is perspectively projected onto the surface of the celestial sphere (region 132). That is, when a straight line is drawn from the coordinates toward the center of the celestial sphere, the point that intersects the sphere is found. Each coordinate is calculated as follows.

In (Equation 3), xsph, ysph, and zsph are coordinates obtained by projecting the coordinates on the coordinate plane 131 onto the coordinates on the surface of the celestial sphere model MT.
Image data projected in a plane can be obtained in this relationship.

The above has been described without considering the processing by the effective cutting area setting unit 34. For example, the effective cutting area for the image projected on the plane by the above method is the step ST16, ST17, ST18 of FIG. It will be set in the process.

As step ST16, the image processing unit 32 performs tracking.
This tracking process is a process of detecting the position of the tracking target specified by the operation information DR1 by the user in the image data PD (#FN) of the current frame.

The operation information DR1 is information on an operation for designating a tracking target by operating the target designation aiming 116 on the framing screen 103 as described with reference to FIG.
When the operation of designating the tracking target is performed, the image processing unit 32 recognizes the designated subject and stores it as a tracking target. After that, the process of determining the position of the tracking target in the screen is performed for each frame.
The position of the tracking target in the screen is determined by performing subject determination (for example, face determination, person determination, etc.) by image analysis, or by using the in-screen position information TPp of the tracking target in the previous frame.
Then, the in-screen position information TP of the tracking target in the determined image data PD (#FN) of the current frame is provided to the process of step S17. The in-screen position information TP may be any information that indicates the range in which the tracking target is imaged by the coordinate values in the coordinate system of the current frame.

In step ST17, the image processing unit 32 specifies an output area. This is a process of setting the requested cropping area CLrq according to the operation information DR2 by the user operation. It can be said that it is a process of designating the area of the image to be finally displayed as an output image as the request cutout area CLrq.
The setting of the required cutout area CLrq here means a temporary setting (candidate for an effective cutout area CL as an area to actually perform the cutout process) before the process of the next step ST18 is performed. This step ST17 is a process of designating the effective cutout area CL based on the user operation and the information of the tracking target, but the effective cutout area CL before being corrected by the protrusion prevention process described later is called a request cutout area CLrq. There is.
For example, when the protrusion prevention process is not performed, or when the protrusion prevention process is performed but no substantial correction is performed, the request cutout area CLrq set in step ST17 may become the effective cutout area CL as it is. be.

The operation information DR2 input by the user includes information about the operation of designating the effective cutout area CL, the operation of selecting the aspect ratio, and the operation of designating the tracking target placement position.
The operation information of the designated operation of the effective cutout area is the information of the operation of designating the range to be included in the display frame 115 on the framing screen 103 as described with reference to FIG.
The operation information of the aspect ratio selection operation is the operation information of the aspect ratio selected on the aspect ratio setting screen 106. By selecting on the aspect ratio setting screen 106, for example, the aspect ratio of the display frame 115 on the framing screen 103 or the like changes, so that the user can specify an arbitrary aspect ratio and then the framing screen 103 is in the state of the aspect ratio. You can specify the effective cutout area with.

The operation information of the operation for designating the tracking target placement position is the information of the operation for designating the position to place the tracking target on the screen as shown in FIG. 13B.
The information on the placement position of the tracking target may be a coordinate value as a coordinate system in the fixed display frame 115 as the selected aspect ratio, not as a coordinate system in the frame of the image data PD.

In step S17, the image processing unit 32 sets the request cutout area CLrq by using these operation information DR2 and the in-screen position information TP of the tracking target.
Specifically, for example, the coordinate range as the cutout area is calculated according to the operation of designating the cutout area and the operation of selecting the aspect ratio in the operation information DR2. Then, the in-screen position information TP (coordinate value in the frame of the image data PD) of the tracking target matches (or is close to) the coordinate value specified as the arrangement position in the coordinate system in the display frame 115 as the effective cutout area. To do.

By doing so, for example, when the face of the person who is the subject is set as the tracking target and the tracking target is placed at the upper right position from the center of the screen as shown in FIG. 13B, the position of the face is set in each frame. The effective cutout area will be set so that is located at the upper right of the center.

It should be noted that all or part of the area (for example, the area of the face image) indicated by the in-screen position information TP (coordinate value in the frame of the image data PD) is strictly corresponded to the tracking target placement position. No need. This is because the position of the face in the frame changes greatly for each frame, the pixel range (the size of the face on the screen) that captures the face changes, and the face part is out of the frame and cannot follow. This is because it may happen. Alternatively, it may not be possible to match the target placement position due to the protrusion prevention process described below. In that sense, the position of the tracking target (position information TP in the screen) may be as close as possible to the tracking target arrangement position (specific position in the display frame 115) designated by the operation.

Regarding the requested cutout area CLrq set as described above, the image processing unit 32 adds the protrusion prevention process in step ST18, and the result is set as the effective cutout area CL actually used in step S19.
In this protrusion prevention process, the quaternion QD (#LN) and the coordinate conversion parameter HP for a plurality of frames before and after the frame including the frame are used.

For example, the requested cutout area CLrq as a candidate set in step ST17 can be used as it is in step S19, but then, the range corresponding to the inside of the display frame 115 is rotated and projected by the shaking correction using the celestial sphere model MT. If the range is exceeded, the shake correction may not function effectively, or if the range exceeds the pixel range of the image sensor 12a, a part of the displayed image may be missing, resulting in image chipping.
Therefore, a protrusion prevention process is performed to limit the cutout area so that a high-quality moving image can be output without causing these problems.

The protrusion prevention process includes associating with the imaging area shown in FIG. 19 (first process), calculating the movable area (hereinafter also referred to as “available area”) (second process), and correcting the display range (third process). It is done in a stepwise process.
In FIG. 19, an imaging region (hereinafter, also referred to as “captured area”) 61, an available area (movable region) 62, a required cutout region CLrq, and an effective cutout region CL are shown.

The captured area (imaging area) 61 is an area in which a subject can be displayed by capturing an image. That is, it is the range in which the subject is captured.
The available area (movable area) 62 is an area that allows cutting in consideration of the shaking of the front and rear frames. That is, it is a range in which the sway correction due to the rotation of the celestial sphere model MT in the sway correction celestial sphere model MT in step ST15 is effectively maintained.

The request cutout area CLrq is an area to be displayed, which is determined according to the user operation and the tracking result.
The effective cut-out area CL is a movement of the required cut-out area CLrq so as to fit in the available cut-out area 62. This effective cutout area CL is the area that is actually cut out.

The first process (process of associating with the image pickup angle of view) in FIG. 19 is a process of setting the captured area 61 on the celestial sphere model MT, which coincides with the image pickup area.
Specifically, it is a process of allocating an imaging region on the celestial sphere model MT in consideration of the angle of view and the distortion of the lens by using the coordinate conversion parameter HP.
As the area to be allocated, the area to which the inverse correction using the coordinate conversion parameter HP is applied is used.
Further, in order to apply the shaking correction on the celestial sphere model MT, the captured area 61 shown in FIG. 19 faces the direction of canceling the shaking on the celestial sphere model MT.

The second process (calculation of the movable area) is a process of calculating the available area 62 (movable area) of the current frame using the information of the frames before and after.
The third process (correction of the display range) is a process of correcting the requested cutout area CLrq to the effective cutout area CL by using the calculated available area 62.

At least finally, if the effective cutout area CL is within the captured area 61 in which the subject exists, no image chipping will occur in the reproduced image or the displayed image. That is, if the cut-out area does not protrude from the captured area 61, no image chipping occurs.
FIG. 20A shows the relationship between the captured area 61 and the cutout area 60 in the frames # i, # i + 1, and # i + 2.
The subject range included in the captured area 61 (that is, captured) fluctuates due to the influence of the shaking. Here, if the cutout area 60 of each frame is set so as to keep the shake correction effective, for example, in the frame # i + 1, the cutout area 60 is shown by the broken line, and image chipping occurs.

On the other hand, if the cutout area 60 is allowed to move in the entire area in the captured area 61 so that the image is not chipped, the corrected shaking motion may appear directly. In other words, the shaking correction does not work effectively. For example, the cutout area 60 of the frame # i + 1 is shown by a solid line, and it is possible to prevent image chipping, but shaking appears.

Therefore, the available area 62 is calculated as a range in which the shaking correction can be effectively maintained, and the cutout area 60 is prevented from protruding from the available area 62 as a measure to prevent the protrusion.
Specifically, using the fluctuation information of the previous and next frames, the available area 62 is set as a frame with a margin so that the cutout area 60 does not go to the part that is likely to protrude in advance, and the cutout area 60 is set only in that range. Try to move.
Since the available area 62 has a meaning as a shake correction margin, it dynamically changes for each frame according to the magnitude of the shake.

FIG. 20B shows a state in which the available area 62 is set in the captured area 61 in the frames # i, # i + 1, and # i + 2 so that the cutout area 60 does not protrude from the available area 62. In this case, no image chipping occurs, and the shake correction is effectively maintained.
In the protrusion prevention process, the required cut-out area CLrq is modified so that the effective cut-out area CL as shown in the cut-out area 60 of FIG. 20B can be obtained for each frame.

An example of calculation of the available area 62 will be described.
The first calculation method is shown in FIG.
The available area 62 is set by using each captured area 61 (imaging area) of a plurality of consecutive frames including the frame to be processed and the frame at a later time in the time axis direction from the frame. In this case, the available area 62 is a range commonly included in the captured area 61 in the state where the shaking is corrected in each frame of the plurality of frames.

FIG. 21A shows a stack of captured areas 61 of a plurality of frames (frame # im to frame # i + m) that are continuous in the front-back direction when the current frame is frame # i.
The captured area 61 of each frame is associated with the celestial sphere model MT to reflect the vibration correction. That is, for each frame, the coordinate conversion parameter HP is used to allocate the imaged angle of view (captured area 61) on the celestial sphere model MT in consideration of the angle of view and lens distortion, and the quarteronion QD (#LN) is used. And rotate. In the figure, the captured areas 61 that are associated with the celestial sphere model MT and face the direction of canceling the shaking correction are shown superimposed.

The range commonly included in the captured area 61 of the plurality of frames, that is, the range in which AND is taken is the range shaded in FIG. 21A, which is referred to as the available area 62 as shown in FIG. 21B.
Since the captured area 61 of each frame is moving by the amount of the vibration correction, by overlapping the captured areas 61 of ± m frames in the front-rear direction and taking AND, the common range is within the captured area 61. You will be compensated for being there. Therefore, this range is referred to as an available area 62.
As a method of taking AND, since it is always a convex polygon, the process of taking the intersection of two convex polygons may be repeated for the number of captured areas 61.

In addition, in order to reduce the contribution of the frame as it is farther from the current frame in time, the amount of shaking correction applied to the captured area 61 may be reduced, or the frame before shaking correction may be increased. good.
If the degree of contribution is not adjusted according to the time distance, the shape of the available area 62 will change discontinuously when the frame in which a large shake occurs is removed from the range of ± m frames. On the other hand, for the captured area 61 of a frame far in time, by reducing the degree of contribution to the available area 62, it is possible to prevent the available area 62 from temporarily fluctuating significantly.

Further, although it was decided to use the captured area 61 in the range of ± m frames, it is also conceivable to use the captured area 61 up to + m frames that are continuous in the later direction on the time axis.

A second calculation method will be described with reference to FIG. 22 as a calculation example of the available area 62. This is the range shown when the available area 62 is set so that the low frequency variation component of the change of each end point of the captured area 61 for each frame of a plurality of frames is contained within the captured area 61 of each frame. It is an example to do.
Specifically, it is a method of calculating an available area 62 based on a quaternion QD to which a low-pass filter is applied.

FIG. 22A is represented in one dimension for the sake of simplification of illustration and description, but the solid line shows the change of the upper end and the lower end of the captured area 61 in each frame. That is, it is a change for each frame due to shaking.
The broken line is the low frequency fluctuation component 61L at the upper end and the lower end of the captured area 61 obtained by applying the low-pass filter.
The alternate long and short dash line indicates the upper end and the lower end as the available area 62.

Here, it can be said that the available area 62 is an area for which it is desired to guarantee that it is within the captured area 61 in a plurality of frames, but it cannot be guaranteed that the available area 62 is inside the captured area 61 only by applying a low-pass filter.
Therefore, the low-frequency fluctuation component 61L of the captured area 61 weights the amount protruding from the captured area 61 by a temporal distance and compensates for it, so that the available area is smoothly contained inside the captured area 61. Ask for 62.
In FIG. 22B, a part of FIG. 22A (range of ± m frame) is enlarged, and the direction of weighting is indicated by an arrow.
FIG. 22C shows in two dimensions the weighting that accommodates the low frequency variation component 61L within the captured area 61.

An example of a specific calculation is as follows.
For each frame (# i), the quaternion (q_crct i) required to accommodate the low frequency fluctuation component 61L in the captured area 61 is obtained.
About j ∈ [-m, m]
Wj = 1- ｜ j / m ｜
q_crct_smooth i = max {slerp (q_crct i + j, q_identity; Wj)}
As a result, the quaternion q_crct_smooth i for smooth correction is obtained for all four endpoints (this equation is a max function for "j").
The quaternion q_crct_smooth i corresponds to the maximum weighting value indicated by the arrow in FIG. 22B.
Note that slerp is spherical linear interpolation, and q_identity is a unit quaternion.

This is calculated and applied for each end point of the low frequency fluctuation component 61L as shown in FIG. 22C.
The quaternion q_crct is obtained as the amount of rotation until it collides with the side of the captured area 61 when rotated in the direction toward the center of the captured area 61 for each end point of the low frequency fluctuation component 61L.

The available area 62 can be calculated by the above first and second calculation methods and the like.
However, when the available area 62 is created by using the movement of the front and rear frames as in the above example, the available area 62 may become too small to fit the required cutout area CLrq when the movement is large.
FIG. 23A shows a state in which there is a violent movement between the frames and the captured area 61 after the vibration correction is greatly deviated. In this case, the available area 62 becomes small as shown in FIG. 23B, and the required cutout area CLrq cannot be accommodated.

Therefore, if the size or shape is such that the required cut-out area CLrq does not fit in the available cut-out area 62, the available-cut area 62 is expanded so that the required cut-out area CLrq fits.

As a method, as shown in FIG. 23C, the required cutout area CLrq is aligned with the center of the available area 62, and the available area 62 is updated so that the protruding portion is accommodated. That is, it is updated as shown in FIG. 23D.
Specifically, in the state of FIG. 23C, a new convex polygon is created by a set of "vertices of the required cutout area CLrq outside the available cutout area 62" and "vertices of the required cutout area 62 outside the required cutout area CLrq". , Let it be the expanded available area 62 of FIG. 23D.

At this time, due to the expansion of the available area 62, it cannot be guaranteed that the available area 62 fits within the captured area 61. For example, as shown in FIG. 23E, the available area 62 may protrude from the captured area 61.
Therefore, as shown in FIG. 23D, the available area 62 is moved so as to fit within the captured area 61.

By expanding the available area 62 as described above, it is possible to cope with a severe shaking.

Subsequently, the modification of the display range shown as the third process in FIG. 19 will be described. That is, it is a process of modifying the requested cutout area CLrq so that it fits within the calculated available area 62.
When the required cutout area CLrq protrudes from the available area 62, the required cutout area CLrq is moved, but the movement directions include a yaw direction, a pitch direction, and a roll direction.

In this example, when the required cutout region CLrq protrudes from the available area 62, priority is given to the correction of the yaw component and the pitch component.
That is, as shown in FIG. 24A, the required cutout region CLrq is moved toward the central CT of the available area 62 by the minimum necessary amount. Here, the roll component is not changed, but is moved by changing the yaw component and the pitch component (arrow yp).
If the requested cutout area CLrq fits in the available area 62 due to this movement, the modification of the display range is completed. That is, the moved request cut-out area CLrq is set as the effective cut-out area CL.

On the other hand, depending on the shape and size of the available area 62, the required cutout region CLrq may not fit well in the available area 62 only by moving by changing the yaw component and the pitch component.
In such a case, first, the center of the required cutout region CLrq is overlapped with the center of the available area 62. Then, as shown in FIGS. 24B and 24C, the roll component is changed. In this case, as shown by an arrow R in each figure, the entire required cutout region CLrq is rotated by the minimum amount of rotation required to fit in the available area 62. Then, the requested cutout area CLrq in the settled state is set as the effective cutout area CL.

FIG. 25 shows the overall flow of the above-mentioned protrusion prevention process.
This figure shows m frames before and after the current frame # i (frame # im to frame # i + m).
For each frame, as step S20, the celestial sphere model MT and the imaging angle of view are associated with each other using the corresponding coordinate conversion parameters HP (HP (# im) ... HP (# i + m)). As a result, the captured area 61 is represented on the ideal celestial sphere.

Then, as step S21, the shaking correction is performed using the corresponding quaternion QD (QD (# im) ... QD (# i + m)).
For example, the quaternion QD (#i) corresponding to the frame #i is the quaternion QD (LN) corresponding to each line in the frame #i.
By this step S21, the captured area 61 facing in the direction of canceling the shaking is represented on the celestial sphere model MT.

The available area 62 is generated in step S22 by using the captured area 61 for each frame. The above-mentioned first calculation method, second calculation method and the like are used.
The calculated available area 62 is expanded as necessary in step S23. That is, the required cutout area CLrq determined by the required cutout area CLrq and the available area 62 are compared, and if they do not fit, the above-mentioned expansion process is performed.

When the available area 62 is determined, the display range is modified in step S24. That is, as described with reference to FIG. 24, the required cutout area CLrq is modified to obtain the effective cutout area CL. This effective cut-out area CL is finally provided as the cut-out area CL corrected by the protrusion prevention process in the plane projection and cut-out process in step ST19 of FIG.

In the protrusion prevention process as shown in FIG. 25, the vibration correction margin is dynamically changed by reflecting the vibration information of the front and rear frames in the available area 62.
In addition, by expressing the aspect ratio, cutout position, and size of the output image in the required cutout area CLrq and putting it inside the available area 62, processing that considers shake correction and cutout position at the same time is realized. ing.

Although the above description has been made assuming an image file MF captured by a normal lens, the same processing can be applied to an image file MF captured by a fisheye lens, for example.
However, in the case of a normal lens and a fisheye lens, the expression of the captured area 61 is different.
FIG. 26A shows the representation on the celestial sphere model MT and the shape of the captured area 61 in the case of a normal lens and in the case of a fisheye lens. In the case of imaging using a fisheye lens, a circular imaging region 66 is formed within the imaging pixel range 65 of the image sensor 12a, so that the captured area 61 is represented by a circle or a polygon as shown in the figure. It will be.

<5. Summary and modification examples of image processing>

The following effects can be obtained by the image processing as the above embodiment.
The image processing device of the embodiment is a frame of image data constituting a moving image, and for each frame that has been pasted to the celestial sphere model MT, the attitude information (IMU data, quarternion QD) corresponding to each frame is used. ) Is provided to provide a shake correction unit 33 for performing shake correction. Further, the image processing device performs a protrusion prevention process for each frame so that the effective cutout area CL from each frame does not protrude from the movable area that keeps the vibration correction effective for the required cutout area CLrq. The effective cut-out area setting unit 34 for setting the effective cut-out area CL to be applied is provided. Further, the image processing device is an image projected from the celestial sphere model MT onto the plane model after the shaking correction is performed by the shaking correction unit 33, and is an image of the effective cutting area CL set by the effective cutting area setting unit 34. It is provided with an effective cut-out area image generation processing unit 35 that performs a process of generating an effective cut-out area image.
That is, the shaking caused by camera shake or the like in the image captured by the image pickup apparatus 1 is corrected by coordinate transformation on the celestial sphere model MT, and the effective cutting area CL of the image projected in a plane with this shaking correction is set.
By rotating the celestial sphere model MT for each frame in consideration of the correction for the shaking to correct the shaking, it is possible to make a correction without trapezoidal distortion. Then, by obtaining a planar projection image according to the required cutout area CLrq set by, for example, user operation or some automatic control, an effective cutout area CL reflecting the required cutout area CLrq is set, and then shake correction is performed. Moreover, output image data oPD without trapezoidal distortion can be obtained. Further, by performing the protrusion prevention processing, it is possible to output a moving image in which the shaking correction is appropriately reflected.
As a result, a high-quality image with less shaking and distortion can be obtained after reflecting the output area suitable for the user and the request of automatic control for the original image file MF.

In the embodiment, an example of cutting out an image of the effective cutout region CL after performing shake correction with the celestial sphere model MT and projecting a plane is described, but the present invention is not limited to this. That is, it is also possible to generate an image of the effective cutout region CL by cutting out an image of the effective cutout region CL with the celestial sphere model MT performing vibration correction and projecting it in a plane.
Further, the shake correction unit 33 may be at least one in which the processing of step ST15 in FIG. 14 is performed, and a part or all of steps ST11, ST12, ST13, and ST14 are outside the function of the shake correction unit 33 (for example, shake). It may be performed by a device different from the image processing device having the correction unit 33).
Further, FIG. 8 shows a case where the RAW data is targeted for image processing, but in that case, the shake correction unit 33 does not need to perform the processing of canceling the internal correction of the image pickup device in step ST12.

In the embodiment, the effective cutout area setting unit 34 gives an example of setting the effective cutout area CL by using the operation information (DR1, DR2) based on the user operation.
As a result, shake correction is performed after appropriately reflecting the user's intention and the output area desired by the user, and output image data oPD without trapezoidal distortion can be obtained.

In the embodiment, the effective cut-out area setting unit 34 gives an example of setting the effective cut-out area CL by using the tracking target information regarding the tracking target.
This makes it possible to set the cutout area in consideration of the tracking target.
Further, in the embodiment, the effective cutout area setting unit 34 gives an example of setting a tracking target using the operation information DR1 based on the user operation.
This makes it possible to set the cutout area in consideration of the subject (tracking target) that the user wants to pay attention to. That is, it is possible to obtain an output image in which shake correction is appropriately performed while capturing the subject to be noticed.

Of course, processing when the tracking target is not specified is also assumed. For example, the user may not specify the tracking target as described in FIG. In such a case, the effective cutting area CL may be set according to the operation described with reference to FIG.
In addition, processing may be performed so that the tracking target is automatically specified regardless of the setting by the user. For example, it is a process that detects a face and automatically sets the face as a tracking target when the face is detected.

In the embodiment, an example is given in which the tracking target is determined by the operation information DR1 that specifies the tracking target, and the effective cutout area CL of each frame constituting the moving image is set according to the position in the image of the determined tracking target. ..
That is, when the subject of the tracking target is specified by the user's operation, the effective cutout area CL is set in consideration of the arrangement of the tracking target in each frame.
This makes it possible to set the cutout position so that the subject (tracking target) that the user wants to pay attention to does not frame out as the moving image progresses.

In the embodiment, an example of setting the effective cutout area CL of each frame constituting the moving image is given by using the operation information DR2 that specifies the arrangement position of the tracking target in the screen.
This allows you to specify where on the screen you want the subject (tracking target) that the user wants to pay attention to, and in each frame, the tracking target is located at the placement position (near the placement position) on the screen. The cutout area can be set. Therefore, it is possible to realize moving image processing in which the moving image progresses in a state where the subject as the tracking target is in the vicinity of the arrangement position desired by the user, for example, the position from the center or the left of the screen.

There may be a processing example in which the placement position of the tracking target is not specified. For example, at least a processing example that prevents the tracking target from being framed out.
Further, the tracking target placement position may be automatically specified. For example, it is automatically set to a predetermined position such as the center of the screen or the position to the right. This makes it possible to set a cutout area so that the tracking target can be captured at a predetermined position without the user having to specify the tracking target placement position.

In the embodiment, an example is given in which the operation information DR2 that specifies the output area of the image is used for setting the effective cutout area CL.
For example, the area of the output image can be specified by the user operation on the framing screen 103 described with reference to FIG. 12, and the operation information DR2 is used for setting the effective cutout area CL.
As a result, it is possible to obtain an output image with appropriate shake correction while cutting out an output area of an image arbitrarily designated by the user.

In the embodiment, an example is given in which the operation information DR2 that specifies the aspect ratio of the image is used for setting the effective cutout area CL.
For example, the aspect ratio of the output image can be specified by the user's operation on the aspect ratio setting screen 106, and the operation information DR2 is used for setting the cutout area.
As a result, it is possible to obtain an output image having an aspect ratio arbitrarily specified by the user and having appropriate shake correction.

In the embodiment, as the protrusion prevention process performed for each frame of the image data, the first process of setting the required cutout area CLrq on the coordinates of the celestial sphere model MT to which the shake correction unit 33 has performed the shake correction, and the imaging region (capture). By the second process of obtaining a movable area (available area 62) that allows fluctuation of the required cutout area CLrq in the data area 61), and by modifying the required cutout area CLrq so that it fits within the movable area (available area 62). An example of performing the third process of setting the effective cutout area CL has been given.
As a result, the tracking target, output area, cut-out area based on the aspect ratio, etc., arbitrarily specified by the user, does not fluctuate indefinitely depending on the image content, etc. To. Therefore, it is possible to obtain the output image data oPD in which the shaking correction is appropriately performed while reflecting various specifications of the user.
Further, in order to properly maintain the shaking correction, an imaging area (captured area 61) is set on the coordinates of the celestial sphere model MT to which the shaking correction is performed, and a movable area (available area 62) is set in the image area (captured area 61). Calculate and modify the request cutout area CLrq so that it fits within the available area 62. Then, the modified request cut-out area CLrq is set as the final effective cut-out area CL. That is, the cutout area is limited by the available area 62. This makes it possible to appropriately set the cutout area that reflects the shaking correction in the celestial sphere model MT.

In the second process in the protrusion prevention process of the embodiment, the available area 62 is divided into a frame to be processed and a plurality of consecutive frames including a frame at a later time point and a frame at a previous time point in the time axis direction from the frame. An example of setting using the captured area 61 is given.
That is, by using the captured area 61 of a plurality of consecutive frames as in the first calculation method of FIG. 21 and the second calculation method of FIG. 22, it occurs in the time axis direction (between frames) for a certain period of time. Even if there is shaking, the image range commonly included in each frame can be detected. This is an area where the image is not chipped during the period of the plurality of frames and the shake correction is effective. This is referred to as an available area 62. As a result, the available area 62 is dynamically changed according to the magnitude of the shaking, and the cutout area can be set so that the image is not chipped due to the shaking and the shaking correction can be maintained on the moving image by the output image data oPD. Therefore, it is possible to easily create a high-quality moving image.
It should be noted that the available area 62 can also be set by using each captured area 61 of a plurality of consecutive frames including the frame to be processed and the frame at a later time in the time axis direction from the frame.
Further, the available area 62 can be set by using each captured area 61 of a plurality of consecutive frames including the frame to be processed and the frame at a time point before the frame in the time axis direction.

In the embodiment, the available area 62 is an example in which the available area 62 is a range commonly included in the captured area 61 in a state where the shake correction is performed for each frame of a plurality of frames.
That is, as described as the first calculation method in FIG. 21, by using the captured area 61 in which the shaking correction is performed for a plurality of consecutive frames, the range in which the image chipping does not occur even if the cutout frame is set. The available area 62 can be set. Since it is a range common to the captured area 61 in the state where the shake correction is performed, the shake correction is also an effective range.
Further, in this case, the captured area 61 may be used as it is, but the degree of contribution is adjusted according to the time distance by changing the degree of vibration correction of the captured area 61 depending on the distance from the current frame. As a result, it is possible to mitigate the influence of large shaking at a distant point and smooth the change of the available area 62. This does not cause a state in which the angle of view changes suddenly in the reproduced moving image.

In the embodiment, the available area 62 is a range shown when the low frequency variation component 61L of the change of each end point of the captured area 61 for each frame of a plurality of frames is accommodated in the captured area 61 of each frame. I gave an example.
That is, it is an example described as the second calculation method in FIG. 22.
By setting the available area 62 of each frame using the low frequency fluctuation component 61L of the change of each end point of the captured area 61, the change of the available area 62 in the time axis direction is smoothed, and the change of the cutout area suddenly changes. You can get a video that does not occur.
However, since it is not possible to compensate for the fact that image chipping does not occur (that is, the cutout area is within the captured area 61) simply by using the low frequency fluctuation component 61L, the image may be contained within the captured area 61 for each frame. The position of each end point defined by the low frequency fluctuation component 61L is corrected. As a result, it is possible to obtain a moving image in which no image chipping occurs and the change in the cutout area is smooth.

The third process in the protrusion prevention process of the embodiment is performed by moving the required cut-out area CLrq toward the center of the available cut-out area 62 without rotating it until the required cut-out area CLrq fits in the available cut-out area 62. did.
That is, as described with reference to FIG. 24 as a modification of the display range, the required cutout area CLrq is moved without rotating by a necessary distance until it fits in the available area 62. As a result, the tracking target, the output area, the cutout area based on the aspect ratio, etc., arbitrarily specified by the user, are moved into the available area 62 so as not to cause rotation as much as possible. It is possible to prevent the image from rotating.

Further, in the third process of the embodiment, by moving the required cutout area CLrq toward the center of the available area 62 without rotating the required cutout area CLrq, if the required cutout area CLrq cannot be accommodated in the availableable area 62, the required cutout area CLrq is cut out. It was decided to rotate the region CLrq so that it fits within the available area 62.
That is, the required cutout region CLrq is rotated only when it does not fit in the available area 62 in the state where it is not rotated. In this way, as a correction of the cutout area, it is decided to rotate it only when it is unavoidable, so that the image chipping is prevented as much as possible.

In the embodiment, the shake correction unit 33 acquires the posture information (quaternion QD) corresponding to each frame based on the metadata generated at the time of imaging the image data (image file MF) to be processed. It was decided to use it for shake correction.
The quaternion QD, which is the attitude information of the image pickup device, is calculated from the IMU data added as the metadata. By using this to perform sway correction on the celestial sphere model MT, it is possible to perform appropriate sway correction according to the situation at the time of imaging. In particular, in the case of the present embodiment, since the internal correction of the camera performed at the time of imaging is canceled, the optimum shaking correction can be performed by reflecting the posture of the imaging device 1 at the time of imaging as it is.

In the embodiment, an example of performing shake correction using the posture information (quaternion QD (LN)) acquired based on the timing of the exposure center of gravity for each line for each frame of the image data is given (step SST14 in FIG. 14). reference).
The relative shaking of the subject and the image pickup device during the exposure period causes so-called blur, which becomes more noticeable as the exposure time increases depending on the shutter speed. In this case, if the posture information of the exposure start timing of each line is used, the relative shaking within the exposure period of each line becomes a one-way shaking when viewed from the posture information, and becomes conspicuous on the image.
On the other hand, if the attitude information is acquired based on the IMU data of the center of gravity position within the exposure period of each line as shown in FIG. 16, the relative shaking during the exposure period is bidirectional shaking when viewed from the state of the posture information. And visually, it seems to be offset in the time direction, which makes the blur less noticeable in the image.

In the embodiment, the shake correction unit 33 performs correction cancellation processing on the image data to be processed using the metadata associated with the image data, and is in a state before the correction processing already performed is performed. It was decided to perform the pasting process on the celestial sphere model MT in the state of returning to (see step SST12 in FIG. 14).
For example, in the image pickup apparatus 1, correction processing such as camera shake correction is performed at the time of imaging, and the image data to be processed is subjected to shake correction after canceling these already performed corrections.
As a result, appropriate shake correction can be performed on the original captured image data, for example, the image data in the state taken out from the image pickup element unit 12, without being affected by the correction at the time of imaging. That is, it is possible to realize high-quality shake correction according to the processing capacity of the image processing device without being affected by the performance of the image pickup device 1 that has taken an image.

In the embodiment, the shake correction unit 33 decides to cancel the correction using the coordinate conversion parameter HP associated with each frame of the image data as metadata.
The coordinate conversion parameter HP is information when changing the image frame (range of the image including the angle of view) of the captured image, such as lens distortion, trapezoidal distortion, focal plane distortion, electronic image stabilization, and optical image stabilization parameters. Is.
As a result, the correction related to the image frame at the time of imaging can be canceled, and when the image frame (range of the image) is corrected by the image pickup apparatus 1, it can be restored.

The image processing apparatus of the embodiment includes a preprocessing unit 31 that extracts and stores the metadata corresponding to each frame of the image data for the image file MF including the image data constituting the moving image and the corresponding metadata.
As a result, the metadata for each frame can be used in the processing of the shaking correction unit 33 and the processing of the effective cutting area setting unit 34.

Note that FIG. 14 shows the processing of steps ST1 to ST4 as the processing by the function of the preprocessing unit 31 of the image processing apparatus of the embodiment, but the processing is not limited to this. The pre-processing unit 31 may at least perform processing for calculating posture information according to the metadata (for example, step ST4) and holding the metadata (for example, step ST3).
Further, as the processing by the function of the image processing unit 32, the processing of steps ST11 to ST19 is shown in FIG. 14, but the processing is not limited to this. For example, the processing of step ST11 may be performed externally, and in that case, the image processing unit 32 may perform the processing of steps ST12 to ST19.
Further, the image processing unit 32 may not perform the process of canceling the correction in the image pickup apparatus in step ST12.
It is also conceivable that the image processing unit 32 does not perform the processing related to the tracking in step ST16.
It is also conceivable that the image processing unit 32 does not perform the synchronization processing of step ST14.
At least in the image processing apparatus of the present disclosure, as processing by the image processing unit 32, step ST13 (pasting to the celestial sphere model MT), step ST15 (shaking correction), step ST17 (output area designation), and step ST18 (prevention of protrusion). Processing) and step ST19 (planar projection and cutting) may be performed.

The technique described in the embodiment can be applied to the following cases as camera shake correction in the image pickup apparatus 1. The optical image stabilization referred to below is the correction of the fluctuation of the inter-frame component. In addition, electronic image stabilization includes the case of only focal plane correction.
・ When optical image stabilization is performed and electronic image stabilization is not performed ・ When neither optical image stabilization nor electronic image stabilization is performed ・ When optical image stabilization is not performed and electronic image stabilization is performed ・ Optical image stabilization When both electronic image stabilization is performed, the processing of the image processing device of the embodiment can be applied to the image captured by the image pickup device 1 in each of these cases.

<6. About UI for image processing>

Subsequently, the details of the display screen related to the image processing whose outline is shown by FIG. 11 and the like will be described.
Here, the acceptance of user operations on various screens described below and the display control according to the operations are realized by the processing of the CPU 71 shown as the reception unit 36a and the display control unit 36b of the UI processing unit 36 described above. Is.

[About the import screen]

FIG. 27 is a diagram showing an example of the import screen 101. As described above, the import screen 101 is a screen that appears on the display unit 77 in response to the activation of the application program for moving image processing, and the user selects the image file MF to be processed.

As shown in FIG. 27, the import screen 101 is provided with a menu button I11, a folder selection area I12, a thumbnail display area I13, and a selection completion button I14.
When the menu button I11 is operated (for example, a touch operation), the menu panel I11a shown outside the screen in the figure is displayed. The menu panel I11a is shown outside the screen for convenience of illustration, and the menu panel I11a is actually displayed as an overlay on the menu button I11.
A setting button I11b is arranged on the menu panel I11a, and when the setting button I11b is operated, the setting screen 107 described above is called.

The application folder button I12a and the "All" button I12b are displayed in the folder selection area I12. The application folder button I12a is a button for selecting a folder for an application program for video processing secured in the mobile terminal 2, and when the application folder button I12a is operated, the thumbnail display area I13 has an application folder. The thumbnail image I13a of the image file MF stored in the image file MF is displayed. Further, the "All" button I12b is a button for selecting a folder in which all the image file MFs are stored in the mobile terminal 2, and when the "All" button is operated, the thumbnail display area I13 is in the folder. The thumbnail image I13a of the stored image file MF is displayed.
In the folder selection area I12, the application folder button I12a is set to the selected state by default.

In the thumbnail display area I13, a part of the frame image in the image file MF is displayed as each thumbnail image I13a, but in this example, the image of the first frame is displayed. In this example, in the case of the image file MF whose playback time exceeds 1 hour, the thumbnail image I13a is grayed out and an error dialog is displayed at the time of tapping.

When a predetermined operation (for example, an operation of tapping a part other than the selection reception button I13b described later) is performed on the thumbnail image I13a displayed in the thumbnail display area I13, the import screen 101 is displayed according to the embodiment shown in FIG. The moving image of the corresponding image file MF is overlaid and played. In the import screen 101 at the time of overlay reproduction, an overlay display area I15 (in this example, the entire area of the import screen 101 is overlaid) is provided, and a moving image display area in which the reproduced moving image is displayed is provided in the overlay display area I15. I15a is provided, and a seek bar I15b, a reproduction time information I15c, a reproduction button I15d, and a close button I15e are displayed.
When the play button I15d is operated, the moving image playback in the moving image display area I15a is started, and when the playing button I15d is operated again, the moving image playback is paused. The seek bar 15b represents the regenerated portion and the other portion in different display modes, and fast forward / rewind is possible by sliding the round portion.
The reproduction time information I15c represents both the total time of the moving image and the time of the portion being reproduced in this example.
When the close button I15e is operated, the overlay reproduction ends, and the import screen 101 is returned to the state before the overlay reproduction (for example, the state shown in FIG. 27).
In the above, an example in which the operation of tapping a portion other than the selection reception button I13b in the thumbnail image I13a is an overlay reproduction instruction operation is given. For example, as shown in the circled broken line at the bottom of FIG. 27, an overlay reproduction instruction is given. It is also possible to provide a play button Pb for the purpose. In this case, the thumbnail selection operation (operation for selecting as an import target) described below is an operation of tapping a predetermined area other than the display area of the play button Pb as shown by the satin area in the figure. You can also.

As shown in FIG. 27, each thumbnail image I13a in the thumbnail display area I13 is provided with a selection reception button I13b. When the selection reception button I13b is operated, the CPU 71 determines whether or not to import based on the meta information of the image file MF. As a specific judgment,
-Whether or not 50 image file MFs are already selected and the 51st image file MF is newly selected.-If the newly selected image file MF is included, the total time of the selected video is included. Whether or not the image exceeds 60 minutes-It is determined whether or not the image file MF is not the target of processing (the image file MF generated by the image pickup device 1 other than the target model).
As a result of the above determination, when it is determined that import is possible (when a negative result is obtained in all the above determinations), the CPU 71 changes the display mode of the operated selection acceptance button I13b to a display mode (for example, specifying) indicating the selected state. Change to (color display) (see FIG. 29A). At the same time, the operated selection reception button I13b is indicated with numerical information for indicating the order of the image file MF in the selected state (see FIG. 29A). For example, in the case of the first selected image file MF, "1" is written).
Here, when the selection reception button I13b in the selection state is operated again, the display mode indicating the selection state of the selection reception button I13b is canceled, and the selection state of the corresponding image file MF is also canceled.
It is not essential to display the numerical information indicating the number of selections on the selection reception button I13b.

On the import screen 101, the selection completion button I14 is inactive due to, for example, gray out display in the state where there is no image file MF selected in the thumbnail display area I13 (see FIG. 27). When at least one image file MF is selected in the thumbnail display area I13, the selection completion button I14 is activated by changing from, for example, a grayout display to a normal display (FIG. 29A).
When the selection completion button I14 activated in this way is operated, the display transitions to, for example, the display of the preview screen 102 illustrated in FIG. 29B.
By operating the selection completion button I14, the selected image file MF is confirmed as the image file MF to be image processed. In this example, the selected image file MFs are concatenated in the selected order according to the operation of the selection completion button I14.

[Preview screen]

The details of the preview screen 102 will be described with reference to FIG. 30.
As shown in the figure, the preview screen 102 has a video list area I21, a menu button I22, a video menu button I23, a preview area I24, a playback button I25, a playback time display area I26, a timeline I27, an aspect ratio setting button I28, and an export. A button I29 is provided, and a frame button B1, a trim button B2, and a speed button B3 are provided at the bottom of the screen.

On the preview screen 102, the thumbnail images of the image file MF selected as the image processing target on the import screen 101 are displayed in the moving image list area I21 in the order of selection. If the number of selected image file MFs is large and cannot be displayed at one time, it is possible to scroll horizontally by swiping. When the thumbnail image in the moving image list area I21 is tapped, the focus frame is displayed in the thumbnail image, and the first frame of the corresponding image file MF is displayed in the preview area I24. At this time, the reproduction position bar I27a on the timeline I27 also moves to that position (when tapped during reproduction, it jumps while the reproduction is continued).
Further, in the moving image list area I21, if the thumbnail image is long-pressed, the order of the moving images can be changed by the drag and drop operation (when the order is changed, the timeline I27 is redrawn).

The moving image menu button I23 is arranged in the vicinity of the moving image list area I21 and functions as a call instruction button of the moving image menu panel I23a.
FIG. 31 illustrates a preview screen 102 when the moving image menu panel I23a is called. As shown in the figure, a moving image addition button I23b, a moving image deletion button I23c, and a moving image copy button I23d are arranged on the moving image menu panel I23a. When the video addition button I23b is operated, the screen transitions to the import screen, and the video to be image processed can be added. When the video deletion button I23b is operated, a confirmation dialog (not shown) is displayed on the preview screen 102, and when OK is tapped on the dialog, the video in focus in the video list area I21 is deleted (excluded from the image processing target). Ru). When the moving image is deleted in this way, the focus frame in the moving image list area I21 shifts to the next moving image (thumbnail image) (when the moving image at the end of the list is deleted, the moving image shifts to the previous one).
When the moving image copy button I23d is operated, the moving image in focus is copied in the moving image list area I21 and inserted one after the moving image in focus. If the moving image to be copied is a moving image that has undergone image processing such as framing, trimming, and speed, which will be described later, the moving image is copied while maintaining the image processing content. When the moving image copy button I23d is operated, if the total time exceeds one hour after copying, an error dialog is displayed according to the operation of the moving image copy button I23d.

In FIG. 30, the menu button I22 functions as a call instruction button for the menu panel I22a as shown outside the screen in the figure. The menu panel I22a is actually overlaid on the screen in the same manner as the previous menu panel I11a (FIG. 27). A new project button I22b, a setting button I22c, and a processing reset button I22d are displayed on the menu panel I22a. When the new project button I22b is operated, a confirmation dialog is displayed, and when OK is tapped on the dialog, all the contents of the image processing performed up to that point are discarded and the process transitions to the import screen 101. When the setting button I22b is operated, the screen transitions to the setting screen 107. When the processing reset button I22d is operated, a confirmation dialog is displayed, and when OK is tapped on the dialog, the image processing contents of framing, trimming, and speed are all discarded. In this example, the addition of the moving image and the change of the aspect ratio (change on the aspect ratio setting screen 106) are not reset.

The preview area I24 is an area in which the moving image being reproduced is displayed and, when the reproduction is paused (paused), the frame image at the corresponding reproduction position is displayed. A display frame 115 is arranged in the preview area I24, and in the preview screen 102 shown in FIG. 30 (that is, the preview screen 102 in which the preview area I24 is in the zoomed-in display state described later), the playback moving image or the pause is formed in the display frame 115. The image inside is displayed.
Here, the display frame 115 corresponds to a frame representing a cutout range from the original image (see the description of the framing screen 103 described later). The area in the display frame 115 is one aspect of the "display area" according to the claim.

The play button I25 functions as a play start instruction button for the currently targeted moving image (the moving image focused in the moving image list area I21). When the play button I25 is operated, the video currently targeted in the preview area I24 starts playing. When the reproduction is started, the display of the reproduction button I25 is switched to the display of the pause button (not shown). When this pause button is operated, the moving image reproduction in the preview area I24 is paused, and the display of the pause button is switched to the display of the reproduction button I25.
In the reproduction time display area I26, the total reproduction time (the total time of the reproduction times of all the moving images selected as the image processing targets) and the time of the portion currently being reproduced are displayed.

On the timeline I27, a timeline in which all the moving images selected as image processing targets are concatenated is displayed. In this example, the reproduction position bar I27a is displayed in the timeline I27, and represents the current reproduction position on the timeline I27. In the preview screen 102 in this example, the reproduction position bar I27a is fixed at the left and right center positions of the timeline I27.
Further, in the timeline I27 on the preview screen 102 of this example, when the current playback position is the beginning of the linked video, the left side of the playback position bar I27a is painted black and the current playback position is linked. If it is the end of the moving image, the right side of the playback position bar I27a is filled in black and displayed.

Here, in the timeline I27 on the preview screen 102 of this example, if there is a moving image section in which the playback speed change processing on the speed screen 105 and the tracking processing on the framing screen 103, which will be described later, are performed, those moving image sections are performed. Is displayed to indicate.
FIG. 32 shows an example of a display on the timeline I27 showing the sections in which the reproduction speed is changed and the tracking processing is performed.
As shown in the figure, in the example of this figure, on the timeline I27 of the preview screen 102, the section where the playback speed is changed is displayed as the speed bar Ss, and the section where the tracking process is performed is set as the tracking bar St. it's shown. In this example, information representing the reproduction speed set in the corresponding section (“× 2” in the illustrated example) is displayed in the speed bar Ss. Further, in the tracking bar St, information indicating that the section is the section where the tracking process has been performed (in the illustrated example, an illustration imitating the upper body of a person and a frame surrounding the person) is displayed.
Further, in order to facilitate the distinction between the respective sections, in this example, the display colors of the speed bar Ss and the tracking bar St are different.

In FIG. 30, the reduction button I30 is a button for switching the display of the preview area I24 to the zoom-out display.
FIG. 33 illustrates a preview screen 102 in which the preview area I24 is set to the zoom-out display state in response to the operation of the reduction button I30.
As shown in the figure, in the preview area I24 in the zoom-out display state, the size of the display frame 115 is reduced, and the image portion outside the display frame 115 is also displayed for the moving image being focused. That is, in the entire image as the captured area 61 (FIG. 19) described above, the image portion outside the display frame 115 (corresponding to the display area 63 in FIG. 19) is also displayed.
Here, the image of the captured area 61 can be rephrased as the image of the cutout source in the case where the cutout is performed by the display frame 115, and is hereinafter referred to as "original image" in this sense.
In this example, the image portion outside the display frame 115 in the original image is referred to as the image portion inside the display frame 115 so that the range cut out from the original image by the display frame 115 and the other ranges can be clearly distinguished. It is supposed to be displayed in a different display mode. Specifically, in this example, the image portion outside the display frame 115 in the original image is grayed out.

In this example, when the reduction button I30 is operated, the preview area I24 is set to the zoom-out display state as described above, and the display of the reduction button I30 is switched to the display of the enlargement button I31 (see FIG. 33). ). By operating the enlargement button I31, the display of the preview area I24 is switched to the zoom-in display shown in FIGS. 30 and 32. In the preview area I24 in the zoomed-in display state, the size of the display frame 115 is enlarged as compared with the zoom-out state, and only the image portion in the display frame 115 in the original image is displayed.
Further, when the enlargement button I31 is operated, the display of the enlargement button I31 is switched to the display of the reduction button I30, which makes it possible to switch the preview area I24 to the zoom-out display state again.

In FIG. 30, the frame button B1, the trim button B2, and the speed button B3 provided on the preview screen 102 function as buttons for instructing the transition to the framing screen 103, the trimming screen 104, and the speed screen 105, respectively. Further, the aspect ratio setting button I28 functions as a button for instructing a transition to the aspect ratio setting screen 106.
The export button I29 functions as a button for instructing a transition to the export screen 108 (display state of the size selection dialog I70 described later).

In this example, the length of the moving image that can be exported is limited to 2 seconds to 1 hour. Therefore, when the export button I29 is operated, the CPU 71 confirms the total time of the moving image to be image-processed, and the export screen is displayed in each case where the total time exceeds 1 hour and less than 2 seconds. A predetermined error dialog is displayed on the preview screen 102 without performing the transition to 108.

[Aspect ratio setting screen]

Subsequently, the aspect ratio setting screen 106 shown in FIG. 34 will be described.
As shown in the figure, the aspect ratio setting screen 106 is provided with a back button I35 and an aspect ratio selection area I36. Further, the aspect ratio setting screen 106 is provided with a preview area I24, a reproduction button I25, a reproduction time display area I26, a timeline I27, and a reduction button I30 similar to the preview screen 102.

The back button I35 functions as a button for instructing the transition to the preview screen 102. When the back button I35 is operated on the aspect ratio setting screen 106, the image processing content on the aspect ratio setting screen 106 is confirmed and the preview screen 102 is displayed.

The aspect ratio selection area I36 is provided with a selection button for selecting the aspect ratio of the display frame 115 (that is, the aspect ratio of the cutout image) for each of the various aspect ratios. When any of these selection buttons is operated, the corresponding aspect ratio is reflected in all the moving images to be processed at once. At this time, the position (cutout position) of the display frame 115 is at the center (for example, at the center of the original image), but it can be changed on the framing screen 103 described later.
The default aspect ratio is the aspect ratio of the previously exported video. If the export has never been performed, the initial value "1: 1" is the default value. Also, after updating the application program, the previous value is forgotten and returned to the initial value.

By providing the reduction button I30 also on the aspect ratio setting screen 106, the display of the preview area I24 can be switched from the zoom-in display shown in FIG. 34 to the zoom-out display as shown in FIG. 35. Also in this case, the display of the reduction button I30 is switched to the display of the enlargement button I31, and the zoom-out display state and the zoom-in display state can be switched between each other.

[About the trimming screen]

FIG. 36 shows an example of the trimming screen 104.
The trimming screen 104 is provided with a video list area I21, a preview area I24, a play button I25, a timeline I27, and a reduction button I30 similar to the preview screen 102, and a return button I35 similar to the aspect ratio setting screen 106. It is provided.
Further, on the trimming screen 104, a start point icon I41 and an end point icon I42 are provided for the timeline I27, and a time comparison display area I43 is provided above the timeline I27. Further, the trimming screen 104 is provided with a reset button I44.

The trimming screen 104 of this example is a screen that accepts the designation of the trimming section for the moving image focused in the moving image list area I21. Therefore, the timeline I27 on the trimming screen 104 displays the timeline for the moving image focused on the moving image list area I21.
Further, the reproduction position bar I27a on the trimming screen 104 can be moved left and right by a swipe operation, and when the reproduction position bar I27a is moved, the display image in the preview area I24 is the reproduction position indicated by the reproduction position bar I27a. It changes to an image. This facilitates the user's search for the trimming start point and end point.

The start point icon I41 functions as an operator for instructing the start point of trimming, and the end point icon I42 functions as an operator for instructing the end point of trimming. The start point icon I41 and the end point icon I42 can be moved by swiping, respectively, and the start point and end point of trimming can be arbitrarily specified. In this example, the initial position of the start point icon I41 is the playback position of the first frame of the video being focused, and the initial position of the end point icon I43 is the playback position of the last frame of the video being focused.

FIG. 37 illustrates a trimming screen 104 when the start point icon I41 and the end point icon I42 are operated.
When the start point icon I41 and the end point icon I42 are operated, the thumbnail of the moving image portion excluded by trimming on the timeline I27 is grayed out as shown in the figure, and the moving image portion extracted by trimming is clarified.
Further, although not shown, for the start point icon I41 and the end point icon I42, information on the reproduction time corresponding to the position of the icon is displayed in the vicinity of the icon such as the upper part of the icon while the icon is tapped.

On the trimming screen 104, in the time comparison display area I43, the playback time of the entire moving image being focused and the playback time after trimming (the playback time of the moving image section instructed by the start point icon I41 and the end point icon I42) are displayed. Is displayed.

Further, when the reset button I44 is operated on the trimming screen 104, the trimming content returns to the initial state (at this time, the icon of the start point / end point moves to the left end / right end), and the return button I35 is operated. Then, the image processing content on the trimming screen 104 is confirmed, and the screen transitions to the preview screen 102.

Also on the trimming screen 104, by providing the reduction button I30, the display of the preview area I24 can be switched from the zoom-in display shown in FIG. 36 to the zoom-out display as shown in FIG. 38. Also in this case, the display of the reduction button I30 is switched to the display of the enlargement button I31, and the zoom-out display state and the zoom-in display state can be switched between each other.

[About speed screen]

FIG. 39 shows an example of the speed screen 105.
The speed screen 105 is provided with a video list area I21, a preview area I24, a play button I25, a timeline I27, and a reduction button I30 similar to the preview screen 102, and a return button I35 similar to the aspect ratio setting screen 106. And the speed selection area I50 is provided.

Similar to the previous trimming screen 104, the speed screen 105 is a screen for the moving image being focused in the moving image list area I21, and the timeline of the moving image being focused is displayed on the timeline I27.
Further, also on the speed screen 105, the reproduction position bar I27a can be moved left and right by a swipe operation as in the case of the trimming screen 104, and in the preview area I24, the image of the reproduction position indicated by the reproduction position bar I27a is displayed. Is displayed. This facilitates the search work of the start point and the end point of the speed setting by the user.

Further, above the timeline I27 in this case, a time comparison display area I43'is provided. In the time comparison display area I43', the playback time before the speed setting and the playback time after the speed setting are displayed for the moving image in focus.

In the speed selection area I50, a selection button for selecting a reproduction speed is provided for each of various reproduction speeds, and a reset button is arranged.
When any of the playback speed-representing selection buttons is operated in the speed selection area I50, the start point icon I51 and the end point icon I52 are displayed on the timeline I27 as illustrated in FIG. 40. As for the start point icon I51 and the end point icon I52, the reproduction time at the position indicated by the icon is displayed while the icon is tapped, as in the case of the previous start point icon I41 and the end point icon I42.
By swiping these icons, the user can specify a moving image section to which the playback speed selected in the speed selection area I50 is to be applied.
Further, when the moving image section is designated in this way, the section bar I53 representing the moving image section is displayed on the timeline I27. In this example, information representing the reproduction speed selected in the speed selection area I50 is displayed in the section bar I53. FIG. 41 illustrates a case where the notation of “x2” is made in the section bar I53 corresponding to the case where the reproduction speed of “x2” is selected. It should be noted that such information display in the section bar I53 is not performed when the section bar I53 is too short.

Further, when any playback speed is selected in the speed selection area I50 and another playback speed is selected in the speed selection area I50 after the section is being specified as shown in FIG. 40. Only the reproduction speed and the reproduction time (the reproduction time after applying the speed setting) in the time comparison display area I43'are changed.
Further, when the reset button in the speed selection area I50 is operated while the speed application section is being set, the speed setting content is reset and the screen state is returned to the speed screen 105 in the initial state shown in FIG. 39.

Also on the speed screen 105, by providing the reduction button I30, the display of the preview area I24 can be switched from the zoom-in display shown in FIG. 39 to the zoom-out display as shown in FIG. 41. Also in this case, the display of the reduction button I30 is switched to the display of the enlargement button I31, and the zoom-out display state and the zoom-in display state can be switched between each other.

[About the framing screen]

Subsequently, the framing screen 103 will be described.
As illustrated in FIG. 42, the framing screen 103 is provided with the same video list area I21, preview area I24, playback button I25, playback time display area I26, timeline I27, and reduction button I30 as the preview screen 102. At the same time, a back button I35 similar to the aspect ratio setting screen 106 is provided.
Further, the framing screen 103 is provided with a rotation button I61, a tracking button I62, and a tracking start button I63, and the target designation aiming 116 described above is displayed in the display frame 115 of the preview area I24.

On the framing screen 103, when a still image of a certain frame in the focused moving image is displayed, that is, when the preview playback in the preview area I24 is in the pause state, the designation of the cutout range of the image from the original image is specified. , Specifying the tracking target, specifying the placement position on the screen of the tracking target, etc. are possible.
As described above, in the framing screen 103 of this example, the range included in the display frame 115 in the original image can be moved or enlarged / reduced by performing a drag operation or a pinch-out / pinch-in operation in the display frame 115. Can be made to. That is, the designation of the cutout range from the original image by moving the original image in the vertical, horizontal, and diagonal directions with respect to the display frame 115, and the designation of the cutout range from the original image by enlarging / reducing the original image with respect to the display frame 115. It can be performed. It is also possible to change the rotation angle of the original image with respect to the display frame 115 by performing a rotation operation using two fingers in the display frame 115. That is, it is also possible to specify the cutout range from the original image by rotating the original image with respect to the display frame 115.

In the framing screen 103 of this example, it is possible to change the rotation angle with respect to the display frame 115 of the original image by operating the rotation button I61, but the rotation by the operation of the rotation button I61 is in units of 90 degrees. It is restricted. Specifically, in this example, each time the rotation button I61 is operated, the original image is rotated 90 degrees clockwise.
At this time, even if the rotation operation with the two fingers described above results in a halfway angle, the rotation angle of the original image is adjusted to an angle that is a multiple of 90 by operating the rotation button I61.

Also on the framing screen 103, by providing the reduction button I30, the display of the preview area I24 can be switched from the zoom-in display shown in FIG. 42 to the zoom-out display as shown in FIG. 43. Also in this case, the display of the reduction button I30 is switched to the display of the enlargement button I31, and the zoom-out display state and the zoom-in display state can be switched between each other.

An example of specifying the cutting range will be described with reference to FIG. 44.
Note that FIG. 44 shows how the cutout range is specified by the operation for the preview area I24 in the zoom-out display state.
For example, it is assumed that a swipe operation to the left with respect to the inside of the display frame 115 as shown as operation Sw in FIG. 44A is performed. In response to such an operation, as shown in FIG. 44B, the original image is moved to the left (swipe direction) with respect to the display frame 115, and as a result, the range included in the display frame 115 in the original image changes. That is, the cutout range from the original image changes.

Here, as illustrated in FIG. 44, in the framing screen 103 of this example, the image inside the display frame 115 and the image outside the display frame 115 are displayed in different display modes in the preview area I24 in the zoom-out display state. I'm letting you.
By doing so, it is possible to make the user clearly recognize what kind of range is set as the cutout range in the original image.

Subsequently, the image processing of tracking will be described.
In the tracking image processing, the user accepts the designation of the placement position on the screen of the tracking target. When the user wants to perform an operation related to tracking, the user operates the tracking button I62 on the framing screen 103 (for example, a tap operation).

FIG. 45 illustrates a framing screen 103 (first screen) in the tracking mode that appears in response to the operation of the tracking button I62.
In this example, on the framing screen 103 in the tracking mode, the display mode is changed, for example, the image in the display frame 115 is displayed in gray out, so that the user can easily perceive that the mode has changed. ing.

In the tracking mode, the operation for the inside of the display frame 115 functions as an operation for moving the target designation aiming 116, that is, an operation for designating the arrangement position of the subject (target subject) as the tracking target in the cutout image.

Specifically, as shown in FIG. 46, the target designated aiming 116 is moved to the tapped position in response to the operation of tapping in the display frame 115 (operation Tc in FIG. 46A) (FIG. 46B). ). This tapped position is recognized as the placement position in the cutout image of the target subject. In addition, this tap operation also functions as an operation of designating a subject to be tracked among the subjects displayed in the original image.
In the framing screen 103 before the tracking start button I63 is operated, the position of the target designated aiming 116 is set to a predetermined initial position (in this example, the center of the display frame 115) (see FIG. 42).
Further, in this example, it is possible to specify the placement position in the cutout image of the target subject not only by tapping the inside of the display frame 115 but also by moving the target designation aiming 116 by a drag operation. Will be done.

When the operation of tapping the inside of the display frame 115 (the operation of designating the position of the target designation aiming 116) is performed as shown in FIG. 46, the tracking start button I63 as illustrated in FIG. 47 is displayed on the framing screen 103. To. When the tracking start button I63 is operated, the reproduction of the tracking image is started in the preview area I24. That is, the moving image display of the cut-out image cut out from the original image is started so that the target subject is arranged at the position designated in the display frame 115 by the target designated aiming 116. At this time, in the moving image to be previewed and reproduced, the display of the tracking frame Ft representing the recognition position of the target subject designated by the target designated aiming 116 is started.
Here, as understood from the above description, in the present embodiment, the process of cutting out from the original image so that the target subject is arranged at the designated position in the display frame 115 is displayed in the display frame 115. According to the designated operation performed on the image, that is, one frame image constituting the moving image, each frame image located after the frame image in order is also performed. That is, in generating a moving image in which the target subject is arranged at the specified position in the display frame 115, the user does not need to perform the operation of specifying the arrangement position for each frame image. In terms of points, the operational burden is reduced.
The moving image is an image group composed of a plurality of images, and can be rephrased as an image group composed of a plurality of consecutive images in a predetermined order.

Further, in response to the operation of the tracking start button I63, the cancel button I64 and the stop button I65 as illustrated in FIG. 48 are displayed on the framing screen 103. The stop button I65 has a function of instructing the stop of preview playback, and is a section of tracking, that is, a section of cutting out from the original image so that the target subject is arranged at a designated position in the display frame 115. It has the function of indicating the end point.
Further, in response to the operation of the tracking start button I63, the display of the section bar I66 indicating the tracking section is also started in the timeline I27.
This section bar I66 gradually extends from the tracking start frame (the frame displayed in the preview area I24 when the tracking start button I63 is operated) as shown in FIG. 49 as the preview playback progresses. The bar is displayed, and the extension of the section bar I66 is stopped when the end point of the tracking section is determined by the operation of the stop button I65 or the like. As a result, the tracking section is shown on the timeline I27.

Here, after the start of tracking according to the operation of the tracking start button I63, tracking may not be possible due to, for example, the target subject being framed out. When tracking becomes impossible in this way, tracking ends, and the frame immediately before the frame in which tracking becomes impossible is determined as the end point of the tracking section.
Also, when the end of the moving image to be tracked is reached, the tracking ends, and in this case, the final frame of the moving image is determined as the end point of the tracking section.

On the framing screen 103, when the stop button I65 is operated, the end of the moving image is reached, or the tracking is terminated due to the inability to track, the reset button I67 as shown at the bottom of FIG. 48 is the framing screen 103. Is displayed in.
When the reset button I67 is operated after the end of tracking or the cancel button I64 is operated during tracking, the framing screen 103 returns to the state before the start of tracking (for example, the state shown in FIG. 47).

Here, as described above, in the present embodiment, the CPU 71 targets the operation on the screen on the first screen as the framing screen 103 in the tracking mode that appears in response to the operation of the tracking button I62. It is recognized as an operation related to the target by the designated aim 116. On the other hand, on the second screen as the framing screen 103 before the tracking button I62 is operated, the CPU 71 recognizes the operation on the screen as an operation for designating an image range to be cut out from the image.
This makes it possible to clearly separate the operation of specifying the cutout range and the operation of specifying the tracking target or its arrangement position on the same screen.

Further, in the present embodiment, a display for changing the range of the image to be the cutout range is performed corresponding to the operation recognized on the second screen (see FIG. 44), and the operation recognized on the first screen is performed. The display for moving the designated position by the target designated aiming 116 is performed corresponding to (see FIG. 46).
As a result, when the operation is performed in the state of the second screen, the user can clearly recognize that the operation is to specify the cutout range by moving, enlarging, reducing, and rotating the image. Similarly, when the operation is performed in the state of the first screen, the target designation aiming 116 is moved, so that the user can clearly recognize that the operation is the designation operation of the tracking target and its arrangement position.

Here, at the time of preview reproduction of the tracking section described with reference to FIG. 47 or the like, image processing (see FIG. 14) is performed by the image processing unit 32 described above for the moving image to be processed. That is, the above-mentioned processing such as shaking correction (ST15), tracking (ST16), display angle of view designation (ST17), protrusion prevention (ST18), plane projection and cutting (ST19), and the like.
The CPU 71 controls to display the moving image obtained by the image processing by the image processing unit 32 in the preview area I24 as a preview moving image.

Further, in this example, such image processing by the image processing unit 32 is also executed for the frame image displayed in the pause state in the preview area I24. That is, the frame image displayed in the preview area I24 on the framing screen 103 before entering the tracking mode as shown in FIG. 42 is an image that has undergone image processing such as shake correction and protrusion prevention.

Therefore, in this example, for the frame image displayed in the preview area I24, the above-mentioned available area 62 (see FIG. 19) is calculated by the protrusion prevention process. In this example, on the framing screen 103, display control is performed in consideration of such an available area 62 at the time of receiving the designation of the cutout range.

FIG. 50 illustrates the relationship between the above-mentioned captured area 61, the available area 62, and the display frame 115 for the original image to be previewed.
In the example of this figure, the lower side of the display frame 115 coincides with the lower side of the available area 62.

Here, in the framing screen 103 shown in FIGS. 42 and 43, that is, the framing screen 103 in a state where the designation of the cutout range can be accepted, the cutout from the original image as shown as the operation Sw of FIG. 50, for example. It is assumed that an instruction operation for moving the range downward is performed. Such an instruction operation is an operation for instructing that the image portion outside the available area 62 is included in the cutout range.

When the instruction operation to include the image portion outside the available area 62 within the cutout range is performed in this way, in this example, as shown in FIG. 51, the original image is placed above the display frame 115 (of the cutout range). It is moved in the movement instruction direction) so that the image portion outside the available area 62 is displayed in the display frame 115. At this time, in this example, the image portion outside the available area 62 to be displayed in the display frame 115 is different from the image portion inside the available area 62, for example, by displaying a grayout (see the shaded area in the figure). Display in the display mode. Further, in this example, when the image portion of the available area 62 is displayed in the display frame 115, a display indicating the boundary line L62 of the available area 62 in the display frame 115 (for example, a thick line in a predetermined color such as red). Is displayed.

Further, in this example, the operation of designating the cutout range is restricted based on the available area 62.
Specifically, in this example, the image portion outside the available area 62 is displayed in the display frame 115 according to the operation Sw shown in FIG. 50 as described above, but such a display state is referred to as the operation Sw. Until the swipe operation is completed (while the contact state with the screen continues), it continues only for a temporary period, and according to the completion of the swipe operation, it is inside the display frame 115 and outside the available area 62. The position of the original image with respect to the display frame 115 is adjusted so that the image portion is not included. That is, for example, as in the positional relationship between the display frame 115 and the original image shown in FIG. 50, the lower side of the display frame 115 coincides with the lower side of the available area 62. In this way, the operation of designating the cutout range is restricted based on the available area 62.

By displaying the available area 62, such as the display representing the boundary line L62 and the grayout display, it becomes easier for the user to understand that the cutout range can be specified in the available area 62.
Further, by displaying the restriction of the cutout range designation operation based on the available area 62 as described above, it becomes easier for the user to understand the situation in which the cutout range change operation is restricted.

In this example, the display control expressing these available areas 62 and the operation restriction for specifying the cutout range based on the available area 62 are not limited to the vertical / horizontal / diagonal range designation by the swipe operation, but the cutout range by the above-mentioned reduction operation or rotation operation. It is also performed during the specified operation of.

The display representing the available area 62 is not limited to the grayout display as shown in FIG. 51, but may be a black-painted display as shown in FIG. 52.
Further, in the preview area I24 in the zoom-out display state (see FIG. 33 and the like), it is possible to display the available area 62 in an area outside the display frame 115.

Further, in the above, an example of temporarily allowing an operation exceeding the available area 62 (allowing until the swipe operation is completed) is given, but for an operation exceeding the available area 62, the movement of the original image with respect to the display frame 115 is performed. It is also possible to make the operation itself beyond the available area 62 impossible by stopping the image portion outside the available area 62 in the display frame 115 so as not to include the image portion.

[About the export screen]

Subsequently, the export screen 108 will be described.
As illustrated in FIG. 53, the export screen 108 is provided with a size selection dialog I70 for selecting the final output size of the cutout image, and the size selection dialog I70 has a dialog title I70a and a full HD button I70b. And the HD button I70c are arranged.
In this example, the export screen 108 in the initial state displayed in response to the operation of the export button I29 is overlaid on the preview screen 102. You can return to the preview screen 102 by tapping the outside of the size selection dialog I70.

In the size selection dialog I70, tapping the full HD button I70b starts rendering in full HD size, and tapping the HD button I70c starts rendering in HD size.
The full HD button I70b is grayed out on models that cannot render in full HD, and an error dialog is displayed when tapped.

FIG. 54 illustrates the export screen 108 during rendering.
On the rendering export screen 108, the progress bar I71a is displayed in the processing status area I71, and the processing wording and the processing animation are displayed in the processing notation area I71b.
Further, the export screen 108 during rendering is provided with a cancel button I72. When the cancel button I72 is operated, a confirmation dialog (not shown) is displayed, and when OK is selected on the dialog, the rendering process is canceled and a preview is performed. Return to the screen 102.

FIG. 55 illustrates the export screen 108 after the rendering is completed.
In this case, the export screen 108 is provided with a preview area I24 (including the display frame 115), a play button I25, a play time display area I26, and a timeline I27, and the rendered moving image is in the play button I25 and the timeline I27. It is possible to perform preview reproduction using the reproduction position bar I27a or the like.
Further, the export screen 108 in this case is provided with a back button I35 and a share button I73. When the back button I35 is operated, the preview screen 102 is returned. When the share button I73 is operated, the SNS selection screen (OS standard) is displayed. Through this selection screen, the user can upload the image-processed moving image to a desired SNS site.

<7. Processing related to UI>

Among the UIs related to the image processing described above, the processing related to the UI of the framing screen 103 will be described with reference to the flowcharts of FIGS. 56 to 58.
The processes described below are processes executed by the CPU 71 as the UI processing unit 36, and the CPU 71 executes these processes according to the above-mentioned application program for moving image processing.

In FIG. 56, first, in step S101, the CPU 71 waits until the framing screen 103 is in a state to be transitioned to. Specifically, in this example, it waits until the frame button B1 on the preview screen 102 is operated.
The CPU 71 executes the first frame output process in step S102 in response to the operation of the frame button B1 and the state in which the transition to the framing screen 103 should be made. That is, as described above, for the moving image being focused in the moving image list area I21, the framing screen 103 in which the image of the first frame is displayed in the preview area I24 is displayed (see FIG. 42). As can be understood from the above description, in the initial state framing screen 103 displayed in response to the operation of the frame button B1, the preview area I24 is in the zoomed-in display state, and the reduction button I30 is displayed on the framing screen 103. To.
Further, in this example, as the frame image displayed in the preview area I24, the image subjected to the above-mentioned shake correction and protrusion prevention processing (calculation of the available area 62) by the editing processing unit 32 is displayed.

Next, in step S103, the CPU 71 determines whether or not the moving image has a tracking section set. That is, it is determined whether or not the above-mentioned tracking section is a designated moving image for the moving image in focus.
If it is determined that the moving image has the tracking section set, the CPU 71 performs a process for displaying the bar of the tracking section in step S104. Specifically, the process of displaying the section bar I66 (see FIG. 48) described above on the timeline I27 of the framing screen 103 is performed. After that, the process proceeds to step S105.
On the other hand, if the moving image does not have a tracking section set, the CPU 71 passes the process of step S104 and proceeds to step S105.

In step S105, the CPU 71 determines whether or not it is in the pause state, that is, whether or not the preview reproduction in the preview area I24 is in the paused state.
If it is in the pause state, the CPU 71 shifts to the process related to the cutout range designation and tracking shown in FIG. 57 (details will be described later).

On the other hand, when it is not in the pause state, that is, in the play state, the CPU 71 proceeds to step S107 and determines whether or not there is a tracking section, that is, whether or not the video to be previewed and reproduced is a video for which the tracking section has been specified. judge.
If there is no tracking section, the CPU 71 proceeds to step S108 to execute the display start processing of the cutout moving image maintaining the current cutout range. As the display start process in step S108, a process of displaying a cut-out image cut out from the original image in the cut-out range designated by the current display frame 115 in the preview area I24 is started. As described above, in this example, the image processing unit 32 also executes the process of correcting the shaking and preventing the protrusion of the moving image to be previewed.

Further, in step S107, if there is a tracking section, the CPU 71 proceeds to step S109 and executes a display start process of the cut-out moving image on which the tracking process is applied in the tracking section. In the display start process of step S109, the tracking process (STST16) by the image processing unit 32 described above is applied in the designated tracking section so that the target subject is arranged at the designated position in the display frame 115. The process of displaying the cut-out image cut out from the original image in the preview area I24 is started.

In response to the execution of the display start processing in step S108 or step S109, the CPU 71 is in a pause state (that is, a state in which the preview playback pause operation is performed) and a playback end state (end frame) by the processing in steps S110 and S111. Wait for the arrival of any of the above (states in which playback has been completed).
If it is determined in step S111 that the reproduction is finished, the CPU 71 returns to step S105. On the other hand, if it is determined in step S110 that the pause state is determined, the CPU 71 pauses the reproduction stop process, that is, the preview reproduction of the video being reproduced in step S112, and returns to step S105.

Subsequently, the process of FIG. 57 will be described.
As described above, the process of FIG. 57 is executed when the pause state is determined in step 105.
First, the CPU 71 performs an operation of the reduction button I30 (S115), an operation of the enlargement button I31 (S116), a movement operation of the cutout range (S117), a rotation operation (S118), and an enlargement / reduction operation by the processing of steps S115 to S120. (S119), the operation of the tracking button I62 (S120) is awaited.
In this example, the swipe operation within the display frame 115 corresponds to the movement operation of the cutout range in step S117. Further, the rotation operation of step S118 comprehensively represents the rotation operation by two fingers in the display frame 115 and the operation of the rotation button I61. Further, the enlargement / reduction operation in step S118 corresponds to the pinch-out / pinch-in operation in the display frame 115 in this example.
Here, if it is determined in step S120 that the tracking button I62 is not operated, the CPU 71 returns to step S105 shown in FIG. 56.

If it is determined in step S115 that the reduction button I30 has been operated, the CPU 71 proceeds to step S121 to execute a process of setting the preview area I24 to the zoom-out display state (see FIG. 43), and then in step S122, the enlargement button I31. Is displayed, and the process returns to step S105 in FIG. Here, in this example, in the zoom-out display process of step S121, the image portion outside the display frame 115 in the original image is grayed out.

If it is determined in step S116 that the enlargement button I31 has been operated, the CPU 71 proceeds to step S123 to execute a process of zooming in on the preview area I24 (see FIG. 42), and then in step S124, the reduction button I30 is performed. Display processing is performed, and the process returns to step S105.

If it is determined in step S117 that the cutout range has been moved, the CPU 71 proceeds to step S125, executes a process of moving the original image according to the operation, and then executes a process of setting the cutout range in step S126. Return to step S105.
Further, when it is determined in step S118 that the rotation operation has been performed, the CPU 71 executes the process of moving the original image according to the operation in step S127, executes the process of setting the cutout range in the subsequent step S128, and returns to step S105.
Further, when it is determined in step S119 that the enlargement / reduction operation has been performed, the CPU 71 executes a process of moving the original image according to the operation in step S129, executes a process of setting the cutout range in the subsequent step S130, and proceeds to step S105. Return.
Here, in this example, in the process of moving the original image according to the operations of steps S125, S127, and S129, the CPU 71 needs to perform the display process for expressing the above-mentioned available area 62 and cut out based on the available area 62. A display process indicating that the range designation operation (specifically, the operation of moving the cutout range, the rotation operation, and the enlargement / reduction operation in this example) is executed is executed.

If it is determined in step S120 that the tracking button I62 has been operated, the CPU 71 executes the tracking corresponding process in step S131.

FIG. 58 is a flowchart of the tracking corresponding process in step S131.
First, the CPU 71 executes the screen content update process in step S201. That is, the process of updating the screen contents of the framing screen 103 to the screen contents in the tracking mode illustrated in FIG. 45 is executed.

In response to the execution of the update process of step S201, the CPU 71 waits for the position designation operation (S202) and the operation of the back button I35 (S203) by the processes of steps S202 and S203. The position designation operation in step S202 is an operation of designating the position of the target designated aiming 116 in this example (in this example, an operation of tapping an arbitrary position in the display frame 115 or an operation of dragging and moving the target designated aiming 116). Applies to.
Here, when the CPU 71 determines that the return button I35 has been operated in step S203, the CPU 71 performs a transition process to the preview screen 102.

When it is determined that the position designation operation has been performed in step S202, the CPU 71 performs the setting process of the position of the target subject and its arrangement position in step S204. That is, the process of setting the position of the target subject for tracking and the arrangement position of the cutout image of the target subject in the image frame is executed.

In step S205 following step S204, the CPU 71 executes the screen content update process. In the update process of step S205, the screen content of the framing screen 103 is updated to the screen content including the tracking start button I63 as illustrated in FIG. 47 above.
Then, in the subsequent step S205, the CPU 71 waits for the tracking start operation, that is, the operation of the tracking start button I63.
When the tracking start button I63 is operated and it is determined that the tracking start operation has been performed, the CPU 71 executes the screen content update process in step S207. In the update process of step S207, the screen contents of the framing screen 103 are updated to the screen contents including the cancel button I64 and the stop button I65 as exemplified in FIG. 48 above.

In step S208 following step S207, the CPU 71 performs display start processing of the cut-out image cut out according to the designated position and the designated arrangement position of the target subject. That is, the display of the cutout image cut out so that the subject at the position designated in step S202 is arranged at the position within the same designated image frame is started in the preview area I24.

Further, the CPU 71 starts the bar stretching process of the tracking section in step S209 following step S208. That is, the display process of extending the section bar I66 as illustrated in FIG. 48 in the timeline I27 is started.

In response to the execution of the stretching start processing in step S209, the CPU 71 reaches the end of the stop button I65 or the end of the moving image (S210), the tracking impossible state arrives (S211), and the cancel button I64 by the processing of steps S210 to S212. Waits for the operation (S212) of.
When it is determined in step S210 that the stop button I65 has been operated or the end of the moving image has been reached, the CPU 71 proceeds to step S213, displays the cutting operation started in step S208, and starts it in step S209. The bar extension stop process is executed, and the tracking section is set in the subsequent step S214.
Further, even when the CPU 71 determines in step S211 that the tracking is impossible (for example, the flare-out of the target subject), the CPU 71 proceeds to step S214 to set the tracking section.
Here, if it is determined in step S212 that the cancel button I64 has been operated, the CPU 71 returns to step S201. As a result, the framing screen 103 is returned to the state before the start of tracking (for example, the state shown in FIG. 47).

In step S215 following the setting process of step S214, the CPU 71 executes a process of updating to a screen including the reset button I67 (see the lower part of FIG. 48), and the operation of the reset button I67 (S216) by the process of steps S216 and S217. The operation of the back button I35 (S217) is awaited. If it is determined in step S216 that the reset button I67 has been operated, the CPU 71 returns to step S201. As a result, the framing screen 103 is returned to the state before the start of tracking (for example, the state shown in FIG. 47) in the same manner as when the cancel button I64 described above is operated.
If it is determined in step S217 that the return button I35 has been operated, the CPU 71 executes a transition process to the preview screen 102.

<8. Modification example about UI>

Here, various modifications can be considered for the details of the UI related to image processing.
For example, as in the framing screen 103'as a modification shown in FIG. 59, an enlargement / viewpoint change mode button B4, a tilt correction mode button B5, and a tracking mode button B6 are arranged in the screen, and in each mode indicated by these buttons. It is also possible to accept different types of operations.
Specifically, the enlargement / viewpoint change mode button B4 is a button for instructing the transition to the enlargement / viewpoint change mode that accepts the operation of specifying the cutout range by enlarging / reducing the original image or moving the original image, and is a tilt correction mode. The button B5 is a button for instructing the transition to the tilt correction mode that accepts the operation of designating the cutout range by rotating the original image. The tracking mode button B6 is a mode for instructing the transition to the tracking mode, similar to the tracking button I62 described above.

As shown in the transition diagram of FIG. 60, when the tracking mode button B6 is operated, a guide dialog describing the operation method for specifying the tracking target and the tracking section is displayed, and the OK displayed in the dialog is displayed. When the button is operated, it is possible to specify the position of the subject to be tracked and the arrangement position in the image frame by touch operation. In this case, tracking can be started by operating the start button B7, and the end point of tracking can be instructed by operating the stop button B8 displayed during tracking. The OK button B9 and the reset button B10 are displayed depending on the end point of the tracking, but the processing content is confirmed by the operation of the OK button B9, and the processing content is discarded by the operation of the reset button B10 for framing. The content of the screen 103'returns to the content in the initial state shown in FIG. 59.

The shape of the target designation aiming 116 for designating the target subject for tracking and designating the position of the target subject within the image frame is not limited to the shapes shown so far, and is as shown in FIG. 61, for example. A shape can also be adopted, and the specific shape is not limited.

FIG. 62 shows an example of screen transition when the enlargement / viewpoint change mode button B4 is operated on the framing screen 103'.
When the enlargement / viewpoint change mode button B4 is operated, a guide dialog describing the operations possible in the enlargement / viewpoint change mode is displayed on the screen, and when the OK button displayed in the dialog is operated, the guide dialog is displayed. By operating the display frame 115, it is possible to enlarge / reduce the original image and specify the cutout range by moving the original image. In this case as well, the screen contents are returned to the initial state by operating the OK button B9 and the reset button B10.

FIG. 63 shows an example of screen transition when the tilt correction mode button B5 is operated on the framing screen 103'.
When the tilt correction mode button B5 is operated, a guide dialog describing the operations possible in the tilt correction mode is displayed on the screen, and when the OK button displayed in the dialog is operated, as shown in the figure. The left rotation button B11 and the right rotation button B12 are displayed on the screen, and the rotation of the original image with respect to the display frame 115 can be instructed by operating these buttons. In this case as well, the screen contents are returned to the initial state by operating the OK button B9 and the reset button B10.

Further, in the description so far, the aspect ratio is specified on the aspect ratio setting screen 106, but for example, the aspect ratio setting button I28'as shown in FIG. 64 is arranged and the aspect ratio setting button I28 is arranged. It is also possible to display an aspect ratio selection dialog as shown in FIG. 65 on the screen in response to the operation of', and accept the aspect ratio designation in the selection dialog.
Here, the aspect ratio setting button I28'shown in FIG. 64 is arranged not only on the preview screen 102 but also on the screens related to various image processing such as the framing screen 103, and can be called on various screens. can do. At this time, when the aspect ratio is specified on the dialog, the dialog can be changed from the display state to the non-display state and returned to the screen state immediately before the dialog is displayed. When the aspect ratio is specified, the display frame 115 is displayed with the specified aspect ratio.
The display form of the aspect ratio setting button I28'is not limited to the display form exemplified above, and is a form showing the aspect ratio currently set as in the aspect ratio setting button I28'' shown in FIG. 66, for example. The image can also be arranged in the screen (in the example in the figure, it is assumed that the image is arranged in the vicinity of the preview area I24). In response to the operation of the aspect ratio setting button I28'', the dialog of FIG. 65 is displayed on various screens and the aspect ratio designation is accepted.

<9. Summary about UI>

As described above, the information processing apparatus (for example, the mobile terminal 2) of the embodiment specifies the arrangement position in the tracking target display area on the first screen for displaying a predetermined image in the display area (display frame 115). It is provided with a reception unit (36a) for receiving.
For example, the user can specify the position (position in the display area) where the tracking target subject is placed while the image content can be recognized on the display screen.
This makes it possible for the user to specify an arbitrary position in the display area of the target subject by a method such as that shown in FIGS. 13 and 46.
Therefore, the degree of freedom of the arrangement position in the image frame of the subject to be tracked is improved, and the degree of freedom of drawing can be improved.

Further, in the information processing apparatus of the embodiment, the predetermined image is an image cut out from the original image.
That is, the image displayed in the display area is a cutout image from the original image.
As a result, when image processing is performed to cut out an image from the original image so that the target subject of tracking is included in the image frame, the user can specify the placement position of the target subject in the image frame at an arbitrary position. It will be possible.
Therefore, the degree of freedom of the arrangement position in the image frame of the subject to be tracked is improved, and the degree of freedom of drawing can be improved.

Further, in the information processing apparatus of the embodiment, the reception unit accepts the designation of the tracking target.
That is, from among the subjects in the image, the designation of the subject as the tracking target is accepted.
As a result, the user can specify an arbitrary subject in the image as the target subject for tracking, and the degree of freedom in drawing can be improved in this respect.

Further, the information processing apparatus of the embodiment includes a display control unit (36b) that controls the display of a predetermined image so that the tracking target is arranged at a position in the display area based on the designated arrangement position. ing.
For example, an image that reflects the placement position of the tracking target subject specified by the user is displayed.
As a result, the user can confirm whether or not the designation of the placement position of the tracking target subject is appropriate, and can provide a good working environment for the work of designating the placement position of the tracking target subject.

Further, in the information processing apparatus of the embodiment, the reception unit accepts the designation of the image range to be cut out from the original image.
For example, on the display screen, the user can specify the range included in the image frame of the cutout image among the original images.
Specifically, for example, as shown in FIGS. 12A to 12B and 44, the original image (the entire frame image) appearing in the display range is moved, and the operation of enlarging / reducing the original image as described in FIG. 12C. , That is, the cutout range CL'can be specified. As a result, after imaging, it is possible to specify an arbitrary cutout range from the original image and edit the image to adopt the desired image range.

Furthermore, in the information processing apparatus of the embodiment, the display control unit displays an operation image (target designation aiming 116) used for the operation of designating the arrangement position of the tracking target on the first screen.
For example, an operation image that serves as a aim for designating an arrangement position is superimposed and displayed on the display area so that the user can specify a position in the image frame by the operation image.
The user can specify the position of the tracking target in the clipped image by moving the target designation aiming 116. In particular, the target designation aiming 116 enables intuitive operation of specifying the placement position, and can improve operability.

Further, in the information processing apparatus of the embodiment, the reception unit accepts the designation of the tracking target and the designation of the arrangement position of the tracking target based on the operation on the operation image.
For example, by an operation of designating a certain place with an operation image on the display area, the subject of that part is set as the tracking target, and the position of the operation image is accepted as the placement position of the tracking target.
By doing so, the user can specify the tracking target and the placement position at the same time by moving the target designation aim 116, and the operability can be improved.

Further, in the information processing apparatus of the embodiment, the predetermined image is an image cut out from the original image, and the reception unit specifies the range to be cut out from the original image and operates the tracking target by the operation image as the operation for the screen. We are accepting.
For example, the range cut out from the original image is displayed on the display screen, and the operation image is displayed on the image. On the screen, the change of the cutout range and the movement of the operation image are accepted.
By doing so, the user can seamlessly specify the cutout range, specify the tracking target by moving the target specification aiming 116, and specify the placement position.
Specifically, for example, by moving the cutout range, the subject to be the tracking target is set to a desired arrangement position within the cutout range. Then, the subject is designated by the target designation aiming 116 (see FIGS. 12, 13, 44, and 46). In this way, the operation of specifying the cutting range and the target placement position while recognizing the target can be made intuitive and easy by the user.

Furthermore, in the information processing apparatus of the embodiment, the reception unit recognizes the operation on the screen as a state of recognizing the operation on the screen as a designated operation of a range cut out from the original image and an operation related to the tracking target by the operation image. Switching between states.
For example, the recognition mode of the user operation performed on the screen is switched by the operation prepared by the icon on the display screen.
For example, in response to the operation of the tracking button I62, the operation input is recognized by separating the operation for designating the cutout range and the operation for moving the target designated aiming 116. By doing so, the user operation on the same screen can be accurately recognized.

Further, in the information processing apparatus of the embodiment, the reception unit recognizes the operation on the screen as the operation for designating the image range to be cut out from the original image on the second screen, and on the first screen, on the screen. The operation is recognized as an operation related to the tracking target by the operation image.
The recognition of the user operation performed on the screen is switched depending on whether the display screen is in the state of the first screen or the state of the second screen.
For example, in the initial state when transitioning to the framing screen 103, the image of the cutout range and the target designated aim 116 are displayed, but the target designated aim 116 is a fixed second screen. In this case, it is assumed that the image is moved and the designation of the cutout range is changed by an operation such as dragging by the user.
When the tracking button I62 is operated from the state of the second screen, the first screen is displayed. The first screen referred to here is a screen in which the target designated aiming 116 can be moved. The operation on the screen of the user in this case is recognized as a movement operation of the target designated aiming 116.
By doing so, it is possible to clearly separate the operation of specifying the cutout range and the operation of specifying the tracking target or its arrangement position on the same screen, and it is less likely that the user's operation is erroneously recognized. Therefore, it is possible to provide the user with an operation environment with less stress.

Further, in the information processing apparatus of the embodiment, the display control unit performs a display for changing the range of the image to be the cutout range in response to the operation recognized by the reception unit on the second screen. On the first screen, the display for moving the designated position by the operation image is performed in response to the operation recognized by the reception unit.
When the display screen is the second screen, for example, the image of the cutout range is displayed in the entire screen or in the display area, but it is changed according to the operation.
When it is the first screen, the designated position by the operation image is moved according to the operation.
When the operation is performed in the state of the second screen, the user can clearly recognize that the operation is to specify the cutout range by moving, enlarging, reducing, and rotating the image. Similarly, when the operation is performed in the state of the first screen, the target designation aiming 116 is moved, so that the user can clearly recognize that the operation is the designation operation of the tracking target and its arrangement position. This makes it possible to provide an operation environment that is easy for the user to understand.

Furthermore, in the information processing apparatus of the embodiment, on the second screen, the image cut out from the original image changes in the display area according to the operation, and on the first screen, the image is displayed according to the operation. The placement position of the operation image changes in the area.
That is, on the second screen, image changes such as movement, enlargement, reduction, and rotation of the original image occur in the display area according to the operation, and on the first screen, for example, in the display area according to the operation. The original image does not change, but the position of the operation image changes.
As a result, the user can clearly recognize that the operation is for specifying the cutout range when the operation is performed in the state of the second screen, and when the operation is performed in the state of the first screen, the tracking target and It can be clearly recognized that the operation is to specify the placement position.
Therefore, it is possible to provide an operation environment that is easy for the user to understand.

Further, in the information processing apparatus of the embodiment, the original image is regarded as one image of an image group composed of a plurality of images.
This makes it possible for the user to specify the arrangement position of the target subject in the image frame of the cutout image at an arbitrary position for each image constituting the image group.
Therefore, the degree of freedom of the arrangement position in the image frame of the subject to be tracked is improved, and the degree of freedom of drawing can be improved.

Further, in the information processing apparatus of the embodiment, among a plurality of consecutive images in a predetermined order, at least one image after the order of the images for which the designation of the placement position is accepted is placed at the designated placement position. The image is cut out so that the tracking target is placed at a position in the display area based on the image.
As a result, the placement position of the tracking target is specified for one image out of a plurality of consecutive images in a predetermined order, and the frame images located after that in order are also tracked to the position based on the specified position. Image cropping processing to position the target will be performed.
Therefore, it is possible to reduce the burden on the user operation when cutting out an image in which the tracking target is positioned at a position based on a designated position for a plurality of consecutive images in a predetermined order.

Furthermore, in the information processing apparatus of the embodiment, the image group is composed of a series of frame images constituting a moving image.
This makes it possible for the user to specify the arrangement position of the target subject in the image frame of the cut-out image at an arbitrary position when the image is cut out for each frame image constituting the moving image.
Therefore, the degree of freedom of the arrangement position in the image frame of the subject to be tracked is improved, and the degree of freedom of drawing can be improved.

Further, in the information processing apparatus of the embodiment, the reception unit is a section that cuts out from the original image so that the tracking target is arranged at a position in the display area based on the designated arrangement position. The designation is accepted.
This allows the user to arbitrarily specify the tracking section as well.
Therefore, the degree of freedom in designating the tracking section can be improved, and the degree of freedom in drawing can also be improved in this respect as well.

Further, in the information processing apparatus of the embodiment, the predetermined image is an image cut out from the original image, and the display control unit sets the image within the cutout range from the original image and the image outside the cutout range to each other. Control is performed to display in different display modes.
For example, the entire original image is displayed on the display screen, but the display mode is such that the inside of the cutout range and the outside of the cutout range can be distinguished.
By doing so, the user can clearly recognize what kind of range is set as the cutout range in the original image, and can provide an image useful for determining the quality of the cutout range and changing the cutout range.

Further, in the information processing apparatus of the embodiment, the shaking correction unit performs the pasting process on the virtual celestial sphere for each frame of the image data constituting the moving image, and performs the shaking correction using the posture information corresponding to the frame. (33) and the cut-out range setting unit (34) that performs a protrusion prevention process that corrects the cut-out range according to the designation of the cut-out range received by the reception unit so that it does not protrude from the area where the shaking correction is effectively maintained. The display control unit controls the display of the cut-out image reflecting the protrusion prevention processing performed by the cut-out range setting unit.
For example, the shake caused by camera shake in the image captured by the image pickup device is corrected by coordinate transformation on the virtual celestial sphere, and the cutout range of the image projected in a plane with this shake correction is set based on the user's operation. do. In this case, the protrusion prevention process is performed so that the cutout range does not exceed the range in which the shake correction is effectively maintained. The display control unit performs display control so that the user can recognize the cutout range of the range limited by this protrusion prevention process.
Here, in the display of the framing screen 103, the limit of the cutout range designation can be clearly communicated to the user by the range of the protrusion prevention process. For example, as shown in FIG. 51 and the like, the user can specify an appropriate cutout range by displaying a display indicating that the cutout range cannot be specified so as to exceed the available area 62 by the protrusion prevention process.

Further, in the information processing apparatus of the embodiment, the display control unit controls so that the display is performed based on the permissible movable range that allows the fluctuation of the cropping range of the image set in the protrusion prevention process.
The protrusion prevention process is a process of keeping the cutout range set based on the operation information within the range in which the shake correction is effective, and at that time, the allowable movable range is set as the range in which the shake correction is effectively maintained. That is, it is a range in which the cutting range may be moved as long as it is within the allowable movable range. On the display, the display is made so that the user can recognize this allowable movable range.
By displaying the available area 62 on the display of the framing screen 103, it can be seen that the cutout range can be specified in the available area 62. This makes it easier for the user to understand the range in which the cutout range can be specified.

Furthermore, in the information processing apparatus of the embodiment, the display control unit controls the display so as to clearly indicate that the operation for designating the cutout range exceeding the allowable movable range is restricted.
For example, when the allowable movable range is exceeded, the display is executed so that the cutout range cannot be changed or the cutout range returns to the allowable movable range.
When the cutout range is specified on the display of the framing screen 103, if the available area 62 is exceeded, the user can display that the operation becomes impossible or the user returns to the available area 62. , Understand the situation where the operation to change the cutout range is restricted. In addition, since the available area 62 is also displayed, it becomes easy to understand that when the designation is not possible, it is not a malfunction of the operation but is restricted.

It should be noted that the effects described in the present specification are merely examples and are not limited, and other effects may be obtained.

<10. Program>

The program of the embodiment is a program for causing, for example, a CPU, a DSP, or a device including these to execute the processes exemplified in FIGS. 56 to 58.
That is, the program of the embodiment is a program that causes the information processing apparatus to execute a process of accepting the designation of the arrangement position in the display area to be tracked on the first screen for displaying a predetermined image in the display area. With such a program, the information processing device of the above-described embodiment can be realized in a device such as a mobile terminal 2, a personal computer 3, or an image pickup device 1.

Such a program can be recorded in advance in an HDD as a recording medium built in a device such as a computer device, a ROM in a microcomputer having a CPU, or the like.
Alternatively, flexible discs, CD-ROMs (Compact Disc Read Only Memory), MO (Magnet optical) discs, DVDs (Digital Versatile Discs), Blu-ray discs (Blu-ray Disc (registered trademark)), magnetic discs, semiconductor memories, It can be temporarily or permanently stored (recorded) on a removable recording medium such as a memory card. Such removable recording media can be provided as so-called package software.
In addition to installing such a program from a removable recording medium on a personal computer or the like, it can also be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.

Further, such a program is suitable for a wide range of provision of the information processing apparatus of the embodiment. For example, by downloading a program to a personal computer, a portable information processing device, a mobile phone, a game device, a video device, a PDA (Personal Digital Assistant), etc., the personal computer, etc. can function as an information processing device of the present technology. Can be done.

<11. This technology>

The present technology can also adopt the following configurations.
(1)
An information processing device including a reception unit that accepts designation of an arrangement position in the display area to be tracked on a first screen for displaying a predetermined image in the display area.
(2)
The information processing apparatus according to (1) above, wherein the predetermined image is an image cut out from the original image.
(3)
The information processing device according to (1) or (2) above, wherein the reception unit receives designation of the tracking target.
(4)
Any of the above (1) to (3) provided with a display control unit that controls the display of the predetermined image so that the tracking target is arranged at a position in the display area based on the designated arrangement position. The information processing device described in.
(5)
The information processing apparatus according to any one of (2) to (4) above, wherein the reception unit receives a designation of a range to be cut out from the original image.
(6)
The information processing device according to (4) or (5) above, wherein the display control unit displays an operation image used for the operation of designating the arrangement position of the tracking target on the first screen.
(7)
The information processing device according to (6), wherein the reception unit receives the designation of the tracking target and the designation of the arrangement position of the tracking target based on the operation on the operation image.
(8)
The predetermined image is an image cut out from the original image, and is
The information processing device according to (6) or (7) above, wherein the reception unit receives a designation of a range to be cut out from the original image and an operation related to the tracking target by the operation image as an operation on the screen.
(9)
The information according to (8) above, wherein the reception unit switches between a state of recognizing an operation on the screen as a designated operation of a range cut out from the original image and a state of recognizing the operation on the tracking target by the operation image. Processing device.
(10)
On the second screen, the reception unit recognizes the operation on the screen as an operation for designating an image range cut out from the original image, and on the first screen, the operation on the screen is tracked by the operation image. The information processing apparatus according to (8) or (9) above, which is recognized as an operation related to an object.
(11)
The display control unit
On the second screen, a display is displayed to change the range of the image to be the cutout range in response to the operation recognized by the reception unit.
The information processing apparatus according to (10), wherein the information processing apparatus according to (10) displays that the designated position is moved by the operation image in response to the operation recognized by the reception unit on the first screen.
(12)
On the second screen, the original image changes in the display area according to the operation.
The information processing apparatus according to (10) or (11), wherein on the first screen, the arrangement position of the operation image changes in the display area according to the operation.
(13)
The information processing apparatus according to any one of (2) to (12) above, wherein the original image is one image of an image group composed of a plurality of images.
(14)
Among the plurality of images consecutive in a predetermined order, at least one image after the order of the images for which the designation of the arrangement position is accepted is placed at a position in the display area based on the designated arrangement position. The information processing apparatus according to (13) above, which cuts out an image so that a tracking target is arranged.
(15)
The information processing apparatus according to (13) or (14), wherein the image group is composed of a series of frame images constituting a moving image.
(16)
The reception unit receives designation of a tracking section which is a section for cutting out from the original image so that the tracking target is arranged at a position in the display area based on the designated arrangement position (15). The information processing device described in.
(17)
The predetermined image is an image cut out from the original image, and is
2. Information processing device.
(18)
A shake correction unit that performs a sticking process to the virtual celestial sphere for each frame of the image data constituting the moving image and corrects the shake using the posture information corresponding to the frame.
A cutout range setting unit that performs a protrusion prevention process that corrects the cutout range according to the designation of the cutout range received by the reception unit so that it does not protrude from the area where the shaking correction is effectively maintained.
Equipped with
The information processing device according to any one of (4) to (16) above, wherein the display control unit controls the display of a cut-out image reflecting the protrusion prevention process performed by the cut-out range setting unit.
(19)
The information processing device according to (18) above, wherein the display control unit controls display based on an allowable movable range that allows fluctuations in the cropping range of an image, which is set in the protrusion prevention process.
(20)
The information processing apparatus according to (19), wherein the display control unit controls the display so as to clearly indicate that the operation for designating a cutting range exceeding the allowable movable range is restricted.

1 image pickup device, 2 mobile terminal, 3 personal computer, 4 server, 5 storage medium, 11 lens system, 12 image pickup element section, 13 camera signal processing section, 14 recording section, 15 display section, 16 output section, 17 operation section, 18 Camera control unit, 19 Memory unit, 22 Driver unit, 23 Sensor unit, 31 Preprocessing unit, 32 Image processing unit, 33 Shake correction unit, 34 Effective cutout area setting unit, 35 Effective cutout area image generation processing unit, 36 UI Processing unit, 36a reception unit, 36b display control unit, 61 captured area, 62 available area, 70 information processing device, 71 CPU, 101 import screen, 102 preview screen, 103 framing screen, 104 trimming screen, 105 speed screen, 106 Aspect ratio setting screen, 107 setting screen, 108 export screen, MF image file, PD, iPD image data, oPD output image data, HP image frame correction parameter, TM timing information, CP camera parameter, QD quaternion, TP, TPp screen Position information, DR1, DR2 operation information, CL, CL'cutout range, MT celestial sphere model, I24 preview area

Claims (22)

An information processing device including a reception unit that accepts designation of an arrangement position in the display area to be tracked on a first screen for displaying a predetermined image in the display area. The information processing apparatus according to claim 1, wherein the predetermined image is an image cut out from the original image. The information processing device according to claim 1, wherein the reception unit receives the designation of the tracking target. The information processing apparatus according to claim 1, further comprising a display control unit that controls the display of the predetermined image so that the tracking target is arranged at a position in the display area based on the designated arrangement position. The information processing device according to claim 2, wherein the reception unit receives a designation of a range to be cut out from the original image. The information processing device according to claim 4, wherein the display control unit displays an operation image used for the operation of designating the arrangement position of the tracking target on the first screen. The information processing device according to claim 6, wherein the reception unit receives the designation of the tracking target and the designation of the arrangement position of the tracking target based on the operation on the operation image. The predetermined image is an image cut out from the original image, and is
The information processing device according to claim 6, wherein the reception unit receives an operation related to the tracking target by designating a range to be cut out from the original image and the operation image as an operation on the screen. The information processing according to claim 8, wherein the reception unit switches between a state of recognizing an operation on the screen as a designated operation of a range cut out from the original image and a state of recognizing the operation on the tracking target by the operation image. Device. On the second screen, the reception unit recognizes the operation on the screen as an operation for designating an image range cut out from the original image, and on the first screen, the operation on the screen is tracked by the operation image. The information processing apparatus according to claim 8, which is recognized as an operation related to an object. The display control unit
On the second screen, a display is displayed to change the range of the image to be the cutout range in response to the operation recognized by the reception unit.
The information processing apparatus according to claim 10, wherein a display is performed in which a designated position is moved by the operation image in response to an operation recognized by the reception unit on the first screen. On the second screen, the original image changes in the display area according to the operation.
The information processing apparatus according to claim 10, wherein in the first screen, the arrangement position of the operation image changes in the display area according to the operation. The information processing apparatus according to claim 2, wherein the original image is one image of an image group composed of a plurality of images. Among the plurality of images consecutive in a predetermined order, at least one image after the order of the images for which the designation of the placement position is accepted is placed at a position in the display area based on the designated placement position. The information processing apparatus according to claim 13, wherein an image is cut out so that a tracking target is arranged. The information processing apparatus according to claim 13, wherein the image group is composed of a series of frame images constituting a moving image. 15. The information processing device described. The predetermined image is an image cut out from the original image, and is
The information processing device according to claim 4, wherein the display control unit controls to display an image within the cutout range from the original image and an image outside the cutout range in different display modes. A shake correction unit that performs a sticking process to the virtual celestial sphere for each frame of the image data constituting the moving image and corrects the shake using the posture information corresponding to the frame.
A cutout range setting unit that performs a protrusion prevention process that corrects the cutout range according to the designation of the cutout range received by the reception unit so that it does not protrude from the area where the shaking correction is effectively maintained.
Equipped with
The information processing device according to claim 4, wherein the display control unit controls the display of a cut-out image reflecting the protrusion prevention process performed by the cut-out range setting unit. The information processing device according to claim 18, wherein the display control unit controls display based on an allowable movable range that allows fluctuations in the cropping range of an image, which is set in the protrusion prevention process. The information processing apparatus according to claim 19, wherein the display control unit controls the display so as to clearly indicate that the operation for designating a cutting range exceeding the allowable movable range is restricted. An information processing method in which an information processing apparatus performs a process of accepting a designation of an arrangement position in the display area to be tracked on a first screen for displaying a predetermined image in a display area. A program that causes an information processing apparatus to execute a process of accepting a designation of an arrangement position in the display area to be tracked on a first screen for displaying a predetermined image in the display area.

Images