JP5948927B2 - Image processing apparatus, image processing method, program, and imaging apparatus - Google Patents

Description

  This technique relates to an image processing device, an image processing method, a program, and an imaging device. Specifically, it is possible to reproduce an image by reproducing the sense of presence at the time of imaging.

  Conventionally, in an electronic still camera in which the imaging range of one time is limited by the angle of view of the lens, it is not possible to take a wide range of images exceeding the angle of view of the lens. For this reason, imaging is performed while moving the imaging direction, and a panoramic overall image (hereinafter referred to as “panorama composite image”) is generated from a plurality of captured images. In addition, by recording the imaging direction and imaging time information from the start of imaging to the end of imaging in association with the panorama composite image, when scrolling playback of a plurality of panorama composite images, the image that the viewer wants to view can be efficiently viewed in a short time. It is possible to browse well (see Patent Document 1).

JP 2009-268037 A

  By the way, the panoramic composite image is generated by combining a plurality of captured images obtained by capturing images while moving in the capturing direction, and is a long image in the moving direction. For this reason, when the display size of the image display device is, for example, the size of one captured image, the image is displayed when the entire panorama composite image is displayed by matching the longitudinal direction of the panorama composite image with the display size of the image display device. Will become smaller. For this reason, scroll playback is performed when displaying a panoramic composite image. However, the scroll speed in the scroll reproduction is fixed at a constant speed, and the scroll reproduction of the panorama composite image is always the same operation regardless of the operation at the time of capturing the panorama composite image.

  Accordingly, an object of the present technology is to provide an image processing apparatus, an image processing method, a program, and an imaging apparatus that can reproduce an image while reproducing the sense of reality at the time of imaging.

The first aspect of this technique performs scroll reproduction of a panoramic composite image having a wider angle of view than the captured image generated by combining a plurality of captured images generated while moving the imaging direction. It includes a scrolling reproducing unit, and a scroll speed control section for controlling the scroll speed in the scrolling reproduced in accordance with the movement of the imaging direction that is the moving, the scroll speed control section, the imaging direction of movement due to the small In the image processing apparatus, the scroll speed is decreased, and the scroll speed is increased as the movement increases .

  In this technique, when a panoramic composite image having a wider angle of view than a captured image obtained by synthesizing a plurality of captured images generated while moving the imaging direction is scroll-reproduced, according to the movement of the moving imaging direction. Scroll speed is controlled. The panorama composite image is associated with, for example, attached information including motion information indicating movement in the imaging direction detected between the captured images. Based on this motion information, the scrolling speed increases as the motion in the imaging direction decreases. The scroll speed is increased as the movement is increased. The attached information includes information indicating the panorama composite image size and the frame rate when multiple captured images are generated. The scroll speed is controlled based on this information and the display size and frame rate in scroll playback. Then, scroll reproduction is performed by reproducing the movement in the imaging direction when generating a plurality of captured images. When there is no attached information associated with the panorama composite image to be scrolled, the scroll speed is set to a constant speed. Further, when an audio signal is associated with the panorama composite image to be scrolled and played, the playback speed of the audio signal is controlled according to the scrolling speed. When the signal level of the audio signal related to the panorama composite image to be scrolled is higher than a preset signal level, the scroll speed is changed from a constant speed to a speed corresponding to the movement in the imaging direction. The playback speed of the audio signal may be controlled according to the speed.

According to a second aspect of the present technology, scroll reproduction of a panoramic composite image having a wider angle of view than the captured image generated by combining a plurality of captured images generated while moving the imaging direction is performed. The image processing method includes a step and a step of slowing down the scroll speed as the movement in the moved imaging direction decreases and increasing the scroll speed as the movement increases .

A third aspect of this technique is a program for causing a computer to execute scroll reproduction of a panoramic composite image, and the imaging generated by combining a plurality of captured images generated while moving the imaging direction. The procedure for scroll playback of a panoramic composite image having a wider angle of view than the image, and the scroll speed is decreased as the movement in the moving imaging direction becomes smaller, and the scroll speed is increased as the movement becomes larger. A program for causing the computer to execute a speed- up procedure.

  Note that the program of the present technology is, for example, a storage medium or a communication medium provided in a computer-readable format to a general-purpose computer that can execute various program codes, such as an optical disk, a magnetic disk, or a semiconductor memory. It is a program that can be provided by a medium or a communication medium such as a network. By providing such a program in a computer-readable format, processing corresponding to the program is realized on the computer.

The fourth aspect of this technology is an imaging unit that generates a captured image and a captured image that is generated by combining a plurality of captured images generated by the imaging unit while moving the imaging direction. Attached information including a panorama compositing processing unit that generates a panoramic composite image having a wide angle of view, a motion detection unit that detects motion in the moving imaging direction, and motion information that indicates motion detected by the motion detection unit Is generated and related to the panorama composite image, a scroll playback unit that performs scroll playback of the panorama composite image, and a scroll speed in the scroll playback is controlled based on the motion detected by the motion detection unit and a scroll speed control section for, the scroll speed control section is to slow down the scrolling speed with the imaging direction of the movement becomes smaller, the movement In the imaging device to increase the scrolling speed with the increase.

  According to this technique, when scroll reproduction of a panoramic composite image having a wider angle of view than a captured image obtained by combining a plurality of captured images generated while moving the imaging direction is performed, the movement in the captured imaging direction The scroll speed is controlled according to the above. For this reason, the scroll speed is slow when the moving speed is slow when generating a plurality of captured images while moving the image capturing direction, and the scroll speed is increased when the moving speed is fast to reproduce the sense of presence at the time of imaging. Scroll playback of the panorama composite image can be performed.

It is the figure which illustrated schematic structure of the imaging device. It is a flowchart which shows the panoramic composite image generation operation | movement of an imaging device. It is a figure for demonstrating a panorama synthetic | combination process. It is a figure for demonstrating the movement operation | movement when moving an imaging direction. It is the figure which illustrated the panorama synthetic image attachment information. It is the figure which illustrated schematic structure of the image processing apparatus. It is a flowchart which shows 1st image reproduction operation | movement. It is a figure for demonstrating fixed speed scroll reproduction | regeneration. It is a figure for demonstrating motion reproduction scroll reproduction | regeneration. It is a flowchart which shows 2nd image reproduction operation | movement. It is a figure which shows other scroll reproduction | regeneration.

Hereinafter, embodiments for carrying out the present technology will be described. The description will be given in the following order.
1. Configuration of imaging apparatus 2. Panorama composite image generation operation 3. Configuration of image processing apparatus 4. First image reproduction operation Second image playback operation

<1. Configuration of Imaging Device>
FIG. 1 illustrates a schematic configuration of an imaging apparatus according to the present technology. The imaging device 10 includes an imaging lens unit 11, an imaging unit 12, a camera signal processing unit 13, a measurement unit 14, an image memory unit 21, a panorama synthesis processing unit 22, an audio acquisition unit 23, a codec unit 24, a recording / playback unit 25, and a communication. Unit 26, display unit 27, audio output unit 28, user interface unit 31, and control unit 35. Each unit is connected via a bus 41.

  The imaging lens unit 11 functions as an imaging optical system for guiding subject light to the imaging unit 12. The imaging lens unit 11 is a focus lens for adjusting the focal point of an optical image formed on the imaging surface of an imaging element provided in the imaging unit 12, and a zoom lens for performing zooming of the optical image. Etc.

  The imaging unit 12 is configured using an imaging element such as a complementary metal-oxide semiconductor (CMOS) or a charge coupled device (CCD). The imaging element generates an imaging signal corresponding to the optical image formed on the light receiving surface by the imaging lens unit 11. In addition, the imaging unit 12 generates image data by performing correlated double sampling (CDS), analog amplification, A / D conversion, and the like on the imaging signal generated by the imaging device. The imaging unit 12 outputs the generated image data to the camera signal processing unit 13 or the image memory unit 21.

  The camera signal processing unit 13 performs camera signal processing such as white balance adjustment processing, gamma correction processing, and color adjustment processing on the image data supplied from the imaging unit 12 or the image data stored in the image memory unit 21. .

  The measurement unit 14 measures an imaging time and the like. The measurement unit 14 may measure a movement in the imaging direction to be moved, for example, a movement direction or a movement speed when the imaging direction is moved, using a gyro sensor or the like. The measurement unit 14 outputs the measurement result to the control unit 35.

  The image memory unit 21 temporarily stores image data output from the imaging unit 12 or the camera signal processing unit 13 and image data obtained by performing decoding processing by a codec unit 24 described later.

  The panorama synthesis processing unit 22 uses the image data of a plurality of captured images output from the imaging unit 12 and the camera signal processing unit 13 and the image data of the plurality of captured images read from the image memory unit 21. Process. In the synthesizing process, image data of a panoramic composite image having a wider angle of view than the captured image is generated by cutting out and connecting the captured images. The cutout of the captured image is performed by adjusting the cutout width according to the moving speed when the imaging direction is moved, for example. In the adjustment of the cutout width according to the moving speed, for example, the cutout width is narrowed when the moving speed is fast, and the cutout width is widened when the moving speed is slow. Further, in the synthesis process, image data of a panorama composite image is generated so that no missing or overlapping images occur at the joints of the images. The panorama composition processing unit 22 outputs the generated image data to the image memory unit 21 or the codec unit 24. The panorama composition processing unit 22 performs composition processing so that no missing or overlapping images occur at image joints. When detecting motion between captured images, the panorama composition processing unit 22 detects detected motion, that is, motion in the image capturing direction to be moved. Output to the control unit 35. Note that the panorama synthesis processing unit 22 may generate a panorama composite image based on the movement in the imaging direction measured by the measurement unit 14 so as not to cause missing or overlapping images. That is, the motion detection unit in the claims corresponds to the measurement unit 14 and the panorama synthesis processing unit 22.

  The voice acquisition unit 23 is configured using a microphone, an A / D converter, or the like. The sound acquisition unit 23 generates sound data indicating the sound collected by the microphone and outputs the sound data to the codec unit 24 or the recording / playback unit 25.

  The codec unit 24 performs data compression processing on the image data output from the imaging unit 12, the camera signal processing unit 13, or the panorama synthesis processing unit 22, and outputs the obtained encoded data to the recording / playback unit 25 or the communication unit 26. To do. The codec unit 24 performs data decompression processing on the encoded data read from the recording medium and the encoded data supplied via the communication unit 26, and the obtained image data is displayed in the image memory unit 21 or the display unit. To 27 etc. In data compression / decompression, for example, a technique such as JPEG (Joint Photographic Coding Experts Group) is applied. In addition, the codec unit 24 performs a data compression process on the audio data supplied from the audio acquisition unit 23 and outputs the obtained encoded data to the recording / playback unit 25 and the communication unit 26. Further, the codec unit 24 performs data decompression processing on the audio encoded data read from the recording medium and the audio encoded data supplied via the communication unit 26, and sends the obtained audio data to the audio output unit 28. Output.

  The recording / reproducing unit 25 records image data read from the image memory unit 21, audio data supplied from the audio acquisition unit 23, encoded data obtained by performing data compression processing in the codec unit 24, and the like. To record. Further, the recording / reproducing unit 25 reads out the image data recorded on the recording medium and outputs it to the image memory unit 21. Further, the recording / reproducing unit 25 reads out the audio data recorded on the recording medium and outputs it to the audio output unit 28. The recording / reproducing unit 25 reads the encoded data recorded on the recording medium and outputs the encoded data to the codec unit 24.

  The communication unit 26 is an interface for performing communication between the imaging device 10 and an external device (not shown). For example, the imaging device 10 transmits image data, audio data, encoded data, various information, and the like to the external device via the communication unit 26. In addition, the imaging apparatus 10 installs a program or the like supplied from an external device via the communication unit 26.

  The display unit 27 is configured using a display element such as a liquid crystal display element or an organic EL. The display unit 27 drives the display element based on the image data output from the image capturing unit 12 or the camera signal processing unit 13 or the image data recorded in the image memory unit 21, The image to be recorded on the recording medium and the recorded image are displayed. In addition, the display unit 27 displays a menu display for selecting a function installed in the imaging apparatus 10, a setting state of the imaging apparatus 10, and the like.

  The audio output unit 28 performs audio output based on the audio data supplied from the audio acquisition unit 23 or the codec unit 24.

  The user interface unit 31 includes operation switches and operation buttons provided in the imaging apparatus 10, a touch panel provided on the display screen of the display unit 27, and the like, and generates and controls an operation signal corresponding to an operation by the user. To the unit 35.

  The control unit 35 is configured using a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. The control unit 35 reads out a program stored in the ROM and executes it by the CPU, and controls the operation of each unit so that the operation of the imaging apparatus 10 becomes an operation corresponding to the user operation based on the operation signal. For example, the control unit 35 generates a captured image and a panorama composite image. In addition, the control unit 35 performs processing for generating panorama composite image attached information including motion information indicating movement in the imaging direction detected by the measurement unit 14 or the panorama synthesis processing unit 22 and relating the panorama composite image. Further, when the display unit 27 performs scroll reproduction of the generated panorama composite image, the control unit 35 outputs a control signal to the image memory unit 21 and sequentially moves the image data reading area in the panorama composite image, Scroll playback of the panorama composite image. Further, the control unit 65 controls the scroll speed in the scroll reproduction according to the movement in the imaging direction to be moved, using the panorama composite image attached information. That is, the scroll reproduction unit in the claims corresponds to the image memory unit 21 and the control unit 35, and the scroll speed control unit and the information generation unit correspond to the control unit 35.

  The control unit 35 may install and execute the program supplied via the communication unit 26 as described above, instead of the program stored in advance. Furthermore, a program provided by various recording media such as a memory card, an optical disk, and a magnetic disk may be installed and executed.

<2. Panorama composite image generation operation>
Next, a panoramic composite image generation operation will be described. FIG. 2 is a flowchart illustrating the panoramic composite image generation operation of the imaging apparatus 10. In step ST1, the imaging device 10 performs a one-frame imaging process. The imaging device 10 performs one-frame imaging processing and generates image data of a one-frame captured image. Further, the imaging apparatus 10 stores the image data of the captured image, for example, in the image memory unit 21, and proceeds to step ST2.

  In step ST <b> 2, the imaging device 10 determines whether the imaging has been completed. The imaging device 10 returns to step ST1 when continuing the imaging operation, and proceeds to step ST3 when ending the imaging operation. The imaging device 10 continues the imaging operation until the imaging end operation is performed, and ends the imaging operation when the imaging end operation is performed. In addition, the imaging device 10 performs the imaging operation when, for example, the captured images for a predetermined number of frames are generated, or when the swing operation or the parallel movement operation of the imaging device 10 performed for generating the panorama composite image is completed. It is good also as an end.

  In step ST <b> 3, the panorama composition processing unit 22 of the imaging device 10 detects a movement vector. The panorama synthesis processing unit 22 detects a movement vector between captured images that are temporally continuous, and proceeds to step ST4. The movement vector may be detected using, for example, the measurement result when the measurement unit 14 measures the movement direction and the movement speed when the imaging direction is moved.

  In step ST4, the imaging apparatus 10 performs panorama composition processing. The imaging device 10 generates a wide-angle composite image by cutting out and joining images from a plurality of captured images stored in the image memory unit 21. For example, as shown in FIG. 3A, imaging is performed while moving the imaging direction of the imaging apparatus 10 from the left DFs to the right DFe, and a plurality of captured images shown in FIG. 3B are generated. In addition, by cutting out and connecting images from a plurality of captured images, a panoramic composite image having a wider angle of view than the captured images is generated as shown in FIG. In the panorama compositing process, the imaging device 10 cuts out and joins images based on the detected movement vector so that the images do not overlap or disappear at the joints. FIG. 4 shows the movement when moving the imaging direction from the left side to the right side. 4A shows the subject, and FIG. 4B shows the temporal change in the imaging direction. For example, the case where the imaging direction is moved from the left DFs to the right DFe shown in FIG. 3A during the period from the imaging start time Trs to the imaging end time Tre is shown. Here, when the imaging direction is moved at a constant speed, the imaging direction moves as indicated by a broken line. In this case, the movement amount of the movement vector is constant. Also, for example, when a person is included in the imaging range, if the moving speed is decreased as the person gets closer to the center of the imaging range and the moving speed is increased as the person moves away from the center of the imaging range, the imaging direction is It changes as shown by the solid line. In this case, the amount of movement of the movement vector decreases as the person approaches the center of the imaging range.

  In step ST5, the imaging device 10 performs a composite image storage process. The imaging apparatus 10 stores the generated panorama composite image data in a recording medium or transmits the image data to an external device, and stores the panorama composite image. Further, when storing the panorama composite image, the imaging device 10 stores information indicating the movement in the imaging direction to be moved in association with the panorama composite image as panorama composite image attached information. Further, the imaging apparatus 10 includes the frame rate and the number of frames used for generating the panorama composite image, the image size of the panorama composite image, and the like in the panorama composite image attached information.

  FIG. 5 illustrates panorama composite image attached information. For example, the frame rate of the image used for generating the panorama composite image is “FR (frame / sec)”, the number of frames is “FNT (frame)”, and the image size of the panorama composite image is “Lpw × Lh (pixel)”. It has been shown. Also, the detected movement vector “MV1, MV2,... (Pixel)” is shown. When the detected movement vector indicates all the movement vectors between adjacent frames in the image used to generate the panorama composite image, the number of frames can be calculated without including information on the number of frames.

  When the panorama composite image is recorded using a file format called Exif (Exchangeable Image File Format), the imaging device 10 provides the panorama composite image attached information as a tag in the Exif data. In Exif, one file is composed of image data and Exif data.

  Also, when using the DCF (Design Rule for Camera File system) standard, which is a camera file system standard using Exif, panorama composite image attached information is filed and associated with the image data and stored in the same folder as the image data. . For example, the image data file “DSC00001.JPG” and the related file “DSC00001.PAM” of panorama composite image attached information are stored in the folder “123ABC”. The extension “JPG” indicates that the image data is compressed by JPEG (Joint Photographic Experts Group), and the extension “PAM” indicates that the information is attached to the panorama composite image. .

  In addition, when sound acquisition is performed at the time of generating a plurality of captured images used for generating a panorama composite image by performing an imaging operation, the sound at the time of imaging is associated with the panorama composite image and stored in the same folder as the image data. Store. For example, the audio data file “DSC00001.WAV” is stored in the folder “123ABC”. The extension “WAV” indicates a file of audio data.

  The generation of the panorama composite image and the panorama composite image attached information may be performed by the image processing apparatus instead of the imaging apparatus. In this case, the imaging device records a plurality of captured images or a plurality of captured images used for generating a panorama composite image and a measurement result of the measurement unit on a recording medium. Further, the imaging device may supply a plurality of captured images used for generating a panoramic composite image or a plurality of captured images and a measurement result of the measurement unit to the image processing device via the communication unit. The image processing apparatus generates a panorama composite image and panorama composite image ancillary information as described above using a plurality of captured images recorded on the recording medium or a plurality of captured images and the measurement result of the measurement unit. . The image processing apparatus may generate a panorama composite image or a panorama composite image attached information using a plurality of captured images supplied via the communication unit or a plurality of captured images and a measurement result of the measurement unit. Good.

<3. Configuration of Image Processing Device>
Next, an image processing apparatus that performs a playback operation of a panorama composite image will be described. FIG. 6 illustrates a schematic configuration of an image processing apparatus that performs scroll reproduction of a panoramic composite image. In the figure, an image processing apparatus having a panorama composite image generation function is illustrated.

  The image processing apparatus 50 includes a recording / playback unit 51, a communication unit 52, a memory unit 53, a codec unit 54, a panorama synthesis processing unit 55, a buffer memory unit 56, a display unit 57, an audio output unit 58, a user interface unit 61, and a control unit. 65. Each unit is connected via a bus 71.

  The recording / reproducing unit 51 reads out image data, encoded data, and the like recorded on the recording medium and outputs them to the memory unit 53.

  The communication unit 52 is an interface for communicating with the imaging device. The communication unit 52 outputs image data, encoded data, and the like supplied from the imaging device to the memory unit 53.

  The memory unit 53 temporarily stores image data, encoded data, and the like supplied from the recording / playback unit 51 and the communication unit 52.

  The codec unit 54 performs a data decompression process on the encoded data stored in the memory unit 53, outputs the obtained image data to the memory unit 53 or the buffer memory unit 56, and outputs the obtained audio data to the memory unit 53 or the audio output. To the unit 58. Further, the codec unit 54 performs data compression processing of the image data output from the panorama synthesis processing unit 55, outputs the obtained encoded data to the recording / reproducing unit 51, and records the data on a recording medium.

  The panorama compositing processing unit 55 performs composite processing of the captured image using the image data stored in the memory unit 53, and generates image data of a panoramic composite image having a wider angle of view than the captured image. The panorama composition processing unit 55 outputs the generated image data to at least one of the memory unit 53, the codec unit 54, and the buffer memory unit 56. In addition, the panorama composition processing unit 22 performs motion detection between captured images, and performs composition processing so that omissions and overlaps do not occur at the joints of the images based on the detection results. Further, the panorama composition processing unit 22 outputs the motion detection result, that is, the detection result of the movement in the imaging direction to the control unit 65.

  The buffer memory unit 56 temporarily stores the image data of the panorama composite image. The image data stored in the buffer memory unit 56 is read out by the display unit 57 for an area where image display is performed by a control signal from the control unit 65 described later, and is output to the display unit 57. The area from which the image data is read is sequentially moved by a control signal from the control unit 65, and the panorama composite image is scrolled and reproduced.

  The display unit 57 is configured using a flat display element such as a liquid crystal display element or an organic EL. The display unit 57 drives the display element based on the image data read from the buffer memory unit 56 to perform scroll display of the panorama composite image. The display unit 57 displays a menu for selecting a function installed in the image processing apparatus 50 and a setting state and an operation state of the image processing apparatus 50.

  Audio output is performed based on audio data supplied from the audio output unit 58, the memory unit 53, and the codec unit 54.

  The user interface unit 61 includes operation switches and operation buttons provided in the image processing apparatus 50, a touch panel provided on the display screen of the display unit 57, and the like, and generates an operation signal corresponding to an operation by the user. Output to the control unit 65.

  The control unit 65 is configured using a CPU, ROM, RAM, and the like. The control unit 65 reads out a program stored in the ROM and executes it by the CPU, and controls the operation of each unit so that the operation of the image processing device 50 becomes an operation corresponding to the user operation based on the operation signal. For example, the control unit 65 outputs a control signal to the buffer memory unit 56, and controls the reading of image data from the buffer memory unit 56, thereby performing the scroll reproduction of the panorama composite image. Further, the control unit 65 controls the scroll speed in the scroll reproduction according to the movement in the imaging direction to be moved, using the panorama composite image attached information. The control unit 65 may control the generation of the panorama composite image using a plurality of captured images stored in the memory unit 53. In this case, the control unit 65 uses the motion detection result output from the panorama synthesis processing unit 55 to control the scroll speed according to the movement in the imaging direction to be moved. That is, the scroll reproduction unit of the claims corresponds to the buffer memory unit 56 and the control unit 65, and the scroll speed control unit corresponds to the control unit 65.

  The control unit 65 may install and execute a program supplied via the communication unit 52 instead of the program stored in advance. Furthermore, a program provided by various recording media such as a memory card, an optical disk, and a magnetic disk may be installed and executed.

<4. First image reproduction operation>
Next, the first image reproduction operation of the image processing apparatus will be described. FIG. 7 is a flowchart showing the first image reproduction operation.

  In step ST11, the image processing apparatus 50 determines whether the image to be reproduced is a panoramic composite image. The image processing apparatus 50 determines whether the image to be reproduced is a panoramic composite image based on, for example, the image size and attached information, and proceeds to step ST12 when the image to be reproduced is a panoramic composite image. If it is not a composite image, the process proceeds to step ST15.

  In step ST12, the image processing apparatus 50 determines whether there is panorama composite image attached information. The image processing apparatus 50 proceeds to step ST13 when there is panorama composite image attached information related to the panorama composite image to be reproduced, and proceeds to step ST14 when there is no panorama composite image attached information.

  In step ST13, the image processing apparatus 50 performs motion reproduction scroll reproduction. Based on the panorama composite image attached information, the image processing device 50 changes the scroll speed in accordance with the movement in the imaging direction that is moved when capturing the plurality of captured images used to generate the panorama composite image, and panorama synthesis. Play the image.

  In step ST14, the image processing apparatus 50 performs fixed speed scroll reproduction. The image processing apparatus 50 reproduces the panorama composite image with the scroll speed fixed.

  When the process proceeds from step ST11 to step ST15, the image processing apparatus 50 reproduces one image. The image processing apparatus 50 displays one image by matching the image size of the image to be displayed with the display size of the display unit 57.

  Next, fixed speed scroll reproduction and motion reproduction scroll reproduction will be described. FIG. 8 is a diagram for explaining the fixed speed scroll reproduction. For example, FIG. 8A illustrates the generated panorama composite image. The image size of the panorama composite image is, for example, “Lpw × Lh”, and the display size of the display unit 57 is, for example, “Ldw × Lh” as shown in FIG.

  In fixed-speed scroll playback, the frame rate during playback is “FP (frame / sec)”, and the playback time is “TP (= Tse−Tss) (sec)”. In this case, the entire panorama composite image can be scroll-reproduced by moving the display area from the left end by (Lpw−Ldw) pixels during the period of time TP. Therefore, by sequentially moving the display area to the right by “(Lpw−Ldw) / (FP × TP)” for each frame, as shown in FIG. 8C, a panorama composite image at a constant scroll speed. The whole can be scrolled. In FIG. 8C, FIG. 9B and FIG. 11B described later, the relationship between time and the display area (left end position) is shown as a continuous characteristic line. A display area (left end position) may be indicated.

  FIG. 9 is a diagram for explaining the motion reproduction scroll reproduction. For example, FIG. 9A illustrates the generated panorama composite image. Further, the image size of the panorama composite image and the display size of the display unit 57 are “Lpw × Lh” and “Ldw × Lh” as described above.

  Here, for example, if the speed of moving the imaging direction is reduced when a person approaches the center of the imaging range, the amount of movement of the movement vector decreases. Accordingly, since a person is included in the central portion of the panoramic composite image shown in FIG. 9A, the movement amount of the movement vector is smaller than that of the other portions.

  In the motion reproduction scroll reproduction, the movement of the display area is determined based on the motion information indicated by the panorama composite image attached information so that the operation at the time of imaging can be reproduced. For example, the amount of movement of the display area is adjusted according to the movement vector used for generating the panorama composite image.

  In the motion reproduction scroll reproduction, when the frame rate during imaging and reproduction is “FR = FP” and the imaging time and reproduction time are “TR = TP”, the number of frames during imaging and reproduction is the same. Here, when the movement vector calculated using the first captured image and the second captured image is MV1, and the total value of the movement vectors is MVt, the movement amount of the display area is expressed as “(Lpw− Ldw) × MV1 / MVt ”. Similarly, when the movement vector calculated using the second captured image and the third captured image is MV2, the movement amount of the display area is “(Lpw−Ldw) × MV2 / MVt”. When the same processing is performed thereafter, as shown in FIG. 9B, scroll playback can be performed by reproducing the operation of capturing the entire panorama composite image. That is, as in the case of imaging, when the person approaches the center of the display area, the scroll speed is reduced, and the operation at the time of imaging can be reproduced. Further, if the relationship between the playback position and time shown in FIG. 9B is expanded or reduced in the time direction according to the playback time, even if the playback time differs from the imaging time, it corresponds to the operation at the time of imaging. It is possible to scroll and reproduce the panorama composite image by changing the scroll speed. Furthermore, by using the relationship between the playback position and time shown in FIG. 9B, even when the frame rate at the time of imaging is different from that at the time of playback, the scrolling speed is varied according to the operation at the time of imaging and the panorama is changed. It is possible to scroll the composite image. For example, if the relationship between time and display area (left end position) is calculated as a continuous characteristic line, the characteristic line is expanded or reduced in the time direction, so that even if the playback time differs from the imaging time, The corresponding display area can be easily identified. Even when the frame rates are different, it is possible to easily determine the display area corresponding to the frame to be displayed. Further, when the display area (left end position) is shown for each frame, the scroll speed can be varied in accordance with the operation at the time of imaging if the display area corresponding to the frame to be displayed is calculated by interpolation or the like. Furthermore, when the short direction of the panorama composite image is different from the size of the display area, the scroll speed may be adjusted according to the ratio between the short direction of the panorama composite image and the size of the display area.

  As described above, in the motion reproduction scroll reproduction, the image processing apparatus 50 controls the scroll speed according to the movement in the imaging direction to be moved. Therefore, for example, when imaging is performed with an emphasis on the subject of interest, scroll playback can be performed with an emphasis on the subject of interest. In other words, when imaging is performed while moving the imaging direction, if the moving speed is reduced when the subject of interest is included in the screen, the scrolling of the panoramic image will scroll when the subject of interest is displayed. Speed is reduced. Therefore, it is possible to reproduce a sense of reality at the time of imaging.

  Further, the image processing device 50 may perform switching between the motion reproduction scroll reproduction and the fixed speed scroll reproduction when there is panorama composite image attached information related to the panorama composite image to be reproduced. The control unit 65 detects a subject, for example, a person in the panorama composite image. If the display area does not include a person, the controller 65 performs fixed-speed scroll playback. If the display area includes a person, the control unit 65 performs a motion reproduction scroll. You may make it perform reproduction | regeneration.

  The image processing apparatus 50 has been described with reference to the case where the display unit 57 displays a panorama composite image or the like. However, the image data for motion reproduction scroll reproduction, the image data for fixed speed scroll reproduction, and the image data for single image display. May be output and displayed on an external device. Further, the image reproducing operation as described above may be performed by the imaging device 10.

<5. Second image reproduction operation>
Next, in the second image reproduction operation of the image processing apparatus, a case where audio data is reproduced together with reproduction of a panorama composite image will be described. FIG. 10 is a flowchart showing the second image reproduction operation.

  In step ST21, the image processing apparatus 50 determines whether the image to be reproduced is a panoramic composite image. The image processing apparatus 50 proceeds to step ST22 when the image to be reproduced is a panorama composite image, and proceeds to step ST29 when the image to be reproduced is not a panorama composite image.

  In step ST <b> 22, the image processing apparatus 50 determines whether there is panorama composite image attached information. The image processing apparatus 50 proceeds to step ST23 when there is panorama composite image attached information associated with the panorama composite image to be reproduced, and proceeds to step ST26 when there is no panorama composite image attached information.

  In step ST23, the image processing apparatus 50 determines whether there is audio data. The image processing apparatus 50 proceeds to step ST24 when there is audio data associated with the panorama composite image, and proceeds to step ST25 when there is no audio data associated therewith.

  In step ST <b> 24, the image processing apparatus 50 performs motion reproduction scroll reproduction with sound. The image processing apparatus 50 performs motion reproduction scroll reproduction as described above. Further, the image processing device 50 sequentially reproduces the audio data when the audio recording time (corresponding to the imaging time) is equal to the scroll reproduction time. Further, when the audio recording time is different from the scroll reproduction time, the audio reproduction speed is adjusted so that the reproduction time of the recorded audio data becomes equal to the scroll reproduction time. Furthermore, the image processing apparatus 50 performs sound processing so that the pitch does not change even if the sound reproduction speed is changed.

  In step ST25, the image processing apparatus 50 performs motion reproduction scroll reproduction. Since the audio data is not associated, the image processing apparatus 50 performs the above-described motion reproduction scroll reproduction.

  When the process proceeds from step ST22 to step ST26, the image processing apparatus 50 determines whether there is audio data. The image processing apparatus 50 proceeds to step ST27 when there is audio data associated with the panorama composite image, and proceeds to step ST28 when there is no audio data associated therewith.

  In step ST27, the image processing apparatus 50 performs fixed speed scroll reproduction with sound. When the audio recording time (corresponding to the imaging time) is equal to the scroll reproduction time, the image processing device 50 sequentially reproduces the audio data. Further, when the audio recording time is different from the scroll reproduction time, the audio reproduction speed is adjusted so that the reproduction time of the recorded audio data becomes equal to the scroll reproduction time. Furthermore, the image processing apparatus 50 performs sound processing so that the pitch does not change even if the sound reproduction speed is changed.

  In step ST28, the image processing apparatus 50 performs fixed speed scroll reproduction. Since the audio data is not associated, the image processing device 50 performs the above-described fixed speed scroll reproduction.

  When the process proceeds from step ST21 to step ST29, the image processing apparatus 50 reproduces one image. The image processing apparatus 50 displays one image by matching the image size of the image to be displayed with the display size of the display unit 57.

  As described above, when performing fixed speed scroll playback with audio, the image processing apparatus 50 changes the scroll speed so that the scroll scroll speed of the panorama composite image is constant and the playback time of the audio is also equal to the scroll playback time. Just do it. Therefore, it is possible to easily perform scroll playback and audio playback of the panorama composite image. In addition, when performing motion reproduction scroll playback with audio, scroll playback of the panorama composite image is performed by changing the scroll speed in accordance with the operation at the time of imaging so that the playback time of audio is equal to the scroll playback time. The scroll speed is changed to. Therefore, for example, when a desired subject is included in the imaging range, the moving speed is reduced and audio such as an explanation is recorded, and in panorama composite image scroll playback, an explanation is given when the desired subject is included in the display range. Will be played.

  Further, the image processing apparatus 50 may perform fixed-speed scroll playback when there is no audio data even if there is panorama composite image attached information related to the panorama composite image to be played back. Furthermore, motion reproduction scroll reproduction with audio may be performed when the reproduced audio is above a predetermined level, and fixed speed scroll reproduction with audio may be performed when the reproduced audio is smaller than the predetermined level. For example, as shown in FIG. 11, until the voice level SL exceeds the threshold SVth at the time Tsa, the fixed speed scroll reproduction with voice is performed, and when the threshold SVth is exceeded, switching to the motion reproduction scroll reproduction with voice is performed. Thereafter, when the audio level SL becomes equal to or less than the threshold value SVth at time Tsb, switching to fixed speed scroll reproduction with audio may be performed.

  When scroll reproduction is performed by the imaging device 20 described above, the control unit 35 controls the image memory unit 21 and the like to perform the first image reproduction operation or the second image reproduction operation. 3, 4, 8, 9, and 11 exemplify cases where the imaging direction is moved in the horizontal direction, the imaging direction is moved in a direction different from the horizontal direction, such as the vertical direction. In the same manner, the scroll reproduction as described above can be performed.

  Furthermore, a series of processes described in the specification can be executed by hardware, software, or a combined configuration of both. When processing by software is executed, a program in which a processing sequence is recorded is installed and executed in a memory in a computer incorporated in dedicated hardware. Alternatively, the program can be installed and executed on a general-purpose computer capable of executing various processes.

  For example, the program can be recorded in advance on a hard disk or ROM as a recording medium. Alternatively, the program is temporarily or permanently stored on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, or a semiconductor memory card. Can be stored (recorded). Such a removable recording medium can be provided as so-called package software.

  In addition to installing the program from the removable recording medium to the computer, the program may be transferred from the download site to the computer wirelessly or by wire via a network such as a LAN (Local Area Network) or the Internet. The computer can receive the program transferred in this way and install it on a recording medium such as a built-in hard disk.

  Note that the present technology should not be interpreted as being limited to the above-described embodiment. The embodiments of this technology disclose the present technology in the form of examples, and it is obvious that those skilled in the art can make modifications and substitutions of the embodiments without departing from the gist of the present technology. In other words, in order to determine the gist of the present technology, the claims should be taken into consideration.

In addition, the image processing apparatus according to the present technology may have the following configuration.
(1) a scroll reproduction unit that performs scroll reproduction of a panoramic composite image having a wider angle of view than the captured image generated by combining a plurality of captured images generated while moving the imaging direction;
An image processing apparatus comprising: a scroll speed control unit that controls a scroll speed in the scroll reproduction according to the movement in the imaging direction that is moved.
(2) The panoramic composite image is associated with attached information including motion information indicating movement in the imaging direction detected between the captured images.
The image processing apparatus according to (1), wherein the scroll speed control unit controls the scroll speed based on motion information included in attached information related to a panoramic composite image to be scroll-played.
(3) The scroll speed control unit according to (1) or (2), wherein the scroll speed control unit decreases the scroll speed as the movement in the imaging direction decreases, and increases the scroll speed as the movement increases. Image processing device.
(4) The attached information includes information indicating a panorama composite image size and a frame rate of the plurality of captured images,
The scroll speed control unit controls the scroll speed based on information indicating the panorama composite image size and the frame rate, and a display size and a frame rate in the scroll reproduction, according to any one of (1) to (3). Image processing apparatus.
(5) The image processing apparatus according to (4), wherein the scroll speed control unit controls the scroll speed to reproduce the movement in the imaging direction when generating the plurality of captured images and perform scroll reproduction. .
(6) The scroll speed control unit according to any one of (1) to (5), wherein the scroll speed is set to a constant speed when there is no attached information related to the panoramic composite image to be scroll-played. Image processing apparatus.
(7) The scroll speed control unit controls the playback speed of the audio signal in accordance with the scroll speed when an audio signal is associated with the panoramic composite image to be scroll-played (1) to (6). ).
(8) When the signal level of the audio signal associated with the panorama composite image to be scroll-played is higher than a preset signal level, the scroll speed control unit changes the scroll speed from a constant speed to the imaging. The image processing device according to any one of (1) to (7), wherein a playback speed of the audio signal is controlled according to the scroll speed as a speed according to a direction motion.

  In the image processing device, the image processing method, the program, and the imaging device of this technique, scrolling of a panoramic composite image having a wider angle of view than the captured image is performed by combining a plurality of captured images generated while moving the imaging direction. Playback is performed, and the scroll speed is controlled according to the movement in the imaging direction. For this reason, the realism at the time of imaging is reproduced by slowing down the scrolling speed when the moving speed is slow when generating multiple captured images while moving the imaging direction, and increasing the scrolling speed when the moving speed is fast. Scroll playback of the panorama composite image can be performed. Therefore, it is suitable for an imaging apparatus having a function of generating a digital photo frame or a panorama composite image.

  DESCRIPTION OF SYMBOLS 10 ... Imaging device, 11 ... Imaging lens part, 12 ... Imaging part, 13 ... Camera signal processing part, 14 ... Measurement part, 21 ... Image memory part, 22, 55. ..Panorama synthesis processing unit, 23 ... Audio acquisition unit, 24,54 ... Codec unit, 25,51 ... Recording / reproducing unit, 26,52 ... Communication unit, 27,57 ... Display Unit, 28 ... audio output unit, 31, 61 ... user interface unit, 35, 65 ... control unit, 41, 71 ... bus, 50 ... image processing device, 53 ... memory , 56... Buffer memory unit, 58... Audio output unit

Claims (10)

A scroll reproduction unit that performs scroll reproduction of a panoramic composite image having a wider angle of view than the captured image generated by combining a plurality of captured images generated while moving the imaging direction;
A scroll speed control unit for controlling a scroll speed in the scroll reproduction according to the movement of the moved imaging direction;
The scroll speed control unit is an image processing apparatus that slows down the scroll speed as the movement in the imaging direction decreases and increases the scroll speed as the movement increases. The panorama composite image is associated with attached information including motion information indicating movement in the imaging direction detected between the captured images,
The image processing apparatus according to claim 1, wherein the scroll speed control unit controls the scroll speed based on motion information included in attached information related to the panoramic composite image to be scroll-played. The attached information includes information indicating an image size of the panoramic composite image and a frame rate of the plurality of captured images,
The image processing apparatus according to claim 2, wherein the scroll speed control unit controls the scroll speed based on information indicating an image size and the frame rate of the panorama composite image and a display size and a frame rate in the scroll reproduction.   The image processing apparatus according to claim 3, wherein the scroll speed control unit controls the scroll speed to reproduce the movement in the imaging direction when the plurality of captured images are generated to perform scroll reproduction.   The image processing apparatus according to claim 1, wherein the scroll speed control unit sets the scroll speed to a constant speed when there is no attached information associated with the panorama composite image to be scroll-played.   The image processing apparatus according to claim 1, wherein the scroll speed control unit controls the playback speed of the audio signal according to the scroll speed when an audio signal is associated with the panoramic composite image to be scroll-played.   The scroll speed control unit moves the scroll speed from a constant speed to the movement in the imaging direction when a signal level of an audio signal related to the panoramic composite image to be scrolled is higher than a preset signal level. The image processing apparatus according to claim 1, wherein a playback speed of the audio signal is controlled according to the scroll speed as a speed corresponding to the scroll speed. A step of performing scroll reproduction of a panoramic composite image having a wider angle of view than the captured image generated by combining a plurality of captured images generated while moving the imaging direction;
An image processing method including a step of reducing a scroll speed in the scroll reproduction as the movement in the moving imaging direction is reduced, and increasing the scroll speed as the movement is increased. A program for causing a computer to execute scroll playback of a panorama composite image,
A procedure for performing scroll reproduction of a panoramic composite image having a wider angle of view than the captured image generated by synthesizing a plurality of captured images generated while moving the imaging direction;
A program for causing the computer to execute a procedure of decreasing a scroll speed in the scroll reproduction as the movement in the moving imaging direction becomes smaller and increasing the scroll speed as the movement becomes larger. An imaging unit for generating a captured image;
A panorama synthesis processing unit that generates a panorama composite image having a wider angle of view than the captured image generated by combining the plurality of captured images generated by the imaging unit while moving the imaging direction;
A motion detector for detecting the movement of the moved imaging direction;
An information generation unit that generates auxiliary information including motion information indicating the motion detected by the motion detection unit and relates to the panorama composite image;
A scroll playback unit that performs scroll playback of the panorama composite image;
A scroll speed control unit that controls a scroll speed in the scroll reproduction based on the motion detected by the motion detection unit;
The scrolling speed control unit is an imaging device that slows down the scrolling speed as the movement in the imaging direction decreases, and increases the scrolling speed as the movement increases.

Images