JP5637298B2 - Information processing apparatus, electronic apparatus, and display control method - Google Patents

Description

  The present technology relates to an information processing apparatus. Specifically, the present invention relates to an information processing apparatus that displays an image, an electronic device, and a display control method.

  2. Description of the Related Art In recent years, imaging apparatuses such as a digital still camera and a digital video camera (for example, a camera-integrated recorder) that have an imaging system for imaging a subject such as a person and record an image generated by the imaging system as an image file have become widespread. ing.

  In addition, there has been proposed an imaging apparatus capable of generating an image (for example, a panoramic image) having a relatively wide imaging range in a specific direction by combining a plurality of images. For example, an imaging apparatus that includes three imaging systems and generates a panoramic image by arranging and synthesizing images output from the imaging systems has been proposed (see, for example, Patent Document 1).

JP 2007-166317 A

  According to the above-described conventional technology, it is possible to easily take a panoramic image.

  Here, when a person stands upright with respect to the ground, the eyes of the person are arranged horizontally with respect to the ground. For this reason, the human-visible area is wide in the left-right direction (horizontal direction (direction parallel to the ground, direction perpendicular to gravity)), and up and down (vertical direction (direction perpendicular to the ground, direction of gravity). Narrow direction)). In addition, when a person takes a panoramic image, an image reflecting a wide vision in the left and right direction of the person (an image that is long in the horizontal direction) is often recorded. However, information processing apparatuses such as smartphones and mobile phones that include an imaging unit are often used in a vertically long state (a state in which the longitudinal direction is parallel to the direction of gravity). For this reason, when the information processing apparatus is used in the vertically long state, it is assumed that the panoramic image cannot be appropriately displayed on the display unit in the vertically long state, and it is difficult to view the panoramic image.

  The present technology has been created in view of such a situation, and aims to appropriately display an image.

  The present technology has been made to solve the above-described problems, and a first side of the present technology includes a plurality of imaging units arranged in a direction substantially orthogonal to the long side direction of the display unit, and the display A display control unit that controls display of an image on the unit, wherein the display control unit is generated by the plurality of imaging units in a state in which a long side direction of the display unit is substantially parallel to a gravity direction. A plurality of regions on a substantially rectangular image in which a plurality of images are connected are displayed on the display unit in a first display form having a step shape and a plurality of steps in the direction of gravity, and a plurality of regions on the substantially rectangular image are displayed. A first image which is an image including a left end region in the long side direction of the substantially rectangular image and a second image which is an image including a right end region in the long side direction of the substantially rectangular image. And both ends of the substantially rectangular image in the long side direction And a third image is a program for executing an information processing apparatus and a display control method, and the method that are displayed on the display unit to the computer is an image including a region existing between. Thereby, in a state where the long side direction of the display unit is substantially parallel to the gravity direction, a plurality of regions on the substantially rectangular image are displayed in a first display form having a plurality of steps and a plurality of steps in the gravity direction. This brings about the effect.

  In addition, a second aspect of the present technology includes a plurality of imaging units arranged side by side in a direction substantially orthogonal to the long side direction of the display unit, and a display control unit that controls display of an image on the display unit. In the state where the long side direction of the display unit is substantially parallel to the direction of gravity, the display control unit includes a plurality of images on a substantially rectangular image in which a plurality of images generated by the plurality of imaging units are connected. A region is displayed on the display unit in a first display form having a step shape and a plurality of steps in the direction of gravity, and the first image, the second image, and the third image are displayed as a plurality of regions on the substantially rectangular image. Are displayed on the display unit, and the third image is a partial region of the substantially rectangular image, and is inward by a finite width from both ends in the long side direction of the substantially rectangular image. The first image is a part of the substantially rectangular image. And the second image is an image that includes a region outside the third image and a region on the left side of the third image, and the second image is a partial region of the substantially rectangular image, An information processing apparatus that is an image including a region outside the third image and a region on the right side of the third image, a display control method thereof, and a program that causes a computer to execute the method. Thereby, in a state where the long side direction of the display unit is substantially parallel to the gravity direction, a plurality of regions on the substantially rectangular image are displayed in a first display form having a plurality of steps and a plurality of steps in the gravity direction. This brings about the effect.

  In the first and second aspects, the display control unit displays the third image on one level, and the first image and the second image are set on different levels from the third image. You may make it display. This brings about the effect that the third image is displayed on one stage, and the first image and the second image are displayed on different stages from the third image.

  In the first and second aspects, the display control unit displays the image in which the ratio of the horizontal size to the vertical size of the image is smaller than the ratio of the substantially rectangular image. A second display mode in which the number of images displayed on the display unit is displayed smaller than that in the first display mode, or a display in which the number of stages displayed on the display unit is smaller than that in the first display mode. You may make it display with the 3rd display form to do. Thereby, when displaying an image in which the ratio of the horizontal size to the vertical size of the image is smaller than the ratio of the substantially rectangular image, the image is displayed in the second display mode or the third display mode. Bring.

  In the first and second aspects, the display control unit determines the number of images to be displayed on the display unit when the long side direction of the display unit is substantially perpendicular to the gravity direction. The display may be performed in the second display form that displays less than the display form, or in the third display form that displays the number of steps displayed on the display unit less than the first display form. Good. Thereby, when the long side direction of the display unit is substantially perpendicular to the gravity direction, the second display form or the third display form is displayed.

  In the first and second aspects, when the display control unit displays on the display unit in the first display mode, the display control unit performs a live view operation, a post view operation, and an image reproduction operation. In each case, a first display method for displaying a region including the center of the substantially rectangular image as the third image, and a region where the specific object included in the substantially rectangular image is present as the third image are displayed. The second image display method, and the region to be displayed as the third image while moving the third image from one end side to the other end side of the both ends of the substantially rectangular image. You may make it display by either of the 3rd display methods to display. Thus, when displaying in the first display mode, the first display method, the second display method, and the third display method are performed in each of the live view operation, the post view operation, and the image reproduction operation. It brings about the effect of displaying by any of the display methods.

  In the first and second aspects, the reproduction operation is an operation for reproducing and displaying an image recorded in the recording unit used by the information processing apparatus on the display unit based on a user operation. It may be. Accordingly, in the reproduction operation for reproducing and displaying the image recorded in the recording unit used by the information processing apparatus on the display unit based on the user operation, the first display method, the second display method, and the third It brings about the effect of displaying by either of the display methods.

  Further, in the first and second aspects, the specific object is detected as an object to be focused at the time of imaging, an object detected as a human face at the time of imaging, and a moving object at the time of imaging. You may make it either one with a thing. Thus, in the second display method, any one of an object that is a focus target at the time of imaging, an object that is detected as a human face at the time of imaging, and an object that is detected as a moving object at the time of imaging is defined as a specific object. The effect of doing.

  In the first and second aspects, the display control unit is configured based on a predetermined setting or a user operation in each of the live view operation, the post view operation, and the playback operation. The image may be displayed by any one of the first to third display methods. As a result, in each of the live view operation, the post view operation, and the playback operation, an image is displayed by any one of the first to third display methods based on a predetermined setting or a user operation. Bring about an effect.

  In the first and second aspects, the display control unit may display the first display method during the live view operation and the post view operation when displaying on the display unit in the first display mode. Alternatively, the image may be displayed by any one of the second display methods, and the image may be displayed by any one of the first to third display methods during the reproduction operation. As a result, when displaying in the first display mode, images are displayed by either the first display method or the second display method during the live view operation and the post view operation, and during the playback operation, the first to third images are displayed. This brings about an effect of displaying an image by any one of the display methods.

  In the first and second aspects, when the display control unit displays on the display unit in the first display mode, the first display method or the second display method is performed during the live view operation. The image may be displayed by any of the above, and the image may be displayed by any one of the first to third display methods during the post-view operation and the reproduction operation. Thus, when displaying in the first display mode, the image is displayed by either the first display method or the second display method during the live view operation, and the first to third during the post view operation and the reproduction operation. This brings about an effect of displaying an image by any one of the display methods.

  In addition, the third aspect of the present technology includes a plurality of imaging units arranged in a direction substantially orthogonal to a long side of the display unit on a surface substantially parallel to the display surface of the display unit, and the display In a state in which the long side of the unit is substantially parallel to the direction of gravity, the plurality of regions on the horizontally long image in which the plurality of images generated by the plurality of imaging units are connected are stepped and the longitudinal direction of the display unit A display control unit that performs control to display on the display unit so that the number of stages is plural, and the display control unit includes a plurality of regions on the horizontally long image in the longitudinal direction of the horizontally long image. Information processing apparatus for displaying first image and second image, which are two images including regions at both ends, and third image, which is an image including a region existing between both ends, on the display unit, and Control method and implementation of the method on a computer Is a program to be. As a result, in a state where the long side of the display unit is substantially parallel to the gravity direction, a plurality of regions on the horizontally long image are displayed in a step shape and a plurality of steps in the longitudinal direction of the display unit. As a plurality of regions on the image, two images (first image and second image) including regions at both ends in the longitudinal direction of the horizontally long image and an image including a region existing between both ends (third image) Image) is displayed.

  According to a fourth aspect of the present technology, the second casing is a casing that is attached to a first casing that houses a display unit, and that has a rotation axis that is substantially parallel to the short side of the display unit. And a plurality of imaging units arranged side by side in a direction substantially parallel to the rotation axis on one surface of the second housing, and a state in which the long side of the display unit is substantially parallel to the direction of gravity Then, a plurality of regions on a horizontally long image in which a plurality of images generated by the plurality of imaging units are connected is displayed on the display unit so as to have a plurality of steps in the longitudinal direction of the display unit. A display control unit that performs control, and the display control unit is a first image that is two images including regions at both ends in the longitudinal direction of the horizontally long image as a plurality of regions on the horizontally long image. Includes the image and second image, and the area that exists between the ends And a third image is an image which is a program for executing the information processing apparatus, information processing method, and the method is displayed on the display unit to the computer. As a result, in a state where the long side of the display unit is substantially parallel to the gravity direction, a plurality of regions on the horizontally long image are displayed in a step shape and a plurality of steps in the longitudinal direction of the display unit. As a plurality of regions on the image, two images (first image and second image) including regions at both ends in the longitudinal direction of the horizontally long image and an image including a region existing between both ends (third image) Image) is displayed.

  In the third and fourth aspects, the display control unit may display the first image and the second image on the same level and display the third image on another level. Good. As a result, the first image and the second image are displayed on the same stage, and the third image is displayed on another stage.

  In the third and fourth aspects, the display control unit may display the third image at a level higher than the level at which the first image and the second image are displayed. This brings about the effect that the third image is displayed in the upper stage than the stage where the first image and the second image are displayed.

  In the third and fourth aspects, the display unit displays the third image in a horizontal size larger than the horizontal size of the first image and the second image. It may be. This brings about the effect that the horizontal size of the third image is displayed larger than the horizontal size of the first image and the second image.

  In the third and fourth aspects, the display control unit displays an image obtained by reducing the entire horizontally long image in a region other than the display region of the first image to the third image. May be. This brings about the effect that an image obtained by reducing the entire horizontally long image is displayed in an area other than the display area of the first image to the third image.

  In the third and fourth aspects, the information processing apparatus further includes an attitude detection unit that detects the attitude of the information processing apparatus, and the display control unit is configured such that the long side of the display unit is substantially perpendicular to the direction of gravity. In other words, the first image to the third image may be controlled to be displayed with a smaller number of steps than when the long side of the display unit is substantially parallel to the direction of gravity. Thus, when the long side of the display unit is substantially perpendicular to the direction of gravity, the first image to the third image are displayed with fewer steps than when the long side of the display unit is substantially parallel to the direction of gravity. It brings about the effect of letting.

  In the third and fourth aspects, the display control unit causes the display unit to display an image in which the ratio of the horizontal size to the vertical size of the image to be displayed is smaller than the horizontally long image. In this case, the display may be controlled with the number of steps reduced compared to the case where the horizontally long image is displayed. As a result, when displaying an image in which the ratio of the horizontal size to the vertical size of the image to be displayed is smaller than the horizontally long image on the display unit, the number of steps is reduced compared to the case of displaying the horizontally long image. It brings about the effect of displaying.

  The third and fourth aspects further include a specific object detection unit that detects a specific object included in the horizontally long image, and the display control unit includes the detected specific object. An image including a region to be displayed may be displayed as the third image. This brings about the effect | action of displaying the image containing the area | region where the detected specific target object exists as a 3rd image.

  Further, in the third and fourth aspects, when recording the horizontally long image, the recording control for recording the specific object information related to the region where the specific object exists in the recording medium in association with the horizontally long image. The display control unit uses the specific object information recorded in association with the horizontally long image when displaying the horizontally long image recorded on the recording medium. You may make it display the image regarding the area | region where a specific target object exists. Thus, when displaying a horizontally long image recorded on the recording medium, an image relating to a region where the specific object exists is displayed using the specific object information recorded in association with the horizontally long image. This brings about the effect.

  In the third and fourth aspects, the display control unit moves the region displayed as the third image from one end of the both ends to the other end. An image may be displayed. This brings about the effect that the third image is displayed while the region displayed as the third image is moved from one end to the other end of both ends of the horizontally long image.

  The third and fourth aspects further include an operation receiving unit that receives a switching instruction operation from a user regarding the display method of the third image, and the display control unit is configured based on the switching instruction operation. A display method of displaying the third image while moving the region displayed as the third image from one end of the both ends to the other end; and a region of the region displayed as the third image The third image may be displayed by selecting one of the display methods for displaying a certain region as the third image without moving. Thus, based on the switching instruction operation related to the display method of the third image, the region displayed as the third image is moved from one end of the horizontally long image to the other end while moving the third image. An operation of selecting either a display method for displaying an image or a display method for displaying a certain region as a third image without moving the region displayed as the third image to display the third image. Bring.

  Further, in the third and fourth aspects, when the display method of the third image is switched, the switching is performed until the display of the third image based on the switching is finished. When the horizontally long image is newly displayed after the display method is retained and the display of the third image based on the switching is finished, the above-described image in the horizontally long image is displayed using the retained display method. A third image may be displayed. As a result, when the display method of the third image is switched, the display method related to the switching is held until the display of the third image based on the switching is completed, and the third method based on the switching is performed. When a horizontally long image is newly displayed after the display of the three images is completed, the third image in the horizontally long image is displayed using the retained display method.

  In the third and fourth aspects, when the reduced image obtained by reducing the entire horizontally long image is displayed in an area other than the display area of the first image to the third image, the reduced image is displayed. An operation accepting unit that accepts a designation operation for designating a position in the display, wherein the display control unit displays an area displayed as the third image when the designation operation is performed. When the third image is displayed by a display method in which the third image is displayed while being moved from one end to the other end, the position in the reduced image specified by the specifying operation and its surroundings An image including a region is displayed as a new third image, and the third image is displayed while performing the movement starting from the position. On the other hand, when the designation operation is performed, the third image is displayed. If the third image is displayed by a display method in which a certain area is displayed as the third image without moving the displayed area, an image including the position and the surrounding area is newly displayed. It may be displayed as a third image. As a result, when the position designation operation in the reduced image is performed, the third image is moved while moving the region displayed as the third image from one end of the horizontally long image to the other end. When the third image is displayed by the display method for displaying the image, an image including the position in the reduced image specified by the specifying operation and the surrounding area is displayed as a new third image, and the position is displayed. The third image is displayed while moving from the starting point. On the other hand, when the designation operation is performed, the region displayed as the third image is not moved, and a certain region is displayed as the third image. When the third image is displayed by the display method, the image including the position and the surrounding area is displayed as a new third image.

  Further, in the third and fourth aspects, the display control unit fixes and displays the position of the third image in the landscape image during live view or post view, and The second display method for displaying an image including an area where the specific object included in the horizontally long image is displayed as the third image is switched based on a user operation, and the area displayed as the third image is displayed during reproduction. The third display method for displaying the third image while moving from one end of the both ends to the other end and the second display method may be switched based on a user operation. . Thus, during live view or post view, the first display method and the second display method are switched based on the user operation, and during playback, the third display method and the second display method are switched based on the user operation. It brings about the effect of switching.

  In the third and fourth aspects, the display control unit may display a first display method in which the position of the third image in the landscape image is fixed during live view, and the landscape image. The second display method for displaying an image including an area where the specific target object is included is displayed as the third image based on a user operation, and the area displayed as the third image is displayed at the time of postview or playback. The third display method for displaying the third image while moving from one end of the both ends to the other end and the second display method may be switched based on a user operation. . Thus, during live view, the first display method and the second display method are switched based on a user operation, and during post view or playback, the third display method and the second display method are switched based on a user operation. It brings about the effect of switching.

  Further, in the third and fourth aspects, the display control unit may include the imaging unit before the user records an image when the information processing apparatus is shipped from a factory or when the user uses the information processing apparatus. Display method by live view operation for displaying the image generated by the above-mentioned display unit, and image display method by post-view operation for displaying the recorded image on the display unit immediately after the user records the image In addition, each of the images may be displayed based on the above setting when the setting is individually made for each of the image display method by the reproduction operation for displaying the image designated by the user on the display unit. Accordingly, when the information processing apparatus is shipped from the factory or when the user uses the information processing apparatus, an image display method using a live view operation, an image display method using a post view operation, and an image display method using a reproduction operation are provided. When each setting is made individually, each image is displayed based on the setting.

  In addition, the fifth aspect of the present technology provides a display unit that displays an image, and a case where a horizontally long image is displayed on the display unit in a state where the long side of the display unit is substantially parallel to the direction of gravity. A display control unit that performs control to display a plurality of regions on the image so as to have a plurality of steps in the longitudinal direction of the display unit, and the display control unit includes a plurality of regions on the horizontally long image. A first image and a second image that are two images including regions at both ends in the longitudinal direction of the horizontally long image, and a third image that is an image including a region existing between the both ends, A display device that performs control for displaying the image, a control method therefor, and a program that causes a computer to execute the method. As a result, in a state where the long side of the display unit is substantially parallel to the gravity direction, a plurality of regions on the horizontally long image are displayed in a step shape and a plurality of steps in the longitudinal direction of the display unit. As a plurality of regions on the image, two images (first image and second image) including regions at both ends in the longitudinal direction of the horizontally long image and an image including a region existing between both ends (third image) Image) is displayed.

  In the fifth aspect, the display control unit may display the first image and the second image on the same level and display the third image on another level. As a result, the first image and the second image are displayed on the same stage, and the third image is displayed on another stage.

  In the fifth aspect, the display control unit may display the third image at a level higher than the level at which the first image and the second image are displayed. This brings about the effect that the third image is displayed in the upper stage than the stage where the first image and the second image are displayed.

  In the fifth aspect, the display unit may display the third image in a horizontal size larger than the horizontal size of the first image and the second image. Good. This brings about the effect that the horizontal size of the third image is displayed larger than the horizontal size of the first image and the second image.

  In the fifth aspect, the display control unit may display an image obtained by reducing the entire horizontally long image in an area other than the display area of the first image to the third image. . This brings about the effect that an image obtained by reducing the entire horizontally long image is displayed in an area other than the display area of the first image to the third image.

  In the fifth aspect, the display device further includes an attitude detection unit that detects the attitude of the display device, and the display control unit is configured so that the long side of the display unit is substantially perpendicular to the direction of gravity. You may make it perform control which displays the said 1st image thru | or a 3rd image by making the said step number small rather than the case where the long side of a display part becomes substantially parallel to the gravity direction. Thus, when the long side of the display unit is substantially perpendicular to the direction of gravity, the first image to the third image are displayed with fewer steps than when the long side of the display unit is substantially parallel to the direction of gravity. It brings about the effect of letting.

  In the fifth aspect, when the display control unit causes the display unit to display an image in which the ratio of the horizontal size to the vertical size of the image to be displayed is smaller than the horizontally long image, You may make it perform control which displays the said number of steps less than the case where the said horizontally long image is displayed. As a result, when displaying an image in which the ratio of the horizontal size to the vertical size of the image to be displayed is smaller than the horizontally long image on the display unit, the number of steps is reduced compared to the case of displaying the horizontally long image. It brings about the effect of displaying.

  According to the present technology, an excellent effect that an image can be appropriately displayed can be obtained.

It is a figure showing an example of appearance composition of information processor 100 in an embodiment of this art. It is a figure showing an example of an internal configuration of information processor 100 in an embodiment of this art. It is a figure showing an example of an internal configuration of imaging part 130 in an embodiment of this art. It is a figure showing an example of arrangement composition of image sensors 134 thru / or 136 which constitute image pick-up part 130 in an embodiment of this art. It is a figure which shows the relationship between the image imaged on the image pick-up element 134 thru | or 136 in embodiment of this technique, and the reading method of image data. It is a figure which shows the relationship between the image imaged on the image pick-up element 134 thru | or 136 in embodiment of this technique, and the reading method of image data. It is a block diagram which shows the example of an internal structure of DSP200 in embodiment of this technique. It is a block diagram which shows the internal structural example of the image signal process part 220 in embodiment of this technique. 3 is a block diagram illustrating an internal configuration example of a clock generation circuit 270 according to an embodiment of the present technology. FIG. FIG. 28 is a block diagram illustrating a modified example of the clock generation circuit 270 according to the embodiment of the present technology. FIG. 28 is a block diagram illustrating a modified example of the clock generation circuit 270 according to the embodiment of the present technology. It is a figure showing an example of an internal configuration of an image sensor in an embodiment of this art. It is a figure which shows typically the holding | maintenance content of the registers 370 and 380 with which the image sensor 134 in embodiment of this technique is provided. It is a timing chart showing typically the situation of the control signal to each pixel in image sensor 134 in the embodiment of this art, and the data outputted from each of these pixels. It is a timing chart showing typically the situation of the control signal to each pixel in image sensor 134 in the embodiment of this art, and the data outputted from each of these pixels. It is a timing chart showing typically the situation of the control signal to each pixel in image sensor 134 in the embodiment of this art, and the data outputted from each of these pixels. It is a figure which shows typically an example of the scanning circuit for performing the thinning-out of the image pick-up element 134 in embodiment of this technique. It is a figure which shows the relationship between the imaging part 130 and a to-be-photographed object in embodiment of this technique. It is a figure which shows typically the relationship between the imaging system in the imaging part 130 in embodiment of this technique, and the to-be-photographed object. It is a figure which shows typically the relationship between the imaging system in the imaging part 130 in embodiment of this technique, and the to-be-photographed object. It is a figure which shows typically the relationship between the imaging system in the imaging part 130 in embodiment of this technique, and the captured image produced | generated by these. It is a figure which shows typically the relationship between the imaging system in the imaging part 130 in embodiment of this technique, and the to-be-photographed object. It is a figure which shows typically the relationship between the imaging system in the imaging part 130 in embodiment of this technique, the captured image produced | generated by these, and the correction image after correction | amendment. It is a figure which shows typically the flow of a synthesis | combination in case the image composition process part 224 in embodiment of this technique produces | generates a synthesized image. It is a figure which shows typically the flow of a synthesis | combination in case the image composition process part 224 in embodiment of this technique produces | generates a synthesized image. It is a figure which shows typically the flow of a synthesis | combination in case the image composition process part 224 in embodiment of this technique produces | generates a synthesized image. It is a figure which shows an example of the file structure of the image file memorize | stored in the recording medium 180 in embodiment of this technique. It is a block diagram showing an example of functional composition of information processor 100 in an embodiment of this art. It is a figure which shows the relationship between the to-be-photographed object and imaging range of the imaging process by the information processing apparatus 100 in embodiment of this technique. It is a figure which shows the relationship between the panoramic image 600 produced | generated by the image generation part 510 in embodiment of this technique, and the image displayed on the display part 140 by the display control part 550. FIG. It is a figure which shows the relationship between the panoramic image 600 produced | generated by the image generation part 510 in embodiment of this technique, and the image displayed on the display part 140 by the display control part 550. FIG. It is a figure which shows the relationship between the panoramic image 600 produced | generated by the image generation part 510 in embodiment of this technique, and the image displayed on the display part 140 by the display control part 550. FIG. It is a figure which shows the relationship between the panoramic image 600 produced | generated by the image generation part 510 in embodiment of this technique, and the image displayed on the display part 140 by the display control part 550. FIG. It is a figure which shows the relationship between the panoramic image 600 produced | generated by the image generation part 510 in embodiment of this technique, and the image displayed on the display part 140 by the display control part 550. FIG. It is a figure which shows the relationship between the panorama image 600 produced | generated by the image generation part 510 in embodiment of this technique, each image used as a display target, and the display example of the display part 140. FIG. It is a figure which shows the example of a display of the panorama image displayed on the display part 140 in embodiment of this technique, and each image used as a display target. It is a figure which shows the relationship between the panorama image 650,655,660 produced | generated by the image generation part 510 in embodiment of this technique, and the middle image 651,656,661. It is a figure which shows the relationship between the panorama image 665,670,675 produced | generated by the image generation part 510 in embodiment of this technique, and the middle image 666,671,676. It is a figure which shows the example of a display of the panorama image produced | generated by the image generation part 510 in embodiment of this technique, and each image (a middle image, a left image, and a right image). It is a figure which shows the example of a display of the panorama image produced | generated by the image generation part 510 in embodiment of this technique, and each image (a middle image, a left image, and a right image). It is a figure which shows the example of a transition of the area | region of the middle image in the panoramic image produced | generated by the image generation part 510 in embodiment of this technique. It is a figure which shows the example of a display of the panorama image produced | generated by the image generation part 510 in embodiment of this technique, and each image (a middle image, a left image, and a right image). It is a figure which shows the example of a display of the panorama image produced | generated by the image generation part 510 in embodiment of this technique, and each image (a middle image, a left image, and a right image). It is a figure which shows the example of a display of the panorama image produced | generated by the image generation part 510 in embodiment of this technique, and each image (a middle image, a left image, and a right image). It is a figure which shows the relationship between the panorama image 600 produced | generated by the image generation part 510 in embodiment of this technique, each image used as a display target, and the display example of the display part 140. FIG. It is a figure which shows the example of a display in the case of displaying the panorama image produced | generated by the image generation part 510 in embodiment of this technique on the display part 140. FIG. It is a figure showing a modification of information processor 100 in an embodiment of this art. It is a figure showing a modification of information processor 100 in an embodiment of this art. It is a figure showing a modification of information processor 100 in an embodiment of this art. It is a figure showing a modification of information processor 100 in an embodiment of this art. It is a figure showing a modification of information processor 100 in an embodiment of this art. It is a figure which shows the example of a display of each image produced | generated by the information processing apparatus in embodiment of this technique. It is a figure which shows the example of a display of each image produced | generated by the information processing apparatus in embodiment of this technique. It is a figure which shows the example of a display of each image produced | generated by the information processing apparatus in embodiment of this technique. It is a figure which shows the example of a display of each image produced | generated by the information processing apparatus in embodiment of this technique. It is a figure which shows the example of a display of each image produced | generated by the information processing apparatus in embodiment of this technique. It is a figure which shows the example of a display of each image produced | generated by the information processing apparatus in embodiment of this technique. 12 is a flowchart illustrating an example of a processing procedure of display control processing by the information processing apparatus 100 according to the embodiment of the present technology. It is a flowchart which shows an example of the imaging | photography process procedure among the display control processing procedures by the information processing apparatus 100 in embodiment of this technique. 14 is a flowchart illustrating an example of a panorama shooting process procedure in a display control process procedure performed by the information processing apparatus 100 according to the embodiment of the present technology. 14 is a flowchart illustrating an example of a reproduction processing procedure in a display control processing procedure performed by the information processing apparatus 100 according to the embodiment of the present technology. 12 is a flowchart illustrating an example of a processing procedure of display control processing by the information processing apparatus 100 according to the embodiment of the present technology. 14 is a flowchart illustrating an example of a panorama shooting process procedure in a display control process procedure performed by the information processing apparatus 100 according to the embodiment of the present technology. 14 is a flowchart illustrating an example of a reproduction processing procedure in a display control processing procedure performed by the information processing apparatus 100 according to the embodiment of the present technology. It is a flowchart which shows an example of the vertically long panorama reproduction | regeneration processing procedure among the display control processing procedures by the information processing apparatus 100 in embodiment of this technique. It is a flowchart which shows an example of the landscape state panorama reproduction | regeneration processing procedure of the display control processing procedure by the information processing apparatus 100 in embodiment of this technique. It is a figure showing an example of appearance composition of information processor 1000 in a 2nd embodiment of this art. It is a figure showing an example of appearance composition of information processor 1000 in a 2nd embodiment of this art. It is a figure showing an example of appearance composition of information processor 1050 in a 2nd embodiment of this art. It is a figure showing an example of appearance composition of information processor 1050 in a 2nd embodiment of this art.

Hereinafter, modes for carrying out the present technology (hereinafter referred to as embodiments) will be described. The description will be made in the following order.
1. First embodiment (display control: an example in which a plurality of regions in a panoramic image are displayed in a plurality of stages in the longitudinal direction of the display unit)
2. Second embodiment (an example of an information processing apparatus in which an imaging unit is movable and arranged)

<1. First Embodiment>
[External configuration example of information processing device]
FIG. 1 is a diagram illustrating an external configuration example of the information processing apparatus 100 according to the first embodiment of the present technology. FIG. 1 a shows a top view of the information processing apparatus 100, and FIG. 1 b shows a front view of the information processing apparatus 100. 1c shows a side view of the information processing apparatus 100, and FIG. 1d shows a rear view of the information processing apparatus 100. Further, FIG. 1 e shows an example of the arrangement of the electronic substrate of the information processing apparatus 100.

  The information processing apparatus 100 is realized by, for example, an information processing apparatus including a plurality of imaging units (for example, a smartphone with a compound eye camera or a mobile phone with a compound eye camera). In FIG. 1, for ease of explanation, the information processing apparatus 100 is illustrated in a simplified manner, and illustration of a power switch and the like provided on the outer surface of the information processing apparatus 100 is omitted. Also, details of the circuits and the like provided in the housing of the information processing apparatus 100 are not shown. The information processing apparatus 100 is an example of an information processing apparatus described in the claims.

  Here, in the embodiment of the present technology, the ratio of the horizontal size to the vertical size of the image is defined as “aspect ratio”. In addition, an image having an aspect ratio larger than the aspect ratio (16: 9) of the digital high-definition image is defined as a “panoramic image”.

  The information processing apparatus 100 includes an electronic substrate 101, a battery storage unit 102, a changeover switch 111, a determination key 112, an imaging unit 130, and a display unit 140.

  The electronic board 101 is an electronic board that mainly manages functions other than the image capturing function. Here, in the embodiment of the present technology, a plurality of imaging systems (imaging units 130) arranged so as to be parallel to the short side of the casing of the information processing apparatus 100 are connected to the outer edge of the casing short side as much as possible. Try to place it close to. That is, an electronic board having functions other than image capturing and display is not arranged between the plurality of imaging systems (imaging unit 130) and the outer edge of the short side of the housing. In addition, the electronic substrate disposed between the plurality of imaging systems (imaging unit 130) and the outer edge of the short side of the housing is an electronic substrate disposed in the information processing apparatus 100, and other than image capturing and display It is preferable to make it 1/2 or less of the area of the electronic substrate that bears the function.

  The battery storage unit 102 is an area where a battery is stored.

  The changeover switch 111 is an operation member (so-called toggle switch) used when switching between two functions. For example, the change-over switch 111 is used when switching between a still image capturing mode for recording a still image and a moving image capturing mode for recording a moving image. Further, for example, the changeover switch 111 is used when changing the scrolling method (the presence or absence of scrolling shown in FIGS. 41 to 43).

  The enter key 112 is an operation member that is pressed when setting various functions by the user. For example, when pressed when the still image capturing mode is set, it functions as a shutter button.

  Note that the numeric keypad and the cross key are appropriately displayed on the display unit 140 by a user operation or automatically. Then, the user can operate the displayed numeric keypad or cross key.

  The imaging unit 130 captures a subject and generates image data. In the embodiment of the present technology, the generation of image data by the imaging unit 130 includes the meaning of shooting by the imaging unit 130. A circle in the imaging unit 130 illustrated in FIG. 1B schematically represents each lens of a plurality of imaging systems included in the imaging unit 130. That is, in the embodiment of the present technology, the imaging unit 130 in which three lens groups are arranged in a specific direction will be described as an example. Also, as shown in FIG. 1a, the optical axis (optical axis that passes through the optical system 131 and enters the image sensor 134) in the middle lens group (optical system 131) of the three lens groups is It is assumed that the display unit 140 is orthogonal (or substantially orthogonal) to the display surface. Also, as shown in FIG. 1a, it is assumed that the optical axes of the three lens groups (optical systems 131 to 133) intersect at one point. Here, the specific direction can be, for example, a direction orthogonal to the longitudinal direction of the display unit 140 (longitudinal direction of the information processing apparatus 100). In addition, about each part which comprises the imaging part 130 shown in FIG. 1, it simplifies and shows for easy description.

  The display unit 140 is a display device that displays various images, and includes, for example, a touch panel. For example, an image generated by an imaging operation is displayed on the display unit 140. As the display unit 140, for example, an LCD (Liquid Crystal Display) panel, an organic EL (Electro Luminescence) panel, or the like can be used. Here, the horizontal to vertical ratio of the display unit 140 included in an information processing apparatus with a camera (for example, a smartphone with a compound eye camera) or a general imaging apparatus is often 4: 3 or 16: 9. Therefore, in the first embodiment of the present technology, an example in which the horizontal to vertical ratio of the display unit 140 is set to 4: 3 will be described.

  Here, as shown in b to d of FIG. 1, a state in which the longitudinal direction of the information processing apparatus 100 is the same as the vertical direction (a direction parallel to the gravity direction) is referred to as a vertically long state of the information processing apparatus 100. Further, the information processing apparatus 100 in the vertically long state with the longitudinal direction of the information processing apparatus 100 being the same as the horizontal direction (direction perpendicular to the gravity direction) (that is, the axis orthogonal to the display surface of the display unit 140 is the rotation axis) Is in a horizontally long state of the information processing apparatus 100.

  In this way, by using the compound eye information processing apparatus 100, for example, when shooting a panoramic image of a still image, a panoramic image is obtained by connecting images captured simultaneously with a plurality of imaging systems immediately after pressing the shutter button. Can be formed. There is a monocular imaging device that can generate a panoramic image by an operation (so-called panning operation) that rotates the imaging device in the horizontal direction around the imaging position (photographer's position) as a rotation center. When a panoramic image is generated by this monocular imaging device, a panning operation is required as described above, and therefore a panoramic image cannot be generated immediately after the shutter button is pressed. In contrast, the information processing apparatus 100 can capture a panoramic image in a shorter time than the monocular imaging apparatus.

[Example of internal configuration of information processing device]
FIG. 2 is a diagram illustrating an internal configuration example of the information processing apparatus 100 according to the embodiment of the present technology.

  The information processing apparatus 100 includes an application processor 11, a digital baseband processing unit 12, an analog baseband processing unit 13, and an RF (Radio Frequency) processing unit 14. In addition, the information processing apparatus 100 includes a battery 15, a microphone 16, a speaker 17, an antenna 18, a changeover switch 111, a determination key 112, an imaging unit 130, and a display unit 140. The information processing apparatus 100 also includes an attitude detection unit 150, a program memory 160, an image memory 170, a recording medium 180, and a DSP (Digital Signal Processor) 200. The RF processing unit 14 includes an antenna 18, and the analog baseband processing unit 13 includes a microphone 16 and a speaker 17.

  The application processor 11 controls each unit of the information processing apparatus 100 based on various programs stored in a built-in memory. The application processor 11 includes, for example, a central processing unit (CPU), a read only memory (ROM), and a random access memory (RAM).

  For example, when a receiving operation is performed, the radio wave received by the antenna 18 is demodulated by the digital baseband processing unit 12 via the RF processing unit 14 and the analog baseband processing unit 13. Then, the demodulation result by the digital baseband processing unit 12 is output from the speaker 17 via the analog baseband processing unit 13.

  When a transmission operation is performed, the voice input from the microphone 16 is modulated by the digital baseband processing unit 12 via the analog baseband processing unit 13. Then, the demodulated audio data is transmitted from the antenna 18 via the analog baseband processing unit 13 and the RF processing unit 14.

  Further, when the user performs an operation to start the imaging operation, the information processing apparatus 100 performs the imaging operation. For example, when the user performs an imaging operation start instruction operation, the application processor 11 instructs each unit (imaging unit 130, DSP 200, etc.) related to the imaging operation to start the imaging operation, and activates each unit. Then, an imaging operation is performed by each activated unit, and the generated image is displayed on the display unit 140. Here, when an image recording instruction operation is performed by the user, the generated image is recorded on the recording medium 180. When the user performs an instruction operation for wirelessly transmitting an image, the generated image is wirelessly transmitted. For example, the generated image data is modulated by the digital baseband processing unit 12 and transmitted from the antenna 18 via the analog baseband processing unit 13 and the RF processing unit 14. The battery 15 is a battery that supplies power to the information processing apparatus 100.

  Note that the selector switch 111, the determination key 112, the imaging unit 130, the display unit 140, the attitude detection unit 150, the program memory 160, the image memory 170, the recording medium 180, and the DSP 200 will be described in detail with reference to FIGS. Explained.

[Example of internal configuration of imaging unit]
FIG. 3 is a diagram illustrating an internal configuration example of the imaging unit 130 according to the embodiment of the present technology. FIG. 3 shows a part of the DSP 200 connected to the imaging unit 130. The overall configuration of the DSP 200 will be described in detail with reference to FIGS.

  The imaging unit 130 includes three imaging systems (first imaging system 191 to third imaging system 193), a power supply control unit 207, and power supply units 208 and 209. The three imaging systems are arranged side by side in a specific direction. That is, the first imaging system 191 is arranged in the center, and the second imaging system 192 and the third imaging system 193 are arranged on both sides of the first imaging system 191.

  The first imaging system 191 includes an optical system 131, an imaging element 134, and an I / F (interface) 137 with the DSP. The second imaging system 192 includes an I / F 138 with the optical system 132, the imaging element 135, and the DSP. The third imaging system 193 includes an I / F 139 with the optical system 133, the imaging element 136, and the DSP. Note that the configurations of the first imaging system 191 to the third imaging system 193 are substantially the same, so here, the configuration of the first imaging system 191 will be mainly described, and the second imaging system 192 and the third imaging system 193 will be described. The description of the imaging system 193 is omitted.

  The optical system 131 includes a plurality of lenses (including a zoom lens and a focus lens) that collect light from the subject. Further, the amount of light (that is, exposure) that has passed through each of these lenses is adjusted by a diaphragm (not shown). Then, the condensed light from the subject enters the image sensor 134.

  The imaging device 134 is an imaging device that forms an image signal by forming a subject image incident via the optical system 131. That is, the image sensor 134 receives light from a subject incident through the optical system 131 and performs photoelectric conversion, thereby generating an analog image signal corresponding to the amount of received light. In this way, the analog image signal generated by the image sensor 134 is supplied to the DSP 200 via the I / F 137 with the DSP. For example, a CCD (Charge Coupled Device) type or CMOS (Complementary Metal Oxide Semiconductor) type solid-state imaging device can be used as the imaging device.

  The DSP I / F 137 is an interface for connecting the image sensor 134 and the DSP 200.

  The power control unit 207 controls the power supply units 208 and 209 based on a power control instruction from the imaging control unit 201 (shown in FIG. 7) of the DSP 200. In other words, when the power supply control unit 207 receives a power control instruction from the imaging control unit 201, the power supply units 208 and 209 adjust the signal amplitude, the signal rising speed and the signal falling speed required as the input specification of the control signal. Create a compliant signal. Then, the generated signal is output to the power supply units 208 and 209 to control the power supply units 208 and 209. In addition, when the output signal of the imaging control unit 201 conforms to the input specification of the control signal of the power supply units 208 and 209, the output signal of the imaging control unit 201 is directly input to the power supply units 208 and 209. You may do it.

  The power supply unit 208 supplies power to the first imaging system 191 based on the control of the power control unit 207. The power supply unit 209 supplies power to the second imaging system 192 and the third imaging system 193 based on the control of the power control unit 207. The power supply units 208 and 209 are realized by, for example, a commercially available power supply IC (Integrated Circuit).

  Each of the first imaging system 191 to the third imaging system 193 is connected to the DSP 200 through one data line and seven types of signal lines. Here, one data line connecting the first imaging system 191 and the DSP 200 will be described as L1, and seven types of signal lines will be described as signal lines L2 to L8. The data lines and signal lines of the second imaging system 192 and the third imaging system 193 are substantially the same as the data lines and signal lines of the first imaging system 191. Therefore, here, mainly the data lines and signal lines of the first imaging system 191 will be described, and descriptions of the second imaging system 192 and the third imaging system 193 will be omitted.

  The data line L1 is a data line that transmits image data from the image sensor 134 to the DSP 200. For example, the data line L1 is preferably composed of a plurality of data lines in order to increase the transmission rate of image data. In order to increase the transmission rate of the image data and to increase the noise resistance on the transmission path, it is preferable to use a high-speed and differential transmission type data line as the data line L1. For example, it is preferable to use LVDS (low voltage differential signaling) or the like as the data line L1.

  The signal line L2 is a bidirectional communication line between the image sensor 134 and the DSP 200, and for example, a four-line serial communication line can be used. The signal line L2 is used, for example, when setting various setting values necessary for using the image sensor 134 from the DSP 200 side. As an example, setting values for thinning out and outputting image data output from the image sensor 134 to the DSP 200 are written from the DSP 200 to the registers 370 and 380 (shown in FIG. 12) via the signal line L2.

  The signal line L3 is a clock signal line that supplies a clock from the DSP 200 to the image sensor 134. The imaging element 134 performs an imaging operation for one pixel per clock cycle using a clock supplied via the signal line L3. Alternatively, a multiplier may be mounted in the image sensor 134, and the clock supplied from the DSP 200 may be multiplied in the image sensor 134 to perform an imaging operation for one pixel per one cycle of the multiplied clock.

  The signal line L4 is a reset signal line that supplies a reset signal from the DSP 200 to the image sensor 134.

  The signal line L5 is a signal line for controlling on / off of the imaging operation of the imaging element 134 from the DSP 200. That is, the signal line L5 is a signal line for notifying the stop and activation of the operation from the DSP 200 to each image sensor. For example, when an imaging mode in which only one image sensor is used among three imaging systems is instructed by the user, power consumption is reduced by stopping the imaging operation of two image sensors that are not used. be able to.

  The signal line L6 is a vertical synchronization signal line. That is, the signal line L6 is a signal line for notifying the synchronization signal indicating the imaging timing for each frame from the DSP 200 to the imaging device 134.

  The signal line L7 is a horizontal synchronization signal line. That is, the signal line L7 is a signal line for notifying the synchronization signal indicating the imaging timing for each line in one frame from the DSP 200 to the imaging device 134.

  The signal line L8 is a shutter signal line. For example, in the information processing apparatus 100, when an operation member (for example, the enter key 112) for recording a captured image is pressed by the user, a shutter signal corresponding to the press is sent from the DSP 200 via the signal line L8. The image sensor 134 is notified.

  In addition, when the imaging mode is set, it is possible to select either a monocular imaging operation or a compound eye imaging operation based on a user operation. When the monocular imaging operation is selected, an imaging operation using image data generated by the first imaging system 191 is performed. On the other hand, when the compound eye imaging operation is selected, the imaging operation using the image data generated by each of the first imaging system 191 to the third imaging system 193 is performed. In addition, when a monocular imaging operation is selected, a normal image is generated, and when a compound eye imaging operation is selected, a panoramic image (for example, shown in FIG. 29b) is generated.

[Example of image sensor arrangement]
FIG. 4 is a diagram illustrating an arrangement configuration example of the imaging elements 134 to 136 constituting the imaging unit 130 according to the embodiment of the present technology.

  FIG. 4 a shows an arrangement configuration of the image sensors 134 to 136. Here, in general, the shape of the region in which the pixels are arranged on the light receiving surface of the image sensor is substantially rectangular. Therefore, in the following, the image sensors 134 to 136 are schematically represented by rectangles.

  FIG. 4A shows an arrangement configuration example in the case where the longitudinal direction of the information processing apparatus 100 is matched with the vertical direction. Specifically, the image sensor 134 is disposed in the center, and the image sensors 135 and 136 are disposed on both sides of the image sensor 134. The image sensors 134 to 136 are arranged so that the longitudinal direction thereof substantially coincides with the arrangement direction. Further, the image sensors 134 to 136 are arranged so that the center positions thereof are on the same plane. That is, the image sensors 134 to 136 are arranged horizontally in the arrangement direction.

  In FIG. 4 a, pixel data readable areas 400 to 402 from which each pixel can be read are schematically shown as rectangles in the imaging elements 134 to 136. For example, the imaging element 134 generates a first captured image whose longitudinal direction is the specific direction of the subject. In addition, the imaging element 135 generates a second captured image including a subject adjacent to one of the subjects included in the first captured image, with the specific direction being the longitudinal direction. In addition, the imaging element 136 generates a third captured image including a subject adjacent to the other of the subjects included in the first captured image with the specific direction as the longitudinal direction.

  Of the three image sensors constituting the imaging unit, the center image sensor may be arranged so that the longitudinal direction of the center image sensor substantially coincides with the direction orthogonal to the arrangement direction (for example, a special feature). (See Kai 2011-44837.)

  FIG. 4B shows an example of a pixel data reading area when image data is generated when the information processing apparatus 100 is in the vertically long state. The pixel data reading areas 403 to 405 are examples of areas for reading out each pixel when generating image data used for display or recording in the pixel data readable areas 400 to 402. In FIG. Shown in bold lines. The pixel data read areas 403 to 405 can be the same as the pixel data readable areas 400 to 402, for example. In addition, the pixel data readout areas 403 to 405 can have a vertical length of V10 and a horizontal length of H10, for example.

  5 and 6 are diagrams illustrating a relationship between an image formed on the imaging elements 134 to 136 and an image data reading method according to the embodiment of the present technology. In general, a so-called inverted image is formed on the image sensor.

  FIG. 5 schematically illustrates the relationship between the subject 409, the optical system 131, and the captured image 410 formed on the image sensor 134 when an inverted image is generated as an image formed on the image sensor. As shown in FIG. 5, the light from the subject 409 that has entered through the optical system 131 is imaged on the image sensor 134, and a captured image 410 is generated. In this case, the vertical direction of the subject 409 and the vertical direction of the captured image 410 are reversed.

  FIG. 6A schematically shows reading start positions 411 to 413 and a reading direction when pixel data is read from the image sensors 134 to 136. In this example, pixel data is read in the pixel data reading areas 403 to 405 shown in FIG. In the following, the reading start position in the pixel data reading area is schematically shown by a rectangle. For example, reading is sequentially started from reading start positions 411 to 413 at the lower right corners of the pixel data reading areas 403 to 405, and pixel data is read in the direction of the arrow. For example, in the pixel data readout region 403, readout of pixel data is started from the readout start position 411, and readout of pixel data is sequentially performed while shifting one pixel at a time in the arrow direction. When a pixel located at the end of one horizontal line (the left end in the pixel data reading area 403 shown in FIG. 6A) is read, the line to be read is moved upward by one pixel. The pixel located at the other end is read out by shifting. Thereafter, pixel readout is sequentially performed in the same manner. When the pixel located at the end of the uppermost line in the pixel data reading area 403 is read, the reading process of the image sensor 134 is terminated. Further, readout processing is also performed on the image sensors 135 and 136 at the same time.

  FIG. 6B schematically shows a composite image 414 on the image memory 170 when the pixel data read from the image sensors 134 to 136 are combined. In this example, an example in which pixel data read by the reading method shown in FIG. It is assumed that the arrow direction in the composite image 414 is the arrangement direction of the image data on the image memory 170.

  FIG. 6 c shows a display example of a composite image when pixel data read from the image sensors 134 to 136 is displayed on the display unit 140. In this example, an example of displaying a composite image 414 shown in FIG. For example, the composite image 414 is displayed in the captured image display area 415, and one-color images (for example, black and white) are displayed in the upper and lower margin image display areas 416 and 417. Note that the arrow direction in the display unit 140 is the scanning direction in the display unit 140.

[DSP configuration example]
FIG. 7 is a block diagram illustrating an internal configuration example of the DSP 200 according to the embodiment of the present technology.

  The DSP 200 includes an imaging control unit 201, a CPU 202, a DMA (Direct Memory Access) controller 203, a data bus 204, an I / F 205 for a program memory, and an I / F 206 for an image memory. The DSP 200 also includes an I / F 210 for the image sensor, image buffers 211 to 219, an image signal processing unit 220, resolution conversion units 231, 241, and 251, and image rotation processing units 232 and 242. The DSP 200 also includes an I / F 233 for a display unit, an I / F 243 for an external display device, an encoding / decoding unit 252, an I / F 253 for a recording medium, oscillation circuits 264 to 266, and clock generation. Circuit 270. The DSP 200 includes an AF (Auto Focus) control unit 281 and a face detection unit 282. The DSP 200 includes an AE (Automatic Exposure) control unit 283 and an AWB (Auto White Balance) control unit 284. The data bus 204 is connected to a CPU 202, a DMA controller 203, an I / F 206 with an image memory, image buffers 211 to 219, an image signal processing unit 220, and the like. In addition, each signal from the changeover switch 111, the enter key 112, and the posture detection unit 150 is input to the imaging control unit 201.

  The posture detection unit 150 detects a change in posture of the information processing device 100 by detecting acceleration, movement, inclination, and the like of the information processing device 100, and a detection result (posture information regarding the detected posture change). Is output to the imaging control unit 201. For example, the posture detection unit 150 detects a rotation angle of three axes (for example, the X axis, the Y axis, and the Z axis) as a change in the posture of the information processing apparatus 100 and outputs the detection result to the imaging control unit 201. . As the attitude detection unit 150, a sensor that can detect the rotation angles of the three axes in the information processing apparatus 100 can be used. However, at least a sensor capable of detecting a rotation angle of one axis or more may be used. For example, as the posture detection unit 150, a fall sensor, a gravity sensor, a gyro sensor, an acceleration sensor that can detect the direction of acceleration, and an angular velocity sensor that can detect rotational motion can be used.

  The imaging control unit 201 controls each unit related to imaging processing. For example, the imaging control unit 201 determines the attitude of the information processing apparatus 100 based on the detection result from the attitude detection unit 150, and performs imaging control of each unit based on the determination result. In addition, the imaging control unit 201 performs imaging control of each unit based on input signals from the changeover switch 111 and the determination key 112.

  Here, in the embodiment of the present technology, the user can set in advance an imaging mode (image size or the like) for recording an image generated by the imaging unit 130. For example, a menu screen for setting the imaging mode is displayed on the display unit 140, and the setting content desired by the user is input using the enter key 112 on the menu screen. Note that the imaging mode includes, for example, the number of imaging elements used during imaging, and the vertical image size and horizontal image size of the image during recording. In addition, the imaging mode includes, for example, a vertical back porch and a vertical front porch that represent the interval between the effective area of the image and the vertical synchronizing signal, and a horizontal back porch and a horizontal front that represent the interval between the effective area of the image and the horizontal synchronizing signal. Including pouch. In addition, the imaging control unit 201, each unit in the DSP 200, and the imaging elements 134 to 136 each include a register that holds an imaging mode.

  Here, when the imaging mode is set by the user, the imaging control unit 201 notifies each of the units in the DSP 200 and the imaging elements 134 to 136 of the set imaging mode, and stores them in a register included in each unit. As described above, by causing the register included in each unit to hold the setting content of the imaging mode set by the user, the user can easily switch and use a plurality of shooting conditions.

  For example, the imaging control unit 201 notifies each unit in the DSP 200 and the imaging elements 134 to 136 of the vertical synchronization signal, the horizontal synchronization signal, and the clock signal based on the setting contents of the imaging mode held in the built-in register. . In addition, the imaging control unit 201 sends, for example, a vertical synchronization signal, a horizontal synchronization signal, and a clock signal to each display-related unit in the DSP 200 and the display unit 140 based on the setting contents of the imaging mode held in the built-in register. Notice. In addition, the imaging control unit 201 outputs a signal for controlling power on / off to the power control unit 207, for example.

  The CPU 202 controls the entire DSP 200 based on various programs stored in the program memory 160. For example, the CPU 202 detects whether the information processing apparatus 100 (display unit 140) is in a horizontally long state or a vertically long state based on a detection result (posture information) from the posture detection unit 150, and the detected state is set. Display control is performed based on this. The details of this control will be described in detail with reference to FIG.

  The DMA controller 203 controls data transfer between the memories based on the control of the CPU 202.

  The program memory I / F 205 is an interface for connecting the program memory 160 and the DSP 200.

  An I / F 206 with the image memory is an interface for connecting the image memory 170 and the DSP 200.

  An I / F 210 with the image sensor is an interface for connecting the image sensors 134 to 136 and the DSP 200. That is, image data generated by the image sensors 134 to 136 is input to the I / F 210 with the image sensor. For example, when the data line L1 for transmitting image data from the image sensors 134 to 136 is a LVDS system with a small amplitude, the image data from the image sensors 134 to 136 is GND in the I / Fs 137 to 139 with the DSP. It is converted into a potential or a power supply potential. Further, three systems of image buffers 211 to 219 corresponding to the image sensors 134 to 136 are provided in the subsequent stage of the I / F 210 with the image sensor.

  The image buffers 211 to 219 are image buffers that hold the image data output from the image sensors 134 to 136, and the held image data is written to the image memory 170 via the data bus 204. For example, each image sensor includes three image buffers, and these image buffers are connected to the data bus 204. For example, the image sensor 134 includes three image buffers 211 to 213. The image sensor 135 is provided with three image buffers 214 to 216. Further, the image sensor 136 is provided with three image buffers 217 to 219. Here, in the embodiment of the present technology, since image data is written into the image memory 170, newly input images from the image sensors 134 to 136 are read even while the image data is being read from the image buffers 211 to 219. Hold data sequentially. For this reason, it is preferable that two or more image buffers are provided for each of the image sensors 134 to 136 as the image buffers 211 to 219.

  One capacity of the image buffers 211 to 219 is preferably larger than the bit width of the data bus 204. For example, when the data bus 204 is 128 bits wide, each image buffer preferably has a capacity of 128 bits or more. Further, it is more preferable that one capacity of the image buffers 211 to 219 is at least twice the bit width of the data bus 204. For example, when the data bus 204 is 128 bits wide, it is more preferable that each image buffer has a capacity of 256 bits or more.

  On the other hand, one capacity of the image buffers 211 to 219 can be equal to or less than the image data amount of one image generated by one image sensor. For example, one capacity of the image buffers 211 to 219 is preferably equal to or less than the data amount of image data generated by pixels for one line of the image sensor 134.

  In the embodiment of the present technology, the bit width of the data line connecting the imaging elements 134 to 136 and the DSP 200 is, for example, 12 bits. For example, the bit width of the data bus 204 of the DSP 200 is 128 bits, and the capacity of one of the image buffers 211 to 219 is 128 bits.

  The image signal processing unit 220 performs various image signal processes on the image data input via the image buffers 211 to 219 and the data bus 204 based on the control of the imaging control unit 201. The internal configuration of the image signal processing unit 220 will be described in detail with reference to FIG.

  The resolution conversion unit 231 performs resolution conversion for displaying each image on the display unit 140 based on the control of the imaging control unit 201 or the CPU 202, and the image data subjected to the resolution conversion is converted into an image rotation processing unit. Output to H.232.

  The resolution conversion unit 241 performs resolution conversion for displaying each image on the external display device 245 based on the control of the imaging control unit 201 or the CPU 202, and performs image rotation processing on the image data subjected to the resolution conversion. To the unit 242.

  The image rotation processing unit 232 performs rotation processing on the image data that has been subjected to resolution conversion based on the control of the imaging control unit 201 or the CPU 202, and the image data that has been subjected to the rotation processing is connected to the display unit. / F233 output.

  The image rotation processing unit 242 performs rotation processing on the image data subjected to resolution conversion based on the control of the imaging control unit 201 or the CPU 202. The image rotation processing unit 242 transmits the image data subjected to the rotation processing to the external display device. Output to the I / F 243.

  The I / F 233 with the display unit is an interface for connecting the display unit 140 and the DSP 200.

  The I / F 243 with the external display device is an interface for connecting the external display device 245 and the DSP 200. The external display device 245 is, for example, a television.

  The resolution conversion unit 251 converts the resolution of each image for recording based on the control of the imaging control unit 201 or the CPU 202, and outputs the image data subjected to the resolution conversion to the encoding / decoding unit 252. . For example, the resolution conversion unit 251 performs a resolution conversion process for converting the resolution to a recording image size desired by the user and a resolution conversion process for generating a thumbnail image.

  The encoding / decoding unit 252 performs encoding for compressing the image data output from the resolution conversion unit 251 based on the control of the imaging control unit 201 or the CPU 202, and converts the encoded image data into Output to the I / F 253 with the recording medium. Also, when displaying the image data recorded on the recording medium 180 on the display unit 140, the encoding / decoding unit 252 passes the image data recorded on the recording medium 180 via the I / F 253 with the recording medium. To input and decrypt. The decoded image data is held in the image memory 170.

  The I / F 253 with the recording medium is an interface for connecting the recording medium 180 and the DSP 200.

  The recording medium 180 is a recording medium that records image data supplied via the I / F 253 with the recording medium. The recording medium 180 may be built in the information processing apparatus 100 or may be detachably attached to the information processing apparatus 100. As the recording medium 180, for example, a tape (for example, a magnetic tape) or an optical disk (for example, a recordable DVD (Digital Versatile Disc)) can be used. Further, as the recording medium 180, for example, a magnetic disk (for example, hard disk), a semiconductor memory (for example, memory card), or a magneto-optical disk (for example, MD (MiniDisc)) may be used. The image data recorded on the recording medium 180 will be described in detail with reference to FIG.

  Note that the oscillation circuits 264 to 266 and the clock generation circuit 270 will be described in detail with reference to FIGS.

  The AF control unit 281 performs focus control on the image data input via the image buffers 211 to 219 and the data bus 204 so as to focus on a subject included in a predetermined area in the image (captured image). is there. Note that the predetermined area may be, for example, an area including a middle area of the captured image, an area specified by the user, or a face position detected by the face detection unit 282. When there are a plurality of predetermined areas, focus control is performed for each predetermined area. Then, the CPU 202 and the imaging control unit 201 output information related to the focused position (in-focus position) in the captured image. Also, information regarding the focus position is held in the AF control unit 281.

  The face detection unit 282 detects the face of a person included in the image (captured image) of the image data input via the image buffers 211 to 219 and the data bus 204, and the detection result is sent to the CPU 202 and The image is output to the imaging control unit 201. Further, the face detection unit 282 may detect the face of a person included in the image data read from the recording medium 180. As a method for detecting a face included in an image, for example, a face detection method (for example, see Japanese Patent Application Laid-Open No. 2004-133637) using matching between a template in which face luminance distribution information is recorded and a content image is used. it can. In addition, a face detection method based on a skin color part included in an image or a feature amount of a human face can be used. With these face detection methods, the position and size of a person's face image can be obtained. Further, the face detection result is held in the face detection unit 282. The face detection unit 282 is an example of a specific object detection unit.

  The AE control unit 283 is an automatic exposure control unit for automatically adjusting the shutter speed and the aperture value of the image data input via the image buffers 211 to 219 and the data bus 204, and the detection result is sent to the CPU 202. And output to the imaging control unit 201.

  The AWB control unit 284 performs auto white balance adjustment on image data input via the image buffers 211 to 219 and the data bus 204, and outputs the detection result to the CPU 202 and the imaging control unit 201.

[Example of Internal Configuration of Image Signal Processing Unit 220]
FIG. 8 is a block diagram illustrating an internal configuration example of the image signal processing unit 220 according to the embodiment of the present technology.

  The image signal processing unit 220 includes a pixel addition processing unit 221, a demosaic processing unit 222, a YC conversion processing unit 223, an image composition processing unit 224, a sharpness processing unit 225, a color adjustment processing unit 226, and an RGB conversion process. Part 227.

  The pixel addition processing unit 221 performs pixel addition processing and pixel thinning processing on the image data generated by the imaging elements 134 to 136.

  The demosaic processing unit 222 performs demosaic processing (interpolation processing) so that the intensities of all the R, G, and B channels are aligned at each pixel position of the image data (mosaic image) generated by the imaging elements 134 to 136. It is. Then, the demosaic processing unit 222 supplies the RGB image subjected to the demosaic processing to the YC conversion processing unit 223. That is, the demosaic processing unit 222 interpolates Bayer data having only pixel data for one color per pixel, and calculates three pixel data of RGB for one pixel.

  The YC conversion processing unit 223 generates a luminance signal (Y) and a color difference signal (Cr, Cb) by performing YC matrix processing and band limitation on chroma components on the RGB image generated by the demosaic processing unit 222. is there. Then, the generated luminance signal (Y image) and color difference signal (C image) are supplied to the image composition processing unit 224.

  The image composition processing unit 224 performs image composition on the image data generated by the YC conversion processing unit 223, and outputs the synthesized image data to the sharpness processing unit 225. This image composition processing will be described in detail with reference to FIGS.

  The sharpness processing unit 225 performs sharpness processing (processing for enhancing the contour of a subject) that extracts and emphasizes a portion with a large signal change in the image data generated by the image composition processing unit 224. Then, the sharpness processing unit 225 supplies the image data subjected to the sharpness processing to the color adjustment processing unit 226.

  The color adjustment processing unit 226 adjusts the hue and saturation of the image data that has been sharpened by the sharpness processing unit 225.

  The RGB conversion processing unit 227 converts the image data that has been adjusted in hue and saturation by the color adjustment processing unit 226 from YCbCr data to RGB data.

  Here, the flow of the image data of the signal of the image signal processing unit 220 will be described. For example, assume that each signal processing unit in the image signal processing unit 220 directly reads image data from the image memory 170 via the data bus 204 and writes the image data after signal processing to the image memory 170 via the data bus 204. To do. In this case, it is advantageous in that the image signal processing unit 220 can read image data at a desired position in the image data at a desired timing. However, since the amount of data that needs to be transmitted via the data bus 204 increases, the operating frequency of the data bus 204 needs to be increased. This makes it difficult to design the data bus 204 and may increase power consumption.

  Further, for example, each signal processing unit in the image signal processing unit 220 receives image data from the preceding signal processing unit without passing through the data bus 204, and the image data after the signal processing is sent to the subsequent signal processing unit in the data bus 204. Assume a case of passing without going through. In this case, since the data bus 204 is not used, it is advantageous in that LSI design is easy and power consumption can be reduced. However, each signal processing unit may not be able to read image data at a desired position in the image data at a desired timing.

  Therefore, in the embodiment of the present technology, between the demosaic processing unit 222 and the color adjustment processing unit 226 having a substantially constant image size, the image data is processed in order to reduce the operating frequency of the data bus 204 and power consumption. Direct transfer between each signal processing unit. In addition, an example in which image data is written to the image memory 170 in the previous stage of a signal processing unit that uses a large amount of image data, such as resolution conversion, and desired image data is read from the image memory 170 when performing resolution conversion. Show.

[Configuration example of clock generation circuit]
FIG. 9 is a block diagram illustrating an internal configuration example of the clock generation circuit 270 according to the embodiment of the present technology.

  The clock generation circuit 270 includes high-frequency clock multipliers 20 and 24, high-frequency clock dividers 21 and 25, low-frequency clock multipliers 22 and 26, and low-frequency clock dividers 23 and 27. Prepare. The clock generation circuit 270 includes clock multipliers 28 and 30 and clock frequency dividers 29 and 31. Here, each multiplier multiplies the frequency of the input clock. Each frequency divider makes the frequency of the input clock 1 / n (n is an arbitrary integer). In this example, the case where the clock generation circuit 270 generates at least six types of clocks according to the connection destination of each unit in the DSP 200 shown in FIG. 7 is shown as an example.

  The oscillators 261 to 263 are vibration sources for generating a clock signal supplied to the DSP 200. For example, crystal oscillators are used.

  The oscillation circuits 264 to 266 generate a clock signal supplied to the DSP 200, and output the generated clock signal to the clock generation circuit 270.

  Two of the six types of clocks generated by the clock generation circuit 270 are clocks supplied to the image sensors 134 to 136. One of the clocks supplied to the image sensors 134 to 136 is a clock having a relatively large frequency for generating an image having a relatively large number of pixels. This clock is generated by the clock output from the oscillation circuit 264 being input to the high-frequency clock multiplier 20 and multiplied, and the multiplied clock being input to the high-frequency clock divider 21 and being divided. Is done. Another type is a clock having a relatively low frequency for generating an image having a relatively small number of pixels. The clock output from the oscillation circuit 264 is input to the low-frequency clock multiplier 22 and multiplied, and the multiplied clock is input to the low-frequency clock divider 23 and divided. Is generated by The clocks divided by the high-frequency clock divider 21 and the low-frequency clock divider 23 are output as clocks generated by the clock generation circuit 270, and the imaging elements 134 to 136 are passed through the DSP 200. To be supplied. Here, the clocks supplied to the imaging elements 134 to 136 are not limited to the two types shown in this example, and it is preferable to generate and use more types according to the image size generated by the imaging operation.

  The other two types of the six types of clocks generated by the clock generation circuit 270 are clocks used in the DSP 200. One type of clock used in the DSP 200 is a clock having a relatively large frequency for generating an image having a relatively large number of pixels. This clock is generated by the clock output from the oscillation circuit 264 being input to the high-frequency clock multiplier 24 and being multiplied, and the multiplied clock being input to the high-frequency clock divider 25 for frequency division. Is done. Another type is a clock having a relatively low frequency for generating an image having a relatively small number of pixels. The clock output from the oscillation circuit 264 is input to the low frequency clock multiplier 26 and multiplied, and the multiplied clock is input to the low frequency clock divider 27 and divided. Is generated by Then, the clock divided by the high frequency clock divider 25 and the low frequency clock divider 27 is output as a clock generated in the clock generation circuit 270 and supplied to the DSP 200. Here, the clocks used in the DSP 200 are not limited to the two types shown in this example, and it is preferable to generate and use more types according to the image size generated by the imaging operation.

  The remaining two types of the six types of clocks generated by the clock generation circuit 270 are a pixel clock for displaying an image on the display unit 140 and a display device outside the information processing apparatus 100 (for example, the external display device 245). This is a pixel clock for displaying an image. The pixel clock for displaying an image on the display unit 140 is multiplied by the clock output from the oscillation circuit 265 being input to the clock multiplier 28, and the multiplied clock is input to the clock divider 29. Generated by frequency division. The pixel clock for displaying an image on a display device outside the information processing apparatus 100 is multiplied by the clock multiplier 30 by the clock output from the oscillation circuit 266, and this multiplied clock is the clock divider. It is generated by being inputted to 31 and divided. The clock divided by the clock divider 29 is output as a clock generated by the clock generation circuit 270 and supplied to the display unit 140 via the DSP 200. The clock divided by the clock divider 31 is output as a clock generated by the clock generation circuit 270 and supplied to a display device outside the information processing apparatus 100 via the DSP 200. Here, the clocks for displaying these images are not limited to the two types shown in this example, and it is preferable to generate and use more types according to the specifications of the connected display device.

  10 and 11 are block diagrams illustrating modifications of the clock generation circuit 270 according to the embodiment of the present technology.

  In FIG. 10, two types of clocks supplied to the imaging devices 134 to 136 share one multiplier (clock multiplier 32), and two types of clocks supplied to the DSP 200 are one. An example of sharing and using a multiplier (clock multiplier 33) is shown.

  In FIG. 11, a plurality of types of clocks supplied to the imaging devices 134 to 136 and a plurality of types of clocks supplied to the DSP 200 share one multiplier (clock multiplier 34). An example is shown.

[Configuration Example of Image Sensor and Pixel Reading Example]
FIG. 12 is a diagram illustrating an internal configuration example of the imaging element according to the embodiment of the present technology. Here, since the internal configurations of the image sensors 134 to 136 are substantially the same, only the image sensor 134 is shown in FIG. 12, and illustration and description of other image sensors are omitted. In FIG. 12, a CMOS type image sensor will be described as an example of the image sensor 134.

  The image sensor 134 includes pixels 40 to 47, a vertical scanning circuit 340, and a horizontal scanning circuit 345. The image sensor 134 includes ADCs (A / D (Analog / Digital) converters) 350 to 353, adders 354 to 357 and 366, and column latches 358 to 361. The image sensor 134 includes switches 362 to 365, an output latch 367, an output circuit 368, registers 370 and 380, and multipliers / dividers 391 and 392. In general, the vertical arrangement of image sensors is referred to as a column, and the horizontal arrangement is referred to as a row. For this reason, in the following description, the names of columns and rows are used as appropriate. Further, in this example, in the image sensor 134, some pixels (pixels 40 to 47) and respective parts related thereto are shown as representatives, and illustration and description of other configurations are omitted.

  In the image sensor 134, vertical control lines 341 to 344 are wired in the row direction, and every other pixel on the same line is connected to the same vertical control line. Further, data readout lines 346 to 349 are wired in the column direction, and pixels on the same line share one readout line.

  The vertical scanning circuit 340 turns on / off the switches between the pixels 40 to 47 and the data readout lines 346 to 349 by the vertical control lines 341 to 344 wired in the row direction. That is, in each pixel in the row direction, every other pixel among the pixels on the same line in the row direction is turned on / off in common by one vertical control line. Further, the image data of the pixels 40 to 47 is output to the data read lines 346 to 349 via the switches between the respective pixels and the corresponding data read lines.

  The horizontal scanning circuit 345 turns on / off switches 362 to 365 between the column latches 358 to 361 and the output data line 369. By selecting ON / OFF of the switch by the vertical scanning circuit 340 and ON / OFF of the switches 362 to 365 by the horizontal scanning circuit 345, signals of all the pixels can be read out in time division while sequentially selecting each pixel. Here, the output data line 369 is an output data line for outputting the output result of each column from the image sensor 134.

  Here, in the image sensor 134, the pixels 40 to 47 are arranged in a two-dimensional square lattice. Since the configurations of the pixels 40 to 47 are the same, the pixel 40 will be described as an example here. The pixel 40 includes a photodiode 51 that is a light receiving unit, an amplifier 52, and a switch 53. The photodiode 51 converts light emitted to the pixel into electric charges corresponding to the amount of light. The amplifier 52 is an amplifier that amplifies the charge signal converted by the photodiode 51. The switch 53 is a switch that controls the charge transfer of the pixel 40 in accordance with on / off of the vertical control line 342.

  Each column includes ADCs 350 to 353, adders 354 to 357, and column latches 358 to 361. Hereinafter, the ADC 350, the adder 354, and the column latch 358 connected to the data read line 346 will be described as an example.

  The ADC 350 is an AD converter that converts image data from each pixel that is an analog value into digital data (digital value).

  Each time digital data is converted by the ADC 350, the adder 354 adds new digital data after the conversion to the digital data stored in the column latch 358.

  The column latch 358 is a column latch that sequentially stores the digital data converted by the ADC 350. Here, the column latch is a name indicating a data storage circuit that stores digital data after AD conversion. As the data storage circuit, in addition to the latch configured by a linear circuit, each circuit capable of storing digital data such as a flip-flop configured by a synchronous circuit can be used.

  For example, the image data output from the pixel 40 passes through the ADC 350, the adder 354, and the column latch 358, and then is output to the output data line 390 via the switch 362 connected to the data read line 346. . Here, in the embodiment of the present technology, the adder 366 and the output latch 367 are provided in the output data line 390 as well as the data read line of each column to add and store the image data. Further, the image data stored in the output latch 367 is output to the output data line 369 via the output circuit 368. Note that the image data from the output data line 369 is output to the data line L1 described above.

  The multipliers / dividers 391 and 392 perform frequency multiplication of the input clock or frequency division of the input clock based on control from the DSP 200. Then, the multiplier / dividers 391 and 392 supply the generated clock to the vertical scanning circuit 340, the horizontal scanning circuit 345, and the output circuit 368.

  The signal line 393 is a vertical synchronization signal line that supplies a vertical synchronization signal from the DSP 200. The signal line 394 is a horizontal synchronization signal line that supplies a horizontal synchronization signal from the DSP 200.

  The signal line 395 is a clock signal line that supplies a clock signal from the DSP 200. The signal line 396 is a signal line for controlling on / off of the imaging operation from the DSP 200 and a signal line for controlling pixel thinning. A signal line 397 is a bidirectional communication line between the image sensor 134 and the DSP 200. A signal line 398 is a power supply line.

  Note that the registers 370 and 380 are registers that hold setting values related to the imaging operation, and an example of the held contents is shown in FIG.

  FIG. 13 is a diagram schematically illustrating the contents held in the registers 370 and 380 included in the image sensor 134 according to the embodiment of the present technology.

  The registers 370 and 380 hold setting values related to the imaging operation and are supplied to the vertical scanning circuit 340 and the horizontal scanning circuit 345. Note that these setting values may be changeable by a user operation.

  14 to 16 are timing charts schematically showing the state of the control signal to each pixel in the image sensor 134 and the data output from each pixel in the embodiment of the present technology.

  Here, the horizontal axis shown in FIGS. 14 to 16 is a time axis. Further, the vertical control lines 341 to 344 and the column latches 358 to 361 shown in FIGS. 14 to 16 will be described with the same reference numerals as those in FIG. Further, the column latch read signal shown in FIGS. 14 to 16 is described with the same reference numerals as the corresponding switches 362 to 365 shown in FIG.

  In the example illustrated in FIG. 14, an imaging operation in the case where all of the pixels in the image sensor 134 are read will be described.

  The vertical control lines 341 to 344 are used to output the pixel data of all the image sensors 134 connected to a certain row (for example, the lines of the pixels 40 to 43) to the data readout lines 346 to 349 of each column. Subsequently, the pixel data output to the data readout lines 346 to 349 are AD converted by the ADCs 350 to 353 of each column. Subsequently, the outputs of the ADCs 350 to 353 are held in the column latches 358 to 361 of the respective columns. For example, the pixel data d1 to d4 are held in the column latches 358 to 361 shown in FIG. Subsequently, the horizontal scanning circuit 345 sequentially turns on the readout switches 362 to 365 from the column latches 358 to 361 to the output data line 390 one column at a time. Thereby, each pixel data in one line can be read in order. For example, the pixel data d2 to d3 are sequentially output after the pixel data d1 is output to the output data line 390 shown in FIG.

  Similarly, every time readout of one line in the horizontal direction is completed, the vertical scanning circuit 340 sequentially turns on the readout switch from each pixel to the vertical signal line one row at a time. As a result, the pixel data of each row is input to the ADCs 350 to 353. After the input pixel data is AD-converted by the ADCs 350 to 353, each pixel data is held in the column latches 358 to 361 of each column. For example, the pixel data d5 to d8 are held in the column latches 358 to 361 shown in FIG. Subsequently, the horizontal scanning circuit 345 sequentially turns on the read switches 362 to 365 from the column latches 358 to 361 to the output data line 390 one column at a time, and sequentially reads each pixel data in one line. For example, the pixel data d5 to d8 are sequentially output to the output data line 390 shown in FIG.

  In the example illustrated in FIG. 15, an imaging operation in the case of performing horizontal thinning and reading out as an example of pixel thinning out of each pixel in the image sensor 134 will be described.

  The vertical scanning circuit 340 turns on the readout switch from each pixel to the vertical signal lines 346 to 349 only in a desired column. As a result, only pixel data of a specific row is input to the ADCs 350 to 353, and AD conversion is performed by the ADCs 350 to 353. The outputs of the ADCs 350 to 353 are held in the column latches 358 to 361 of the respective columns. For example, by turning on the readout switch connected to the vertical control lines 342 and 344, the pixel data d1 and d3 are held in the column latches 358 and 360 shown in FIG. After the readout of one row is completed, for example, by turning on the readout switch connected to the vertical control lines 342 and 344, the pixel data d5 and d7 are held in the column latches 358 and 360 shown in FIG. .

  In addition, the horizontal scanning circuit 345 turns on a read switch from the column latches 358 to 361 to the output data line 390 only in a desired column. Thereby, only specific pixel data in one line can be read in order.

  For example, when one pixel data is read out from N pieces of pixel data in the horizontal direction, 1 / N thinning-out reading is performed in the horizontal direction. For example, when one pixel data is read out of two pixels, 1/2 thinning readout is performed in the horizontal direction, and when one pixel data is read out of four pixels, 1/4 thinning out reading is performed in the horizontal direction. It becomes.

  A thinning operation in the vertical direction (that is, the column direction) can be performed simultaneously with a thinning operation in the horizontal direction (that is, the row direction). For example, when reading pixel data of one line out of M lines in the vertical direction, 1 / M decimation reading is performed in the vertical direction. For example, when reading out pixel data of one row in two lines, 1/2 decimation reading is performed in the vertical direction, and when reading out pixel data of one row in four lines, 1 is read out in the vertical direction. / 4 thinning out reading.

  In the example illustrated in FIG. 16, an imaging operation in the case of performing pixel addition reading among the pixels in the imaging element 134 will be described. In this example, an imaging operation in the case of pixel addition readout of 1/2 in the horizontal direction and pixel addition readout of 1/2 in the vertical direction will be described as pixel addition readout.

  As in the case of all pixel readout, the vertical control lines are used to output the data of all the image sensors 134 connected to a certain row to the data readout lines provided in each column, AD conversion, and column latch To hold. Here, unlike the case of performing all-pixel readout, using another vertical control line, the data of all the image sensors 134 connected to different rows are output to the data readout line provided in each column, A / D conversion is performed. And it adds to the data of a column latch using an adder. The value of each pixel data is added by this method for the desired number in the vertical direction, and the added data is held in each column latch. For example, the pixel data d1 + d5, d2 + d6, d3 + d7, d4 + d8 are held in the column latches 358 to 361. As described above, after adding N pieces of pixel data in the vertical direction, the addition result is output as one piece of pixel data, whereby 1 / N pixel addition reading in the vertical direction is performed.

  Next, the horizontal scanning circuit 345 turns on the readout switch from the column latch to the output data line 390 one by one in order. In this case, the data read from each column latch to the output data line 390 is added by the adder 366 of the output data line 390 and held in the output latch 367. Then, the addition process is repeated for the desired number of columns in the horizontal direction, and the added data is output to the image sensor 134. For example, data d1 + d5 + d2 + d6 obtained by adding the pixel data d1 + d5 and the pixel data d2 + d6 is output to the image sensor 134 via the output data line 369. Thus, by adding M row pixel data in the horizontal direction, 1 / M pixel addition reading is performed in the horizontal direction. By performing the above processing, addition processing in the horizontal direction (row direction) and addition processing in the vertical direction (column direction) can be performed.

  Here, in the case of performing the decimation readout shown in FIG. 15 and the pixel addition readout shown in FIG. 16, it is desirable to operate the image sensor corresponding to a plurality of types of decimation rates and pixel addition rates. Therefore, hereinafter, a scanning circuit that operates the image sensor corresponding to a plurality of types of thinning rates and pixel addition rates will be described.

  FIG. 17 is a diagram schematically illustrating an example of a scanning circuit for performing thinning of the image sensor 134 in the embodiment of the present technology. 17A shows three types of scanning circuits 104 to 106, and FIG. 17B shows a configuration example of the 1 / N thinning scanning circuit. In the example shown in FIG. 17, the number of pixels in the horizontal direction of the image sensor for the scanning circuit of the image sensor for supporting three types of pixel decimation rates of all pixel readout, 1/2 thinning readout, and 1/4 thinning readout. A case where there are 1024 will be described as an example.

  As shown in FIG. 17a, the scanning circuit 104 includes 1024 output signal lines (scan_a [n (0 ≦ n ≦ 1023: n is an integer)]), and each signal line is set to an enable state one by one. After that, the 1024 to 1 scanning circuit is brought into a disable state. In addition, the scanning circuit 105 includes 512 output signal lines (scan_b [n (0 ≦ n ≦ 511: n is an integer)]), and each signal line is set to the disable state after being set to the disable state. This is a scanning circuit. The scanning circuit 106 includes 256 output signal lines (scan_c [n (0 ≦ n ≦ 255: n is an integer)]). Each signal line is set to the disable state after being set to the disable state one by one. This is a scanning circuit.

  As shown in b of FIG. 17, among the 1024 output signal lines of the scanning circuit 104, the signal line having a number that is a multiple of 4 includes the output signal line of the scanning circuit 105, the output signal line of the scanning circuit 106, and the like. A control line indicating which of the three types of pixel thinning rates is selected is connected. These three types of control lines are control lines corresponding to all pixel readout (sel_1per1), 1/2 decimation readout (sel_1per2), and 1/4 decimation readout (sel_1per4).

  Further, out of the 1024 output signal lines of the scanning circuit 104, the signal lines having a number that is not a multiple of 4 and that is a multiple of 2 include the output signal line of the scanning circuit 105 and two types of pixel decimation rates. A control line indicating which one is selected is connected. These two types of control lines are control lines corresponding to all pixel readout and 1/2 thinning readout.

  In addition, among the 1024 output signal lines of the scanning circuit 104, a control line indicating whether all pixel readout is selected is connected to signal lines other than the signal lines described above.

  By the output from the scanning circuit shown in FIG. 17b (scan_out [n (0 ≦ n ≦ 1023: n is an integer)]), the image sensor 134 reads out all pixels, reads out 1/2, and reads out 1/4. It is possible to perform the thinning process.

[Imaging system layout configuration example]
FIG. 18 is a diagram illustrating a relationship between the imaging unit 130 and the subject in the embodiment of the present technology. 18 shows only the optical systems 131 to 133 and the imaging elements 134 to 136 among the first imaging system 191 to the third imaging system 193 in the imaging unit 130 shown in FIG. Omitted. 18 schematically illustrates a case where the information processing apparatus 100 is arranged such that the subject to be imaged is the subject plane 300 and the optical axis 194 of the first imaging system 191 is orthogonal to the subject plane 300. Shown in Further, the optical axis 194 of the first imaging system 191 is orthogonal (or substantially orthogonal) to the display surface of the display unit 140. In addition, it is assumed that the three optical axes 194 to 196 intersect at the intersection point P0.

  In the imaging unit 130, an angle formed by the optical axis 194 of the first imaging system 191 and the optical axis 195 of the second imaging system 192 is θ0. Similarly, an angle formed by the optical axis 194 of the first imaging system 191 and the optical axis 196 of the third imaging system 193 is θ0. Note that the third imaging system 193 is arranged so as to be in the position of the line target with the second imaging system 192 across the optical axis 194 of the first imaging system 191.

  In the first imaging system 191, an angle formed by the outermost line of the incident optical path incident on the imaging element 134 and the optical axis 194 is θ1. In the second imaging system 192, the angle formed by the outermost line of the incident optical path incident on the imaging device 135 and the optical axis 195 is θ2. Similarly, in the third imaging system 193, the angle formed by the outermost line of the incident optical path incident on the imaging device 136 and the optical axis 196 is θ2. For example, as shown in FIG. 18, the horizontal width of the light receiving surface of the image sensor 134 and the horizontal width of the light receiving surfaces of the image sensors 135 and 136 are the same (or substantially the same). Therefore, the angle of view 2 × θ1 incident on the image sensor 134 and the angle of view 2 × θ2 incident on the image sensors 135 and 136 are the same (or substantially the same).

  Here, the imaging range 301 of the subject plane 300 is specified by the angle of view 2 × θ1 incident on the imaging element 134. Similarly, the imaging range 302 of the subject plane 300 is specified by the angle of view 2 × θ2 incident on the imaging element 135, and the imaging range 303 of the subject plane 300 is determined by the angle of view 2 × θ2 incident on the imaging element 136. Identified. In the embodiment of the present technology, images generated by the imaging elements 134 to 136 are combined to generate a panoramic image. For this reason, the angle formed by each optical axis is set so that the imaging range 301 of the subject plane 300 and the imaging range 302 of the subject plane 300 partially overlap. Specifically, the angle θ0 formed by the optical axis 194 of the first imaging system 191 and the optical axis 195 of the second imaging system 192 is set so that a part of the imaging ranges 301 and 302 overlap with each other. An angle θ0 formed by the optical axis 194 of the imaging system 191 and the optical axis 196 of the third imaging system 193 is set. The optical axes 194 to 196 are included in the same plane. Further, the first imaging system 191 to the third imaging system 193 are arranged so that the optical axes 194 to 196 intersect at one point (intersection point P0).

  Here, the lens center of the first imaging system 191 is R1, the lens center of the second imaging system 192 is R2, and the lens center of the third imaging system 193 is R3. Further, the distance between the lens center R1 and the intersection point P0 is L11, the distance between the lens center R2 and the intersection point P0 is L21, and the distance between the lens center R3 and the intersection point P0 is L31. In this case, it is preferable to arrange the first imaging system 191 to the third imaging system 193 so that the distances L11, L21, and L31 are equal.

[Example of keystone correction]
19 and 20 are diagrams schematically illustrating a relationship between an imaging system in the imaging unit 130 and subjects to be imaged in the embodiment of the present technology.

  FIG. 19 a schematically illustrates a subject 310 to be imaged by the imaging unit 130. The subject 310 is assumed to be a subject corresponding to the subject plane 300 shown in FIG. In addition, a range corresponding to the subject 310 is indicated by a rectangle, and the inside of the rectangle is schematically shown as a lattice.

  FIG. 20 a schematically shows subjects 311 to 313 to be imaged by the imaging unit 130. The subject 311 is assumed to be a subject corresponding to the first imaging system 191 in the subject 310 shown in FIG. Similarly, it is assumed that the subject 312 is a subject corresponding to the second imaging system 192 of the subject 310 and the subject 313 is a subject corresponding to the third imaging system 193 of the subject 310. Of the subjects 311 to 313, both ends in the horizontal direction of the subject 311 overlap with one end in the horizontal direction of the subjects 312 and 313.

  In FIG. 19 b and FIG. 20 b, an imaging system in the imaging unit 130 is shown. Note that the examples shown in FIG. 19 b and FIG. 20 b are the same as the example shown in FIG. 18 except that the reference numerals and the like are omitted.

  FIG. 21 is a diagram schematically illustrating a relationship between an imaging system in the imaging unit 130 and a captured image generated by these in the embodiment of the present technology. FIG. 21 a schematically illustrates captured images 314 to 316 generated by the imaging unit 130. These captured images 314 to 316 are captured images corresponding to the subjects 311 to 313 shown in a of FIG. 20, and schematically show a grid-like rectangle shown in the subjects 311 to 313.

  As shown in FIG. 21a, the optical axes 195 and 196 of the second imaging system 192 and the third imaging system 193 are not orthogonal to the object plane 300, so that the captured images 315 and 316 have a trapezoidal shape. Distortion occurs. This trapezoidal distortion will be described in detail with reference to FIG.

  FIG. 22 is a diagram schematically illustrating a relationship between the imaging system in the imaging unit 130 and the subject to be imaged in the embodiment of the present technology. In FIG. 22, for ease of explanation, the configuration relating to the third imaging system 193 is omitted, and only the first imaging system 191 and the second imaging system 192 are shown. The example shown in FIG. 22 is substantially the same as the example shown in FIG. 18 except that the configuration relating to the third imaging system 193 is omitted. Further, the intersection point P0, angles θ0, θ1 and θ2, lens centers R1 and R2, and distance L11 are the same as those shown in FIG.

  In FIG. 22, the intersection of the optical axis 194 of the first imaging system 191 and the subject plane 300 is defined as S11. Further, the intersection point between the right outline of the angle of view of the first imaging system 191 and the object plane 300 is S12, and the intersection of the left outline of the angle of view of the first imaging system 191 and the object plane 300 is S13. And

  A planar subject region that includes the intersection S11 and is orthogonal to the optical axis 194 and is incident on the first imaging system 191 is defined as a subject surface S10.

  The intersection of the optical axis 195 of the second imaging system 192 and the object plane 300 is S21, the intersection of the right outline of the angle of view of the second imaging system 192 and the object plane 300 is S32, and the second Let S43 be the intersection of the left outline of the angle of view of the second imaging system 192 and the subject plane 300.

  A planar subject area that includes the intersection S21 and is orthogonal to the optical axis 195 and is incident on the second imaging system 192 is defined as a subject surface S20.

  A planar subject region that includes the intersection S32 and is orthogonal to the optical axis 195 and is incident on the second imaging system 192 is referred to as a subject surface S30.

  A planar subject region that includes the intersection S43 and is orthogonal to the optical axis 195 and is incident on the second imaging system 192 is referred to as a subject surface S40.

  Further, an intersection point between the subject surface S30 and the optical axis 195 is set as S31, and an intersection point between the subject surface S40 and the optical axis 195 is set as S41.

  Further, the intersection point between the right outline of the angle of view of the second imaging system 192 and the subject surface S20 is S22, and the intersection point of the right outline of the angle of view of the second imaging system 192 and the subject plane S40 is S42. And

  Further, the intersection point between the left outline of the field angle of the second imaging system 192 and the subject surface S20 is S23, and the intersection point between the right outline of the field angle of the second imaging system 192 and the subject surface S30 is S33. And

  In addition, an intersection of a line segment 197 that passes through the lens center R2 of the second imaging system 192 and is perpendicular to the subject surface 300 and the subject surface 300 is defined as S51.

  For example, when comparing the subject surface S40 including the leftmost point S43 of the angle of view with the subject surface S30 including the rightmost point S32 of the angle of view, the subject surface S40 is located farther from the lens center R2 than the subject surface S30. To do. For this reason, when the subject surface S40 is imaged, a wider area is imaged than when the subject surface S30 is imaged. For example, a case is assumed where line segments having the same length are arranged as subjects on the subject surface S40 and on the subject surface S30. In this case, when the captured image generated for the subject surface S30 is compared with the captured image generated for the subject surface S40, the line segment included in the captured image generated for the subject surface S40 is shorter.

  For this reason, for example, when the subject 312 shown in FIG. 20A is imaged by the second imaging system 192, the area corresponding to the subject 312 is trapezoidal like the captured image 315 shown in FIG. It becomes a shape. That is, in the example illustrated in FIG. 21A, in the captured image 315, the left side of the rectangle corresponding to the subject 312 is shorter than the right side.

  Similarly, when the subject 313 shown in FIG. 20a is imaged by the third imaging system 193, the area corresponding to the subject 313 has a trapezoidal shape as in the captured image 316 shown in FIG. Become. In this way, trapezoidal distortion occurs in the captured image generated by the three-lens imaging unit. Therefore, a trapezoidal distortion correction method for correcting the trapezoidal distortion of the captured image will be described below.

  Here, the distance between the intersection S11 and the lens center R1 is L12, the distance between the intersection S13 and the lens center R1, and the distance between the intersection S12 and the lens center R1 is L13.

  The distance between the intersection S21 and the lens center R2 is L22, the distance between the intersection S31 and the lens center R2 is L30, and the distance between the intersection S41 and the lens center R2 is L40.

  The distance between the intersection S32 and the lens center R2 is L23, the distance between the intersection S43 and the lens center R2 is L24, and the distance between the intersection S51 and the lens center R2 is L51. The distances L61 to L66 will be described with reference to FIG.

Here, the following formula is established by the formula of the trigonometric function.
L21 + L22 = L11 + L22 = (L11 + L12) / cos θ0

From this equation, the following equation 1 is obtained. Note that L11 = L21.
L22 = {(L11 + L12) / cos θ0} −L11 Formula 1

For the distance L51, the following formula 2 is obtained from the trigonometric formula and formula 1.
L51 = L22 × cos θ0
= [{(L11 + L12) / cos θ0} −L11)] × cos θ0 Equation 2

As for the distance L23, the following formula 3 is obtained from the trigonometric formula and formula 2.
L23 = L51 / cos (θ2-θ0)
= (L22 × cos θ0) / cos (θ2−θ0) Equation 3

As for the distance L30, the following formula 4 is obtained from the trigonometric formula and formula 3.
L30 = L23 × cos θ2
= {(L22 × cos θ0) / cos (θ2−θ0)} × cos θ2
= ([({(L11 + L12) / cos θ0} −L11) × cos θ0] / cos (θ2−θ0)]) × cos θ2 Formula 4

  Here, when the distances L11 and L12 on the optical axis 194 of the first imaging system 191 are determined, the distance L30 can be obtained by using Expression 4. Thus, by determining the distance L30, the value XR (= L12 / L30) of the ratio of the distance L30 to the distance L12 can be determined. Note that XR <1.

Further, for the distance L24, the following formula 5 is obtained from the trigonometric formula and formula 2.
L24 = L51 / cos (θ2 + θ0)
= (L22 × cos θ0) / cos (θ2 + θ0) Equation 5

Further, for the distance L40, the following formula 6 is obtained from the trigonometric formula and formulas 1 and 5.
L40 = L24 × cos θ2
= {(L22 × cos θ0) / cos (θ2 + θ0)} × cos θ2
= [[{(L11 + L12) / cos θ0−L11} × cos θ0] / cos (θ2 + θ0)] × cos θ2 Formula 6

  Here, when the distances L11 and L12 on the optical axis 194 of the first imaging system 191 are determined, the distance L40 can be obtained by using Expression 6. Thus, by determining the distance L40, the value XL (= L12 / L40) of the ratio of the distance L40 to the distance L12 can be determined. Note that XL> 1.

  Next, a correction method for correcting trapezoidal distortion using the ratio values XR and XL will be described.

  In the captured image 315 shown in FIG. 21a, each coordinate is converted so that the length of the right side is XR times and the length of the left side is XL times. That is, the right side in the captured image 315 is reduced in the direction of the arrow 321 so that the length of the right side is XR times. On the other hand, the left side in the captured image 315 is enlarged in the direction of the arrow 322 so that the length of the left side is XL times. The corrected image 317 thus corrected is shown in FIG.

  FIG. 23 is a diagram schematically illustrating a relationship between the imaging system in the imaging unit 130 according to the embodiment of the present technology, the captured image generated by these, and the corrected image after correction. FIG. 23 a schematically illustrates a captured image 314 generated by the imaging unit 130 and corrected images 317 and 318 obtained by correcting the captured images 315 and 316 illustrated in FIG. The example shown in b of FIG. 23 is the same as the example shown in FIG. 18 except that the reference numerals and the like are omitted.

  As described above, in the captured image 315 shown in FIG. 21a, the coordinates are converted so that the length of the right side is XR times and the length of the left side is XL times. A corrected image 317 is generated. The corrected image 317 generated in this way is an image having a trapezoidal outer shape. Therefore, an image 325 (indicated by a thick line) in which the trapezoidal distortion is corrected can be acquired by cutting out the central portion of the corrected image 317 into a rectangle.

  Similarly, for the captured image 316 shown in a of FIG. 21, by correcting each coordinate so that the length of the left side is XR times and the length of the right side is XL times, a corrected image shown in a of FIG. 318 is generated. Then, an image 326 (indicated by a thick line) in which the trapezoidal distortion is corrected can be acquired by cutting out the central portion of the corrected image 318 into a rectangle. Note that the trapezoidal distortion correction processing of the captured image is performed by the image composition processing unit 224.

  When performing such trapezoidal distortion correction, for example, the coordinates of each pixel in a captured image distorted in a trapezoid are measured, and the ratio values XR and XL are obtained in advance. Then, using the ratio values XR and XL obtained in advance, the trapezoidal distortion correction process can be obtained by software using an arithmetic unit such as a CPU built in the information processing apparatus 100.

  In this example, an example of the correction method for correcting the trapezoidal distortion of the captured image generated by the trinocular imaging operation has been described. However, other trapezoidal distortion correction methods (for example, see Japanese Patent Application Laid-Open No. 08-307770). Correction may be performed.

[Example of composition of captured images]
FIG. 24 is a diagram schematically illustrating a synthesis flow when the image synthesis processing unit 224 according to the embodiment of the present technology generates a synthesized image. In this example, an example in which three captured images generated by three imaging systems are synthesized based on an angle (convergence angle) formed by optical axes of the three imaging systems. That is, of the two captured images generated by the two imaging systems, one of the overlapping portions is excised based on the convergence angle, and the two captured images are synthesized after the excision.

  FIG. 24 a shows a captured image 314 and corrected images 317 and 318 generated by the first imaging system 191 to the third imaging system 193 in the imaging unit 130. The captured image 314 and the corrected images 317 and 318 are the same as those shown in FIG. As shown in a of FIG. 24, the captured image 314 and the corrected images 317 and 318 in which the trapezoidal distortion is corrected are acquired by the trapezoidal distortion correction process.

  Here, as described above, the same subject is included in the right end area of the corrected image 317 and the left end area of the captured image 314. Further, the same subject is included in the left end area of the corrected image 318 and the right end area of the captured image 314. Therefore, in the following, a method for calculating a region including these same subjects will be described.

  Here, in FIG. 22, the distance between the intersection S11 and the intersection S21 is L61, the distance between the intersection S11 and the intersection S13 is L62, and the distance between the intersection S51 and the intersection S21 is L63. The distance between the intersection S13 and the intersection S51 is L64, the distance between the intersection S32 and the intersection S51 is L65, and the distance between the intersection S32 and the intersection S13 is L66. The distance L66 is a distance for specifying a region including the same subject among the captured image generated by the first imaging system 191 and the captured image generated by the second imaging system 192. is there. This is a common area between the left end area of the captured image generated by the first imaging system 191 and the right end area of the captured image generated by the second imaging system 192.

Here, the following formulas 7 and 8 are established by the formula of the trigonometric function.
L61 = (L11 + L12) × tan θ0 Equation 7
L62 = L12 × tan θ1 Equation 8

Further, the following formula 9 is obtained from the trigonometric function formula and formula 2.
L63 = L51 × tan θ0
= [{(L11 + L12) / cos θ0} −L11] × cos θ0 × tan θ0 (Equation 9)

Further, using Expression 7 to Expression 8, the following Expression 10 is obtained.
L64 = L62 + L63-L61
= (L12 × tan θ1) + ([{(L11 + L12) / cos θ0} −L11] × cos θ0 × tan θ0) − {(L11 + L12) × tan θ0} Expression 10

Further, the following formula 11 is obtained from the trigonometric function formula and formula 2.
L65 = L51 × tan (θ2−θ0)
= [{(L11 + L12) / cos θ0} −L11] × cos θ0 × tan (θ2−θ0) Equation 11

The following equation 12 is obtained using the equations 10 and 11 obtained above.
L66 = L65 + L64
= {[{(L11 + L12) / cos θ0} −L11] × cos θ0 × tan (θ2−θ0)} + (L12 × tan θ1) + ([{(L11 + L12) / cos θ0} −L11] × cos θ0 × tan θ0) − [{ (L11 + L12) × tan θ0}] Equation 12

  Here, when the distances L11 and L12 on the optical axis 194 of the first imaging system 191 are determined, the distance L66 can be obtained by using Expression 12. Further, a common area between the right end area of the captured image generated by the first image capturing system 191 and the left end area of the captured image generated by the third image capturing system 193 can be obtained in the same manner. .

  FIG. 24B shows a region to be combined among the captured image 314 and the corrected images 317 and 318. For example, an area corresponding to the distance L66 calculated using Expression 12 is deleted from the captured image 314 generated by the first imaging system 191. Similarly, for the captured image 314, the common area calculated for the right end of the captured image 314 is deleted. In FIG. 24b, the outer shape of the image 327 after the common area at both ends is deleted is indicated by a thick line.

  FIG. 24 c shows a panoramic image 330 generated using the captured image 314 and the corrected images 317 and 318. As shown in FIG. 24b, after the images at both ends are deleted from the captured image 314, a panoramic image is generated using the deleted image 327 and the corrected images 317 and 318. For example, the correction image 317 is connected to the left end of the image 327 and the correction image 318 is connected to the right end of the image 327 to generate the panoramic image 330.

  FIG. 25 is a diagram schematically illustrating a synthesis flow when the image synthesis processing unit 224 according to the embodiment of the present technology generates a synthesized image. The example shown in FIG. 25 is a modification of FIG. 24, and an image including a region to be deleted at the time of image composition is different. That is, as shown in FIG. 25b, for the corrected image 317 corresponding to the captured image generated by the second imaging system 192, the region corresponding to the distance L66 calculated using Expression 12 (the region at the right end) ) Is deleted. Similarly, the common area calculated for the left end of the corrected image 318 is deleted. In FIG. 25b, the outlines of the images 332 and 333 after the common area is deleted are indicated by bold lines.

  FIG. 25C shows a panoramic image 330 generated using the images 331 to 333. As illustrated in FIG. 25b, for example, the image 332 is connected to the left end of the image 331, and the image 333 is connected to the right end of the image 331, so that a panoramic image 330 is generated.

  FIG. 26 is a diagram schematically illustrating a composition flow when the image composition processing unit 224 according to the embodiment of the present technology generates a composite image. The example shown in FIG. 26 is a modification of FIG. 24, and an image including a region to be deleted at the time of image composition is different. That is, as shown in FIG. 26 b, an area corresponding to half of the distance L66 calculated using Expression 12 is deleted from the captured image 314 generated by the first imaging system 191. In addition, an area corresponding to half of the distance L66 calculated by using Expression 12 (right end area) is deleted from the corrected image 317 corresponding to the captured image generated by the second imaging system 192.

  Similarly, for the captured image 314, half of the common area calculated for the right end of the captured image 314 is deleted, and for the corrected image 318 corresponding to the captured image generated by the third imaging system 193, half of the common area. The area corresponding to (the area at the left end) is deleted. In FIG. 26b, the outlines of the images 334 to 336 after the common area is deleted are indicated by bold lines.

  FIG. 26 c shows a panoramic image 330 generated using the images 334 to 336. As shown in FIG. 26b, for example, the image 335 is connected to the left end of the image 334, and the image 336 is connected to the right end of the image 334, so that a panoramic image 330 is generated. In this way, by deleting a part of each image, it is possible to appropriately combine the images.

  Note that these image composition processes are performed by the image composition processing unit 224. Further, FIG. 8 shows an example in which the image composition processing unit 224 is arranged after the YC conversion processing unit 223 and before the sharpness processing unit 225 in the image signal processing unit 220. However, the image composition processing may be performed at another stage of the image signal processing unit 220. For example, the trapezoidal distortion correction process and the image composition process may be performed before the demosaic processing unit 222. Further, for example, the trapezoidal distortion correction process and the image composition process may be performed after the demosaic processing unit 222 and before the YC conversion processing unit 223. Further, for example, the trapezoidal distortion correction process and the image composition process may be performed after the color adjustment processing unit 226.

  In addition, when performing such image composition processing, for example, for an image after trapezoidal distortion correction, an overlapping area of each image is measured in advance. Then, using this measured value, it is possible to obtain the deletion processing of the overlapping region of the image by software using an arithmetic device such as a CPU built in the information processing apparatus 100.

  In this example, an example in which three captured images are combined based on the convergence angle has been described. However, for example, an image combining process may be performed using another image combining method. For example, for two overlapping images generated by two imaging systems, an image combining method can be used in which two images are pattern-matched and two images are combined by pattern matching. Further, it is possible to use an image composition method in which a change in density level in two images generated by two imaging systems is obtained, and an overlapping portion is obtained based on the change in density level and the two images are synthesized.

[Image file configuration example]
Next, image data (image file) recorded on the recording medium 180 by the still image recording process in the information processing apparatus 100 will be described in detail with reference to the drawings.

  FIG. 27 is a diagram illustrating an example of a file structure of an image file stored in the recording medium 180 according to the embodiment of the present technology. The example shown in FIG. 27 schematically shows the file structure of a still image file recorded according to the DCF (Design Rule for Camera File system) standard.

  DCF is a file system standard for realizing mutual use of images via recording media between devices such as digital still cameras and printers. In addition, in DCF, file name assignment and folder configuration are defined when recording on a recording medium based on Exif (Exchangeable image file format). Exif is a standard for adding image data and camera information to an image file, and defines a format (file format) for recording an image file. 27A shows a configuration example of the image file 181, FIG. 27B shows a configuration example of the attached information 182, and FIG. 27C shows a configuration example of the manufacturer note 185.

  The image file 181 is a still image file recorded according to the DCF standard, and is composed of attached information 182 and image information 183 as shown in FIG. For example, the image information 183 is generated by the imaging unit 130 and subjected to various types of image signal processing by the image signal processing unit 220. The resolution conversion unit 251 converts the resolution of the image data, and the encoding / decoding unit 252 compresses the image information. Image data. This image data is compressed in, for example, JPEG (Joint Photographic Experts Group) format.

  The attached information 182 includes attribute information 184 and a maker note 185, as shown in FIG. The attribute information 184 is attribute information related to the image file 181, and includes, for example, GPS information, orientation information, imaging unit characteristic information (for example, device characteristics), imaging update date / time, image size, color space information, manufacturer name, and the like. included.

  The maker note 185 is generally an area where user-specific data is recorded, and is an extended area where each maker can freely record information (TAGID = 37500, MakerNote). As shown in c of FIG. 27, the manufacturer note 185 includes, for example, monocular / compound eye 186, presence / absence of a panoramic image 187, in-focus position information 188, face information 189, as information about the captured image. The moving object information 190 is recorded.

  Whether the monocular imaging / compound imaging 186 is image data generated using only the image sensor 134 (monocular imaging), or is image data composed of image data generated using the imaging elements 134 to 136. This is information indicating (compound eye photography). For example, “0” is stored in the case of image data generated by monocular imaging, and “1” is stored in the case of image data generated by compound eye imaging.

  The presence / absence 187 of the panoramic image is information indicating whether the image has an aspect ratio exceeding a certain value (panoramic image) or other image (normal image). For example, “0” is stored for a normal image, and “1” is stored for a panoramic image.

  The in-focus position information 188 is information related to the in-focus position in the captured image. For example, information about the in-focus position detected by the AF control unit 281 is stored.

  The face information 189 is information including the position and size of the face included in the image generated by the imaging unit 130. For example, the position (coordinates) of the upper left corner of the rectangular area including the face in the captured image is stored as the face position, and the vertical and horizontal lengths (vertical width) of the rectangular area in the captured image are stored as the face size. And width) are stored. The face information is detected by the face detection unit 282.

  The moving object information 190 is information including the position and size of a moving object (moving object (for example, a running car)) included in the image generated by the imaging unit 130. For example, the position (coordinates) of the upper left corner of the rectangular area including the moving object in the captured image is stored as the moving object position, and the vertical and horizontal lengths (vertical width) of the rectangular area in the captured image are stored as the moving object size. And width) are stored. Such moving body information is detected by the CPU 202.

  Various images can be displayed at the time of reproduction using the attached information 182 (attribute information 184 and maker note 185) recorded in the image file 181.

[Example of still image display timing]
Here, the timing at which image display is required in an information processing apparatus that handles still images will be described. There are mainly four timings at which image display is required in an information processing apparatus that handles still images: live view display, post view display, playback display, and recorded image list display (so-called thumbnail display). Therefore, each of these image displays will be described.

  Note that, as described above, there is a monocular imaging device capable of generating a panoramic image by an operation (so-called panning operation) that rotates the imaging device in the horizontal direction with the imaging position (photographer's position) as the rotation center. Exists. When a panoramic image is generated by this monocular imaging device, the image displayed on the display unit at the start of panoramic image shooting becomes one end of the panoramic image, and the image displayed on the display unit at the end of panoramic image shooting Is the other end of the panoramic image. For this reason, the user can grasp comparatively easily where the panoramic image shooting starts and where.

  Further, when a panoramic image is generated by this monocular imaging device, the range of the panoramic image to be photographed can be changed while photographing. For example, the image corresponding to the position where the panoramic image shooting is started is one end of the panoramic image, and the image corresponding to the position where the panoramic image shooting is stopped is the other end of the panoramic image.

  In contrast, the information processing apparatus 100 can capture a panoramic image in an instant when a shutter button (for example, the enter key 112) is pressed. That is, a panoramic image can be taken in a desired moment desired by the user. In this way, in the information processing apparatus 100, panoramic image shooting ends in an instant, and thus it is necessary to determine the angle of view (for example, a horizontal range desired by the user) before shooting. For example, it is necessary to check on the display unit 140 during live view display before panoramic image shooting. For example, it is necessary to confirm whether the main subject is captured in the picture or whether the subject is in good condition (whether the person is not closing his eyes or the butterfly is caught in the flower). Also, it is necessary to check how far the image is taken as the right end and the left end of the image (whether the desired subject is captured).

(1) Live View Display In live view display, a captured image generated by the imaging unit 130 is displayed on the display unit 140 before a still image recording instruction operation is performed in a state where the still image recording mode is set. Is the action. In other words, the live view display is a moving image display in which an image input to the image sensor at that time is continuously displayed so that the user confirms the subject before the image capturing operation (that is, before the shutter button is pressed). An image (captured image) displayed in the live view display is referred to as a live view image.

The purpose of the live view display in the information processing apparatus is mainly the following (a) to (c).
(A) Confirming whether the user of the information processing apparatus 100 captures the main subject within the angle of view (in other words, the area to be recorded as an image) while viewing the live view image.
(B) Check how far the left and right area is recorded as the angle of view (in other words, how far the area is recorded as an image, especially how far the left and right area is recorded in the case of a panoramic image) While adjusting this area to determine this.
(C) Checking the status of the main subject (for example, checking whether the person has closed his eyes or whether the butterfly has just stayed on the flower) and determining the timing for shooting.
Further, the adjustment of each area can be performed while the adjustment result is fed back to the live view display and displayed.

(2) Post-view display When the still-image recording instruction operation is performed in a state where the still-image recording mode is set, the post-view display is performed for a certain period after the recording process based on the recording instruction operation is completed. This is an operation for automatically displaying the recorded captured image. That is, the post-view display is a display for the user to confirm the recorded image after the imaging operation (that is, after the panoramic image recording process by pressing the shutter button is completed). An image (captured image) displayed in the post-view display is referred to as a post-view image.

  For example, in order to reduce the power consumption of the information processing apparatus 100, it is assumed that the clock frequency is intentionally lowered based on an instruction from the user or information on the remaining battery level. In this case, there is a possibility that the image signal processing and the image recording processing of the still image do not end within a certain time. In such a case, the monitoring image is not displayed on the display unit 140 for a certain period after the user performs a still image recording instruction operation. Specifically, since the image signal processing and the image recording process are performed after the recording instruction operation is performed and until the recording process of the captured image is completed in the information processing apparatus 100, a new captured image is captured. Is displayed on the display unit 140. For example, a monochrome solid image (for example, black or amber) is displayed on the display unit 140. On this monochrome solid image, characters such as “processing” and “please wait”, and processing inside the apparatus A sign (for example, an hourglass) indicating that is running is displayed. Thus, an image indicating that a new captured image cannot be taken is referred to as a “black image”, and this display operation is referred to as a “black image display operation”. That is, when the user performs a still image recording instruction operation, a black image is displayed on the display unit 140 by a black image display operation, and after a certain period of time, a post-view image is displayed on the display unit 140 by a post-view operation. .

The purpose of the postview display in the information processing apparatus is mainly the following (d) and (e).
(D) Confirmation of an image recorded by photographing and determination of whether or not retaking of the image is necessary.
For example, whether or not the angle of view (that is, the area of the image recorded by shooting) is satisfactory, and whether or not the situation of the subject is satisfactory (for example, at the moment when the butterfly just stops on the flower) It is to check if there is. Also, along with this confirmation, it is determined whether it is necessary to retake the image.

  Here, for confirmation of the angle of view, it is important to confirm which area is captured and recorded as a panoramic image, and the right end and the left end of the image. In order to confirm the state of the subject, it is important to confirm in detail how the main subject is photographed and recorded.

(E) Appreciating images recorded by shooting.
The purpose of image viewing is the same as that of the following image reproduction display. In addition, the confirmation of the captured image and the determination of re-taking are only for the post-view display, and if the opportunity is missed, re-taking is often impossible. For this reason, the main purpose of post-view display is (d) determination of image confirmation and re-shooting rather than (e) image viewing. The post-view display is intended to confirm the image recorded by the still image recording process, so that the recorded still image is displayed without performing the real-time moving image display like the live view.

(3) Reproduction display Reproduction display is an operation of reproducing content (for example, a panoramic image) stored in the recording medium 180 by a user operation at an arbitrary time.

  Also, the purpose of image reproduction and display in the information processing apparatus is mainly to appreciate images recorded by photographing. When viewing a panoramic image, it is preferable to be able to view both the entire image and the details of the main subject in detail.

  Note that the playback display shown here is for the purpose of viewing the image recorded by the still image recording process, so the recorded still image is displayed without the real-time moving image display as in the live view. .

(4) List display of recorded images (so-called thumbnail display)
When a plurality of images are captured and recorded, it is convenient to be able to see at a glance what kind of images have been recorded. However, since a plurality of recorded images are displayed on the display unit 140, the size of one sheet must be reduced.

  In this state, it is preferable that the user can visually recognize what each of the images (particularly, the panoramic image) is an image of which each is captured.

  Note that the thumbnail display is intended to display a list of recorded images, so only a still image display is performed.

[Video display timing example]
Next, the timing at which image display is required in an information processing apparatus that handles moving images will be described. There are mainly three timings at which image display is required in an information processing apparatus that handles moving images: real-time display, playback display, and list display of recorded images (so-called thumbnail display). Therefore, each of these image displays will be described.

(1) Real-time display The purpose of the real-time display in the information processing apparatus is mainly to check the angle of view of the image to be recorded and the state of the subject before and during shooting in a state where the moving image recording mode is set. It is confirmation.

  Since the real-time display is intended for real-time confirmation of moving images before and during recording, only moving image display is performed.

(2) Reproduction Display The purpose of the image reproduction display in the information processing apparatus is mainly to appreciate the image recorded by photographing. When viewing a panoramic image, it is preferable to be able to view both the entire image and the details of the main subject in detail.

  Note that the reproduction display shown here is for the purpose of appreciating the recorded moving image, so only the moving image display is performed.

(3) List display of recorded images (so-called thumbnail display)
When multiple images are taken and recorded, you can see at a glance a representative image (for example, the image when you started taking a movie) as a still image, what kind of image was recorded It is convenient to be able to. However, since a plurality of recorded images are displayed on the display unit 140, the size of one sheet must be reduced. In this state, it is preferable that the user can visually recognize what each of the images (particularly, the panoramic image) is an image of which each is captured.

  Note that the thumbnail display is intended to display a list of the top images of the recorded moving images, so only a still image display is performed.

  In the embodiment of the present technology, the term “image” includes both the meaning of the image itself and the image data for displaying the image.

[Functional configuration example of information processing device]
FIG. 28 is a block diagram illustrating a functional configuration example of the information processing apparatus 100 according to the embodiment of the present technology.

  The information processing apparatus 100 includes a display unit 140, a posture detection unit 150, an image generation unit 510, an image memory 520, a recording control unit 530, a storage unit 540, a display control unit 550, and an operation reception unit 560. Is provided.

  The image generation unit 510 captures a subject and generates a captured image, and causes the image memory 520 to hold the generated captured image. This captured image is, for example, a captured image generated using one image sensor 134 or a captured image generated by combining three captured images generated using three image sensors 134 to 136 (for example, Panoramic image). These captured images are generated in response to a user operation received by the operation receiving unit 560. Further, the image generation unit 510 outputs each information at the time of generating the captured image to the display control unit 550 and the recording control unit 530. Each information at the time of generating the captured image is, for example, information about the face (face information) included in the captured image or information about the focus position (focus position information) (for example, each information shown in c of FIG. 27). ). That is, the image generation unit 510 functions as a detection unit that detects a specific object in the panoramic image. Note that the image generation unit 510 corresponds to, for example, the imaging unit 130 and the DSP 200 (the image signal processing unit 220, the AF control unit 281, the face detection unit 282, and the like illustrated in FIG. 7) illustrated in FIG. The image generation unit 510 is an example of an object detection unit.

  The image memory 520 holds a captured image generated by the image generation unit 510 or a captured image (image file) acquired from the storage unit 540 by the display control unit 550. Then, the image memory 520 supplies the held captured image to the recording control unit 530 or the display control unit 550. Note that the image memory 520 corresponds to, for example, the image memory 170 illustrated in FIG.

  The recording control unit 530 causes the storage unit 540 to record the captured image generated by the image generation unit 510 and held in the image memory 520 in accordance with a user operation received by the operation reception unit 560 as an image file. is there. Also, the recording control unit 530 records the information output from the image generation unit 510 (each information at the time of generating the captured image) in the image file when recording the captured image. For example, when the recording control unit 530 records a panoramic image (a horizontally long image), the specific target object information (for example, the face shown in FIG. 27c) related to the region where the specific target object (for example, a human face) exists. Information 189) is recorded in the storage unit 540 in association with the panoramic image. The recording control unit 530 corresponds to, for example, the DSP 200 (the imaging control unit 201, the resolution conversion unit 251, the encoding / decoding unit 252 and the like illustrated in FIG. 7) illustrated in FIG.

  The storage unit 540 stores the captured image generated by the image generation unit 510 as an image file, and supplies the stored image file to the display control unit 550. Note that the storage unit 540 corresponds to, for example, the recording medium 180 illustrated in FIG.

  The display control unit 550 causes the display unit 140 to display a captured image generated by the image generation unit 510 and held in the image memory 520 in response to a user operation received by the operation reception unit 560. In addition, the display control unit 550 acquires an image file stored in the storage unit 540 in accordance with a user operation received by the operation receiving unit 560, holds the image file in the image memory 520, and acquires a captured image of the image file. It is displayed on the display unit 140. For example, the display control unit 550 displays images of a plurality of regions in the captured image held in the image memory 520 based on information recorded in the acquired image file (each information at the time of generating the captured image). Can be displayed. Further, for example, when the display control unit 550 displays a panoramic image stored in the storage unit 540, an area where the specific target exists using the specific target information recorded in association with the panoramic image. The image regarding can be displayed. For example, an image of an area where a human face exists can be displayed using face information (face information 189 shown in FIG. 27C).

  In addition, the display control unit 550 displays an image on the display unit 140 at the time of live view display, post view display, and playback display, for example. In this case, for example, the display control unit 550 performs control to display a plurality of regions in the panoramic image generated by the image generation unit 510 in a plurality of stages in the longitudinal direction of the display unit 140.

  For example, as shown in FIG. 32 and the like, the display control unit 550 divides the panorama image with an orthogonal direction orthogonal to the longitudinal direction of the panoramic image as a boundary, and a plurality of the divided images in the longitudinal direction of the display unit 140 are divided. Display in a row. In this case, the display control unit 550, for example, includes two images (left image and right image) that include regions at both ends in the longitudinal direction of the panoramic image, and an image (middle image) that exists between the both ends. Image) is displayed in two stages in the longitudinal direction of the display unit 140. Further, the display control unit 550 displays, for example, the left image and the right image on the same line, and displays the middle image on another line. Further, the display control unit 550 displays the left image, the right image, and the middle image with the same size in the vertical direction, for example. In addition, the display control unit 550 displays, for example, the horizontal size of the middle image larger than the horizontal size of the left image and the right image.

  Further, for example, as shown in FIGS. 37 and 38, the display control unit 550 displays an image including a region where a specific object (for example, a person or a butterfly) detected from a panoramic image is present as a middle image. It may be.

  Further, for example, as shown in FIGS. 42 and 43, the display control unit 550 moves the region displayed as the middle image from one end of the both ends of the panoramic image to the other end. A middle image may be displayed.

  In addition, the display control unit 550 may display a reduced image of the panoramic image in a region other than the display region of the plurality of regions in the panoramic image, for example, as illustrated in FIGS.

  Further, for example, as illustrated in FIGS. 39 and 40, the display control unit 550 displays display information (for example, a frame) representing the positions of a plurality of regions in the panoramic image in association with the reduced image of the panoramic image. You may make it make it.

  For example, as shown in FIG. 46, the display control unit 550 determines whether or not the longitudinal direction of the panoramic image matches the longitudinal direction of the display unit 140. A panoramic image may be displayed on the display unit 140 without performing the above. On the other hand, when the longitudinal direction of the panoramic image and the longitudinal direction of the display unit 140 do not coincide with each other, the above-described controls are performed so that a plurality of regions in the panoramic image have a plurality of stages in the longitudinal direction of the display unit 140. You may make it display.

  Further, the display control unit 550 displays each setting screen on the display unit 140 in accordance with the user operation received by the operation receiving unit 560. Note that the display control unit 550 corresponds to, for example, the DSP 200 (the imaging control unit 201, the CPU 202, the encoding / decoding unit 252, the resolution conversion unit 231, the image rotation processing unit 232, and the like illustrated in FIG. 7) illustrated in FIG.

  The display unit 140 displays each image based on the control of the display control unit 550. The display unit 140 corresponds to, for example, the display unit 140 illustrated in FIG.

  The operation accepting unit 560 is an operation accepting unit that accepts an operation input by a user, and outputs the contents of the accepted operation input to each unit. For example, when a user performs a still image recording instruction operation, the operation receiving unit 560 outputs a message to that effect to the image generation unit 510, the recording control unit 530, and the display control unit 550. In addition, when a display instruction operation for displaying an image file stored in the storage unit 540 is performed, the operation reception unit 560 outputs that fact to the display control unit 550. In addition, when an image file stored in the storage unit 540 is displayed and an instruction operation for changing the display state is performed, the operation reception unit 560 notifies the display control unit 550 of that fact. Output. The operation receiving unit 560 corresponds to, for example, each operation member (the changeover switch 111, the determination key 112, and the like) illustrated in FIGS.

  The posture detection unit 150 detects the posture of the information processing apparatus 100 (display unit 140), and outputs the detection result to the display control unit 550. For example, the posture detection unit 150 detects whether the posture of the information processing apparatus 100 is in a horizontally long state or a vertically long state. The posture detection unit 150 corresponds to the posture detection unit 150 illustrated in FIG.

[Panorama image display example]
Next, an example in which image data is read from the imaging elements 134 to 136 and the image is displayed on the display unit 140 will be described.

  FIG. 29 is a diagram illustrating a relationship between a subject that is an imaging target of an imaging process performed by the information processing apparatus 100 according to the embodiment of the present technology and an imaging range.

  FIG. 29 a schematically illustrates the relationship between the subject and the information processing apparatus 100 during the imaging operation. 29b shows a panoramic image 600 corresponding to the imaging range (indicated by a dotted rectangle 590) in the subject shown in FIG.

  Here, a conventional information processing apparatus will be described. For example, a compound-eye imaging device (for example, a compound-eye digital still camera) composed of a single casing is generally displayed on the casing such that the longitudinal direction of the rectangular casing coincides with the longitudinal direction of the rectangular display apparatus. The device is arranged. Here, since one image to be recorded or displayed is a rectangle having a horizontal to vertical ratio of 4: 3 or 16: 9, it is important to make these images easily visible to the user as much as possible. It becomes. Therefore, in order to display the entire image as large as possible on the rectangular display device, the optical center arrangement line (connecting the optical centers of the plurality of imaging systems) is used for a plurality of imaging systems including an optical system (lens, etc.) and an imaging device. Line) and the longitudinal direction of the casing are arranged in the casing.

  Similarly, in the monocular imaging apparatus including one imaging system composed of one optical system and one imaging element with one optical center, the longitudinal direction of the rectangular casing and the rectangular display A display device is arranged in the housing so that the longitudinal direction of the device matches. Such an arrangement is also necessary for a device that handles a panoramic image having a large horizontal to vertical ratio.

  In this way, when shooting a panoramic image (a panoramic image extending in the horizontal direction) using an information processing device in which a plurality of imaging systems are arranged in a direction parallel to the longitudinal direction of the information processing device, the information processing device Are assumed to be used in a landscape state. However, in the case of an information processing device such as a smartphone or a mobile phone, a rectangular casing and a rectangular display device are often used in a vertically long state. For these information processing devices, if the imaging system is arranged so that the long side of the display device and the optical center arrangement line are parallel to each other, it is necessary to be in a horizontally long state every time a panoramic image is taken. It may be difficult for a user who is unfamiliar with shooting to use the device.

  Thus, for example, when using functions of an information processing device other than image shooting, a user often uses the information processing device in a vertically long state. On the other hand, it can be considered that the panoramic image is used in the landscape state when the panoramic image is captured and the panoramic image is displayed. Thus, changing the orientation of the housing depending on the function to be used is complicated for the user.

  Further, in such an information processing apparatus, the optical center arrangement line (line connecting the optical centers of the plurality of imaging systems) is parallel to the longitudinal direction of the casing, and the short side of the casing (the longitudinal direction of the casing) It is assumed that the optical center arrangement line is arranged in the vicinity of the center on the end side in FIG. However, when the imaging system (optical system) is arranged in this way, it may be difficult to design an electronic board that performs functions other than photography. That is, many of the imaging system parts including the lens and the imaging element are high in the information processing apparatus parts. For this reason, if an imaging system component is arranged near the center of the short side of the casing (the end side in the longitudinal direction of the casing) so that the optical center alignment line is parallel to the longitudinal direction of the casing, There is a risk of disturbing the placement of the substrate. For example, the concave portion on the upper side of the electronic substrate 101 shown in FIG.

  Therefore, in the embodiment of the present technology, an example in which a plurality of imaging units are arranged side by side in an orthogonal direction orthogonal to the longitudinal direction of the display unit is shown. In addition, an example is shown in which a plurality of regions in a panoramic image composed of a plurality of images generated by a plurality of imaging units are displayed in a plurality of stages in the longitudinal direction of the display unit.

  In the embodiment of the present technology, when a panoramic image generated using a plurality of imaging systems is displayed in a state in which the longitudinal direction and the vertical direction of the display unit 140 of the information processing apparatus 100 substantially match. An example in which a panoramic image is divided and displayed is shown. In the example shown below, the longitudinal direction of the display unit 140 may be referred to as the Y coordinate direction.

[Panorama image display example]
30 to 32 are diagrams illustrating a relationship between the panoramic image 600 generated by the image generation unit 510 and the image displayed on the display unit 140 by the display control unit 550 according to the embodiment of the present technology. Note that the timing for displaying the panoramic image on the display unit 140 is assumed to be live view display, post view display, reproduction display, or the like, as described above.

  FIG. 30 a shows a panoramic image 600 generated by the image generation unit 510 and images 601 to 603 to be displayed. The panoramic image 600 is the same as the panoramic image 600 shown in FIG.

  FIG. 30 b shows images 604 to 606 displayed on the display unit 140 by the display control unit 550. Here, the image 604 corresponds to the image 601 shown in FIG. 30A, the image 605 corresponds to the image 602 shown in FIG. 30A, and the image 606 corresponds to the image 603 shown in FIG. 30A. .

  In this manner, the image (medium image) 604 arranged in the upper stage in the longitudinal direction of the display unit 140 is set as an image near the center of the panoramic image. Further, the image (left image) 605 arranged on the left side of the lower stage in the longitudinal direction of the display unit 140 is used as the image on the left end of the panorama image 600, and the image (right image) 606 arranged on the right side of the lower stage is panorama. Assume that the image is an image on the right end of the image 600.

  Thus, at least three images are displayed at at least two positions in the longitudinal direction of the display unit 140. In the embodiment of the present technology, as described above, the image arranged in the upper stage in the longitudinal direction of the display unit 140 will be described as a middle image. Further, an image arranged on the left side of the lower stage in the longitudinal direction of the display unit 140 will be referred to as a left image, and an image arranged on the right side of the lower stage will be referred to as a right image.

  Note that FIG. 30 shows a display example when the horizontal distance between the middle image and the left image and the middle image and the right image is separated, but the middle image and the left image, the middle image, and the right image are displayed. Each of the ranges may be partially overlapped. An example of this display is shown in FIG. Further, the horizontal boundary between the middle image and the left image, and the middle image and the right image may be in contact with each other. An example of this display is shown in FIG.

  FIG. 31 a shows the relationship between the panoramic image 600 and the images 611 to 613 to be displayed. FIG. 31B shows images 614 to 616 displayed on the display unit 140 by the display control unit 550.

  FIG. 32 a shows the relationship between the panoramic image 600 and the images 621 to 623 to be displayed. FIG. 32 b shows images 624 to 626 displayed on the display unit 140 by the display control unit 550.

  As described above, in the embodiment of the present technology, the region on the left side of the middle image that is not displayed on the middle image is displayed as the whole or a part of the left image. In addition, an area that is not displayed in the middle image and is on the right side of the middle image is displayed as the whole or a part of the right image. Further, a part of the area on the right side of the left image and a part of the area on the right side of the left image are displayed as the whole or part of the middle image.

  In addition, when displaying a left image, a middle image, and a right image, it is preferable that the display magnification of each of these images (the left image, the middle image, and the right image) is approximately matched. In addition, it is preferable that the vertical widths (vertical sizes) of these images are substantially matched.

  Thus, by displaying three images in two stages in the longitudinal direction on the display unit 140, each image can be displayed with a smaller width (horizontal size) than when displaying the entire panoramic image on the display unit 140. it can. That is, an enlarged image can be displayed on the display unit 140 having a certain width as compared with the case where the entire panoramic image is displayed. Thereby, for example, a car (existing at the left end) and a dog (existing at the right end) included in the panoramic image 600 are displayed as images of a relatively large size, so that the user can easily check them. .

  Here, the main purposes in performing live view display are the above-described purposes (a) to (c). For example, (a) confirming whether or not the main subject is captured within the angle of view and (b) confirming how far the left and right areas are recorded as the angle of view, adjusting this area and determining this And (c) confirming the status of the main subject and determining the timing of shooting.

  As described above, for the purposes (a) to (c), it is not always necessary to display the entire panoramic image on the display unit 140. The main subject and the left and right ends of the angle of view are enlarged. It is important to display it. Therefore, as shown in FIG. 30, the middle image, the left image, the middle image, and the right image are not overlapped. Instead, the middle image, the left image, and the right image are within a range that can be displayed on the display unit 140. Is preferably displayed as large as possible.

  Further, when displaying the middle image, the left image, and the right image, if the display magnification (magnification ratio) of the middle image that displays the main subject is different from the left image and the right image that display the left and right edges of the angle of view, The user may be confused when confirming.

  Also, if the vertical width (vertical size) of the middle image displaying the main subject and the left image and right image displaying the left and right ends of the angle of view are different, the user may instantly be aware that the enlargement ratios of the two images are different. There is a risk of illusion. For this reason, it is preferable that the display magnification (enlargement ratio) of the middle image and the left and right images is the same, and the vertical width (vertical size) of the middle image and the left and right images is preferably the same.

  Here, for example, in the case of various information processing apparatuses (for example, mobile phones and smartphones), the display content on the display unit is often displayed so that the user reads from top to bottom. For this reason, the user is often accustomed to seeing the display unit from the top to the bottom.

  Further, for example, when performing live view display, the user often first confirms whether the main subject is properly captured within the angle of view. For this reason, it is conceivable to view the middle image before the left image and the right image. Therefore, in the display unit 140, when the middle image, the left image, and the right image are arranged at different positions on the Y coordinate, the middle image is preferably arranged above the left image and the right image. Moreover, it is preferable to arrange | position the left image and the right image in the same position on a Y coordinate.

  Therefore, FIGS. 30 to 32 show an example in which the middle image is displayed above the left image and the right image in the longitudinal direction of the display unit 140.

[Example of displaying the horizontal width (horizontal size) of the middle image larger than other images]
30 to 32 show an example in which the middle image is arranged on the upper side in the longitudinal direction of the display unit 140 and the left image and the right image are arranged and displayed on the lower side. In addition, an example is shown in which the middle image, the left image, and the right image are displayed with the same size. When displaying each image in this way, only one middle image is arranged on the upper side in the longitudinal direction of the display unit 140, so that an empty area exists on the upper side in the longitudinal direction of the display unit 140. Therefore, in the following, an example in which the empty area on the upper side in the longitudinal direction in the display unit 140 is effectively used will be described.

  33 and 34 are diagrams illustrating a relationship between the panoramic image 600 generated by the image generation unit 510 and the image displayed on the display unit 140 by the display control unit 550 according to the embodiment of the present technology.

  FIG. 33 shows an example in which the aspect ratio and width (horizontal size) of the middle image are displayed larger than those of the left image and the right image. Note that a and b in FIG. 33 are the same as those in FIG. 30 except that the aspect ratio and width (horizontal size) of the middle image are larger than those of the other images (left image and right image). It is the same as the relationship. For example, in FIG. 33, other points (the display magnification of each image is substantially the same, and the vertical width (vertical size) of each image are the same) are the same as those in FIG.

  Thereby, for example, during live view display, the user can check not only the main subject situation but also the surrounding situation in the middle image, so that the user can easily check the vicinity of the center of the panoramic image. be able to. Therefore, the user can easily check the background around the person. Thus, convenience for the user is improved.

  Also, the relationship between the middle image, the left image, and the right image may be a size other than the size shown in FIG. An example of this display is shown in FIG.

  34A shows the relationship between a panoramic image 640 having a horizontal size larger than that of the panoramic image 600 and images 641 to 643 displayed on the display unit 140 by the display control unit 550. FIG.

  FIG. 34 b shows the relationship between the panoramic image 645 whose horizontal size is smaller than that of the panoramic image 600 and the images 646 to 648 displayed on the display unit 140 by the display control unit 550. The horizontal size can be appropriately changed by a user operation.

[Example of displaying the entire panorama image]
In the above, an example in which a panoramic image is divided and displayed has been shown. Here, the details of the panoramic image can be confirmed by each image (the middle image, the left image, and the right image), but it is assumed that there are users who desire confirmation of the entire panoramic image. Therefore, in the following, an example in which an entire panoramic image or a similar image (for example, a panoramic image lacking a part of the entire panoramic image) is displayed simultaneously with each image (a middle image, a left image, and a right image) will be described.

  FIG. 35 is a diagram illustrating a relationship between the panorama image 600 generated by the image generation unit 510 and each image to be displayed and the display example of the display unit 140 according to the embodiment of the present technology.

  FIG. 35 a shows a panoramic image 600 generated by the image generation unit 510. FIG. 35 shows an example in which a reduced image of the entire panoramic image (or an image similar thereto) is displayed together with each image (medium image, left image, and right image). Specifically, in a of FIG. 35, the relationship between the panoramic image 600 and a reduced image 637 obtained by reducing the panoramic image 600 is schematically indicated by four arrows. FIG. 35 b shows a display example of a reduced image 638 (corresponding to the reduced image 637 shown in FIG. 35 a) displayed on the display unit 140. The relationship between a and b in FIG. 35 is the same as the relationship between a and b in FIG. 33 except that the entire panorama image 600 is displayed.

  In particular, when an image is displayed in a state where the longitudinal direction and the vertical direction in the display unit 140 of the information processing apparatus 100 substantially match, it is preferable to display the image as shown in FIG. For example, the panorama image (or the similar image) 638 is preferably smaller than the display magnification of the middle image 634, the left image 635, and the right image 636. That is, it is preferable that the vertical width (vertical size) of the middle image 634, the left image 635, and the right image 636 is larger than that of the entire panoramic image (or similar image) 638.

  Thus, FIG. 35 shows an example in which the entire panoramic image is displayed between the middle image stage and the left and right image stages. It should be noted that the entire panoramic image may be displayed in the display examples shown in FIGS.

  In this way, by displaying the entire panorama image (or an image similar thereto) together with the middle image, the left image, and the right image, the entire panorama image can be easily confirmed. That is, by displaying the middle image, the left image, and the right image, it is possible to confirm the status of the main subject and the left and right angles of view, which are necessary for live view display, for example. Furthermore, the entire panorama image can be easily confirmed by displaying the entire panorama image (or an image similar thereto).

  FIG. 36 is a diagram illustrating a display example of a panoramic image displayed on the display unit 140 and each image to be displayed in the embodiment of the present technology. FIG. 36 shows an example in which the entire panorama image (or an image similar thereto) is displayed outside the display area of the middle image, the left image, and the right image.

  FIG. 36A shows an example in which an entire panoramic image (or an image similar thereto) 638 is displayed below the display area of the left image 635 and the right image 636.

  FIG. 36 b illustrates an example in which the entire panoramic image (or an image similar thereto) 638 is displayed above the display area of the middle image 634.

  Thus, the display position of the entire panoramic image (or an image similar thereto) 638 may be changed as appropriate. Further, the display position may be changed according to the user's preference. For example, the display position may be changed by a user operation (for example, movement by a touch operation).

[Example of determining the middle image based on the subject]
In the above, an example in which the middle image of the panoramic image (the middle portion in the horizontal direction) is the middle image has been shown. As described above, for example, the main purpose in performing live view display is the above-described three points (objects (a) to (c)).

  Here, for example, a case is assumed where the main subject is a person and a panoramic image is taken against a background of a distant landscape. In this case, an image may be shot not only with a composition in which the person who is the main subject is placed at the center of the image, but also with a composition in which the person is placed at a position shifted to the left or right from the center of the image. . In such a case, it is assumed that an inconvenience occurs when the area of the image displayed as the middle image is fixed at the center of the panoramic image (that is, the center of the angle of view). For example, it is assumed that the main subject (for example, a person) is not included in the middle image displayed as the live view display before shooting. In this case, there exists a possibility that the objective (a) and objective (c) mentioned above cannot be achieved.

  In addition, for example, when shooting a butterfly that is trying to stay on a flower as the main subject, it is assumed that the butterfly is shot not in the vicinity of the center of the panoramic image but in a composition that exists on either the left or right end side. To do. In this case, there is a possibility that it is not possible to determine the timing for photographing by confirming whether it is the moment when the butterfly stops on the flower. Therefore, in the following, an example in which the middle image is determined based on the subject is shown.

[Example of determining the middle image based on the position of the face included in the panoramic image]
FIG. 37 is a diagram illustrating a relationship between the panoramic images 650, 655, and 660 generated by the image generation unit 510 and the middle images 651, 656, and 661 according to the embodiment of the present technology.

  FIG. 37 shows an example in which the middle image is determined around the face position included in the panoramic image. Here, rectangles 654, 659, and 664 shown in a to c of FIG. 37 represent the positions of the faces detected by the image generation unit 510 (face detection unit 282). In this way, when a human face is detected from the panoramic image, the display control unit 550 generates a middle image centered on the detected face position (position in the longitudinal direction of the panoramic image).

  For example, as illustrated in a of FIG. 37, when a human face (shown in a rectangle 654) is detected from the panoramic image 650, the display control unit 550 generates a middle image 651 centered on the face. To do. For example, as illustrated in FIG. 37 b, when a human face (shown in a rectangle 659) is detected from the panoramic image 655, the display control unit 550 performs a middle image 656 centered on the face. Is generated. For example, as illustrated in FIG. 37 c, when a human face (shown in a rectangle 664) is detected from the panoramic image 660, the display control unit 550 performs a middle image 661 centered on the face. Is generated.

  FIG. 37 shows an example in which an area including a human face area detected by the face detection unit 282 at the time of shooting and an area including the periphery thereof are used as a middle image. However, an area that is a focus target at the time of shooting and an area that includes the periphery thereof may be used as a middle image. In addition, a moving object area detected by a moving object detection unit (for example, the CPU 202 shown in FIG. 7) at the time of photographing and an area including the periphery thereof may be used as a middle image. FIG. 38 shows an example in which a moving object area detected from a panoramic image and an area including its periphery are used as a middle image.

  Also, a singular point is detected based on the result of image analysis other than these (face detection, moving object detection, focusing) at the time of shooting, and the region including the detected singular point region and its surrounding area is used as a middle image. You may do it.

  FIG. 38 is a diagram illustrating a relationship between panoramic images 665, 670, and 675 generated by the image generation unit 510 and middle images 666, 671, and 676 according to the embodiment of the present technology.

  FIG. 38 shows an example in which the middle image is determined around the position of the butterfly included in the panoramic image. The butterfly may be detected as a moving object, or may be detected as a specific object by object detection.

  As described above, in order to appropriately check the main subject, an image including a singular point detected based on image analysis is used instead of fixing the middle image at the center of the panoramic image (the center of the angle of view). And this periphery can be displayed as a middle image.

[Example of recording coordinates of singular points]
Here, as described above, the main purpose of performing the post-view display is, for example, confirmation of an image recorded by photographing and determination of whether or not retaking of the image is necessary. Note that the confirmation of the image recorded by photographing is, for example, confirmation of whether or not the situation where the main subject is recorded is satisfactory (for example, when photographing a butterfly that is going to stay on a flower, Check if you can shoot the moment when the flower stays in the flower.). Further, for example, it is confirmation whether or not the angle of view (that is, the area of the image recorded by photographing) is satisfactory.

  Also, for example, assume that the main subject is a person and an image with a background of a distant landscape is taken. In this case, as described above, instead of shooting with the composition in which the main subject (person) is placed at the center of the image, the image is placed with a composition in which the person is placed at a position shifted to the left or right from the center of the image. I sometimes shoot.

  In such a case, if the area of the middle image displayed on the display unit 140 is fixed at the center of the angle of view, the image cannot be appropriately confirmed, for example, in post-view display after shooting. There is a fear. For example, when confirming whether or not the situation where the main subject is recorded is satisfactory, and determining whether or not re-taking is necessary, the determination is made because the main subject is not included in the middle image. It is assumed that there is a disadvantage that it is difficult to do.

  For this reason, in order to accurately perform these checks, as described above, the region displayed in the middle image includes a singular point detected based on image analysis, rather than being fixed at the center of the angle of view. It is preferable to display the area and its periphery.

  Here, the detection of the position of the subject by image analysis is often performed for an image input to the image sensor during a live view display period before shooting and an image input to the image sensor during shooting recording. . Also, this detection result is often erased after shooting. Thus, if the detection result of the subject position by image analysis is deleted, it cannot be used at the time of post-view display or playback display.

  Therefore, in the embodiment of the present technology, the position of the singular point (coordinate of the singular point in the panoramic image) detected by image analysis at the time of shooting is recorded in association with the panoramic image. For example, it can be recorded in an area other than the pixel data of the generated image (or an area other than the compressed data). For example, it can be recorded in the header of Exif format image data of a recorded panorama image, the header of a jpeg format image file in the image data, or the like. For example, it can be recorded in a maker note 185 shown in FIG. Then, during post-view display or playback display at an arbitrary time, using the coordinates of the singular points recorded in association with the panoramic image to be displayed and the surrounding area, a middle image including the singular points is obtained. Can be displayed.

  In addition, for example, separately from the panoramic image, an area including a singular point detected at the time of shooting and an image around the area (for example, a middle image) may be recorded with a resolution as a display image. . For example, file identification information (for example, a file name) of the display image can be recorded in the header of a jpeg format image file of the panoramic image. Then, during post-view display or playback display at an arbitrary time, the display image recorded in association with the panoramic image to be displayed can be read and displayed as a middle image.

  In the embodiment of the present technology, an example in which a person's face or butterfly is detected as a specific target has been described, but an object other than a person's face or butterfly may be detected and used as a specific target. . For example, you may make it detect and use specific objects, such as animals (for example, a dog, a cat, a cow, a horse), a car, an airplane, such as a mammal, a reptile, and fish.

[Example of displaying each image area in a panorama image]
In the above, the example which displays the whole image (or image similar to this) of a panoramic image with the middle image, the left image, and the right image was shown. Here, for example, if the position (position in the longitudinal direction) of each displayed image (the middle image, the left image, and the right image) in the panoramic image can be grasped, the user can easily confirm. Therefore, in the following, an example in which a region in a panoramic image of each displayed image (middle image, left image, and right image) is displayed will be described.

  FIG. 39 is a diagram illustrating a display example of a panoramic image generated by the image generation unit 510 and each image (a middle image, a left image, and a right image) according to the embodiment of the present technology. Each image shown in FIG. 39 and its arrangement are the same as the example shown in FIG.

  FIG. 39 a shows an example in which a region in the panorama image 638 of the middle image 634 is represented by a frame 681 on the panorama image 638. 39B, an example in which the region of the panorama image 638 of the left image 635 is represented by a frame 682 on the panorama image 638 and the region of the panorama image 638 of the right image 636 is represented by a frame 683 on the panorama image 638. Show. Note that the display control unit 550 displays these frames 681 to 693 based on the position of each image to be displayed (middle image, left image, and right image).

  In FIG. 39, an example in which the area of the panorama image 638 of each image is represented by a frame on the panorama image 638 is shown, but other display modes may be used. For example, a region of the panorama image 638 of each image may be covered with a semi-transparent shade on the panorama image 638. Further, a region of each image in the panorama image 638 may be covered with a semi-transparent shading on the panorama image 638 and the shading may be blinked. In addition, an area of each image in the panoramic image 638 may be represented on the panoramic image 638 by an underline or an arrow. As shown in FIG. 39b, when the areas of the left image 635 and the right image 636 in the panorama image 638 are displayed on the panorama image 638, the display indicating the area of the right image and the area of the left image are displayed. You may make it make it display different from the display to represent. In addition, the display showing the area | region of the left image and the right image is a frame, a semi-transparent shading, etc., for example. For example, when the area of each image is represented by a frame on the panoramic image (entire image), the display color of the frame may be changed on the left and right.

  FIG. 40 is a diagram illustrating a display example of a panoramic image generated by the image generation unit 510 and each image (a middle image, a left image, and a right image) according to the embodiment of the present technology. FIG. 40 shows an example in which each region in the panoramic image of each image (medium image, left image, and right image) is displayed as a frame on the panoramic image (entire image). FIG. 40 shows a display example in which a panoramic image 685 is added to the display example shown in FIG.

  Specifically, the area of the middle image 604 in the panorama image 685 is represented by a frame 686 on the panorama image 685. In addition, an example in which a region in the panorama image 685 of the left image 605 is represented by a frame 687 on the panorama image 685 and a region in the panorama image 685 of the right image 606 is represented by a frame 688 on the panorama image 685 is shown.

  In FIG. 40, an example in which the area of the panorama image 685 of each image is represented by a frame on the panorama image 685 is shown, but other display modes may be used as in FIG. For example, the area of each image in the panoramic image 685 may be covered with a semi-transparent shade on the panoramic image 685. Further, a region of each image in the panorama image 685 may be covered with a semi-transparent shading on the panorama image 685 and the shading may be blinked. Further, an area of each image in the panorama image 685 may be represented on the panorama image 685 by an underline or an arrow.

  In addition, it is preferable to set it as a different display mode about the display showing the area | region of a middle image, and the display showing the area | region of a left image and a right image. For example, the area of the middle image can be covered with translucent shading, and the area of the left image and the right image can be surrounded by a frame. In this case, it is preferable that the display representing the areas of the left image and the right image have different display modes. For example, the display color of the frame representing the area of the left image and the right image can be changed on the left and right.

[Example of changing the area display according to the mode]
As described above, at the time of shooting, it is preferable to display the area of each image so that the user can easily grasp the composition of each image. However, during playback, it is assumed that emphasis is placed on viewing panoramic images rather than checking the composition of each image. Thus, the contents that the user wants to confirm on the display image differ depending on the mode. For example, the content that the user wants to confirm on the display image differs between live view display, post view display, and playback display. Therefore, here, an example in which a panoramic image is displayed so as to meet the user's desire in each mode when displaying a panoramic image is shown. That is, an example in which the area display is changed according to the display mode is shown.

  For example, in live view display and postview display whose main purpose is image confirmation, display information (for example, frames 681 to 683 and 686 to 688 shown in FIGS. 39 and 40) is displayed as a panoramic image. Display above.

  On the other hand, in reproduction display mainly for viewing images, display information representing the area of each image is not displayed on the panoramic image.

  These display methods may be set when the information processing apparatus 100 is shipped from the factory. Further, in each of the live view display, the post view display, and the playback display, the user may be able to individually set whether or not display information indicating the area of each image is displayed on the panoramic image. For example, an item setting screen may be displayed on the display unit 140 of the information processing apparatus 100, and the user may set the item setting screen.

[Example of scrolling the middle image]
In the above, the example in which the position (position in the longitudinal direction) in the panoramic image of each image (the middle image, the left image, and the right image) displayed on the display unit 140 is fixed and displayed has been shown. Here, it is also possible to display the middle image while moving the position (position in the longitudinal direction) in the panoramic image. Therefore, in the following, an example in which display (so-called automatic scroll display) is performed while moving the position (position in the longitudinal direction) of the middle image in the panoramic image will be described.

  FIG. 41 is a diagram illustrating a transition example of the region of the middle image in the panoramic image generated by the image generation unit 510 according to the embodiment of the present technology. Note that the panoramic image shown in FIG. 41 is the same as the panoramic image 600 shown in FIG. In FIG. 41, the middle image area in the panoramic image is surrounded by a frame 690.

  In FIG. 41, only four regions are shown as transitions of the middle image region in the panoramic image for ease of explanation. However, the movement interval can be advanced by, for example, one pixel or several pixels. . Moreover, about a moving speed, it can be set as the speed which a user can see easily.

  42 and 43 are diagrams illustrating a display example of a panoramic image generated by the image generation unit 510 and each image (a middle image, a left image, and a right image) according to the embodiment of the present technology. The panoramic image 685, the left image 605, and the right image 606 shown in FIGS. 42 and 43 are the same as those in FIG. However, FIG. 42 and FIG. 43 show an example in which only display information (frames 691, 693, 695, 697) relating to the middle image is displayed as the display information representing the area of each image.

  The transition of the middle image area in the panoramic image 685 corresponds to the transition shown in FIG.

  As described above, the middle image is displayed while automatically moving the area in the panoramic image. A so-called automatic scroll display of the middle image is performed.

  Here, whether or not to perform automatic scroll display of the middle image according to the mode may be set. For example, when performing live view display for confirming a subject before photographing and post view display for confirming a recorded image after photographing, automatic scroll display of a middle image is not performed. That is, the position (position in the longitudinal direction) in the panoramic image of the middle image is fixed and displayed. On the other hand, when an image is reproduced and displayed at an arbitrary time, an automatic scroll display of the middle image is performed.

  In addition, for example, the display control unit 550 may switch between the first display method and the second display method based on a user operation during live view or post view. Here, the first display method is, for example, a display method in which the position of the middle image in the panoramic image is fixed and displayed. The second display method is a display method in which an image including a region where a specific object included in a panoramic image is present is displayed as a middle image. On the other hand, the display control unit 550 may switch between the third display method (automatic scroll display) and the second display method based on a user operation during reproduction.

  Alternatively, a specific region (for example, a region of the middle image) may be displayed without performing the automatic scroll display of the middle image only when performing live view display for confirming the subject before shooting. In this case, the automatic scroll display of the middle image may be performed when performing the post-view display for confirming the recorded image after shooting and when performing the reproduction display of the image at an arbitrary time. Or, when performing live view display and post view display, a specific area is displayed without automatic scroll display of the middle image, and when performing playback display of the image at any time, automatic scroll display of the middle image May be performed.

  For example, the display control unit 550 may switch between the first display method and the second display method based on a user operation during live view. On the other hand, the display control unit 550 may switch between the third display method and the second display method based on a user operation during postview or playback.

  These display method settings may be performed when the information processing apparatus 100 is shipped from the factory or when the user uses the information processing apparatus 100. Further, in each of the live view display, the post view display, and the reproduction display, the user may be able to individually set whether or not to perform the automatic scroll display of the middle image. In addition, for example, it is possible to individually set an image display method by a live view operation, an image display method by a post-view operation, and an image display method by a reproduction operation. When such a setting is made, the display control unit 550 can display each image based on the setting. Each of these settings can be performed manually by the user by displaying an item setting screen on the display unit 140 of the information processing apparatus 100 and within the item setting screen. For example, when each function of live view display, post view display, and playback display is activated, a middle image can be displayed according to the contents set in the item setting screen.

  Further, an input unit (for example, a touch panel) for inputting an instruction from the user may be provided on the surface of the housing of the information processing apparatus 100. When an instruction is input from the input unit during each of the live view display, post view display, and playback display, the intermediate image is displayed in the still image display and the automatic scroll display for each display period. And may be switched. Here, the automatic scroll display is an example of a display method in which the middle image is displayed while the region displayed as the middle image is moved from one end of the both ends of the panoramic image to the other end. Still image display is an example of a display method for displaying a certain area as a middle image without moving the area displayed as the middle image.

  For example, a case is assumed in which a switching instruction operation from a user regarding a middle image display method (still image display, automatic scroll display) is received. In this case, the display control unit 550 can display either the automatic scroll display or the still image display and display the middle image based on the switching instruction operation.

  Further, when the still image display or automatic scroll display of the intermediate image is finished, which display method was used as the intermediate image display method may be stored inside the information processing apparatus 100. Then, when each function of live view display, post view display, and playback display is activated after the storage, the middle image may be displayed according to the stored content.

  For example, when the display method of the middle image is switched, the display method related to the switching is held until the display of the middle image based on the switching is completed. When the panorama image is newly displayed after the display of the middle image based on the switching is completed, the middle image in the panorama image can be displayed using the held display method.

[Example of displaying medium image based on user operation]
In the above, an example in which each image (middle image, left image, and right image) is automatically displayed on the display unit 140 has been described. Here, regarding the middle image, it is also important to quickly display a position (position in the longitudinal direction) that the user desires to see. Therefore, in the following, an example in which a middle image is displayed based on a user operation is shown. That is, an example in which the display area of the middle image is changed based on a user operation is shown.

  FIG. 44 is a diagram illustrating a display example of a panoramic image generated by the image generation unit 510 and each image (a middle image, a left image, and a right image) according to the embodiment of the present technology. The panoramic image 685, the left image 605, and the right image 606 shown in FIG. 44 are the same as those in FIG. However, FIG. 44 shows an example in which only the display information (frames 701 and 703) regarding the middle image is displayed as the display information representing the area of each image.

  Here, it is assumed that the display unit 140 of the information processing apparatus 100 is configured with a touch panel. In this case, for example, when the user presses the finger 10 on the display surface of the display unit 140, the pressing operation is detected as a change in electrical characteristics such as a resistance change or a capacitance change. When a pressing operation is detected while a specific area is displayed as one display form of the middle image, a middle image centered on the detected position is generated and displayed. Further, when a touch panel that can detect the proximity of the display unit 140 to the display surface by the user (for example, the proximity of a part of a body such as a hand or a finger) is used, the proximity may also be detected. The middle image can be displayed in the same manner as when the pressing operation is detected.

  Here, it is assumed that the pressing operation is detected during the above-described automatic scroll display as another display mode of the middle image. In this case, a middle image centered on the detected position is displayed. After this display, the middle image may be continuously displayed as a still image, or automatic scroll display may be resumed starting from that position.

  For example, it is assumed that a designation operation for designating a position in a panoramic image is performed. In this case, when the middle image is displayed by the automatic scroll display, the display control unit 550 uses the image including the position in the panorama image designated by the designation operation and the surrounding area as a new middle image. Display. Then, the display control unit 550 displays the middle image while moving from the position as a starting point. On the other hand, if the middle image is displayed by still image display when the designation operation is performed, the display control unit 550 displays an image including the position and the surrounding area as a new middle image. Let

  As the touch panel, for example, an electrostatic (capacitance type) touch panel, a pressure-sensitive (resistance film pressure type) touch panel, an optical touch panel, or the like can be used.

  In addition, the position where the middle image is to be displayed may be designated using an operation member (for example, a cross key) other than the touch panel.

[Modified example of two-stage display]
The example in which the middle image, the left image, and the right image are displayed in two stages in the longitudinal direction of the display unit 140 has been described above. However, depending on the size of each image, the middle image, the left image, and the right image may be displayed in one row in the longitudinal direction of the display unit 140. Therefore, in the following, an example in which the middle image, the left image, and the right image are displayed in one row in the longitudinal direction of the display unit 140 will be described.

  FIG. 45 is a diagram illustrating a relationship between the panoramic image 600 generated by the image generation unit 510 and each image to be displayed and the display example of the display unit 140 according to the embodiment of the present technology.

  FIG. 45 a shows a panoramic image 600 generated by the image generation unit 510. The relationship shown in FIG. 45a is the same as the relationship shown in FIG. FIG. 45 b shows a display example of the panoramic image 711 and each image (the middle image 712, the left image 713, and the right image 714) displayed on the display unit 140.

  As shown in a and b of FIG. 45, by reducing the horizontal width of each image (the middle image 712, the left image 713, and the right image 714), each of these images is displayed in one stage in the longitudinal direction of the display unit 140. Can be displayed. Accordingly, each image that matches the horizontal direction of the panoramic image 711 can be displayed.

  Here, when performing operations such as a call operation of an information processing apparatus such as a smartphone or a mobile phone, creation of e-mail text, and text reading, the casing is often used in a vertically long shape. For this reason, for example, even when shooting is performed using an information processing apparatus with a camera, shooting is often performed with the casing and the display unit in a vertically long state, and a vertically long image is recorded.

  Therefore, in the embodiment of the present technology, in the information processing apparatus 100 such as a smartphone or a mobile phone, a plurality of imaging systems are arranged side by side in an orthogonal direction orthogonal to the longitudinal direction of the information processing apparatus 100. As a result, a panoramic image can be recorded even if the information processing apparatus 100 is photographed in a vertically long state.

  When displaying a panoramic image with the information processing apparatus 100 in a vertically long state during live view display, post view display, or playback display, a plurality of images obtained by dividing the panorama image are displayed in the longitudinal direction of the display unit 140. In FIG. Thereby, in the portable information processing apparatus 100 with a compound eye camera, it is possible to easily shoot a compound eye panoramic image in a state where the user holds the information processing apparatus 100 vertically long. Even if the panorama image is displayed with the information processing apparatus 100 in the vertically long state, the panorama image can be appropriately displayed.

[Display example when changing the state of the information processing device (portrait state, landscape state)]
The panoramic image display example when the information processing apparatus 100 is in the vertically long state has been described above. However, depending on the user, it may be assumed that the panoramic image is viewed with the information processing apparatus 100 in a horizontally long state.

  Here, for example, the same number of display stages may be considered in the horizontally long state and the vertically long state. For example, when displaying a panoramic image in a horizontally long state, the panoramic image can be displayed in a large size by displaying the panoramic image to the full width of the display unit 140. However, with respect to the vertical height of the margin areas that can be formed above and below the panoramic image, the panoramic image is displayed in the horizontal state of the display unit 140 in the horizontal state, compared to the case where the panoramic image is displayed in the horizontal direction of the display unit 140 in the vertical state. Narrower when is displayed. As described above, in the horizontally long state, the margin areas formed above and below the panoramic image are narrower than in the vertically long state. For this reason, it is preferable to reduce the number of display stages in the horizontally long state than in the vertically long state. Therefore, in the following, a display example in the case where the state (vertically long state, horizontally long state) of the information processing apparatus 100 is changed is shown.

  FIG. 46 is a diagram illustrating a display example when the panorama image generated by the image generation unit 510 according to the embodiment of the present technology is displayed on the display unit 140.

  46A to 46C, panoramic image display examples when the information processing apparatus 100 is in the landscape orientation are shown. On the right side of FIG. 46A to 46C, the information processing apparatus 100 is in the portrait orientation. The example of a display of the panorama image in a case is shown.

  For example, when the information processing apparatus 100 is in the landscape state, only the panorama image 720 or the panorama image 720 and the middle image 721 are displayed on the display unit 140. Further, for example, when the information processing apparatus 100 is in a vertically long state, each of the middle image 722, the left image 723, and the right image 724, or a reduced image 725 of a panoramic image is displayed together with these images. Displayed on the section 140.

  As described above, when the user has the information processing apparatus 100 so that the display unit 140 is in the vertically long state, the number of stages of each image is set so that the user has the information processing apparatus 100 so that the display unit 140 is in the horizontally long state. The number of stages of each image is larger than the number of stages. Thus, even when the state of the information processing apparatus (vertically long state, horizontally long state) is changed, the user can easily display the panoramic image.

[Modification of information processing apparatus]
Next, a modified example of the information processing apparatus including a plurality of imaging systems will be described.

  47 and 48 are diagrams illustrating a modification of the information processing apparatus 100 according to the embodiment of the present technology.

  47 and 48 show an information processing apparatus 800 in which the display unit 140 and the imaging unit 130 are provided in different housings (first housing 811 and second housing 812). 47a shows a top view of the information processing apparatus 800, and FIG. 47b shows an arrangement example of the electronic substrate of the information processing apparatus 800. 47a corresponds to a in FIG. 1, and b in FIG. 47 corresponds to e in FIG. 48a shows a front view of the information processing apparatus 800, FIG. 48b shows a side view of the information processing apparatus 800, and FIG. 48c shows a rear view of the information processing apparatus 800. Indicates. 48 corresponds to b in FIG. 1, b in FIG. 48 corresponds to c in FIG. 1, and c in FIG. 48 corresponds to d in FIG. In addition, portions common to the information processing apparatus 100 illustrated in FIG. 1 are denoted by the same reference numerals.

  The information processing apparatus 800 is realized by, for example, an information processing apparatus including a plurality of imaging systems (for example, a folding mobile phone with a compound eye camera).

  The information processing apparatus 800 includes a rotating member 802, a first housing 811, and a second housing 812. The first housing 811 and the second housing 812 are connected so as to be foldable with the rotating member 802 as a rotating shaft. In other words, the operation member 801 and the display unit 140 can be folded so that the rotation member 802 is a rotation axis.

  The first housing 811 includes an imaging unit 130 and a display unit 140. The second housing 812 includes an operation unit 801, an electronic substrate 815, and a battery storage unit 816.

  The operation unit 801 is an operation unit including a numeric keypad, a cross key, a determination key, and the like.

  As described above, the embodiments of the present technology can be applied to a foldable information processing apparatus 800 including a plurality of imaging systems. In this modification, an example in which three imaging systems are arranged side by side according to a predetermined rule is shown. However, the embodiment of the present technology can be applied to an information processing apparatus (for example, an imaging apparatus such as a digital still camera or a digital video camera (for example, a camera-integrated recorder)) including two or four or more imaging systems. . That is, the embodiment of the present technology can be applied by arranging two or four or more imaging systems side by side according to a certain rule. An example in which two imaging systems are arranged side by side according to a certain rule is shown in FIGS.

  49 to 51 are diagrams illustrating modifications of the information processing apparatus 100 according to the embodiment of the present technology.

  FIG. 49 illustrates an information processing device 820 in which the display unit 140 and the imaging unit 830 are provided in one housing. 49a shows a top view of the information processing apparatus 820, and FIG. 49b shows a front view of the information processing apparatus 820. 49 shows a side view of the information processing apparatus 820, and FIG. 49d shows a rear view of the information processing apparatus 820. FIG. 49e shows an example of the arrangement of the electronic boards of the information processing apparatus 820. 49 correspond to a to e in FIG. 1. In addition, portions common to the information processing apparatus 100 illustrated in FIG. 1 are denoted by the same reference numerals.

  49, the optical axes of the two imaging systems (optical systems 831 and 832 and imaging elements 833 and 834) constituting the imaging unit 830 are optical axes 835 and 836. In this case, an angle formed between the optical axis 835 and a perpendicular line 837 orthogonal (or substantially orthogonal) to the display surface of the display unit 140 is θ, and an angle formed between the optical axis 836 and the perpendicular line 837 is θ. That is, the two imaging systems (optical system 831, optical system 831, 832, imaging elements 833, 834) are arranged so that the optical axes 835 and 836 are symmetrical with respect to the perpendicular 837. 832 and image sensors 833 and 834) are arranged.

  The information processing device 820 is realized by, for example, an information processing device including a plurality of imaging systems (for example, a smartphone with a compound eye camera or a mobile phone with a compound eye camera).

  The information processing device 820 includes an electronic substrate 101, a battery storage unit 102, a changeover switch 111, a determination key 112, a display unit 140, and an imaging unit 830. The information processing device 820 is the same as the information processing device 100 shown in FIG. 1 except that the number of imaging systems (optical systems 831 and 832 and imaging elements 833 and 834) constituting the imaging unit 830 is two. For this reason, detailed description here is abbreviate | omitted.

  50 and 51 show an information processing device 840 provided in different housings (first housing 811 and second housing 812) in which the display unit 140 and the imaging unit 830 are provided. FIG. 50 a shows a top view of the information processing apparatus 840, and FIG. 50 b shows an arrangement example of the electronic substrate of the information processing apparatus 840. 51a shows a front view of the information processing apparatus 840, FIG. 51b shows a side view of the information processing apparatus 840, and FIG. 51c shows a rear view of the information processing apparatus 840. Indicates. Note that a and b in FIG. 50 and a to c in FIG. 51 correspond to a and b in FIG. 47 and a to c in FIG. 48. Further, portions common to the information processing device 800 shown in FIGS. 47 and 48 and the information processing device 820 shown in FIG. 49 are denoted by the same reference numerals.

  The information processing apparatus 840 is realized by, for example, an information processing apparatus including a plurality of imaging systems (for example, a folding mobile phone with a compound eye camera).

  The information processing device 840 includes a rotating member 802, a first housing 811, and a second housing 812. Note that the information processing apparatus 840 is the same as the information processing apparatus 800 shown in FIGS. 47 and 48 except that two image pickup systems (optical systems 831 and 832 and image pickup devices 833 and 834) that constitute the image pickup unit 830 are provided. It is. For this reason, detailed description here is abbreviate | omitted.

[Display example of panorama image and other images]
As described above, a panoramic image generated by an information processing apparatus including a plurality of imaging systems can be displayed. Here, since an information processing apparatus including a plurality of imaging systems can generate a normal image (for example, an image having an aspect ratio of 3: 4), each of these images can be appropriately displayed. is important.

  Therefore, in the following, an example in which each image generated by an information processing apparatus including a plurality of imaging systems is appropriately displayed will be described.

[Display example of information processing apparatus including three imaging units]
52 to 54 are diagrams illustrating display examples of each image generated by the information processing device according to the embodiment of the present technology. 52 to 54 show the relationship between image readout areas (indicated by bold lines) in the three image sensors and images displayed on the display unit 140 by the display control unit 550. FIG. Each of these images is generated by the information processing apparatus 100 shown in FIG. 1 or the information processing apparatus 800 shown in FIGS. 47 and 48, for example.

  52A shows a display example when an image (for example, a vertically long 3: 4 image) 851 is generated using a part of one of the three image sensors (the middle image sensor). . In this case, as shown in FIG. 52b, the entire image 852 generated by one image sensor is displayed. In other words, the number of stages for displaying an image (for example, a vertically long 3: 4 image) on the display screen of the display unit 140 can be displayed with a smaller number of stages (for example, one stage) than that for displaying a panoramic image. .

  FIG. 53A shows a display example when an image (for example, a horizontally long 4: 3 image) 853 is generated using all of one of the three image sensors (middle image sensor). In this case, as shown in FIG. 53b, the entire image 854 generated by one image sensor is displayed. In other words, the number of stages for displaying an image (for example, landscape 4: 3 image) on the display screen of the display unit 140 can be displayed with a number of stages (for example, one stage) smaller than the number of stages for displaying a panoramic image. .

  FIG. 54a shows a display example when a panoramic image 850 is generated using three image sensors. In this case, as shown in b of FIG. 54, the panoramic image 850 is divided and displayed. For example, a plurality of images 855 to 857 obtained by dividing the panoramic image 850 are displayed as a middle image 858, a left image 859, and a right image 860.

[Display example of information processing apparatus including two imaging units]
55 to 57 are diagrams illustrating display examples of each image generated by the information processing apparatus according to the embodiment of the present technology. 55 to 57 show the relationship between the image reading areas (indicated by bold lines) in the two image sensors and the images displayed on the display unit 140 by the display control unit 550. FIG. Each of these images is generated by, for example, the information processing device 820 shown in FIG. 49 or the information processing device 840 shown in FIGS.

  FIG. 55 a shows a display example when an image (for example, a vertically long 3: 4 image) 871 is generated using a part of two image sensors. In this case, as shown in FIG. 55b, the entire image 872 generated by a part of the two image sensors is displayed. In other words, the number of stages for displaying an image (for example, a vertically long 3: 4 image) on the display screen of the display unit 140 can be displayed with a number of stages (for example, one stage) smaller than the number of stages for displaying a panoramic image. .

  FIG. 56a shows a display example when an image (for example, a horizontally long 4: 3 image) 873 is generated using a part of two image sensors. In this case, as shown in FIG. 56b, the entire image 874 generated by a part of the two image sensors is displayed. In other words, the number of stages for displaying an image (for example, landscape 4: 3 image) on the display screen of the display unit 140 can be displayed with a number of stages (for example, one stage) smaller than the number of stages for displaying a panoramic image. .

  FIG. 57a shows a display example when a panoramic image 870 is generated using two imaging elements. In this case, as shown in FIG. 57b, the panoramic image 870 is divided and displayed. For example, a plurality of images 875 to 877 obtained by dividing the panoramic image 870 are displayed as a middle image 878, a left image 879, and a right image 880.

[Operation example of information processing device]
FIG. 58 is a flowchart illustrating an example of a processing procedure of display control processing by the information processing apparatus 100 according to the embodiment of the present technology.

  First, it is determined whether the shooting mode or the playback mode is set as the operation mode (step S901). When an operation mode other than the shooting mode and the playback mode is set, processing according to the set operation mode is performed.

  If the shooting mode is set (step S901), the shooting function is activated (step S902), and shooting processing is performed (step S910). This photographing process will be described in detail with reference to FIG.

  Subsequently, it is determined whether or not the shooting mode has been released (step S903). If the shooting mode has not been released, the process returns to step S910. On the other hand, when the shooting mode is canceled (step S903), it is determined whether an operation mode other than the shooting mode and the playback mode has been set (step S904). If no operation mode other than the shooting mode and the playback mode is set (step S904), the process returns to step S901. On the other hand, when an operation mode other than the shooting mode and the playback mode is set (step S904), the display control processing operation ends.

  If the playback mode is set (step S901), the playback function is activated (step S905), and playback processing is performed (step S940). This reproduction process will be described in detail with reference to FIG. Note that step S940 is an example of a display control procedure described in the claims.

  Subsequently, it is determined whether or not the playback mode has been released (step S906). If the playback mode has not been released, the process returns to step S940. On the other hand, when the playback mode is canceled (step S906), it is determined whether an operation mode other than the shooting mode and the playback mode has been set (step S904).

  FIG. 59 is a flowchart illustrating an example of an imaging processing procedure (processing procedure in step S910 illustrated in FIG. 58) in the display control processing procedure by the information processing apparatus 100 according to the embodiment of the present technology.

  First, the display control unit 550 determines whether or not the aspect ratio of the image generated by the image generation unit 510 is that of a panoramic image (step S911). If the aspect ratio of the image is that of a panoramic image (step S911), panorama shooting processing is performed (step S930). The panorama shooting process will be described in detail with reference to FIG.

  If the aspect ratio of the image is not that of the panoramic image (step S911), the display control unit 550 determines whether the aspect ratio of the image is 3: 4 (step S912). When the aspect ratio of the image is 3: 4 (step S912), the display control unit 550 causes the display unit 140 to display the image (3: 4 image) generated by the image generation unit 510 ( Step S913). That is, 3: 4 images are displayed in live view.

  Subsequently, it is determined whether or not a shutter button (for example, the enter key 112) has been pressed (step S914). If the shutter button has not been pressed, the operation of the photographing process is terminated. On the other hand, when the shutter button is pressed (step S914), the recording control unit 530 records the image (3: 4 image) generated at the pressing timing in the storage unit 540 (step S915). . Subsequently, the display control unit 550 causes the display unit 140 to display an image (3: 4 image) to be recorded in the storage unit 540 (step S916). That is, a 3: 4 image is post-view displayed.

  When the aspect ratio of the image is not 3: 4 (4: 3) (step S912), the display control unit 550 displays the image (4: 3 image) generated by the image generation unit 510. It is displayed on the display unit 140 (step S917). That is, a 4: 3 image is displayed in live view.

  Subsequently, it is determined whether or not the shutter button has been pressed (step S918). If the shutter button has not been pressed, the operation of the photographing process is terminated. On the other hand, when the shutter button is pressed (step S918), the recording control unit 530 records the image (4: 3 image) generated at the pressing timing in the storage unit 540 (step S919). . Subsequently, the display control unit 550 causes the display unit 140 to display an image (4: 3 image) to be recorded in the storage unit 540 (step S920). That is, a 4: 3 image is displayed in post view.

  FIG. 60 is a flowchart illustrating an example of a panorama shooting processing procedure (processing procedure of step S930 illustrated in FIG. 59) in the display control processing procedure by the information processing apparatus 100 according to the embodiment of the present technology.

  First, the display control unit 550 displays the image (panoramic image) generated by the image generation unit 510 on the display unit 140 (step S931). In this case, a plurality of images obtained by dividing the panoramic image are displayed in a live view in a plurality of stages (a plurality of stages in the longitudinal direction of the display unit 140) (step S931). For example, a live view image is displayed as shown in FIG. In this case, when a singular point is detected, as shown in FIGS. 37 and 38, an image including the singular point may be displayed as a middle image.

  Subsequently, it is determined whether or not the shutter button has been pressed (step S932). If the shutter button has not been pressed, the panorama shooting processing operation is terminated. On the other hand, when the shutter button is pressed (step S932), the recording control unit 530 records the image (panorama image) generated at the timing of the pressing in the storage unit 540 (step S933). In this case, when a singular point is detected, information regarding the singular point (for example, coordinates of the singular point in the panoramic image) is recorded in association with the panoramic image. In addition to the panoramic image, a peripheral image including the singular point may be recorded in association with the panoramic image. The peripheral image including the singular point can be recorded as a display image (for example, a thumbnail image), for example.

  Subsequently, the display control unit 550 causes the display unit 140 to display an image (panorama image) to be recorded in the storage unit 540 (step S934). In this case, a plurality of images obtained by dividing the panoramic image are post-view displayed in a plurality of stages (a plurality of stages in the longitudinal direction of the display unit 140) (step S934). For example, as shown in FIG. 32 etc., a post-view image is displayed. In this case, when a singular point is detected, as shown in FIGS. 37 and 38, an image including the singular point may be displayed as a middle image. Step S933 is an example of an imaging procedure. Steps S931 and S934 are an example of a display control procedure described in the claims.

  FIG. 61 is a flowchart illustrating an example of a reproduction processing procedure (processing procedure in step S940 illustrated in FIG. 58) in the display control processing procedure performed by the information processing apparatus 100 according to the embodiment of the present technology.

  First, the display control unit 550 determines whether the aspect ratio of the image instructed to be reproduced by the user (the image stored in the storage unit 540) is that of a panoramic image (step S941). If the aspect ratio of the image is that of a panoramic image (step S941), the display control unit 550 determines whether or not the display unit 140 is in the vertically long state based on the posture information from the posture detection unit 150. Is determined (step S942).

  When the display unit 140 is in the vertically long state (step S942), the display control unit 550 displays on the display unit 140 the panorama image (the panorama image stored in the storage unit 540) instructed to be reproduced by the user. It is displayed (step S943). In this case, a plurality of images obtained by dividing the panoramic image are displayed in a plurality of stages (a plurality of stages in the longitudinal direction of the display unit 140) (step S943). For example, an image is displayed as shown in FIG. In this case, when a singular point is recorded in association with a panoramic image, an image including the singular point may be displayed as a middle image as shown in FIGS.

  When the display unit 140 is not in the vertically long state (in the case of the horizontally long state) (step S942), the display control unit 550 displays a panorama image (a panorama image stored in the storage unit 540) instructed to be reproduced by the user. ) Is displayed on the display unit 140 (step S944). In this case, a panoramic image is displayed with a lower number of steps (the number of steps in the longitudinal direction of the display unit 140) than when displayed in the vertically long state (step S944). For example, images are displayed as shown on the left side of FIGS.

  If the aspect ratio of the image is not that of the panoramic image (step S941), the display control unit 550 determines whether the aspect ratio of the image is 3: 4 (step S945). When the aspect ratio of the image is 3: 4 (step S945), the display control unit 550 determines whether or not the display unit 140 is in the portrait state based on the posture information from the posture detection unit 150. Is determined (step S946).

  When the display unit 140 is in the vertically long state (step S946), the display control unit 550 displays an image (a 3: 4 image stored in the storage unit 540) instructed to be reproduced by the user. (Step S947). In this case, the 3: 4 image instructed to be reproduced is displayed upright (step S947). For example, as shown in FIGS. 52 and 55, images are displayed.

  When the display unit 140 is not in the vertically long state (in the case of the horizontally long state) (step S946), the display control unit 550 displays an image (3: 4 stored in the storage unit 540) instructed to be reproduced by the user. Image) is displayed on the display unit 140 (step S948). In this case, the 3: 4 image instructed to be reproduced is displayed upright (step S948).

  When the aspect ratio of the image is not 3: 4 (when the aspect ratio is 4: 3) (step S949), the display control unit 550 is based on the posture information from the posture detection unit 150. It is determined whether or not the display unit 140 is in the vertically long state (step S949).

  When the display unit 140 is in the vertically long state (step S950), the display control unit 550 displays an image (4: 3 image stored in the storage unit 540) instructed to be reproduced by the user. (Step S950). In this case, the 4: 3 image instructed to be reproduced is displayed upright (step S950). For example, images are displayed as shown in FIGS.

  When the display unit 140 is not in the vertically long state (in the case of the horizontally long state) (step S949), the display control unit 550 displays the image instructed to be reproduced by the user (4: 3 stored in the storage unit 540). Image) is displayed on the display unit 140 (step S951). In this case, the 4: 3 image instructed to be reproduced is displayed upright (step S951).

[Operation example when scrolling the middle image]
The operation example in the case where the position (position in the longitudinal direction) in the panoramic image of each image (the middle image, the left image, and the right image) displayed on the display unit 140 is fixed and displayed has been described above. In the following, an operation example in the case of displaying (so-called automatic scroll display) while moving the position (position in the longitudinal direction) of the middle image in the panoramic image will be shown.

  FIG. 62 is a flowchart illustrating an example of a processing procedure of display control processing by the information processing apparatus 100 according to the embodiment of the present technology. 62 is a modification of FIG. 58, and portions common to FIG. 58 are denoted by the same reference numerals and description thereof is partially omitted.

  First, the scroll method is set (step S907). In the setting of the scroll method, presence / absence of scroll display at each display is set. The scroll method is set by, for example, manual setting or automatic setting by the user. These setting contents are held in the display control unit 550, for example.

  If the shooting mode is set (step S901), the shooting function is activated (step S902), and shooting processing is performed (step S910). The panorama shooting process (step S930 shown in FIG. 59) in this shooting process (shown in FIG. 59) will be described in detail with reference to FIG.

  If the playback mode is set (step S901), the playback function is activated (step S905), and playback processing is performed (step S960). This reproduction process will be described in detail with reference to FIG.

  FIG. 63 is a flowchart illustrating an example of a panorama shooting processing procedure (processing procedure of step S930 illustrated in FIG. 59) in the display control processing procedure by the information processing apparatus 100 according to the embodiment of the present technology. 63 is a modification of FIG. 60, and portions common to FIG. 60 are denoted by the same reference numerals, and a part of the description is omitted.

  First, the display control unit 550 reads the setting of the scroll method at the time of live view display (step S921). Subsequently, the display control unit 550 causes the display unit 140 to display the image (panoramic image) generated by the image generation unit 510 according to the read scroll method setting (step S922). For example, when the live view display is set to have scrolling, as shown in FIGS. 42 and 43, the middle image is displayed while moving the position of the middle image in the panoramic image (position in the longitudinal direction). Done. On the other hand, when the setting for no scrolling is performed during live view display, a middle image with a fixed position in the panoramic image is displayed.

  Subsequently, it is determined whether or not the scroll method setting has been changed (step S923). For example, at the time of live view display, the scroll method setting is changed by manual operation by the user (for example, change of scroll display from yes to no, no from yes to yes). When the scroll method setting is changed (step S923), the scroll method during live view display is changed (step S924), and the changed scroll method is saved (step S925).

  Subsequently, the display control unit 550 reads the setting of the scroll method at the time of postview display (step S926). Subsequently, the display control unit 550 causes the display unit 140 to display an image (panorama image) to be recorded in the storage unit 540 in accordance with the read scroll method setting (step S927). For example, when scrolling is set during postview display, as shown in FIGS. 42 and 43, the middle image is displayed while moving the position (position in the longitudinal direction) of the middle image in the panoramic image. Done. On the other hand, when no scroll is set during postview display, a middle image with a fixed position in the panoramic image is displayed.

  Subsequently, it is determined whether or not the scroll method setting has been changed (step S928). For example, at the time of post-view display, the scroll method setting is changed by manual operation by the user (for example, changing the scroll display from yes to no and no to yes). When the scroll method setting is changed (step S928), the scroll method during post-view display is changed (step S929), and the changed scroll method is saved (step S935).

  FIG. 64 is a flowchart illustrating an example of a reproduction processing procedure (processing procedure in step S960 illustrated in FIG. 62) in the display control processing procedure performed by the information processing apparatus 100 according to the embodiment of the present technology. 64 is a modification of FIG. 61, and portions common to FIG. 61 are denoted by the same reference numerals, and a part of the description is omitted.

  When the display unit 140 is in the vertically long state (step S942), the vertically long state panorama reproduction process is performed (step S970). This vertically long panorama reproduction process will be described in detail with reference to FIG.

  If the display unit 140 is not in the vertically long state (in the case of the horizontally long state) (step S942), a horizontally long state panorama reproduction process is performed (step S980). This landscape-oriented panorama playback process will be described in detail with reference to FIG.

  FIG. 65 is a flowchart illustrating an example of the vertically long panorama reproduction processing procedure (the processing procedure of step S970 illustrated in FIG. 64) in the display control processing procedure by the information processing apparatus 100 according to the embodiment of the present technology.

  First, the display control unit 550 reads the setting of the scroll method at the time of reproduction display (step S971). Subsequently, the display control unit 550 causes the display unit 140 to display an image (a panoramic image stored in the storage unit 540) instructed to be reproduced by the user according to the read scroll method setting (step S972). . For example, when the display is set to be scrolled during playback display, as shown in FIGS. 42 and 43, the middle image is displayed while moving the position of the middle image in the panoramic image (position in the longitudinal direction). Is called. On the other hand, if the setting for no scrolling is made during playback display, a middle image with a fixed position in the panoramic image is displayed.

  Subsequently, it is determined whether or not the scroll method setting has been changed (step S973). For example, at the time of playback display, the scroll method setting is changed by manual operation by the user (for example, change of scroll display from yes to no, no from yes to yes). If the scroll method setting has been changed (step S973), the scroll method during playback display is changed (step S974), and the changed scroll method is saved (step S975).

  FIG. 66 is a flowchart illustrating an example of the landscape state panorama reproduction processing procedure (the processing procedure of step S980 illustrated in FIG. 64) in the display control processing procedure performed by the information processing apparatus 100 according to the embodiment of the present technology.

  First, the display control unit 550 reads the setting of the scroll method at the time of reproduction display (step S981). Subsequently, the display control unit 550 causes the display unit 140 to display an image (a panoramic image stored in the storage unit 540) instructed to be reproduced by the user according to the read scroll method setting (step S982). . For example, when the display is set to be scrolled during playback display, as shown in FIGS. 42 and 43, the middle image is displayed while moving the position of the middle image in the panoramic image (position in the longitudinal direction). Is called. On the other hand, if the setting for no scrolling is made during playback display, a middle image with a fixed position in the panoramic image is displayed.

  Subsequently, it is determined whether or not the scroll method setting has been changed (step S983). For example, at the time of playback display, the scroll method setting is changed by manual operation by the user (for example, change of scroll display from yes to no, no from yes to yes). If the scroll method setting has been changed (step S983), the scroll method during playback display is changed (step S984), and the changed scroll method is saved (step S985).

<2. Second Embodiment>
In 1st Embodiment of this technique, the example which fixes and arrange | positions an imaging part to information processing apparatus was shown. However, the first embodiment of the present technology can also be applied to an information processing apparatus in which the imaging unit is movable.

  Therefore, in the second embodiment of the present technology, an example of an information processing apparatus in which the imaging unit is arranged to be movable is shown.

  67 and 68 are diagrams illustrating an external configuration example of the information processing apparatus 1000 according to the second embodiment of the present technology.

  67 and 68 show an information processing apparatus 1000 in which an imaging unit 1030 is movably arranged. 67A shows a top view of the information processing apparatus 1000, and FIG. 67B shows a front view of the information processing apparatus 1000. 67c shows a side view of the information processing apparatus 1000, and FIG. 67d shows a rear view of the information processing apparatus 1000. FIG. 67 e shows an example of the arrangement of the electronic boards of the information processing apparatus 1000. 67 correspond to a to e in FIG. 1. In addition, portions corresponding to the information processing apparatus 100 illustrated in FIG. 1 are denoted by the same reference numerals. 68a shows a front perspective view of the information processing apparatus 1000, and FIG. 68b shows a rear perspective view of the information processing apparatus 1000.

  The information processing apparatus 1000 is realized by, for example, an information processing apparatus including a plurality of imaging systems (for example, a mobile phone with a movable compound eye camera).

  The information processing apparatus 1000 includes a first housing 1010, a second housing 1020, and a rotating member 1021. The second housing 1020 is a housing attached to the first housing 1010 that houses the display unit 140, and is a rotatable housing having a rotation axis that is parallel (or substantially parallel) to the short side of the display unit 140. Is the body. In addition, the three imaging systems (the optical systems 131 to 133 and the imaging elements 134 to 136) constituting the imaging unit 1030 are parallel (or substantially) to the rotation axis of the second casing 1020 on one surface of the second casing 1020. Are arranged side by side in the direction of (parallel). As described above, by arranging the imaging unit 1030 to be movable, the imaging unit 1030 is rotated in the directions of arrows 1022 and 1023 shown in FIG. 68 within a range of 180 degrees with the rotation member 1021 as the rotation axis. Can be made. For example, when the user has the information processing apparatus 1000 in the state illustrated in FIG. 68 b, the user can take a horizontally long image including himself / herself while viewing the display unit 140. Further, for example, when the user has the information processing apparatus 1000 in a state where the optical axis of the imaging unit 1030 is parallel (or substantially parallel) to the gravity direction, It is also possible to take a horizontally long image with (for example, the sky) as a subject.

  As described above, the first embodiment of the present technology can be applied to the information processing apparatus 1000 that includes a plurality of imaging systems and in which these imaging systems are movably arranged. In this example, an example in which three imaging systems are arranged side by side according to a predetermined rule is shown. However, the first embodiment of the present technology is also applied to an information processing apparatus (for example, an imaging apparatus such as a digital still camera or a digital video camera (for example, a camera-integrated recorder)) including two or four or more imaging systems. can do. An example in which two imaging systems are arranged side by side according to a certain rule is shown in FIGS.

  69 and 70 are diagrams illustrating an external configuration example of the information processing device 1050 according to the second embodiment of the present technology.

  69 and 70 show an information processing apparatus 1050 in which an imaging unit 1080 including two imaging systems is movably arranged. 69 a shows a top view of the information processing apparatus 1050, and FIG. 69 b shows a front view of the information processing apparatus 1050. 69c shows a side view of the information processing apparatus 1050, and FIG. 69d shows a rear view of the information processing apparatus 1050. In addition, FIG. 69 e illustrates an example of the arrangement of the electronic substrate of the information processing apparatus 1050. Note that a to e in FIG. 69 correspond to a to e in FIG. 49. In addition, portions corresponding to the information processing device 820 illustrated in FIG. 49 are denoted by the same reference numerals. Further, FIG. 70 a shows a front perspective view of the information processing apparatus 1050, and FIG. 70 b shows a rear perspective view of the information processing apparatus 1050.

  The information processing device 1050 is realized, for example, by an information processing device (for example, a mobile phone with a movable compound eye camera) including two imaging systems.

  The information processing apparatus 1050 includes a first housing 1060, a second housing 1070, and a rotating member 1071. The second housing 1070 is a housing attached to the first housing 1060 that houses the display unit 140, and is a rotatable housing having a rotation axis that is parallel (or substantially parallel) to the short side of the display unit 140. Is the body. In addition, two imaging systems (optical systems 831 and 832 and imaging elements 833 and 834) constituting the imaging unit 1080 are parallel (or substantially) to the rotation axis of the second casing 1070 on one surface of the second casing 1070. Are arranged side by side in the direction of (parallel). As described above, by arranging the image pickup unit 1080 to be movable, the image pickup unit 1080 is rotated in the directions of arrows 1072 and 1073 shown in FIG. 70 within a range of 180 degrees with the rotation member 1071 as the rotation axis. Can be made. Thereby, for example, as in the example shown in FIGS. 67 and 68, the user can view a horizontally long image including himself / herself while viewing the display unit 140 or the upper side (for example, the sky) while viewing the display unit 140. It is also possible to take a horizontally long image.

  As described above, the first embodiment of the present technology can be applied to the information processing device 1050 that includes two imaging systems and these imaging systems are movably disposed. 67 to 70 show examples of the information processing apparatus that can rotate the imaging unit within a range of 180 degrees with the rotation member as the rotation axis. However, the rotation range may be a range other than 180 degrees. For example, the imaging unit 1030 may be arranged so that it can be rotated within the range of 90 degrees around the rotation member 1071 in the direction of the arrow 1022 from the state shown in FIG. Good.

  The embodiment of the present technology is applied to an information processing device (electronic device, image processing device, display device, display control device) capable of displaying a panoramic image on a display device (built-in display device or external display device). can do. For example, the present invention can be applied to devices such as a compound eye imaging device, a digital photo frame, a tablet terminal, a digital signage terminal (for example, a rotary type), a navigation device, a personal computer, and a portable media player. The present invention can also be applied to a device that does not have an imaging function (for example, a tablet terminal that does not have an imaging function (for example, a device that displays a panoramic image on the Web)). The compound-eye imaging device is, for example, a compound-eye digital still camera or a compound-eye digital video camera (for example, a camera-integrated recorder).

  The display method according to the embodiment of the present technology can also be applied to an imaging apparatus other than a compound-eye imaging apparatus. For example, the present invention can be applied to a monocular imaging device including a wide-angle lens and an imaging device (for example, a monocular camera capable of capturing a panoramic image that is longer than 16: 9 in the short side direction of the housing).

  The above-described embodiment shows an example for embodying the present technology, and the matters in the embodiment and the invention-specific matters in the claims have a corresponding relationship. Similarly, the invention-specific matters in the claims and the matters in the embodiments of the present technology having the same names as those have a corresponding relationship. However, the present technology is not limited to the embodiment, and can be embodied by making various modifications to the embodiment without departing from the gist thereof.

  Further, the processing procedure described in the above embodiment may be regarded as a method having a series of these procedures, and a program for causing a computer to execute these series of procedures or a recording medium storing the program. You may catch it. As this recording medium, for example, a CD (Compact Disc), an MD (MiniDisc), a DVD (Digital Versatile Disk), a memory card, a Blu-ray (registered trademark) disc, or the like can be used.

In addition, this technique can also take the following structures.
(1)
A plurality of imaging units arranged side by side in a direction substantially orthogonal to the long side of the display unit on a surface substantially parallel to the display surface of the display unit;
In a state in which the long side of the display unit is substantially parallel to the direction of gravity, a plurality of regions on a horizontally long image in which a plurality of images generated by the plurality of imaging units are connected to each other in a step shape and the display unit A display control unit that performs control to display on the display unit so that there are a plurality of stages in the longitudinal direction;
The display control unit includes a first image and a second image that are two images including regions at both ends in the longitudinal direction of the horizontally long image as a plurality of regions on the horizontally long image, and between the both ends. The information processing apparatus which displays the 3rd image which is an image including the area | region which exists in the said display part.
(2)
A housing that is attached to a first housing that houses a display unit, and that is housed in a rotatable second housing having a rotation axis substantially parallel to a short side of the display unit, and the second housing A plurality of imaging units arranged side by side in a direction substantially parallel to the rotation axis on one surface;
In a state in which the long side of the display unit is substantially parallel to the direction of gravity, a plurality of regions on a horizontally long image in which a plurality of images generated by the plurality of imaging units are connected to each other in a step shape and the display unit A display control unit that performs control to display on the display unit so that there are a plurality of stages in the longitudinal direction;
The display control unit includes a first image and a second image that are two images including regions at both ends in the longitudinal direction of the horizontally long image as a plurality of regions on the horizontally long image, and between the both ends. The information processing apparatus which displays the 3rd image which is an image including the area | region which exists in the said display part.
(3)
The information processing apparatus according to (1) or (2), wherein the display control unit displays the first image and the second image on the same stage and displays the third image on another stage.
(4)
The information processing apparatus according to (3), wherein the display control unit causes the third image to be displayed at a level higher than a level at which the first image and the second image are displayed.
(5)
The information processing unit according to (3) or (4), wherein the display unit displays the third image in a horizontal size larger than the horizontal size of the first image and the second image. apparatus.
(6)
The display control unit according to any one of (1) to (5), wherein an image obtained by reducing the entire horizontally long image is displayed in an area other than the display area of the first image to the third image. Information processing device.
(7)
Further comprising an attitude detection unit for detecting the attitude of the information processing apparatus;
When the long side of the display unit is substantially perpendicular to the gravitational direction, the display control unit reduces the number of steps as compared with the case where the long side of the display unit is substantially parallel to the gravitational direction. The information processing apparatus according to any one of (1) to (6), wherein control for displaying an image to the third image is performed.
(8)
In the case where the display unit displays an image in which the ratio of the horizontal size to the vertical size of the image to be displayed is smaller than the horizontally long image on the display unit, the display control unit displays the horizontally long image. The information processing apparatus according to any one of (1) to (7), wherein the display is controlled with a reduced number of stages.
(9)
A specific object detection unit for detecting a specific object included in the horizontally long image;
The information processing apparatus according to any one of (1) to (8), wherein the display control unit displays an image including an area where the detected specific target object is present as the third image.
(10)
When recording the horizontally long image, further comprising a recording control unit that records the specific object information related to the region where the specific object exists in a recording medium in association with the horizontally long image,
When the display control unit displays the horizontally long image recorded on the recording medium, the specific object exists using the specific object information recorded in association with the horizontally long image. The information processing apparatus according to (9), wherein an image relating to a region is displayed.
(11)
The display control unit displays the third image while moving the region displayed as the third image from one end of the both ends to the other end. (1) to (10) The information processing apparatus according to any one of the above.
(12)
An operation receiving unit that receives a switching instruction operation from a user regarding the display method of the third image;
The display control unit displays the third image while moving a region displayed as the third image from one end of the both ends to the other end based on the switching instruction operation. (1) Displaying the third image by selecting one of a display method and a display method for displaying a certain region as the third image without moving the region displayed as the third image To (11).
(13)
When the display method of the third image is switched, the display method related to the switching is held until the display of the third image based on the switching is completed, and the third method based on the switching is performed. In the item (12), when the horizontally long image is newly displayed after the display of the three images is finished, the third image in the horizontally long image is displayed using the held display method. Information processing device.
(14)
An operation receiving unit that receives a designation operation for designating a position in the reduced image when a reduced image obtained by reducing the entire horizontally long image is displayed in an area other than the display area of the first image to the third image. Further comprising
When the designation operation is performed, the display control unit moves the region displayed as the third image from one end of the both ends to the other end while moving the third image. When the third image is displayed by the display method to be displayed, an image including the position in the reduced image specified by the specifying operation and the surrounding area is displayed as a new third image, and the position The third image is displayed while performing the movement from the starting point. On the other hand, when the designation operation is performed, the area displayed as the third image is not moved, and a certain area is set as the third area. When the third image is displayed by a display method for displaying as an image, any one of (1) to (11), wherein an image including the position and its surrounding area is displayed as a new third image. Information described in Processing apparatus.
(15)
The display control unit includes a first display method for fixing and displaying the position of the third image in the landscape image during live view or post view, and a specific object included in the landscape image. The second display method for displaying an image including a region as the third image is switched based on a user operation, and at the time of playback, the region displayed as the third image is changed from one end of the both ends to the other. The information processing apparatus according to any one of (1) to (11), wherein a third display method for displaying the third image while being moved to an end of the display and a second display method are switched based on a user operation .
(16)
The display control unit is configured to display a first display method in which the position of the third image in the landscape image is fixed during live view, and an image including a region where the specific object included in the landscape image exists. Is switched based on a user operation, and the area displayed as the third image is changed from one end of the both ends to the other during post-view or playback. The information processing apparatus according to any one of (1) to (11), wherein a third display method for displaying the third image while being moved to an end of the display and a second display method are switched based on a user operation .
(17)
The display control unit displays an image generated by the imaging unit on the display unit before the user records an image when the information processing apparatus is shipped from a factory or when the user uses the information processing apparatus. An image display method by a live view operation, an image display method by a post-view operation in which the recorded image is displayed on the display unit immediately after the user records an image, and an image designated by the user is displayed on the display unit. The information processing apparatus according to any one of (1) to (16), wherein each image is displayed based on the setting when individually set for each of the image display methods by the reproduction operation to be displayed.
(18)
A display for displaying an image;
When a horizontally long image is displayed on the display unit in a state where the long side of the display unit is substantially parallel to the direction of gravity, the plurality of regions on the image are stepped and the number of steps in the longitudinal direction of the display unit. Including a display control unit that performs control to display a plurality of
The display control unit includes a first image and a second image that are two images including regions at both ends in the longitudinal direction of the horizontally long image as a plurality of regions on the horizontally long image, and between the both ends. The display apparatus which performs control which displays the 3rd image which is an image including the area | region which exists in.
(19)
The display device according to (18), wherein the display control unit displays the first image and the second image on the same stage and displays the third image on another stage.
(20)
The display device according to (19), wherein the display control unit displays the third image at a level higher than a level at which the first image and the second image are displayed.
(21)
The display device according to (19) or (20), wherein the display unit displays a size of the third image in a horizontal direction larger than a horizontal size of the first image and the second image. .
(22)
The display control unit according to any one of (18) to (21), wherein an image obtained by reducing the entire horizontally long image is displayed in an area other than the display area of the first image to the third image. Display device.
(23)
Further comprising an attitude detection unit for detecting the attitude of the display device;
When the long side of the display unit is substantially perpendicular to the gravitational direction, the display control unit reduces the number of steps as compared with the case where the long side of the display unit is substantially parallel to the gravitational direction. The display device according to any one of (18) to (22), wherein control for displaying an image to the third image is performed.
(24)
In the case where the display unit displays an image in which the ratio of the horizontal size to the vertical size of the image to be displayed is smaller than the horizontally long image on the display unit, the display control unit displays the horizontally long image. The display device according to any one of (18) to (23), wherein the display is controlled with a reduced number of stages.
(25)
An imaging procedure for generating a plurality of images by a plurality of imaging units arranged in a direction substantially orthogonal to the long side of the display unit on a surface substantially parallel to the display surface of the display unit;
In a state where the long side of the display unit is substantially parallel to the direction of gravity, the plurality of regions on the horizontally long image in which the plurality of images are connected have a step shape and a plurality of steps in the longitudinal direction of the display unit. And a display control procedure for performing control to be displayed on the display unit,
The display control procedure includes a first image and a second image that are two images including regions at both ends in the longitudinal direction of the horizontally long image as a plurality of regions on the horizontally long image, and between the both ends. A control method for an information processing apparatus that causes a display to display a third image that is an image including an area existing in the display.
(26)
An imaging procedure for generating a plurality of images by a plurality of imaging units arranged in a direction substantially orthogonal to the long side of the display unit on a surface substantially parallel to the display surface of the display unit;
In a state where the long side of the display unit is substantially parallel to the direction of gravity, the plurality of regions on the horizontally long image in which the plurality of images are connected have a step shape and a plurality of steps in the longitudinal direction of the display unit. And a program for causing a computer to execute a display control procedure for performing control to be displayed on the display unit,
The display control procedure includes a first image and a second image that are two images including regions at both ends in the longitudinal direction of the horizontally long image as a plurality of regions on the horizontally long image, and between the both ends. A program for causing the display unit to display a third image that is an image including a region existing in the display.

DESCRIPTION OF SYMBOLS 11 Application processor 12 Digital baseband process part 13 Analog baseband process part 14 RF process part 15 Battery 16 Microphone 17 Speaker 18 Antenna 100, 800, 820, 840 Information processing apparatus 101, 815 Electronic board 102, 816 Battery storage part 111 Switching Switch 112 Determination key 130, 830 Imaging unit 131-133 Optical system 134-136 Image sensor 137-139 I / F with DSP
140 Display Unit 150 Attitude Detection Unit 160 Program Memory 170 Image Memory 180 Recording Medium 191 First Imaging System 192 Second Imaging System 193 Third Imaging System 201 Imaging Control Unit 202 CPU
203 DMA controller 204 Data bus 205 I / F with program memory
206 I / F with image memory
207 Power supply control unit 208 Power supply unit 209 Power supply unit 200 DSP
210 I / F with image sensor
211 to 219 Image buffer 220 Image signal processing unit 221 Pixel addition processing unit 222 Demosaic processing unit 223 YC conversion processing unit 224 Image composition processing unit 225 Sharpness processing unit 226 Color adjustment processing unit 227 RGB conversion processing unit 231, 241, 251 Resolution conversion Unit 232, 242 Image rotation processing unit 233 I / F with display unit
243 I / F with external display
245 External display device 252 Encoding unit 253 I / F with recording medium
261 to 263 Oscillators 264 to 266 Oscillation circuit 270 Clock generation circuit 340 Vertical scanning circuit 345 Horizontal scanning circuit 510 Image generation unit 520 Image memory 530 Recording control unit 540 Storage unit 550 Display control unit 560 Operation reception unit 801 Operation unit 802 Rotation Member 811 First housing 812 Second housing

Claims (25)

A plurality of imaging units arranged side by side in a direction substantially orthogonal to the long side direction of the display unit;
A display control unit that controls display of an image on the display unit,
The display control unit includes a plurality of regions on a substantially rectangular image in which a plurality of images generated by the plurality of imaging units are connected in a state where a long side direction of the display unit is substantially parallel to a gravity direction. In the first display form having a plurality of steps and a plurality of steps in the direction of gravity, the display unit displays them,
A first image which is an image including a left end region in the long side direction of the substantially rectangular image as a plurality of regions on the substantially rectangular image, and a right end region in the long side direction of the substantially rectangular image An information processing apparatus in which a second image that is an image including a second image and a third image that is an image including a region existing between both ends in the long side direction of the substantially rectangular image are displayed on the display unit. A plurality of imaging units arranged side by side in a direction substantially orthogonal to the long side direction of the display unit;
A display control unit that controls display of an image on the display unit,
The display control unit includes a plurality of regions on a substantially rectangular image in which a plurality of images generated by the plurality of imaging units are connected in a state where a long side direction of the display unit is substantially parallel to a gravity direction. In the first display form having a plurality of steps and a plurality of steps in the direction of gravity, the display unit displays them,
As the plurality of regions on the substantially rectangular image, the first image, the second image, and the third image are displayed on the display unit,
The third image is a partial region of the substantially rectangular image, and is an image of a region that is inward by a finite width from both ends in the long side direction of the substantially rectangular image;
The first image is an image that is a partial area of the substantially rectangular image, and includes an area outside the third image and a left area of the third image.
The information processing apparatus, wherein the second image is a partial area of the substantially rectangular image and includes an area outside the third image and a right area of the third image.   The information processing apparatus according to claim 1, wherein the display control unit displays the third image on one level and displays the first image and the second image on a level different from the third image. .   The display control unit displays the number of images to be displayed on the display unit when displaying an image in which a ratio of a horizontal size to a vertical size of the image is smaller than the ratio of the substantially rectangular image. The display is performed in a second display form that is displayed in a smaller number than the first display form, or in a third display form in which the number of stages to be displayed on the display unit is displayed smaller than in the first display form. The information processing apparatus according to any one of 1 to 3.   The display control unit displays a second image with a smaller number of images to be displayed on the display unit than in the first display mode when the long side direction of the display unit is substantially perpendicular to the gravity direction. The information processing apparatus according to any one of claims 1 to 3, wherein the information is displayed in a display form or in a third display form in which the number of steps displayed on the display unit is displayed smaller than the first display form.   The display control unit includes a center of the substantially rectangular image in each of a live view operation, a post view operation, and an image reproduction operation when displaying on the display unit in the first display form. A first display method for displaying a region as the third image, a second display method for displaying a region in which the specific object included in the substantially rectangular image exists as the third image, and the third image. As a third display method in which the third image is displayed while the region to be displayed is moved from one end side of the both ends of the substantially rectangular image to the other end side. The information processing apparatus according to claim 1, which is displayed.   The information processing apparatus according to claim 6, wherein the reproduction operation is an operation for reproducing and displaying an image recorded in a recording unit used by the information processing apparatus on the display unit based on a user operation.   The specific object is any one of an object that is a focus target at the time of imaging, an object that is detected as a human face at the time of imaging, and an object that is detected as a moving object at the time of imaging. The information processing apparatus described.   The display control unit may select any one of the first to third display methods based on a predetermined setting or a user operation in each of the live view operation, the post view operation, and the playback operation. The information processing apparatus according to any one of claims 6 to 8, wherein an image is displayed depending on whether or not.   When the display control unit displays the image on the display unit in the first display form, the image is displayed by either the first display method or the second display method during the live view operation and the post view operation. The information processing apparatus according to claim 6, wherein the information processing apparatus displays the image and displays an image by any one of the first to third display methods during the reproduction operation.   When the display control unit displays the image on the display unit in the first display form, during the live view operation, the display control unit displays an image by either the first display method or the second display method. The information processing apparatus according to claim 6, wherein an image is displayed by any one of the first to third display methods during operation and during the reproduction operation. In a state in which the long side direction of the display unit is substantially parallel to the gravity direction, a plurality of images generated by a plurality of imaging units arranged side by side in a direction substantially orthogonal to the long side direction of the display unit are connected. Comprising a display control procedure for displaying a plurality of regions on a rectangular image on the display unit in a first display form having a stepped shape and a plurality of steps in the direction of gravity;
A first image which is an image including a left end region in the long side direction of the substantially rectangular image as a plurality of regions on the substantially rectangular image, and a right end region in the long side direction of the substantially rectangular image Display method for displaying on the display unit a second image that is an image including the image and a third image that is an image including a region existing between both ends in the long side direction of the substantially rectangular image In a state in which the long side direction of the display unit is substantially parallel to the gravity direction, a plurality of images generated by a plurality of imaging units arranged side by side in a direction substantially orthogonal to the long side direction of the display unit are connected. Comprising a display control procedure for displaying a plurality of regions on a rectangular image on the display unit in a first display form having a stepped shape and a plurality of steps in the direction of gravity;
As the plurality of regions on the substantially rectangular image, the first image, the second image, and the third image are displayed on the display unit,
The third image is a partial region of the substantially rectangular image, and is an image of a region that is inward by a finite width from both ends in the long side direction of the substantially rectangular image;
The first image is an image that is a partial area of the substantially rectangular image, and includes an area outside the third image and a left area of the third image.
The display control method, wherein the second image is a partial area of the substantially rectangular image and includes an area outside the third image and a right area of the third image.   The display control method according to claim 12 or 13, wherein the display control procedure causes the third image to be displayed in one stage, and causes the first image and the second image to be displayed in a stage different from the third image. .   In the display control procedure, when displaying an image in which the ratio of the horizontal size to the vertical size of the image is smaller than the ratio of the substantially rectangular image, the number of images to be displayed on the display unit is set to the first display unit. 13. The display is displayed in a second display form that is displayed in a smaller number than the first display form, or in a third display form in which the number of steps displayed on the display unit is displayed in a smaller number than in the first display form. The display control method according to any one of 1 to 14.   In the display control procedure, when the long side direction of the display unit is substantially perpendicular to the gravitational direction, the second display unit displays the number of images displayed on the display unit less than the first display form. The display control method according to any one of claims 12 to 14, wherein display is performed in a display form or a third display form in which the number of steps displayed on the display unit is smaller than that in the first display form.   The display control procedure includes a center of the substantially rectangular image in each of a live view operation, a post view operation, and an image reproduction operation when displaying on the display unit in the first display form. A first display method for displaying a region as the third image, a second display method for displaying a region in which the specific object included in the substantially rectangular image exists as the third image, and the third image. As a third display method in which the third image is displayed while the region to be displayed is moved from one end side of the both ends of the substantially rectangular image to the other end side. The display control method according to claim 12, wherein display is performed.   The display control method according to claim 17, wherein the reproduction operation is an operation for reproducing and displaying an image recorded in a recording unit used by the information processing apparatus on the display unit based on a user operation.   The specific object is any one of an object that is a focus object during imaging, an object that is detected as a human face during the imaging, and an object that is detected as a moving object during the imaging. The display control method described. A plurality of imaging units arranged side by side in a direction substantially orthogonal to the long side direction of the display unit;
A display control unit that controls display of an image on the display unit,
The display control unit includes a plurality of regions on a substantially rectangular image in which a plurality of images generated by the plurality of imaging units are connected in a state where a long side direction of the display unit is substantially parallel to a gravity direction. In the first display form having a plurality of steps and a plurality of steps in the direction of gravity, the display unit displays them,
A first image which is an image including a left end region in the long side direction of the substantially rectangular image as a plurality of regions on the substantially rectangular image, and a right end region in the long side direction of the substantially rectangular image An electronic device in which a second image that is an image including the image and a third image that is an image including an area existing between both ends in the long side direction of the substantially rectangular image are displayed on the display unit. A plurality of imaging units arranged side by side in a direction substantially orthogonal to the long side direction of the display unit;
A display control unit that controls display of an image on the display unit,
The display control unit includes a plurality of regions on a substantially rectangular image in which a plurality of images generated by the plurality of imaging units are connected in a state where a long side direction of the display unit is substantially parallel to a gravity direction. In the first display form having a plurality of steps and a plurality of steps in the direction of gravity, the display unit displays them,
As the plurality of regions on the substantially rectangular image, the first image, the second image, and the third image are displayed on the display unit,
The third image is a partial region of the substantially rectangular image, and is an image of a region that is inward by a finite width from both ends in the long side direction of the substantially rectangular image;
The first image is an image that is a partial area of the substantially rectangular image, and includes an area outside the third image and a left area of the third image.
The second apparatus is an electronic device that is a partial area of the substantially rectangular image and includes an area outside the third image and a right area of the third image.   The electronic device according to claim 20 or 21, wherein the display control unit displays the third image on one level, and displays the first image and the second image on a level different from the third image.   The display control unit displays the number of images to be displayed on the display unit when displaying an image in which a ratio of a horizontal size to a vertical size of the image is smaller than the ratio of the substantially rectangular image. 21. The display is performed in a second display form that is displayed in a smaller number than the first display form, or in a third display form in which the number of stages to be displayed on the display unit is displayed smaller than that in the first display form. The electronic apparatus in any one of thru | or 22.   The display control unit displays a second image with a smaller number of images to be displayed on the display unit than in the first display mode when the long side direction of the display unit is substantially perpendicular to the gravity direction. The electronic device according to any one of claims 20 to 22, wherein the electronic device is displayed in a display form or a third display form in which the number of stages displayed on the display unit is displayed smaller than the first display form.   The display control unit includes a center of the substantially rectangular image in each of a live view operation, a post view operation, and an image reproduction operation when displaying on the display unit in the first display form. A first display method for displaying a region as the third image, a second display method for displaying a region in which the specific object included in the substantially rectangular image exists as the third image, and the third image. As a third display method in which the third image is displayed while the region to be displayed is moved from one end side of the both ends of the substantially rectangular image to the other end side. The electronic device according to any one of claims 20 to 24 to be displayed.

Images