JP2014082768A - Imaging apparatus and control method thereof - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging apparatus capable of easily photographing user's preferred images.SOLUTION: The imaging apparatus includes: an imaging unit group having plural imaging units disposed being aligned in a shorter edge of a housing on one plane of the housing; and a control section that controls an image range. The image range is any of an image range in which a part of the imaging units is used and an image range as a display target, a recording target or a transmission target. The control section selects and sets one of plural ranges (plural ranges in which the ratio of a longer edge with respect to a shorter edge of a generally rectangular shape is different from each other) as the image range.

Description

  The present invention relates to an imaging apparatus, and more particularly to an imaging apparatus including a plurality of imaging systems and a control method thereof.

  2. Description of the Related Art In recent years, an imaging system for imaging a subject such as a person is provided, and imaging such as a digital still camera and a digital video camera (for example, a camera-integrated recorder) that records an image (captured image) generated by the imaging system as an image file. Equipment is widespread.

  There has also been proposed an imaging apparatus that includes a plurality of imaging systems and that can generate a panoramic image by synthesizing images generated by these imaging systems. For example, an imaging apparatus that includes three imaging systems and generates a panoramic image by arranging and synthesizing images output from the imaging systems has been proposed (see, for example, Patent Document 1).

JP 2007-166317 A

  According to the above-described conventional technology, it is possible to easily take a panoramic image. For this reason, for example, even when shooting a state where a plurality of people are scattered with a famous building in the background of a tourist destination, it is easy to shoot panoramic images including these persons. It can be carried out. In addition, for example, a user who desires to take a picture with a composition including a person located near the center and a high building existing in the background of the person included in the panoramic image described above is also assumed. In this case, for example, the image can be taken with the desired composition by rotating the image pickup apparatus 90 degrees about the optical axis as a rotation center so as to form an image that is long in the vertical direction.

  Here, it is assumed that a panoramic image and an image that is long in the vertical direction are captured. In this case, for example, after taking a panoramic image, the image pickup device can be rotated 90 degrees to take a long image in the vertical direction. However, for example, it is assumed that it is difficult for a person unfamiliar with the handling of the imaging device to rotate the imaging device 90 degrees after taking a panoramic image to obtain a desired composition. For example, as described above, when shooting is performed at a tourist destination, the shooting timing is important because each person may move. However, if the time required to obtain the desired composition after rotating the imaging device is long, there is a risk of missing the imaging timing.

  The present invention has been made in view of such a situation, and an object of the present invention is to easily capture a user-preferred image.

  The present invention has been made to solve the above-described problems, and the first side surface of the present invention is that a plurality of imaging units are arranged side by side in the short side direction of the casing on one surface of the casing. An image capturing unit group, and a control unit that controls the image range. The image range includes an image range captured using at least a part of the image capturing unit group, a display target, and A range of images to be recorded, a range of images to be recorded, or a range of images to be transmitted. The imaging unit group includes a first imaging unit whose optical axis is orthogonal to the display surface of the display unit, and an optical axis. Comprises two second imaging units that are arranged in line symmetry with respect to the optical axis of the first imaging unit and that are symmetrical about the optical axis of the first imaging unit. Is the range of the rectangle as the range of the image, and the length of the short side of the rectangle is One of a plurality of ranges having different side length ratios is selected and set, and the first range constituting the plurality of ranges is within the range of the image obtained from the first imaging unit. And a substantially rectangular range that is long in a direction orthogonal to the short side direction of the housing, and the second range that constitutes the plurality of ranges is the first imaging unit and the two second imaging units. The imaging device, the electronic apparatus, the control method thereof, and the program for causing the computer to execute the method are within the range of the image obtained from the above and within the range of a substantially rectangular shape extending in the short side direction of the casing. Thereby, as an image range, one of a plurality of ranges having different ratios of the length of the long side to the length of the short side of the substantially rectangular shape is selected and set.

  In the first aspect, the ratio of the length of the short side of the first range to the length of the short side of the second range is greater than the ratio of the length of the long side of the first range to the length of the short side of the first range. You may make it enlarge the ratio of the length of the long side of 2 range. Thereby, about the 1st range and the 2nd range, it is to the length of the short side of the 2nd range rather than the ratio of the length of the long side of the 1st range to the length of the short side of the 1st range. This brings about the effect of increasing the ratio of the lengths of the long sides of the second range.

  In the first aspect, the first range may be one side of the first range and the length of the side in the short side direction of the housing, and one side of the first range. The ratio of the length of the side in the direction orthogonal to the short side direction of the housing is a substantially rectangular shape of 3: 4 or 9:16, and the second range is one side of the second range and The ratio of the length of the side in the short side direction of the housing to the length of a side that is one side of the second range and is orthogonal to the short side direction of the housing is a substantially rectangular shape larger than 16: 9. You may do it. Thereby, the first range is a substantially rectangular shape with the ratio of the lengths of the sides being 3: 4 or 9:16, and the second range is the ratio of the lengths of the sides with the ratio of 16: 9. Also has the effect of forming a large, generally rectangular shape.

  In the first aspect, the range in the direction perpendicular to the short side direction of the casing in the first range is greater than the range in the direction perpendicular to the short side direction of the casing in the second range. You may make it wide. This brings about the effect that the range in the direction orthogonal to the short side direction of the casing in the first range is wider than the range in the direction orthogonal to the short side direction of the casing in the second range.

  In the first aspect, each of the plurality of imaging units includes an imaging element, and the imaging element has a substantially rectangular portion used for imaging, and the imaging element included in the first imaging unit includes the above-described imaging element. The two imaging elements included in the two second imaging units are arranged so that the part is substantially rectangular in a direction orthogonal to the short side direction of the casing, and the part is in the short side direction of the casing. You may make it arrange | position so that it may become a long substantially rectangular shape. Thereby, the imaging device included in the first imaging unit is arranged such that a part used for imaging is a substantially rectangular shape that is long in a direction orthogonal to the short side direction of the housing, and two imaging units included in the two second imaging units are included. The element has an effect that the part used for imaging is arranged so as to be a substantially rectangular shape that is long in the short side direction of the housing.

  According to a second aspect of the present invention, there is provided a first captured image having a specific direction of a subject as a longitudinal direction and a captured image having a longitudinal direction orthogonal to the specific direction as a longitudinal direction. A second captured image including a subject adjacent to one of the orthogonal directions of the subject included in the image, and a captured image having the orthogonal direction as a longitudinal direction and the orthogonality of the subject included in the first captured image An imaging unit that generates a third captured image including a subject adjacent to the other of the directions by using different imaging elements, a detection unit that detects a state of the imaging device, and the first that is generated by the imaging unit An imaging device including a control unit that determines a range of a target image to be displayed or recorded among the three captured images based on the detected state, a control method thereof, and a computer that uses the method. Is a program to be executed by the. This brings about the effect that the range of the target image to be displayed or recorded among the first to third captured images generated by the imaging unit is determined based on the detected state of the imaging device.

  In the second aspect, the first casing including the imaging unit, the second casing including the display unit that displays the target image, the first casing, and the second casing. A rotation member that rotatably connects the body, and the detection unit detects a rotation state of the second housing relative to the first housing as a state of the imaging device. It may be. This brings about the effect | action of detecting the rotation state of the 2nd housing | casing with respect to a 1st housing | casing as a state of an imaging device.

  In the second aspect, in the detection unit, the longitudinal direction of the display unit and the longitudinal direction of the first imaging element that generates the first captured image are substantially parallel as the state of the imaging device. The first specific state may be detected, and the control unit may determine that only the first captured image is used as the target image when the first specific state is detected. Thereby, when the 1st specific state is detected, it brings about the effect | action of determining using only a 1st captured image as a target image.

  In the second aspect, the control unit may determine to use all of the first captured image as the target image when the first specific state is detected. . Thereby, when the first specific state is detected, there is an effect that all the first captured images are used as the target images.

  Further, in the second aspect, the detection unit includes an orthogonal direction orthogonal to a longitudinal direction of the display unit as a state of the imaging device, and a longitudinal direction of the first imaging element that generates the first captured image. Is detected, and the control unit determines to use the first to third captured images as the target image when the second specific state is detected. You may do it. As a result, when the second specific state is detected, the first to third captured images are determined to be used as the target image.

  In the second aspect, the control unit combines a part of the first captured image with the second and third captured images when the second specific state is detected. It may be determined that the generated composite image is used as the target image. Thereby, when the second specific state is detected, a combined image generated by combining a part of the first captured image with the second and third captured images is used as the target image. It brings about the effect of deciding.

  In the second side surface, the rotating member is coupled so that one surface of the first housing and one surface of the second housing face each other, and the first housing A lens group corresponding to each of the imaging elements constituting the imaging unit is arranged side by side on the surface opposite to the one surface of the body, and on the surface opposite to the one surface of the second casing. May be provided with the display unit. Thereby, the effect | action of displaying the picked-up image produced | generated by the imaging part on the display part with which the surface of the other side is provided is brought about.

  In the second aspect, the rotating member has a display surface of the display unit and an optical axis direction of the lens group corresponding to the first imaging element that generates the first captured image. You may make it connect the said 1st housing | casing and the said 2nd housing | casing so that it may become perpendicular | vertical. This brings about the effect | action of producing | generating the 1st captured image containing the to-be-photographed object which exists in the optical axis direction which becomes substantially perpendicular | vertical with respect to a display part.

  In the second aspect, the image processing device further includes an image composition unit that generates a composite image by combining a part of the first captured image and the second and third captured images. When the detected state is a specific state, it may be determined to use the composite image as the target image. As a result, when the detected state of the imaging device is the specific state, there is an effect of determining to use the composite image as the target image.

  In the second aspect, the image composition unit may generate a substantially rectangular image as the composite image. This brings about the effect | action of producing | generating the substantially rectangular image as a synthesized image.

  In the second aspect, the control unit generates a first picked-up image and a compound-eye image pick-up operation performed using a plurality of image pickup elements constituting the image pickup unit based on the detected state. The range of the target image may be determined by switching between a monocular imaging operation performed using only one imaging element. Thereby, based on the detected state of the imaging device, there is an effect that the range of the target image is determined by switching between the compound eye imaging operation and the monocular imaging operation.

  In the second aspect, the control unit may stop the operation of the image sensor that generates the second and third captured images when the operation is switched to the monocular imaging operation. As a result, when switching to the monocular imaging operation, the operation of the imaging device that generates the second and third captured images is stopped.

  In the second aspect, the control unit uses a clock having a predetermined frequency for the compound-eye imaging operation when switching to the compound-eye imaging operation, and the monocular when switching to the monocular imaging operation. It may be determined to use a clock having a frequency lower than the predetermined frequency for the imaging operation. As a result, when switching to a compound eye imaging operation, a clock with a predetermined frequency is used for the compound eye imaging operation, and when switching to a monocular imaging operation, a clock with a frequency lower than that frequency is used for the monocular imaging operation. It has the effect of deciding what to do.

  In the second aspect, the control unit rotates the target image when the rotation state of the second housing with respect to the first housing is a specific state, and the display unit You may make it display on. Thereby, when the rotation state of the second housing with respect to the first housing is a specific state, the target image is rotated and displayed on the display unit.

  The third aspect of the present invention is a first captured image having a specific direction of a subject as a longitudinal direction and a captured image having a longitudinal direction orthogonal to the specific direction as a longitudinal direction. A second captured image including a subject adjacent to one of the orthogonal directions of the subject included in the image, and a captured image having the orthogonal direction as a longitudinal direction and the orthogonality of the subject included in the first captured image Generated by an imaging unit that generates a third captured image including a subject adjacent to the other of the directions by different imaging elements, an operation reception unit that receives an instruction operation for instructing a change in the imaging range, and the imaging unit An imaging apparatus comprising: a control unit that determines a range of a target image to be displayed or recorded among the first to third captured images to be displayed based on the received instruction operation; and a control method thereof Is a program for causing a computer to execute the method in rabbi. This brings about the effect that the range of the target image to be displayed or recorded among the first to third captured images generated by the imaging unit is determined based on the received instruction operation.

  According to the present invention, it is possible to achieve an excellent effect that it is possible to easily capture a user-preferred image.

It is a figure which shows the example of an external appearance structure of the mobile telephone apparatus 100 in the 1st Embodiment of this invention. It is a figure which shows the internal structural example of the mobile telephone apparatus 100 in the 1st Embodiment of this invention. It is a figure which shows the internal structural example of the imaging part 130 in the 1st Embodiment of this invention. It is a figure which shows the example of arrangement configuration of the image pick-up elements 134 thru | or 136 which comprise the image pick-up part 130 in the 1st Embodiment of this invention. It is a figure which shows the relationship between the image imaged on the image pick-up element 134 thru | or 136 in the 1st Embodiment of this invention, and the reading method of image data. It is a figure which shows the relationship between the image imaged on the image pick-up element 134 thru | or 136 in the 1st Embodiment of this invention, and the reading method of image data. It is a block diagram which shows the example of an internal structure of DSP200 in the 1st Embodiment of this invention. It is a block diagram which shows the internal structural example of the image signal processing part 220 in the 1st Embodiment of this invention. 3 is a block diagram illustrating an example of an internal configuration of a clock generation circuit 270 according to the first embodiment of the present invention. FIG. It is a block diagram which shows the modification of the clock generation circuit 270 in the 1st Embodiment of this invention. It is a block diagram which shows the modification of the clock generation circuit 270 in the 1st Embodiment of this invention. It is a figure which shows the internal structural example of the image pick-up element in the 1st Embodiment of this invention. It is a figure which shows typically the holding | maintenance content of the registers 370 and 380 with which the image sensor 134 in the 1st Embodiment of this invention is provided. 4 is a timing chart schematically showing a state of control signals to each pixel in the image sensor 134 and data output from these pixels in the first embodiment of the present invention. 4 is a timing chart schematically showing a state of control signals to each pixel in the image sensor 134 and data output from these pixels in the first embodiment of the present invention. 4 is a timing chart schematically showing a state of control signals to each pixel in the image sensor 134 and data output from these pixels in the first embodiment of the present invention. It is a figure which shows typically an example of the scanning circuit for performing the thinning-out of the image pick-up element 134 in the 1st Embodiment of this invention. It is a figure which shows the relationship between the imaging part 130 and a to-be-photographed object in the 1st Embodiment of this invention. It is a figure which shows typically the relationship between the imaging system in the imaging part 130 in the 1st Embodiment of this invention, and the to-be-photographed object. It is a figure which shows typically the relationship between the imaging system in the imaging part 130 in the 1st Embodiment of this invention, and the to-be-photographed object. It is a figure which shows typically the relationship between the imaging system in the imaging part 130 in the 1st Embodiment of this invention, and the captured image produced | generated by these. It is a figure which shows typically the relationship between the imaging system in the imaging part 130 in the 1st Embodiment of this invention, and the to-be-photographed object. It is a figure which shows typically the relationship between the imaging system in the imaging part 130 in the 1st Embodiment of this invention, the captured image produced | generated by these, and the correction image after correction | amendment. It is a figure which shows typically the flow of a synthesis | combination in case the image composition process part 224 in the 1st Embodiment of this invention produces | generates a synthesized image. It is a figure which shows typically the flow of a synthesis | combination in case the image composition process part 224 in the 1st Embodiment of this invention produces | generates a synthesized image. It is a figure which shows typically the flow of a synthesis | combination in case the image composition process part 224 in the 1st Embodiment of this invention produces | generates a synthesized image. It is a figure which shows the to-be-photographed object 500 used as the imaging target of the imaging process by the mobile telephone apparatus 100 in the 1st Embodiment of this invention. It is a figure which shows typically an example (1st reading method) of the reading method of the image data in the image pick-up elements 134 thru | or 136 in the 1st Embodiment of this invention. It is a figure which shows the example of a display of the image in the display part 140 in the 1st Embodiment of this invention. It is a figure which shows typically an example (2nd reading method) of the reading method of the image data in the image pick-up elements 134 thru | or 136 in the 1st Embodiment of this invention. It is a figure which shows the example of a display of the image in the display part 140 in the 1st Embodiment of this invention. It is a figure which shows typically an example (3rd reading method) of the reading method of the image data in the image pick-up elements 134 thru | or 136 in the 1st Embodiment of this invention. It is a figure which shows the example of a display of the image in the display part 140 in the 1st Embodiment of this invention. It is a figure which shows typically an example (4th reading method) of the reading method of the image data in the image pick-up elements 134 thru | or 136 in the 1st Embodiment of this invention. It is a figure which shows the example of a display of the image in the display part 140 in the 1st Embodiment of this invention. It is a figure which shows typically an example (5th reading method) of the reading method of the image data in the image pick-up elements 134 thru | or 136 in the 1st Embodiment of this invention. It is a figure which shows the example of a display of the image in the display part 140 in the 1st Embodiment of this invention. It is a figure which shows the example of a display of the image in the display part 140 in the 1st Embodiment of this invention. It is a figure which shows typically the rotation process which rotates the image displayed on the display part 140 in the 1st Embodiment of this invention. It is a figure which shows the example of a display of the image in the display part 140 in the 1st Embodiment of this invention. It is a figure which shows the example of a display of the image in the display part 140 in the 1st Embodiment of this invention. It is a figure which shows the example of a display of the image in the display part 140 in the 1st Embodiment of this invention. It is a figure which shows typically the flow of the image data when pixel thinning | decimation and pixel addition are performed by the image pick-up element 134 in the 1st Embodiment of this invention. It is a figure which shows typically the flow of the image data in the case of changing the number of pixels by the pixel addition process part 221 in the 1st Embodiment of this invention. It is a figure which shows typically the flow of the image data in the case of changing the number of pixels, when reading image data from the image memory 170 in the 1st Embodiment of this invention. It is a figure which shows typically the flow of the image data in the case of changing a read-out area | region with the image pick-up element 134 in the 1st Embodiment of this invention. It is a figure which shows typically the flow of the image data in the case of changing a read-out area | region when reading image data from the image memory 170 in the 1st Embodiment of this invention. It is a block diagram which shows the function structural example of the mobile telephone apparatus 100 in the 1st Embodiment of this invention. It is a block diagram which shows the function structural example of the mobile telephone apparatus 100 in the 1st Embodiment of this invention. It is a figure which shows the relationship between the control method for stopping the operation | movement of the imaging system in the 1st Embodiment of this invention, and each signal line. 3 is a timing chart schematically showing the output timing from the image sensor and the state of writing to the image buffer in the first embodiment of the present invention. It is a figure which shows the relationship between the clock frequency for reading each pixel in the image pick-up element in the 1st Embodiment of this invention, and writing in an image buffer, and the clock frequency for reading image data from an image buffer. It is a figure which shows the relationship between the clock frequency for reading each pixel in the image pick-up element in the 1st Embodiment of this invention, and writing in an image buffer, and the clock frequency for reading image data from an image buffer. It is a figure which shows the relationship between the clock frequency for reading each pixel in the image pick-up element in the 1st Embodiment of this invention, and writing in an image buffer, and the clock frequency for reading image data from an image buffer. It is a figure which shows typically the flow of the image data produced | generated by the image pick-up element 134 in the 1st Embodiment of this invention. It is a figure which shows typically the relationship between the process which occupies the data bus 204 in the 1st Embodiment of this invention, and time. It is a figure which shows the parameter for determining the operating frequency of the data bus 204 about each imaging operation of the mobile telephone apparatus 100 in the 1st Embodiment of this invention. It is a figure which shows an example of the parameter for determining the operating frequency of the data bus 204 about each imaging operation of the mobile telephone apparatus 100 in the 1st Embodiment of this invention. It is a figure which shows typically the time division process performed by the image signal process part 220 in the 1st Embodiment of this invention. It is a figure which shows typically the time division process performed by the image signal process part 220 in the 1st Embodiment of this invention. It is a figure which shows an example of the parameter for determining the operating frequency of the data bus | bath about the still image recording operation | movement of the mobile telephone apparatus 100 in the 1st Embodiment of this invention. It is a figure which shows an example of the parameter for determining the operating frequency of the data bus 204 about the moving image recording operation | movement of the mobile telephone apparatus 100 in the 1st Embodiment of this invention. 6 is a timing chart schematically showing the timing of writing to the image buffers 211 to 219 and the timing of taking in the DSP 200 according to the first embodiment of the present invention. 6 is a timing chart schematically showing the timing of writing to the image buffers 211 to 219 and the timing of taking in the DSP 200 according to the first embodiment of the present invention. It is a flowchart which shows the process sequence of the imaging control process by the mobile telephone apparatus 100 in the 1st Embodiment of this invention. It is a flowchart which shows a horizontal state imaging process in the process sequence of the imaging control process by the mobile telephone apparatus 100 in the 1st Embodiment of this invention. It is a flowchart which shows the still image imaging process in the process sequence of the imaging control process by the mobile telephone apparatus 100 in the 1st Embodiment of this invention. It is a flowchart which shows the moving image imaging process in the process sequence of the imaging control process by the mobile telephone apparatus 100 in the 1st Embodiment of this invention. It is a flowchart which shows the vertically long state imaging process in the process sequence of the imaging control process by the mobile telephone apparatus 100 in the 1st Embodiment of this invention. It is a figure which shows the modification of the mobile telephone apparatus 100 in the 1st Embodiment of this invention. It is a figure which shows the modification of the mobile telephone apparatus 100 in the 1st Embodiment of this invention. It is a figure which shows the external appearance structure of the imaging device 1000 in the 2nd Embodiment of this invention.

Hereinafter, modes for carrying out the present invention (hereinafter referred to as embodiments) will be described. The description will be made in the following order.
1. First Embodiment (Imaging Control: Example of Performing Monocular Imaging Operation and Compound Eye Imaging Operation in Cellular Phone Device)
2. Second Embodiment (Imaging Control: Example of Performing Monocular Imaging Operation and Compound Eye Imaging Operation in Imaging Apparatus)

<1. First Embodiment>
[External configuration example of mobile phone device]
FIG. 1 is a diagram illustrating an external configuration example of the mobile phone device 100 according to the first embodiment of the present invention. FIG. 1A shows a front side of one mode when the mobile phone device 100 is used, and FIG. 1B shows a back side of the mode. Further, FIG. 1C shows a front side of another mode when the mobile phone device 100 is used, and FIG. 1D shows a back side of the mode.

  The mobile phone device 100 includes a first housing 110 and a second housing 120. Further, the first casing 110 and the second casing 120 are coupled so as to be rotatable with the rotation member 101 as a rotation reference. The mobile phone device 100 is realized by, for example, a mobile phone device having a plurality of imaging functions (so-called camera-equipped mobile phone device). In FIG. 1, for ease of explanation, the mobile phone device 100 is illustrated in a simplified manner, and illustration of a power switch and the like provided on the outer surface of the mobile phone device 100 is omitted.

  The first housing 110 includes an imaging range changeover switch 111, a still image / moving picture changeover switch 112, a numeric keypad 113, an enter key 114, a cross key 115, and an imaging unit 130. Here, when the user uses the cellular phone device 100 by holding it in his hand, it is necessary to hold any part of the cellular phone device 100 with his / her hand. For example, in normal times, the mobile phone device 100 is often used in a state where a user holds any part of the first housing 110 (so-called main body housing) with a hand.

  The imaging range changeover switch 111 is an operation member for switching the imaging range when image data is generated by the imaging unit 130, and the imaging range is sequentially switched every time it is pressed by a user operation. The switching of the imaging range will be described in detail with reference to FIGS. 28 to 42. The imaging range changeover switch 111 is an example of an operation reception unit.

  The still image / moving image switching switch 112 is an operation member used when switching between a still image capturing mode for recording a still image and a moving image capturing mode for recording a moving image. The imaging range switch 111 and the still image / moving image switch 112 are so-called toggle switches.

  The numeric keypad 113 is an operation member for inputting numbers and symbols.

  The enter key 114 is an operation member that is pressed when setting various functions by the user. For example, when pressed when the still image capturing mode is set, it functions as a shutter button.

  The cross key 115 is an operation key that is pressed when the selection state of each item displayed on the display screen is changed, or when the object displayed on the display screen is moved in the vertical and horizontal directions.

  The imaging unit 130 captures a subject and generates image data. In FIG. 1A, the position on the front side corresponding to the imaging unit 130 shown in FIG. Moreover, the circle in the imaging unit 130 illustrated in FIG. 1B schematically represents each lens of a plurality of imaging systems included in the imaging unit 130. That is, in the first embodiment of the present invention, the imaging unit 130 in which three lens groups are arranged in a specific direction will be described as an example. Here, the specific direction can be, for example, the horizontal direction when the longitudinal direction of the first casing 110 is matched with the vertical direction.

  The second housing 120 includes a display unit 140. The display unit 140 is a display device that displays various images. For example, an image generated by an imaging operation is displayed on the display unit 140 as a monitoring image. As the display unit 140, for example, an LCD (Liquid Crystal Display) panel, an organic EL (Electro Luminescence) panel, or the like can be used. Here, the horizontal to vertical ratio of a display device included in a camera-equipped mobile phone device or a general imaging device is often 4: 3 or 16: 9. Therefore, in the first embodiment of the present invention, an example will be described in which the horizontal to vertical ratio of the display unit 140 is 4: 3 when the longitudinal direction of the second casing 120 is aligned with the horizontal direction.

  As described above, the first casing 110 and the second casing 120 are rotatably connected. That is, the second casing 120 can be rotated with respect to the first casing 110 with the rotation member 101 (indicated by a dotted line) as a rotation reference. Thereby, the relative positional relationship of the 2nd housing | casing 120 with respect to the 1st housing | casing 110 can be changed. For example, FIGS. 1C and 1D show a mode in which the second casing 120 is rotated 90 degrees in the direction of the arrow 102 shown in FIGS.

  1C and 1D, the mobile phone device 100 rotates the second housing 120 by 90 degrees with respect to the first housing 110 with the rotation member 101 as a rotation reference. Except for these points, the example is the same as the example shown in FIGS. Further, in the state shown in FIGS. 1C and 1D, when the second casing 120 is further rotated 90 degrees in the direction of the arrow 103, it becomes an unused state (so-called closed state). As described above, by rotating the second casing 120 with respect to the first casing 110, the relative positional relationship of the second casing 120 with respect to the first casing 110 is at least two or more. The positional relationship can be obtained. Note that the second housing 120 can be rotated in the direction opposite to the direction of the arrow 102 with the rotation member 101 as a rotation reference. However, in the first embodiment of the present invention, Illustration and description of the form are omitted.

  Here, as shown in FIGS. 1A and 1B, the longitudinal direction of the first casing 110 and the longitudinal direction of the second casing 120 are set to the same direction, and the display unit 140 and the numeric keypad 113 are rotated. A state in which the members 101 are opposed to each other is referred to as a vertically long state of the second casing 120. In this state, the imaging operation in the case where the longitudinal direction of the first casing 110 and the second casing 120 is made to coincide with the vertical direction is referred to as a portrait imaging operation.

  In addition, as shown in FIGS. 1C and 1D, a state in which the longitudinal direction of the first casing 110 and the longitudinal direction of the second casing 120 are substantially orthogonal to each other is set to be horizontally long. This is called a state. In this state, the imaging operation in the case where the longitudinal direction of the first casing 110 is matched with the vertical direction and the longitudinal direction of the second casing 120 is matched with the horizontal direction is referred to as a horizontally long imaging operation.

[Example of internal configuration of mobile phone device]
FIG. 2 is a diagram illustrating an internal configuration example of the mobile phone device 100 according to the first embodiment of the present invention. The cellular phone device 100 includes an application processor 11, a digital baseband processing unit 12, an analog baseband processing unit 13, and an RF (Radio Frequency) processing unit 14. In addition, the mobile phone device 100 includes a battery 15, a microphone 16, a speaker 17, an antenna 18, an imaging range changeover switch 111, a still image / video changeover switch 112, a determination key 114, an imaging unit 130, And a display unit 140. In addition, the mobile phone device 100 includes a rotation state detection unit 150, a program memory 160, an image memory 170, a recording medium 180, and a DSP (Digital Signal Processor) 200. The RF processing unit 14 includes an antenna 18, and the analog baseband processing unit 13 includes a microphone 16 and a speaker 17.

  The application processor 11 controls each unit of the mobile phone device 100 based on various programs stored in a built-in memory. The application processor 11 includes, for example, a central processing unit (CPU), a read only memory (ROM), and a random access memory (RAM).

  For example, when a receiving operation is performed, the radio wave received by the antenna 18 is demodulated by the digital baseband processing unit 12 via the RF processing unit 14 and the analog baseband processing unit 13. Then, the demodulation result by the digital baseband processing unit 12 is output from the speaker 17 via the analog baseband processing unit 13.

  When a transmission operation is performed, the voice input from the microphone 16 is modulated by the digital baseband processing unit 12 via the analog baseband processing unit 13. Then, the demodulated audio data is transmitted from the antenna 18 via the analog baseband processing unit 13 and the RF processing unit 14.

  Further, when the user performs an operation for instructing the start of an imaging operation, the imaging operation is performed in the mobile phone device 100. For example, when the user performs an imaging operation start instruction operation, the application processor 11 instructs each unit (imaging unit 130, DSP 200, etc.) related to the imaging operation to start the imaging operation, and activates each unit. Then, an imaging operation is performed by each activated unit, and the generated image is displayed on the display unit 140. Here, when an image recording instruction operation is performed by the user, the generated image is recorded on the recording medium 180. When the user performs an instruction operation for wirelessly transmitting an image, the generated image is wirelessly transmitted. For example, the generated image data is modulated by the digital baseband processing unit 12 and transmitted from the antenna 18 via the analog baseband processing unit 13 and the RF processing unit 14. The battery 15 is a battery that supplies power to the mobile phone device 100.

  Note that the switches 111 and 112, the determination key 114, the imaging unit 130, the display unit 140, the rotation state detection unit 150, the program memory 160, the image memory 170, the recording medium 180, and the DSP 200 are described with reference to FIGS. And will be described in detail.

[Example of internal configuration of imaging unit]
FIG. 3 is a diagram illustrating an internal configuration example of the imaging unit 130 according to the first embodiment of the present invention. FIG. 3 shows a part of the DSP 200 connected to the imaging unit 130. The overall configuration of the DSP 200 will be described in detail with reference to FIGS.

  The imaging unit 130 includes three imaging systems (first imaging system 191 to third imaging system 193), a power supply control unit 207, and power supply units 208 and 209. The three imaging systems are arranged side by side in a specific direction. That is, the first imaging system 191 is arranged in the center, and the second imaging system 192 and the third imaging system 193 are arranged on both sides of the first imaging system 191.

  The first imaging system 191 includes an optical system 131, an imaging element 134, and an I / F (interface) 137 with the DSP. The second imaging system 192 includes an I / F 138 with the optical system 132, the imaging element 135, and the DSP. The third imaging system 193 includes an I / F 139 with the optical system 133, the imaging element 136, and the DSP. Note that the configurations of the first imaging system 191 to the third imaging system 193 are substantially the same, so here, the configuration of the first imaging system 191 will be mainly described, and the second imaging system 192 and the third imaging system 193 will be described. The description of the imaging system 193 is omitted.

  The optical system 131 includes a plurality of lenses (including a zoom lens and a focus lens) that collect light from the subject. Further, the amount of light (that is, exposure) that has passed through each of these lenses is adjusted by a diaphragm (not shown). Then, the condensed light from the subject enters the image sensor 134.

  The imaging device 134 is an imaging device that forms an image signal by forming a subject image incident via the optical system 131. That is, the image sensor 134 receives light from a subject incident through the optical system 131 and performs photoelectric conversion, thereby generating an analog image signal corresponding to the amount of received light. In this way, the analog image signal generated by the image sensor 134 is supplied to the DSP 200 via the I / F 137 with the DSP. For example, a CCD (Charge Coupled Device) type or CMOS (Complementary Metal Oxide Semiconductor) type solid-state imaging device can be used as the imaging device.

  The DSP I / F 137 is an interface for connecting the image sensor 134 and the DSP 200.

  The power control unit 207 controls the power supply units 208 and 209 based on a power control instruction from the imaging control unit 201 (shown in FIG. 7) of the DSP 200. In other words, when the power supply control unit 207 receives a power control instruction from the imaging control unit 201, the power supply units 208 and 209 adjust the signal amplitude, the signal rising speed and the signal falling speed required as the input specification of the control signal. Create a compliant signal. Then, the generated signal is output to the power supply units 208 and 209 to control the power supply units 208 and 209. In addition, when the output signal of the imaging control unit 201 conforms to the input specification of the control signal of the power supply units 208 and 209, the output signal of the imaging control unit 201 is directly input to the power supply units 208 and 209. You may do it.

  The power supply unit 208 supplies power to the first imaging system 191 based on the control of the power control unit 207. The power supply unit 209 supplies power to the second imaging system 192 and the third imaging system 193 based on the control of the power control unit 207. The power supply units 208 and 209 are realized by, for example, a commercially available power supply IC (Integrated Circuit).

  Each of the first imaging system 191 to the third imaging system 193 is connected to the DSP 200 through one data line and seven types of signal lines. Here, one data line connecting the first imaging system 191 and the DSP 200 will be described as L1, and seven types of signal lines will be described as signal lines L2 to L8. The data lines and signal lines of the second imaging system 192 and the third imaging system 193 are substantially the same as the data lines and signal lines of the first imaging system 191. Therefore, here, mainly the data lines and signal lines of the first imaging system 191 will be described, and descriptions of the second imaging system 192 and the third imaging system 193 will be omitted.

  The data line L1 is a data line that transmits image data from the image sensor 134 to the DSP 200. For example, the data line L1 is preferably composed of a plurality of data lines in order to increase the transmission rate of image data. In order to increase the transmission rate of the image data and to increase the noise resistance on the transmission path, it is preferable to use a high-speed and differential transmission type data line as the data line L1. For example, it is preferable to use LVDS (low voltage differential signaling) or the like as the data line L1.

  The signal line L2 is a bidirectional communication line between the image sensor 134 and the DSP 200, and for example, a four-line serial communication line can be used. The signal line L2 is used, for example, when setting various setting values necessary for using the image sensor 134 from the DSP 200 side. As an example, setting values for thinning out and outputting image data output from the image sensor 134 to the DSP 200 are written from the DSP 200 to the registers 370 and 380 (shown in FIG. 12) via the signal line L2.

  The signal line L3 is a clock signal line that supplies a clock from the DSP 200 to the image sensor 134. The imaging element 134 performs an imaging operation for one pixel per clock cycle using a clock supplied via the signal line L3. Alternatively, a multiplier may be mounted in the image sensor 134, and the clock supplied from the DSP 200 may be multiplied in the image sensor 134 to perform an imaging operation for one pixel per one cycle of the multiplied clock.

  The signal line L4 is a reset signal line that supplies a reset signal from the DSP 200 to the image sensor 134.

  The signal line L5 is a signal line for controlling on / off of the imaging operation of the imaging element 134 from the DSP 200. That is, the signal line L5 is a signal line for notifying the stop and activation of the operation from the DSP 200 to each image sensor. For example, when an imaging mode in which only one image sensor is used among three imaging systems is instructed by the user, power consumption is reduced by stopping the imaging operation of two image sensors that are not used. be able to.

  The signal line L6 is a vertical synchronization signal line. That is, the signal line L6 is a signal line for notifying the synchronization signal indicating the imaging timing for each frame from the DSP 200 to the imaging device 134.

  The signal line L7 is a horizontal synchronization signal line. That is, the signal line L7 is a signal line for notifying the synchronization signal indicating the imaging timing for each line in one frame from the DSP 200 to the imaging device 134.

  The signal line L8 is a shutter signal line. For example, in the cellular phone device 100, when an operation member (for example, the enter key 114) for recording a captured image is pressed by the user, a shutter signal corresponding to the press is sent from the DSP 200 via the signal line L8. The image sensor 134 is notified.

[Example of image sensor arrangement]
FIG. 4 is a diagram illustrating an arrangement configuration example of the imaging elements 134 to 136 included in the imaging unit 130 according to the first embodiment of the present invention. FIG. 4A shows an arrangement configuration of the image sensors 134 to 136. Here, in general, the shape of the region in which the pixels are arranged on the light receiving surface of the image sensor is substantially rectangular. Therefore, in the following, the image sensors 134 to 136 are schematically represented by rectangles.

  FIG. 4A shows an arrangement configuration example in the case where the longitudinal direction of the first casing 110 is aligned with the vertical direction so that the rotating member 101 side is on the upper side. Specifically, the image sensor 134 is disposed in the center, and the image sensors 135 and 136 are disposed on both sides of the image sensor 134. Further, the image sensor 134 disposed at the center is disposed such that the longitudinal direction thereof coincides with the direction orthogonal to the arrangement direction. On the other hand, the image sensors 135 and 136 disposed on both sides of the image sensor 134 are disposed such that the longitudinal direction thereof coincides with the arrangement direction. Further, the image sensors 134 to 136 are arranged so that the center positions thereof are on the same plane. That is, in the arrangement direction, the image sensors 135 and 136 are arranged horizontally and the image sensor 134 is arranged vertically. In FIG. 4A, pixel data readable areas 400 to 402 from which each pixel can be read are schematically shown as rectangles in the image sensors 134 to 136. For example, the imaging element 134 generates a first captured image whose longitudinal direction is the specific direction of the subject. In addition, the imaging element 135 generates a second captured image including a subject adjacent to one of the orthogonal directions of the subject included in the first captured image, with the orthogonal direction orthogonal to the specific direction as the longitudinal direction. In addition, the imaging element 136 generates a third captured image including a subject adjacent to the other of the orthogonal directions of the subject included in the first captured image with the orthogonal direction as the longitudinal direction. As described above, in the first embodiment of the present invention, the aspect ratio (horizontal / vertical) of the central image sensor 134 is made smaller than the aspect ratios of the image sensors 135 and 136 arranged on both sides thereof. Thereby, for example, even when a vertically long image is generated using only the image sensor 134, a sufficient resolution can be maintained. On the other hand, for wide-angle images (for example, panoramic images) generated using the image sensors 134 to 136, the image frame of the image sensor 134 can be used effectively.

  FIG. 4B shows an example of a pixel data reading area when image data is generated when the second casing 120 is in the vertically long state. The pixel data reading areas 403 to 405 are examples of areas for reading out each pixel when generating image data used for display or recording in the pixel data readable areas 400 to 402. In FIG. Is indicated by a bold line. The pixel data reading areas 404 and 405 can be the same as the pixel data reading areas 401 and 402, for example. The pixel data read area 403 has, for example, the same vertical length V10 as the vertical length V11 of the pixel data readable areas 401 and 402, and the horizontal length H10 can be read. The length of the region 400 in the horizontal direction can be the same.

  FIG. 4C shows an example of a pixel data reading area when image data is generated when the second casing 120 is in the landscape state. In this example, a pixel data reading area 406 in the case where image data is generated only by the image sensor 134 among the image sensors 134 to 136 is shown. The pixel data reading area 406 is an example of an area from which each pixel is read out when generating a composite image used for display or recording in the pixel data readable areas 400 to 402. In FIG. It shows with. The pixel data read area 406 can be the same as the pixel data readable area 400, for example. Note that these image generation examples will be described in detail with reference to FIGS.

  Here, for example, an imaging apparatus is assumed in which the imaging elements 134 to 136 are arranged so that the longitudinal directions of the imaging elements 134 to 136 coincide with the specific direction. That is, in the state shown in FIG. 4A, it is assumed that the image sensor 134 is rotated 90 degrees and the longitudinal direction and the arrangement direction of the image sensors 134 to 136 are the same. In this case, when shooting a normal vertically long still image, the maximum number of pixels in the vertical direction can be taken only in the short side direction of the image sensor 134. Further, the angle of view in the vertical direction is also limited to a region incident in the short side direction of the image sensor 134. For this reason, in order to increase the angle of view in the vertical direction and increase the number of pixels in the vertical direction, it is necessary for the user to perform imaging by rotating the imaging apparatus 90 degrees.

  On the other hand, according to the first embodiment of the present invention, even when a normal vertically long still image is captured, the user lays down and shoots a mobile phone device including one imaging system. It is possible to shoot with the same number of pixels and the same angle of view. For this reason, it is possible to save the user from having to turn the imaging device to the side.

  Next, a reading method for reading out one line of data in the image sensors 134 to 136 will be described. For example, the following two methods can be used as a data reading method for reading out one line of data in the imaging elements 134 to 136.

  The first data reading method is a method in which when one line of data is read from the image sensor 134, the direction of the one line is set to the short side direction of a rectangular region corresponding to the image sensor 134. In the first data reading method, when one line of data is read from the image sensors 135 and 136, the direction of the one line is set to the long side direction of the rectangular region corresponding to the image sensors 135 and 136. An example of this is shown in FIG. In the first data reading method, the image data read from the three image pickup devices 134 to 136 can be written in the image memory 170 in the read order. Further, even when the image signal processing is performed by reading out this data, the data can be read out in the same order, so that the data can be easily written into and read out from the memory. However, in general, since the readout line direction of the image sensor is the long side direction, it is necessary to prepare a new image sensor whose readout line direction is the short side direction.

  The second data reading method is a method in which when one line of data of the image sensor 134 is read, the direction of the one line is set to the long side direction like the image sensors 135 and 136. In this case, it is not necessary to newly prepare an image sensor whose reading direction is the short side direction. However, the image data reading direction of the image sensor 134 is rotated by 90 ° from the image data reading direction of the image sensors 135 and 136. For this reason, when image signal processing is performed using image data read from the image sensor 134, the image is rotated by 90 ° and oriented in the same direction as the images generated by the image sensors 135 and 136. It is preferable to perform image signal processing. In the first embodiment of the present invention, an example using the first data reading method is shown. That is, in the following, an example in which the line direction when reading out one line of pixel data in the image sensor 134 is the arrangement direction of the image sensors 134 to 136 will be described. Similarly, an example in which the line direction when reading out one line of pixel data in the image sensors 135 and 136 is the arrangement direction of the image sensors 134 to 136 is shown.

  FIGS. 5 and 6 are diagrams showing the relationship between the image formed on the image sensors 134 to 136 and the image data reading method according to the first embodiment of the present invention. In general, a so-called inverted image is formed on the image sensor.

  FIG. 5 schematically illustrates the relationship between the subject 409, the optical system 131, and the captured image 410 formed on the image sensor 134 when an inverted image is generated as an image formed on the image sensor. As shown in FIG. 5, the light from the subject 409 that has entered through the optical system 131 is imaged on the image sensor 134, and a captured image 410 is generated. In this case, the vertical direction of the subject 409 and the vertical direction of the captured image 410 are reversed.

  FIG. 6A schematically shows reading start positions 411 to 413 and a reading direction when pixel data is read from the imaging elements 134 to 136. In this example, pixel data is read out in the pixel data reading areas 403 to 405 shown in FIG. In the following, the reading start position in the pixel data reading area is schematically shown by a rectangle. For example, reading is sequentially started from reading start positions 411 to 413 at the lower right corners of the pixel data reading areas 403 to 405, and pixel data is read in the direction of the arrow. For example, in the pixel data readout region 403, readout of pixel data is started from the readout start position 411, and readout of pixel data is sequentially performed while shifting one pixel at a time in the arrow direction. When a pixel located at the end of one horizontal line (the left end in the pixel data reading area 403 shown in FIG. 6A) is read, the line to be read is moved up by one pixel. The pixel located at the other end is read out after shifting to. Thereafter, pixel readout is sequentially performed in the same manner. When the pixel located at the end of the uppermost line in the pixel data reading area 403 is read, the reading process of the image sensor 134 is terminated. Further, readout processing is also performed on the image sensors 135 and 136 at the same time.

  FIG. 6B schematically shows a composite image 414 on the image memory 170 when the pixel data read from the image sensors 134 to 136 are combined. In this example, an example in which pixel data read by the reading method shown in FIG. It is assumed that the arrow direction in the composite image 414 is the arrangement direction of the image data on the image memory 170.

  FIG. 6C shows a display example of a composite image when the pixel data read from the imaging elements 134 to 136 is displayed on the display unit 140. In this example, an example in which the composite image 414 shown in FIG. 6B is displayed is shown. For example, the composite image 414 is displayed in the captured image display area 415, and one-color images (for example, black and white) are displayed in the upper and lower margin image display areas 416 and 417. Note that the arrow direction in the display unit 140 is the scanning direction in the display unit 140.

[DSP configuration example]
FIG. 7 is a block diagram showing an example of the internal configuration of the DSP 200 in the first embodiment of the present invention. The DSP 200 includes an imaging control unit 201, a CPU 202, a DMA (Direct Memory Access) controller 203, a data bus 204, an I / F 205 for a program memory, and an I / F 206 for an image memory. The DSP 200 also includes an I / F 210 for the image sensor, image buffers 211 to 219, an image signal processing unit 220, resolution conversion units 231, 241, and 251, and image rotation processing units 232 and 242. The DSP 200 also includes an I / F 233 for a display unit, an I / F 243 for an external display device, an encoding unit 252, an I / F 253 for a recording medium, oscillation circuits 264 to 266, and a clock generation circuit 270. With. The data bus 204 is connected to a CPU 202, a DMA controller 203, an I / F 206 with an image memory, image buffers 211 to 219, an image signal processing unit 220, and the like. In addition, the imaging control unit 201 receives signals from the imaging range switch 111, the still image / moving image switch 112, the enter key 114, and the rotation state detection unit 150.

  The rotation state detection unit 150 detects the rotation state of the second housing 120 with respect to the first housing 110 and outputs a detection result to the imaging control unit 201. The rotation state detection unit 150 detects, for example, an angle formed by the first case 110 and the second case 120 as the rotation state of the second case 120 with respect to the first case 110 and detects the angle. The result is output to the imaging control unit 201. For example, when the rotation angle of the second housing 120 with respect to the first housing 110 is less than a certain value, the angle detection is not performed and is depressed when the rotation angle becomes a certain value or more. A switch is provided on any of the rotating members 101. And the rotation state detection part 150 detects the angle which the 1st housing | casing 110 and the 2nd housing | casing 120 make by an angle detection switch. The rotation state detection unit 150 is an example of a detection unit.

  The imaging control unit 201 controls each unit related to imaging processing. For example, the imaging control unit 201 determines the rotation state of the second housing 120 relative to the first housing 110 based on the detection result from the rotation state detection unit 150, and based on the determination result, The imaging control of each part is performed. For example, the imaging control unit 201 determines the range of the target image to be displayed or recorded among the image data generated by the imaging elements 134 to 136 based on the determination result. In addition, the imaging control unit 201 performs imaging control of each unit based on input signals from the imaging range switch 111, the still image / moving image switch 112, and the enter key 114. These imaging controls will be described in detail with reference to FIGS. 28 to 42 and the like. The imaging control unit 201 is an example of a control unit described in the claims.

  Here, in the first embodiment of the present invention, the user can set in advance an imaging mode (image size or the like) for recording an image generated by the imaging unit 130. For example, a menu screen for setting the imaging mode is displayed on the display unit 140, and the setting content desired by the user is input using the enter key 114 and the cross key 115 on the menu screen. Note that the imaging mode includes, for example, the number of imaging elements used during imaging, and the vertical image size and horizontal image size of the image during recording. In addition, the imaging mode includes, for example, a vertical back porch and a vertical front porch that represent the interval between the effective area of the image and the vertical synchronizing signal, and a horizontal back porch and a horizontal front that represent the interval between the effective area of the image and the horizontal synchronizing signal. Including pouch. In addition, the imaging control unit 201, each unit in the DSP 200, and the imaging elements 134 to 136 each include a register that holds an imaging mode.

  Here, when the imaging mode is set by the user, the imaging control unit 201 notifies each of the units in the DSP 200 and the imaging elements 134 to 136 of the set imaging mode, and stores them in a register included in each unit. As described above, by causing the register included in each unit to hold the setting content of the imaging mode set by the user, the user can easily switch and use a plurality of shooting conditions.

  For example, the imaging control unit 201 notifies each unit in the DSP 200 and the imaging elements 134 to 136 of the vertical synchronization signal, the horizontal synchronization signal, and the clock signal based on the setting contents of the imaging mode held in the built-in register. . In addition, the imaging control unit 201 sends, for example, a vertical synchronization signal, a horizontal synchronization signal, and a clock signal to each display-related unit in the DSP 200 and the display unit 140 based on the setting contents of the imaging mode held in the built-in register. Notice. In addition, the imaging control unit 201 outputs a signal for controlling power on / off to the power control unit 207, for example.

  The CPU 202 controls the entire DSP 200 based on various programs stored in the program memory 160.

  The DMA controller 203 controls data transfer between the memories based on the control of the CPU 202.

  The program memory I / F 205 is an interface for connecting the program memory 160 and the DSP 200.

  An I / F 206 with the image memory is an interface for connecting the image memory 170 and the DSP 200.

  An I / F 210 with the image sensor is an interface for connecting the image sensors 134 to 136 and the DSP 200. That is, image data generated by the image sensors 134 to 136 is input to the I / F 210 with the image sensor. For example, when the data line L1 for transmitting image data from the image sensors 134 to 136 is a LVDS system with a small amplitude, the image data from the image sensors 134 to 136 is GND in the I / Fs 137 to 139 with the DSP. It is converted into a potential or a power supply potential. Further, three systems of image buffers 211 to 219 corresponding to the image sensors 134 to 136 are provided in the subsequent stage of the I / F 210 with the image sensor.

  The image buffers 211 to 219 are image buffers that hold the image data output from the image sensors 134 to 136, and the held image data is written to the image memory 170 via the data bus 204. For example, each image sensor includes three image buffers, and these image buffers are connected to the data bus 204. For example, the image sensor 134 includes three image buffers 211 to 213. The image sensor 135 is provided with three image buffers 214 to 216. Further, the image sensor 136 is provided with three image buffers 217 to 219. Here, in the first embodiment of the present invention, since image data is written to the image memory 170, new input from the image sensors 134 to 136 is made even while the image data is being read from the image buffers 211 to 219. The obtained image data is sequentially held. For this reason, it is preferable that two or more image buffers are provided for each of the image sensors 134 to 136 as the image buffers 211 to 219.

  One capacity of the image buffers 211 to 219 is preferably larger than the bit width of the data bus 204. For example, when the data bus 204 is 128 bits wide, each image buffer preferably has a capacity of 128 bits or more. Further, it is more preferable that one capacity of the image buffers 211 to 219 is at least twice the bit width of the data bus 204. For example, when the data bus 204 is 128 bits wide, it is more preferable that each image buffer has a capacity of 256 bits or more.

  On the other hand, one capacity of the image buffers 211 to 219 can be equal to or less than the image data amount of one image generated by one image sensor. For example, one capacity of the image buffers 211 to 219 is preferably equal to or less than the data amount of image data generated by pixels for one line of the image sensor 134.

  In the first embodiment of the present invention, the bit width of the data line connecting the image sensors 134 to 136 and the DSP 200 is, for example, 12 bits. For example, the bit width of the data bus 204 of the DSP 200 is 128 bits, and the capacity of one of the image buffers 211 to 219 is 128 bits.

  The image signal processing unit 220 performs various image signal processes on the image data input via the image buffers 211 to 219 and the data bus 204 based on the control of the imaging control unit 201. The internal configuration of the image signal processing unit 220 will be described in detail with reference to FIG.

  The resolution conversion unit 231 performs resolution conversion for displaying each image on the display unit 140 based on the control of the imaging control unit 201, and the image data subjected to the resolution conversion is supplied to the image rotation processing unit 232. Output.

  The resolution conversion unit 241 performs resolution conversion for displaying each image on the external display device 245 based on the control of the imaging control unit 201, and the image rotation processing unit 242 converts the image data subjected to the resolution conversion. Output to.

  The image rotation processing unit 232 performs rotation processing on the image data that has undergone resolution conversion based on the control of the imaging control unit 201, and the image data that has been subjected to the rotation processing is an I / F 233 with the display unit. Output to.

  The image rotation processing unit 242 performs rotation processing on the image data subjected to resolution conversion based on the control of the imaging control unit 201. The image data subjected to the rotation processing is subjected to I / O with an external display device. Output to F243.

  The I / F 233 with the display unit is an interface for connecting the display unit 140 and the DSP 200.

  The I / F 243 with the external display device is an interface for connecting the external display device 245 and the DSP 200. The external display device 245 is, for example, a television.

  The resolution conversion unit 251 converts the resolution of each image for recording based on the control of the imaging control unit 201, and outputs the image data subjected to the resolution conversion to the encoding unit 252. For example, the resolution conversion unit 251 performs a resolution conversion process for converting the resolution to a recording image size desired by the user and a resolution conversion process for generating a thumbnail image.

  The encoding unit 252 performs encoding for compressing the image data output from the resolution conversion unit 251 and outputs the encoded image data to the I / F 253 with the recording medium.

  The I / F 253 with the recording medium is an interface for connecting the recording medium 180 and the DSP 200.

  The recording medium 180 is a recording medium that records image data supplied via the I / F 253 with the recording medium. The recording medium 180 may be built in the mobile phone device 100 or may be detachably attached to the mobile phone device 100. As the recording medium 180, for example, a tape (for example, a magnetic tape) or an optical disk (for example, a recordable DVD (Digital Versatile Disc)) can be used. Further, as the recording medium 180, for example, a magnetic disk (for example, hard disk), a semiconductor memory (for example, memory card), or a magneto-optical disk (for example, MD (MiniDisc)) may be used.

  Note that the oscillation circuits 264 to 266 and the clock generation circuit 270 will be described in detail with reference to FIGS.

[Example of Internal Configuration of Image Signal Processing Unit 220]
FIG. 8 is a block diagram illustrating an internal configuration example of the image signal processing unit 220 according to the first embodiment of the present invention. The image signal processing unit 220 includes a pixel addition processing unit 221, a demosaic processing unit 222, a YC conversion processing unit 223, an image composition processing unit 224, a sharpness processing unit 225, a color adjustment processing unit 226, and an RGB conversion process. Part 227.

  The pixel addition processing unit 221 performs pixel addition processing and pixel thinning processing on the image data generated by the imaging elements 134 to 136. The pixel addition processing unit 221 will be described in detail with reference to FIG.

  The demosaic processing unit 222 performs demosaic processing (interpolation processing) so that the intensities of all the R, G, and B channels are aligned at each pixel position of the image data (mosaic image) generated by the imaging elements 134 to 136. It is. Then, the demosaic processing unit 222 supplies the RGB image subjected to the demosaic processing to the YC conversion processing unit 223. That is, the demosaic processing unit 222 interpolates Bayer data having only pixel data for one color per pixel, and calculates three pixel data of RGB for one pixel.

  The YC conversion processing unit 223 generates a luminance signal (Y) and a color difference signal (Cr, Cb) by performing YC matrix processing and band limitation on chroma components on the RGB image generated by the demosaic processing unit 222. is there. Then, the generated luminance signal (Y image) and color difference signal (C image) are supplied to the image composition processing unit 224.

  The image composition processing unit 224 performs image composition on the image data generated by the YC conversion processing unit 223, and outputs the synthesized image data to the sharpness processing unit 225. This image composition processing will be described in detail with reference to FIGS. The image composition processing unit 224 is an example of an image composition unit.

  The sharpness processing unit 225 performs sharpness processing (processing for enhancing the contour of a subject) that extracts and emphasizes a portion with a large signal change in the image data generated by the image composition processing unit 224. Then, the sharpness processing unit 225 supplies the image data subjected to the sharpness processing to the color adjustment processing unit 226.

  The color adjustment processing unit 226 adjusts the hue and saturation of the image data that has been sharpened by the sharpness processing unit 225.

  The RGB conversion processing unit 227 converts the image data that has been adjusted in hue and saturation by the color adjustment processing unit 226 from YCbCr data to RGB data.

  Here, the flow of the image data of the signal of the image signal processing unit 220 will be described. For example, assume that each signal processing unit in the image signal processing unit 220 directly reads image data from the image memory 170 via the data bus 204 and writes the image data after signal processing to the image memory 170 via the data bus 204. To do. In this case, it is advantageous in that the image signal processing unit 220 can read image data at a desired position in the image data at a desired timing. However, since the amount of data that needs to be transmitted via the data bus 204 increases, the operating frequency of the data bus 204 needs to be increased. This makes it difficult to design the data bus 204 and may increase power consumption.

  Further, for example, each signal processing unit in the image signal processing unit 220 receives image data from the preceding signal processing unit without passing through the data bus 204, and the image data after the signal processing is sent to the subsequent signal processing unit in the data bus 204. Assume a case of passing without going through. In this case, since the data bus 204 is not used, it is advantageous in that LSI design is easy and power consumption can be reduced. However, each signal processing unit may not be able to read image data at a desired position in the image data at a desired timing.

  Therefore, in the first embodiment of the present invention, between the demosaic processing unit 222 and the color adjustment processing unit 226 whose image size is substantially constant, in order to reduce the operating frequency of the data bus 204 and power consumption, The image data is directly transferred between the signal processing units. In addition, an example in which image data is written to the image memory 170 in the previous stage of a signal processing unit that uses a large amount of image data, such as resolution conversion, and desired image data is read from the image memory 170 when performing resolution conversion. Show.

[Configuration example of clock generation circuit]
FIG. 9 is a block diagram illustrating an internal configuration example of the clock generation circuit 270 according to the first embodiment of the present invention. The clock generation circuit 270 includes high-frequency clock multipliers 20 and 24, high-frequency clock dividers 21 and 25, low-frequency clock multipliers 22 and 26, and low-frequency clock dividers 23 and 27. Prepare. The clock generation circuit 270 includes clock multipliers 28 and 30 and clock frequency dividers 29 and 31. Here, each multiplier multiplies the frequency of the input clock. Each frequency divider makes the frequency of the input clock 1 / n (n is an arbitrary integer). In this example, the case where the clock generation circuit 270 generates at least six types of clocks according to the connection destination of each unit in the DSP 200 shown in FIG. 7 is shown as an example.

  The oscillators 261 to 263 are vibration sources for generating a clock signal supplied to the DSP 200. For example, crystal oscillators are used.

  The oscillation circuits 264 to 266 generate a clock signal supplied to the DSP 200, and output the generated clock signal to the clock generation circuit 270.

  Two of the six types of clocks generated by the clock generation circuit 270 are clocks supplied to the image sensors 134 to 136. One of the clocks supplied to the image sensors 134 to 136 is a clock having a relatively large frequency for generating an image having a relatively large number of pixels. This clock is generated by the clock output from the oscillation circuit 264 being input to the high-frequency clock multiplier 20 and multiplied, and the multiplied clock being input to the high-frequency clock divider 21 and being divided. Is done. Another type is a clock having a relatively low frequency for generating an image having a relatively small number of pixels. The clock output from the oscillation circuit 264 is input to the low-frequency clock multiplier 22 and multiplied, and the multiplied clock is input to the low-frequency clock divider 23 and divided. Is generated by The clocks divided by the high-frequency clock divider 21 and the low-frequency clock divider 23 are output as clocks generated by the clock generation circuit 270, and the imaging elements 134 to 136 are passed through the DSP 200. To be supplied. Here, the clocks supplied to the imaging elements 134 to 136 are not limited to the two types shown in this example, and it is preferable to generate and use more types according to the image size generated by the imaging operation.

  The other two types of the six types of clocks generated by the clock generation circuit 270 are clocks used in the DSP 200. One type of clock used in the DSP 200 is a clock having a relatively large frequency for generating an image having a relatively large number of pixels. This clock is generated by the clock output from the oscillation circuit 264 being input to the high-frequency clock multiplier 24 and being multiplied, and the multiplied clock being input to the high-frequency clock divider 25 for frequency division. Is done. Another type is a clock having a relatively low frequency for generating an image having a relatively small number of pixels. The clock output from the oscillation circuit 264 is input to the low frequency clock multiplier 26 and multiplied, and the multiplied clock is input to the low frequency clock divider 27 and divided. Is generated by Then, the clock divided by the high frequency clock divider 25 and the low frequency clock divider 27 is output as a clock generated in the clock generation circuit 270 and supplied to the DSP 200. Here, the clocks used in the DSP 200 are not limited to the two types shown in this example, and it is preferable to generate and use more types according to the image size generated by the imaging operation.

  The remaining two types of the six types of clocks generated by the clock generation circuit 270 are a pixel clock for displaying an image on the display unit 140 and a display device (for example, the external display device 245) outside the mobile phone device 100. This is a pixel clock for displaying an image. The pixel clock for displaying an image on the display unit 140 is multiplied by the clock output from the oscillation circuit 265 being input to the clock multiplier 28, and the multiplied clock is input to the clock divider 29. Generated by frequency division. Further, the pixel clock for displaying an image on a display device outside the mobile phone device 100 is multiplied by the clock multiplier 30 by the clock output from the oscillation circuit 266, and this multiplied clock is the clock divider. It is generated by being inputted to 31 and divided. The clock divided by the clock divider 29 is output as a clock generated by the clock generation circuit 270 and supplied to the display unit 140 via the DSP 200. Further, the clock frequency-divided by the clock frequency divider 31 is output as a clock generated by the clock generation circuit 270 and supplied to the display device outside the mobile phone device 100 via the DSP 200. Here, the clocks for displaying these images are not limited to the two types shown in this example, and it is preferable to generate and use more types according to the specifications of the connected display device.

  10 and 11 are block diagrams showing modifications of the clock generation circuit 270 according to the first embodiment of the present invention.

  In FIG. 10, two types of clocks supplied to the imaging devices 134 to 136 share one multiplier (clock multiplier 32), and two types of clocks supplied to the DSP 200 are one. An example of sharing and using a multiplier (clock multiplier 33) is shown.

  In FIG. 11, a plurality of types of clocks supplied to the imaging devices 134 to 136 and a plurality of types of clocks supplied to the DSP 200 share one multiplier (clock multiplier 34). An example is shown.

[Configuration Example of Image Sensor and Pixel Reading Example]
FIG. 12 is a diagram illustrating an example of the internal configuration of the image sensor according to the first embodiment of the present invention. Here, since the internal configurations of the image sensors 134 to 136 are substantially the same, only the image sensor 134 is shown in FIG. 12, and illustration and description of other image sensors are omitted. In FIG. 12, a CMOS type image sensor will be described as an example of the image sensor 134.

  The image sensor 134 includes pixels 40 to 47, a vertical scanning circuit 340, and a horizontal scanning circuit 345. The image sensor 134 includes ADCs (A / D (Analog / Digital) converters) 350 to 353, adders 354 to 357 and 366, and column latches 358 to 361. The image sensor 134 includes switches 362 to 365, an output latch 367, an output circuit 368, registers 370 and 380, and multipliers / dividers 391 and 392. In general, the vertical arrangement of image sensors is referred to as a column, and the horizontal arrangement is referred to as a row. For this reason, in the following description, the names of columns and rows are used as appropriate. Further, in this example, in the image sensor 134, some pixels (pixels 40 to 47) and respective parts related thereto are shown as representatives, and illustration and description of other configurations are omitted.

  In the image sensor 134, vertical control lines 341 to 344 are wired in the row direction, and every other pixel on the same line is connected to the same vertical control line. Further, data readout lines 346 to 349 are wired in the column direction, and pixels on the same line share one readout line.

  The vertical scanning circuit 340 turns on / off the switches between the pixels 40 to 47 and the data readout lines 346 to 349 by the vertical control lines 341 to 344 wired in the row direction. That is, in each pixel in the row direction, every other pixel among the pixels on the same line in the row direction is turned on / off in common by one vertical control line. Further, the image data of the pixels 40 to 47 is output to the data read lines 346 to 349 via the switches between the respective pixels and the corresponding data read lines.

  The horizontal scanning circuit 345 turns on / off switches 362 to 365 between the column latches 358 to 361 and the output data line 369. By selecting ON / OFF of the switch by the vertical scanning circuit 340 and ON / OFF of the switches 362 to 365 by the horizontal scanning circuit 345, signals of all the pixels can be read out in time division while sequentially selecting each pixel. Here, the output data line 369 is an output data line for outputting the output result of each column from the image sensor 134.

  Here, in the image sensor 134, the pixels 40 to 47 are arranged in a two-dimensional square lattice. Since the configurations of the pixels 40 to 47 are the same, the pixel 40 will be described as an example here. The pixel 40 includes a photodiode 51 that is a light receiving unit, an amplifier 52, and a switch 53. The photodiode 51 converts light emitted to the pixel into electric charges corresponding to the amount of light. The amplifier 52 is an amplifier that amplifies the charge signal converted by the photodiode 51. The switch 53 is a switch that controls the charge transfer of the pixel 40 in accordance with on / off of the vertical control line 342.

  Each column includes ADCs 350 to 353, adders 354 to 357, and column latches 358 to 361. Hereinafter, the ADC 350, the adder 354, and the column latch 358 connected to the data read line 346 will be described as an example.

  The ADC 350 is an AD converter that converts image data from each pixel that is an analog value into digital data (digital value).

  Each time digital data is converted by the ADC 350, the adder 354 adds new digital data after the conversion to the digital data stored in the column latch 358.

  The column latch 358 is a column latch that sequentially stores the digital data converted by the ADC 350. Here, the column latch is a name indicating a data storage circuit that stores digital data after AD conversion. As the data storage circuit, in addition to the latch configured by a linear circuit, each circuit capable of storing digital data such as a flip-flop configured by a synchronous circuit can be used.

  For example, the image data output from the pixel 40 passes through the ADC 350, the adder 354, and the column latch 358, and then is output to the output data line 390 via the switch 362 connected to the data read line 346. . Here, in the first embodiment of the present invention, the output data line 390 includes the adder 366 and the output latch 367 as well as the data read line of each column, and performs addition and storage of image data. . Further, the image data stored in the output latch 367 is output to the output data line 369 via the output circuit 368. Note that the image data from the output data line 369 is output to the data line L1 described above.

  The multipliers / dividers 391 and 392 perform frequency multiplication of the input clock or frequency division of the input clock based on control from the DSP 200. Then, the multiplier / dividers 391 and 392 supply the generated clock to the vertical scanning circuit 340, the horizontal scanning circuit 345, and the output circuit 368.

  The signal line 393 is a vertical synchronization signal line that supplies a vertical synchronization signal from the DSP 200. The signal line 394 is a horizontal synchronization signal line that supplies a horizontal synchronization signal from the DSP 200.

  The signal line 395 is a clock signal line that supplies a clock signal from the DSP 200. The signal line 396 is a signal line for controlling on / off of the imaging operation from the DSP 200 and a signal line for controlling pixel thinning. A signal line 397 is a bidirectional communication line between the image sensor 134 and the DSP 200. A signal line 398 is a power supply line.

  Note that the registers 370 and 380 are registers that hold setting values related to the imaging operation, and an example of the held contents is shown in FIG.

  FIG. 13 is a diagram schematically showing the contents held in the registers 370 and 380 included in the image sensor 134 according to the first embodiment of the present invention. The registers 370 and 380 hold setting values related to the imaging operation and are supplied to the vertical scanning circuit 340 and the horizontal scanning circuit 345. Note that these setting values may be changeable by a user operation. These holding contents and the imaging operation performed based on them will be described in detail with reference to FIG. 28 and FIG.

  FIGS. 14 to 16 are timing charts schematically showing the state of the control signal to each pixel in the image sensor 134 and the data output from each pixel in the first embodiment of the present invention. . Here, the horizontal axis shown in FIGS. 14 to 16 is a time axis. Further, the vertical control lines 341 to 344 and the column latches 358 to 361 shown in FIGS. 14 to 16 will be described with the same reference numerals as those in FIG. Further, the column latch read signal shown in FIGS. 14 to 16 is described with the same reference numerals as the corresponding switches 362 to 365 shown in FIG.

  In the example illustrated in FIG. 14, an imaging operation in the case where all the pixels in the imaging element 134 are read will be described.

  The vertical control lines 341 to 344 are used to output the pixel data of all the image sensors 134 connected to a certain row (for example, the lines of the pixels 40 to 43) to the data readout lines 346 to 349 of each column. Subsequently, the pixel data output to the data readout lines 346 to 349 are AD converted by the ADCs 350 to 353 of each column. Subsequently, the outputs of the ADCs 350 to 353 are held in the column latches 358 to 361 of the respective columns. For example, the pixel data d1 to d4 are held in the column latches 358 to 361 shown in FIG. Subsequently, the horizontal scanning circuit 345 sequentially turns on the readout switches 362 to 365 from the column latches 358 to 361 to the output data line 390 one column at a time. Thereby, each pixel data in one line can be read in order. For example, the pixel data d2 to d3 are sequentially output after the pixel data d1 is output to the output data line 390 shown in FIG.

  Similarly, every time readout of one line in the horizontal direction is completed, the vertical scanning circuit 340 sequentially turns on the readout switch from each pixel to the vertical signal line one row at a time. As a result, the pixel data of each row is input to the ADCs 350 to 353. After the input pixel data is AD-converted by the ADCs 350 to 353, each pixel data is held in the column latches 358 to 361 of each column. For example, the pixel data d5 to d8 are held in the column latches 358 to 361 shown in FIG. Subsequently, the horizontal scanning circuit 345 sequentially turns on the read switches 362 to 365 from the column latches 358 to 361 to the output data line 390 one column at a time, and sequentially reads each pixel data in one line. For example, the pixel data d5 to d8 are sequentially output to the output data line 390 shown in FIG.

  In the example illustrated in FIG. 15, an imaging operation in the case of performing horizontal thinning and reading out as an example of pixel thinning out of each pixel in the image sensor 134 will be described.

  The vertical scanning circuit 340 turns on the readout switch from each pixel to the vertical signal lines 346 to 349 only in a desired column. As a result, only pixel data of a specific row is input to the ADCs 350 to 353, and AD conversion is performed by the ADCs 350 to 353. The outputs of the ADCs 350 to 353 are held in the column latches 358 to 361 of the respective columns. For example, by turning on the readout switch connected to the vertical control lines 342 and 344, the pixel data d1 and d3 are held in the column latches 358 and 360 shown in FIG. After the readout of one row is completed, for example, by turning on the readout switch connected to the vertical control lines 342 and 344, the pixel data d5 and d7 are held in the column latches 358 and 360 shown in FIG. .

  In addition, the horizontal scanning circuit 345 turns on a read switch from the column latches 358 to 361 to the output data line 390 only in a desired column. Thereby, only specific pixel data in one line can be read in order.

  For example, when one pixel data is read out from N pieces of pixel data in the horizontal direction, 1 / N thinning-out reading is performed in the horizontal direction. For example, when one pixel data is read out of two pixels, 1/2 thinning readout is performed in the horizontal direction, and when one pixel data is read out of four pixels, 1/4 thinning out reading is performed in the horizontal direction. It becomes.

  A thinning operation in the vertical direction (that is, the column direction) can be performed simultaneously with a thinning operation in the horizontal direction (that is, the row direction). For example, when reading pixel data of one line out of M lines in the vertical direction, 1 / M decimation reading is performed in the vertical direction. For example, when reading out pixel data of one row in two lines, 1/2 decimation reading is performed in the vertical direction, and when reading out pixel data of one row in four lines, 1 is read out in the vertical direction. / 4 thinning out reading.

  In the example illustrated in FIG. 16, an imaging operation in a case where pixel addition reading is performed among the pixels in the imaging element 134 will be described. In this example, an imaging operation in the case of pixel addition readout of 1/2 in the horizontal direction and pixel addition readout of 1/2 in the vertical direction will be described as pixel addition readout.

  As in the case of all pixel readout, the vertical control lines are used to output the data of all the image sensors 134 connected to a certain row to the data readout lines provided in each column, AD conversion, and column latch To hold. Here, unlike the case of performing all-pixel readout, using another vertical control line, the data of all the image sensors 134 connected to different rows are output to the data readout line provided in each column, A / D conversion is performed. And it adds to the data of a column latch using an adder. The value of each pixel data is added by this method for the desired number in the vertical direction, and the added data is held in each column latch. For example, the pixel data d1 + d5, d2 + d6, d3 + d7, d4 + d8 are held in the column latches 358 to 361. As described above, after adding N pieces of pixel data in the vertical direction, the addition result is output as one piece of pixel data, whereby 1 / N pixel addition reading in the vertical direction is performed.

  Next, the horizontal scanning circuit 345 turns on the readout switch from the column latch to the output data line 390 one by one in order. In this case, the data read from each column latch to the output data line 390 is added by the adder 366 of the output data line 390 and held in the output latch 367. Then, the addition process is repeated for the desired number of columns in the horizontal direction, and the added data is output to the image sensor 134. For example, data d1 + d5 + d2 + d6 obtained by adding the pixel data d1 + d5 and the pixel data d2 + d6 is output to the image sensor 134 via the output data line 369. Thus, by adding M row pixel data in the horizontal direction, 1 / M pixel addition reading is performed in the horizontal direction. By performing the above processing, addition processing in the horizontal direction (row direction) and addition processing in the vertical direction (column direction) can be performed.

  Here, in the case of performing the decimation readout shown in FIG. 15 and the pixel addition readout shown in FIG. 16, it is desirable to operate the image sensor corresponding to a plurality of types of decimation rates and pixel addition rates. Therefore, hereinafter, a scanning circuit that operates the image sensor corresponding to a plurality of types of thinning rates and pixel addition rates will be described.

  FIG. 17 is a diagram schematically illustrating an example of a scanning circuit for thinning out the image sensor 134 in the first embodiment of the present invention. FIG. 17A shows three types of scanning circuits 104 to 106, and FIG. 17B shows a configuration example of the 1 / N thinning scanning circuit. In the example shown in FIG. 17, the number of pixels in the horizontal direction of the image sensor for the scanning circuit of the image sensor for supporting three types of pixel decimation rates of all pixel readout, 1/2 thinning readout, and 1/4 thinning readout. A case where there are 1024 will be described as an example.

  As shown in FIG. 17A, the scanning circuit 104 includes 1024 output signal lines (scan_a [n (0 ≦ n ≦ 1023: n is an integer)]), and each signal line is in an enable state. This is a 1024 to 1 scanning circuit that is placed in a disabled state after being turned on. In addition, the scanning circuit 105 includes 512 output signal lines (scan_b [n (0 ≦ n ≦ 511: n is an integer)]), and each signal line is set to the disable state after being set to the disable state. This is a scanning circuit. The scanning circuit 106 includes 256 output signal lines (scan_c [n (0 ≦ n ≦ 255: n is an integer)]). Each signal line is set to the disable state after being set to the disable state one by one. This is a scanning circuit.

  As shown in FIG. 17B, among the 1024 output signal lines of the scanning circuit 104, the signal line having a number that is a multiple of 4 includes the output signal line of the scanning circuit 105 and the output signal line of the scanning circuit 106. And a control line indicating which of the three types of pixel thinning rates is selected. These three types of control lines are control lines corresponding to all pixel readout (sel_1per1), 1/2 decimation readout (sel_1per2), and 1/4 decimation readout (sel_1per4).

  Further, out of the 1024 output signal lines of the scanning circuit 104, the signal lines having a number that is not a multiple of 4 and that is a multiple of 2 include the output signal line of the scanning circuit 105 and two types of pixel decimation rates. A control line indicating which one is selected is connected. These two types of control lines are control lines corresponding to all pixel readout and 1/2 thinning readout.

  In addition, among the 1024 output signal lines of the scanning circuit 104, a control line indicating whether all pixel readout is selected is connected to signal lines other than the signal lines described above.

  With the output from the scanning circuit shown in FIG. 17B (scan_out [n (0 ≦ n ≦ 1023: n is an integer)]), the image sensor 134 reads out all pixels, reads out 1/2, and extracts 1/4. It is possible to perform a thinning process for each reading.

[Imaging system layout configuration example]
FIG. 18 is a diagram illustrating a relationship between the imaging unit 130 and the subject in the first embodiment of the present invention. 18 shows only the optical systems 131 to 133 and the imaging elements 134 to 136 among the first imaging system 191 to the third imaging system 193 in the imaging unit 130 shown in FIG. Omitted. Also, in FIG. 18, a case is schematically shown in which the mobile phone device 100 is arranged such that the subject to be imaged is the subject surface 300 and the optical axis 194 of the first imaging system 191 is orthogonal to the subject surface 300. Shown in

  In the imaging unit 130, an angle formed by the optical axis 194 of the first imaging system 191 and the optical axis 195 of the second imaging system 192 is θ0. Similarly, an angle formed by the optical axis 194 of the first imaging system 191 and the optical axis 196 of the third imaging system 193 is θ0. Note that the third imaging system 193 is arranged so as to be in the position of the line target with the second imaging system 192 across the optical axis 194 of the first imaging system 191.

  In the first imaging system 191, an angle formed by the outermost line of the incident optical path incident on the imaging element 134 and the optical axis 194 is θ1. In the second imaging system 192, the angle formed by the outermost line of the incident optical path incident on the imaging device 135 and the optical axis 195 is θ2. Similarly, in the third imaging system 193, the angle formed by the outermost line of the incident optical path incident on the imaging device 136 and the optical axis 196 is θ2. For example, as shown in FIG. 18, the horizontal width of the light receiving surface of the image sensor 134 is smaller than the horizontal width of the light receiving surfaces of the image sensors 135 and 136. Therefore, the angle of view 2 × θ1 incident on the image sensor 134 is smaller than the angle of view 2 × θ2 incident on the image sensors 135 and 136.

  Here, the imaging range 301 of the subject plane 300 is specified by the angle of view 2 × θ1 incident on the imaging element 134. Similarly, the imaging range 302 of the subject plane 300 is specified by the angle of view 2 × θ2 incident on the imaging element 135, and the imaging range 303 of the subject plane 300 is determined by the angle of view 2 × θ2 incident on the imaging element 136. Identified. In the first embodiment of the present invention, images generated by the imaging elements 134 to 136 are combined to generate a panoramic image. For this reason, the angle formed by each optical axis is set so that the imaging range 301 of the subject plane 300 and the imaging range 302 of the subject plane 300 partially overlap. Specifically, the angle θ0 formed by the optical axis 194 of the first imaging system 191 and the optical axis 195 of the second imaging system 192 is set so that a part of the imaging ranges 301 and 302 overlap with each other. An angle θ0 formed by the optical axis 194 of the imaging system 191 and the optical axis 196 of the third imaging system 193 is set. The optical axes 194 to 196 are included in the same plane. Further, the first imaging system 191 to the third imaging system 193 are arranged so that the optical axes 194 to 196 intersect at one point (intersection point P0).

  Here, the lens center of the first imaging system 191 is R1, the lens center of the second imaging system 192 is R2, and the lens center of the third imaging system 193 is R3. Further, the distance between the lens center R1 and the intersection point P0 is L11, the distance between the lens center R2 and the intersection point P0 is L21, and the distance between the lens center R3 and the intersection point P0 is L31. In this case, it is preferable to arrange the first imaging system 191 to the third imaging system 193 so that the distances L11, L21, and L31 are equal.

[Example of keystone correction]
19 and 20 are diagrams schematically illustrating the relationship between the imaging system in the imaging unit 130 and the subjects to be imaged in the first embodiment of the present invention. FIG. 19A schematically shows a subject 310 to be imaged by the imaging unit 130. The subject 310 is assumed to be a subject corresponding to the subject plane 300 shown in FIG. In addition, a range corresponding to the subject 310 is indicated by a rectangle, and the inside of the rectangle is schematically shown as a lattice.

  FIG. 20A schematically shows subjects 311 to 313 to be imaged by the imaging unit 130. The subject 311 is assumed to be a subject corresponding to the first imaging system 191 in the subject 310 shown in FIG. Similarly, it is assumed that the subject 312 is a subject corresponding to the second imaging system 192 of the subject 310 and the subject 313 is a subject corresponding to the third imaging system 193 of the subject 310. Of the subjects 311 to 313, both ends in the horizontal direction of the subject 311 overlap with one end in the horizontal direction of the subjects 312 and 313.

  FIG. 19B and FIG. 20B show an imaging system in the imaging unit 130. The example shown in FIGS. 19B and 20B is the same as the example shown in FIG. 18 except that the reference numerals and the like are omitted.

  FIG. 21 is a diagram schematically illustrating a relationship between the imaging system in the imaging unit 130 and the captured image generated by the imaging unit 130 according to the first embodiment of the present invention. FIG. 21A schematically shows captured images 314 to 316 generated by the imaging unit 130. These captured images 314 to 316 are captured images corresponding to the subjects 311 to 313 shown in FIG. 20A, and schematically show a grid-like rectangle shown in the subjects 311 to 313.

  As shown in FIG. 21A, since the optical axes 195 and 196 of the second imaging system 192 and the third imaging system 193 are not orthogonal to the subject plane 300, the captured images 315 and 316 include Trapezoidal distortion occurs. This trapezoidal distortion will be described in detail with reference to FIG.

  FIG. 22 is a diagram schematically illustrating the relationship between the imaging system in the imaging unit 130 and the subject to be imaged in the first embodiment of the present invention. In FIG. 22, for ease of explanation, the configuration relating to the third imaging system 193 is omitted, and only the first imaging system 191 and the second imaging system 192 are shown. The example shown in FIG. 22 is substantially the same as the example shown in FIG. 18 except that the configuration relating to the third imaging system 193 is omitted. Further, the intersection point P0, angles θ0, θ1 and θ2, lens centers R1 and R2, and distance L11 are the same as those shown in FIG.

  In FIG. 22, the intersection of the optical axis 194 of the first imaging system 191 and the subject plane 300 is defined as S11. Further, the intersection point between the right outline of the angle of view of the first imaging system 191 and the object plane 300 is S12, and the intersection of the left outline of the angle of view of the first imaging system 191 and the object plane 300 is S13. And

  A planar subject region that includes the intersection S11 and is orthogonal to the optical axis 194 and is incident on the first imaging system 191 is defined as a subject surface S10.

  The intersection of the optical axis 195 of the second imaging system 192 and the object plane 300 is S21, the intersection of the right outline of the angle of view of the second imaging system 192 and the object plane 300 is S32, and the second Let S43 be the intersection of the left outline of the angle of view of the second imaging system 192 and the subject plane 300.

  A planar subject area that includes the intersection S21 and is orthogonal to the optical axis 195 and is incident on the second imaging system 192 is defined as a subject surface S20.

  A planar subject region that includes the intersection S32 and is orthogonal to the optical axis 195 and is incident on the second imaging system 192 is referred to as a subject surface S30.

  A planar subject region that includes the intersection S43 and is orthogonal to the optical axis 195 and is incident on the second imaging system 192 is referred to as a subject surface S40.

  Further, an intersection point between the subject surface S30 and the optical axis 195 is set as S31, and an intersection point between the subject surface S40 and the optical axis 195 is set as S41.

  Further, the intersection point between the right outline of the angle of view of the second imaging system 192 and the subject surface S20 is S22, and the intersection point of the right outline of the angle of view of the second imaging system 192 and the subject plane S40 is S42. And

  Further, the intersection point between the left outline of the field angle of the second imaging system 192 and the subject surface S20 is S23, and the intersection point between the right outline of the field angle of the second imaging system 192 and the subject surface S30 is S33. And

  In addition, an intersection of a line segment 197 that passes through the lens center R2 of the second imaging system 192 and is perpendicular to the subject surface 300 and the subject surface 300 is defined as S51.

  For example, when comparing the subject surface S40 including the leftmost point S43 of the angle of view with the subject surface S30 including the rightmost point S32 of the angle of view, the subject surface S40 is located farther from the lens center R2 than the subject surface S30. To do. For this reason, when the subject surface S40 is imaged, a wider area is imaged than when the subject surface S30 is imaged. For example, a case is assumed where line segments having the same length are arranged as subjects on the subject surface S40 and on the subject surface S30. In this case, when the captured image generated for the subject surface S30 is compared with the captured image generated for the subject surface S40, the line segment included in the captured image generated for the subject surface S40 is shorter.

  For this reason, for example, when the subject 312 shown in FIG. 20A is imaged by the second imaging system 192, an area corresponding to the subject 312 is present as in the captured image 315 shown in FIG. It has a trapezoidal shape. In other words, in the example illustrated in FIG. 21A, in the captured image 315, the left side of the rectangle corresponding to the subject 312 is shorter than the right side.

  Similarly, when the subject 313 shown in FIG. 20A is imaged by the third imaging system 193, the area corresponding to the subject 313 is trapezoidal like the captured image 316 shown in FIG. It becomes a shape. In this way, trapezoidal distortion occurs in the captured image generated by the three-lens imaging unit. Therefore, a trapezoidal distortion correction method for correcting the trapezoidal distortion of the captured image will be described below.

  Here, the distance between the intersection S11 and the lens center R1 is L12, the distance between the intersection S13 and the lens center R1, and the distance between the intersection S12 and the lens center R1 is L13.

  The distance between the intersection S21 and the lens center R2 is L22, the distance between the intersection S31 and the lens center R2 is L30, and the distance between the intersection S41 and the lens center R2 is L40.

  The distance between the intersection S32 and the lens center R2 is L23, the distance between the intersection S43 and the lens center R2 is L24, and the distance between the intersection S51 and the lens center R2 is L51. The distances L61 to L66 will be described with reference to FIG.

Here, the following formula is established by the formula of the trigonometric function.
L21 + L22 = L11 + L22 = (L11 + L12) / cos θ0
From this equation, the following equation 1 is obtained. Note that L11 = L21.
L22 = {(L11 + L12) / cos θ0} −L11 Formula 1

For the distance L51, the following formula 2 is obtained from the trigonometric formula and formula 1.
L51 = L22 × cos θ0

          = [{(L11 + L12) / cos θ0} −L11)] × cos θ0 Equation 2

As for the distance L23, the following formula 3 is obtained from the trigonometric formula and formula 2.
L23 = L51 / cos (θ2-θ0)
= (L22 × cos θ0) / cos (θ2−θ0) Equation 3

As for the distance L30, the following formula 4 is obtained from the trigonometric formula and formula 3.
L30 = L23 × cos θ2
= {(L22 × cos θ0) / cos (θ2−θ0)} × cos θ2
= ([({(L11 + L12) / cos θ0} −L11) × cos θ0] / cos (θ2−θ0)]) × cos θ2 Formula 4

  Here, when the distances L11 and L12 on the optical axis 194 of the first imaging system 191 are determined, the distance L30 can be obtained by using Expression 4. Thus, by determining the distance L30, the value XR (= L12 / L30) of the ratio of the distance L30 to the distance L12 can be determined. Note that XR <1.

Further, for the distance L24, the following formula 5 is obtained from the trigonometric formula and formula 2.
L24 = L51 / cos (θ2 + θ0)
= (L22 × cos θ0) / cos (θ2 + θ0) Equation 5

Further, for the distance L40, the following formula 6 is obtained from the trigonometric formula and formulas 1 and 5.
L40 = L24 × cos θ2
= {(L22 × cos θ0) / cos (θ2 + θ0)} × cos θ2
= [[{(L11 + L12) / cos θ0−L11} × cos θ0] / cos (θ2 + θ0)] × cos θ2 Formula 6

  Here, when the distances L11 and L12 on the optical axis 194 of the first imaging system 191 are determined, the distance L40 can be obtained by using Expression 6. Thus, by determining the distance L40, the value XL (= L12 / L40) of the ratio of the distance L40 to the distance L12 can be determined. Note that XL> 1.

  Next, a correction method for correcting trapezoidal distortion using the ratio values XR and XL will be described.

  In the captured image 315 shown in FIG. 21A, each coordinate is converted so that the length of the right side is XR times and the length of the left side is XL times. That is, the right side in the captured image 315 is reduced in the direction of the arrow 321 so that the length of the right side is XR times. On the other hand, the left side in the captured image 315 is enlarged in the direction of the arrow 322 so that the length of the left side is XL times. FIG. 23A shows the corrected image 317 corrected in this way.

  FIG. 23 is a diagram schematically illustrating a relationship between the imaging system in the imaging unit 130 according to the first embodiment of the present invention, the captured image generated by these, and the corrected image after correction. FIG. 23A schematically illustrates a captured image 314 generated by the imaging unit 130 and corrected images 317 and 318 obtained by correcting the captured images 315 and 316 illustrated in FIG. Note that the example shown in FIG. 23B is the same as the example shown in FIG. 18 except that the reference numerals and the like are omitted.

  As described above, in the captured image 315 shown in FIG. 21A, by converting each coordinate so that the length of the right side is XR times and the length of the left side is XL times, FIG. A corrected image 317 shown in FIG. The corrected image 317 generated in this way is an image having a trapezoidal outer shape. Therefore, an image 325 (indicated by a thick line) in which the trapezoidal distortion is corrected can be acquired by cutting out the central portion of the corrected image 317 into a rectangle.

  Similarly, for the captured image 316 shown in FIG. 21A, the coordinates are converted so that the length of the left side is XR times and the length of the right side is XL times, which is shown in FIG. A corrected image 318 is generated. Then, an image 326 (indicated by a thick line) in which the trapezoidal distortion is corrected can be acquired by cutting out the central portion of the corrected image 318 into a rectangle. Note that the trapezoidal distortion correction processing of the captured image is performed by the image composition processing unit 224.

  When performing such trapezoidal distortion correction, for example, the coordinates of each pixel in a captured image distorted in a trapezoid are measured, and the ratio values XR and XL are obtained in advance. Then, using the ratio values XR and XL obtained in advance, the trapezoidal distortion correction process can be obtained by software using an arithmetic device such as a CPU built in the mobile phone device 100.

  In this example, an example of the correction method for correcting the trapezoidal distortion of the captured image generated by the trinocular imaging operation has been described. However, other trapezoidal distortion correction methods (for example, see Japanese Patent Application Laid-Open No. 08-307770). Correction may be performed.

[Example of composition of captured images]
FIG. 24 is a diagram schematically showing a composition flow when the image composition processing unit 224 generates a composite image according to the first embodiment of the present invention. In this example, an example in which three captured images generated by three imaging systems are synthesized based on an angle (convergence angle) formed by optical axes of the three imaging systems. That is, of the two captured images generated by the two imaging systems, one of the overlapping portions is excised based on the convergence angle, and the two captured images are synthesized after the excision.

  FIG. 24A shows a captured image 314 and corrected images 317 and 318 generated by the first imaging system 191 to the third imaging system 193 in the imaging unit 130. The captured image 314 and the corrected images 317 and 318 are the same as those shown in FIG. As shown in FIG. 24A, the captured image 314 and the corrected images 317 and 318 in which the trapezoidal distortion is corrected are acquired by the trapezoidal distortion correction process.

  Here, as described above, the same subject is included in the right end area of the corrected image 317 and the left end area of the captured image 314. Further, the same subject is included in the left end area of the corrected image 318 and the right end area of the captured image 314. Therefore, in the following, a method for calculating a region including these same subjects will be described.

  Here, in FIG. 22, the distance between the intersection S11 and the intersection S21 is L61, the distance between the intersection S11 and the intersection S13 is L62, and the distance between the intersection S51 and the intersection S21 is L63. The distance between the intersection S13 and the intersection S51 is L64, the distance between the intersection S32 and the intersection S51 is L65, and the distance between the intersection S32 and the intersection S13 is L66. The distance L66 is a distance for specifying a region including the same subject among the captured image generated by the first imaging system 191 and the captured image generated by the second imaging system 192. is there. This is a common area between the left end area of the captured image generated by the first imaging system 191 and the right end area of the captured image generated by the second imaging system 192.

Here, the following formulas 7 and 8 are established by the formula of the trigonometric function.
L61 = (L11 + L12) × tan θ0 Equation 7
L62 = L12 × tan θ1 Equation 8

Further, the following formula 9 is obtained from the trigonometric function formula and formula 2.
L63 = L51 × tan θ0
= [{(L11 + L12) / cos θ0} −L11] × cos θ0 × tan θ0 (Equation 9)

Further, using Expression 7 to Expression 8, the following Expression 10 is obtained.
L64 = L61−L62−L63
= {(L11 + L12) × tan θ0} − (L12 × tan θ1) − {(L11 + L12) / cos θ0} −L11) × cos θ0 × tan θ0 (Equation 10)

Further, the following formula 11 is obtained from the trigonometric function formula and formula 2.
L65 = L51 × tan (θ2−θ0)
= [{(L11 + L12) / cos θ0} −L11] × cos θ0 × tan (θ2−θ0) Equation 11

The following equation 12 is obtained using the equations 10 and 11 obtained above.
L66 = L65-L64
= {[{(L11 + L12) / cos θ0} −L11] × cos θ0 × tan (θ2−θ0)} − {{(L11 + L12) × tan θ0} − (L12 × tan θ1) − {(L11 + L12) / cos θ0} −L11) × cos θ0 × tan θ0} Equation 12

  Here, when the distances L11 and L12 on the optical axis 194 of the first imaging system 191 are determined, the distance L66 can be obtained by using Expression 12. Further, a common area between the right end area of the captured image generated by the first image capturing system 191 and the left end area of the captured image generated by the third image capturing system 193 can be obtained in the same manner. .

  FIG. 24B shows a region to be synthesized among the captured image 314 and the corrected images 317 and 318. For example, an area corresponding to the distance L66 calculated using Expression 12 is deleted from the captured image 314 generated by the first imaging system 191. Similarly, for the captured image 314, the common area calculated for the right end of the captured image 314 is deleted. In FIG. 24B, the outer shape of the image 327 after the common area at both ends is deleted is indicated by a bold line.

  FIG. 24C shows a panoramic image 330 generated using the captured image 314 and the corrected images 317 and 318. As shown in FIG. 24B, after the images at both ends of the captured image 314 are deleted, a panoramic image is generated using the deleted image 327 and the corrected images 317 and 318. For example, the correction image 317 is connected to the left end of the image 327 and the correction image 318 is connected to the right end of the image 327 to generate the panoramic image 330.

  FIG. 25 is a diagram schematically showing a composition flow when the image composition processing unit 224 generates a composite image according to the first embodiment of the present invention. The example shown in FIG. 25 is a modification of FIG. 24, and an image including a region to be deleted at the time of image composition is different. That is, as shown in FIG. 25B, for the corrected image 317 corresponding to the captured image generated by the second imaging system 192, an area corresponding to the distance L66 calculated using Expression 12 (at the right end) Area). Similarly, the common area calculated for the left end of the corrected image 318 is deleted. In FIG. 25B, the outlines of the images 332 and 333 after the common area is deleted are indicated by bold lines.

  FIG. 25C shows a panoramic image 330 generated using the images 331 to 333. As shown in FIG. 25B, for example, the image 332 is connected to the left end of the image 331, and the image 333 is connected to the right end of the image 331, so that a panoramic image 330 is generated.

  FIG. 26 is a diagram schematically illustrating a composition flow when the image composition processing unit 224 generates a composite image according to the first embodiment of the present invention. The example shown in FIG. 26 is a modification of FIG. 24, and an image including a region to be deleted at the time of image composition is different. That is, as shown in FIG. 26B, an area corresponding to half of the distance L66 calculated using Expression 12 is deleted from the captured image 314 generated by the first imaging system 191. In addition, an area corresponding to half of the distance L66 calculated by using Expression 12 (right end area) is deleted from the corrected image 317 corresponding to the captured image generated by the second imaging system 192.

  Similarly, for the captured image 314, half of the common area calculated for the right end of the captured image 314 is deleted, and for the corrected image 318 corresponding to the captured image generated by the third imaging system 193, half of the common area. The area corresponding to (the area at the left end) is deleted. In FIG. 26B, the outlines of the images 334 to 336 after the common area is deleted are indicated by bold lines.

  FIG. 26C shows a panoramic image 330 generated using the images 334 to 336. As shown in FIG. 26B, for example, the image 335 is connected to the left end of the image 334, and the image 336 is connected to the right end of the image 334, so that a panoramic image 330 is generated. In this way, by deleting a part of each image, it is possible to appropriately combine the images.

  Note that these image composition processes are performed by the image composition processing unit 224. Further, FIG. 8 shows an example in which the image composition processing unit 224 is arranged after the YC conversion processing unit 223 and before the sharpness processing unit 225 in the image signal processing unit 220. However, the image composition processing may be performed at another stage of the image signal processing unit 220. For example, the trapezoidal distortion correction process and the image composition process may be performed before the demosaic processing unit 222. Further, for example, the trapezoidal distortion correction process and the image composition process may be performed after the demosaic processing unit 222 and before the YC conversion processing unit 223. Further, for example, the trapezoidal distortion correction process and the image composition process may be performed after the color adjustment processing unit 226.

  In addition, when performing such image composition processing, for example, for an image after trapezoidal distortion correction, an overlapping area of each image is measured in advance. Then, using this measured value, it is possible to obtain the deletion processing of the overlapping region of the image by software using an arithmetic device such as a CPU built in the mobile phone device 100.

  In this example, an example in which three captured images are combined based on the convergence angle has been described. However, for example, an image combining process may be performed using another image combining method. For example, for two overlapping images generated by two imaging systems, an image combining method can be used in which two images are pattern-matched and two images are combined by pattern matching. Further, it is possible to use an image composition method in which a change in density level in two images generated by two imaging systems is obtained, and an overlapping portion is obtained based on the change in density level and the two images are synthesized.

[Image mode control example]
Next, an example in which image data is read from the imaging elements 134 to 136 and the image is displayed on the display unit 140 will be described.

  FIG. 27 is a diagram illustrating a subject 500 that is an imaging target of imaging processing by the mobile phone device 100 according to the first embodiment of the present invention. The subject 500 is a subject including, for example, people 501 and 502 standing in the background of mountains, cars 503 and a house 504 in the vicinity thereof.

[Example of control of imaging mode in landscape orientation of second casing]
First, a method for reading image data from each image sensor and a method for displaying the read image data when the second casing 120 is in the horizontally long state will be described. In the first embodiment of the present invention, five types of imaging modes are shown as imaging modes when the second casing 120 is in the horizontally long state.

[Control example of three-lens landscape image mode]
FIG. 28 is a diagram schematically showing an example (first reading method) of reading image data in the imaging devices 134 to 136 according to the first embodiment of the present invention.

  FIG. 28A shows pixel data reading areas 403 to 405 from which pixel data is read in the image sensors 134 to 136. The pixel data reading areas 403 to 405 are the same as the pixel data reading areas 403 to 405 shown in FIG. For this reason, the description about the common part which attaches | subjects the same code | symbol is abbreviate | omitted. In the following, the outline of the pixel data readout region in the image sensors 134 to 136 is indicated by a bold line. Here, the pixel data reading areas 403 to 405 are areas determined based on the setting contents held in the registers 370 and 380 shown in FIG. In this example, an example in which all pixels are read out from the pixels included in the pixel data reading areas 403 to 405 will be described.

  Here, in the first embodiment of the present invention, for example, as the image sensor 134, an image sensor having the number of pixels of horizontal 1440 pixels × vertical 1920 pixels and the horizontal to vertical ratio of the pixels being 3: 4 is used. An example of use will be shown. In addition, an example is shown in which as the imaging devices 135 and 136, imaging devices having a number of pixels of horizontal 1920 pixels × vertical 1440 pixels and a horizontal to vertical ratio of the pixels of 4: 3 are used.

Each set value relating to the reading of the pixel data of the image sensors 134 to 136 when performing this reading operation is held in the registers 370 and 380 shown in FIG. Specifically, the following setting values (11) to (17) are held in the vertical direction imaging region setting register 374 at the time of compound-eye all-pixel readout imaging and the horizontal direction imaging region setting register 384 at the time of compound-eye all-pixel readout imaging. Has been.
(11) The number of pixels H10 in the horizontal direction of the area (pixel data reading area 403) to be read by the image sensor 134
(12) Number of pixels V11 in the vertical direction of the area (pixel data reading area 403) to be read by the image sensor 134
(13) The number of pixels H11 in the horizontal direction of the area (pixel data reading areas 404 and 405) to be read by the image sensors 135 and 136
(14) Number of pixels V11 in the vertical direction of the area (pixel data reading areas 404 and 405) to be read by the image sensors 135 and 136
(15) In the imaging devices 134 to 136, the number of pixels in the horizontal direction and the number of pixels in the vertical direction from the array end where the pixels are arranged to the reading start position. (16) In the imaging devices 134 to 136, from the vertical synchronization signal. Vertical back porch period until start of pixel readout in vertical direction (17) Horizontal back porch period from start of horizontal sync signal to start of pixel readout in horizontal direction in image sensors 134 to 136

  These set values may be set in advance, and by a user operation, the vertical direction imaging region setting register 374 at the time of compound-eye all-pixel reading and imaging and the compound-eye all-pixel reading and imaging at the time of compound eye reading and imaging through a signal line L2. It may be set in the horizontal imaging area setting register 384.

  FIG. 28B shows readout start positions 411 to 413 and readout scanning directions in the pixel data readout areas 403 to 405. The readout start positions 411 to 413 and the readout scanning direction are the same as the readout start positions 411 to 413 and the readout scanning direction shown in FIG. For this reason, the description about the common part which attaches | subjects the same code | symbol is abbreviate | omitted.

  As described above, the image data read from the image sensors 134 to 136 are combined to generate overlapping images of the images as described above. However, in the following, for ease of explanation, explanation will be made without considering overlapping portions of each image. In this example, a part of the image data (horizontal 1440 pixels × vertical 1440 pixels) is read out from the imaging device 134 of horizontal 1440 pixels × vertical 1920 pixels, and the imaging devices 135 and 136 of horizontal 1920 pixels × vertical 1440 pixels. All of the image data is read out. When the image data read out in this way are combined, an image of approximately 7.6 million pixels (5280 pixels × 1440 pixels) having a horizontal to vertical ratio of 11: 3 is generated. A display example of the composite image generated in this way is shown in FIG.

  FIG. 29 is a diagram illustrating a display example of an image on the display unit 140 according to the first embodiment of the present invention. The example shown in FIG. 29 is a display example for displaying the image data read by the first reading method shown in FIG.

  FIG. 29A shows a subject 500 that is an imaging target of an imaging process by the mobile phone device 100. The subject 500 is the same as that shown in FIG. Here, it is assumed that the imaging ranges 511 to 513 in the subject 500 are imaging ranges corresponding to the pixel data readable areas 400 to 402 of the imaging elements 134 to 136 shown in FIG. That is, the imaging range 511 corresponds to the pixel data readable area 400 of the imaging element 134, the imaging range 512 corresponds to the pixel data readable area 401 of the imaging element 135, and the imaging range 513 is the pixel of the imaging element 136. It corresponds to the data readable area 402. In addition, the outlines of the areas corresponding to the pixel data reading areas 403 to 404 shown in FIG. 28 in the imaging ranges 511 to 513 are indicated by bold lines.

  FIG. 29B shows a display example in which the image data read out by the first reading method shown in FIG. 28 is displayed on the display unit 140 when the subject 500 is an imaging target. Note that, in FIG. 29B, the first casing 110 is omitted from the first casing 110 and the second casing 120 included in the mobile phone device 100. In the following display examples, the first housing 110 is omitted.

  As described above, since the horizontal to vertical ratio of the display unit 140 is 4: 3, an image having a horizontal to vertical ratio of 11: 3 cannot be displayed on the entire screen of the display unit 140. Therefore, when each pixel in the display unit 140 is a square lattice, for example, resolution conversion is performed on an image having a horizontal to vertical ratio of 11: 3 that is read and generated by the first reading method illustrated in FIG. To display. For example, the resolution of the image is converted in the horizontal direction and the vertical direction so that the number of pixels in the horizontal direction of the image is substantially the same as the number of pixels in the horizontal direction of the display unit 140 and the horizontal to vertical ratio is 11: 3. .

  As shown in FIG. 29 (b), the composite image after the resolution conversion is displayed on the central portion (captured image display area 521) in the vertical direction of the display unit 140. Here, for example, a single color image can be displayed in the upper and lower margin portions (margin image display areas 522 and 523) of the displayed image.

  In addition, when each pixel of the display unit 140 is not a square lattice, the resolution is determined by using the ratio of the vertical and horizontal lengths of the pixels of the display unit 140 so that the shape of the image does not become abnormal when displayed on the display unit 140. The conversion magnification can be converted, and the converted image can be displayed.

  FIG. 29C shows a scanning direction in the display unit 140 when displaying pixel data read from the image sensors 134 to 136. Since this scanning direction is the same as the example shown in FIG. 6C, description thereof is omitted here.

[Control example of three-lens horizontal long-angle imaging mode]
FIG. 30 is a diagram schematically illustrating an example (second reading method) of reading image data in the image sensors 134 to 136 according to the first embodiment of the present invention.

  FIG. 30A shows pixel data readout areas 421 to 423 from which pixel data is read out in the image sensors 134 to 136. The second reading method is a method of reading a horizontally long image that is widely used by users of image pickup apparatuses such as digital still cameras. For example, a part of the image pickup element 134 is read and the image pickup elements 135 and 136 are read out. Read some area.

  The pixel data reading area 421 is, for example, the same as the pixel data reading area 403 shown in FIG. 28 (for example, an area of 1440 pixels wide × 1440 pixels high). In addition, the pixel data read areas 422 and 423 have, for example, the same vertical length V21 as the vertical length V20 of the pixel data readable areas 401 and 402. Further, for example, the pixel data reading areas 422 and 423 have a horizontal length H21 that is 1/6 of the horizontal length of the pixel data readable areas 401 and 402. That is, the pixel data reading areas 422 and 423 are, for example, areas of horizontal 240 pixels × vertical 1440 pixels. Further, the pixel data readout areas 421 to 423 are areas determined based on the setting contents held in the registers 370 and 380 shown in FIG. In this example, an example in which all pixels are read out from the pixels included in the pixel data reading areas 421 to 423 will be described. Here, since so-called inverted images are formed on the image sensors 134 to 136, the pixel data reading areas 422 and 423 of the image sensors 135 and 136 are areas opposite to the image sensor 134.

Each set value relating to the reading of the pixel data of the image sensors 134 to 136 when performing this reading operation is held in the registers 370 and 380 shown in FIG. Specifically, the following setting values (21) to (27) are held in the vertical direction imaging region setting register 374 at the time of compound-eye all-pixel readout imaging and the horizontal direction imaging region setting register 384 at the time of compound-eye all-pixel readout imaging. Has been.
(21) The number H20 of pixels in the horizontal direction of the area (pixel data reading area 421) to be read by the image sensor 134
(22) The number of pixels V21 in the vertical direction of the area (pixel data reading area 421) to be read by the image sensor 134
(23) The number of pixels H21 in the horizontal direction of the area (pixel data reading areas 422 and 423) to be read by the image sensors 135 and 136
(24) Number of pixels V21 in the vertical direction of the area (pixel data reading areas 422 and 423) to be read by the image sensors 135 and 136
(25) In the image sensors 134 to 136, the number of pixels in the horizontal direction and the number of pixels in the vertical direction from the array end where the pixels are arranged to the reading start position. (26) In the image sensors 134 to 136, from the vertical synchronization signal. Vertical back porch period until start of pixel readout in vertical direction (27) Horizontal back porch period from horizontal synchronization signal to start of pixel readout in horizontal direction in imaging devices 134 to 136

  These set values may be set in advance, and by a user operation, the vertical direction imaging region setting register 374 at the time of compound-eye all-pixel reading and imaging and the compound-eye all-pixel reading and imaging at the time of compound eye reading and imaging through a signal line L2. It may be set in the horizontal imaging area setting register 384.

  FIG. 30B shows the reading start positions 424 to 426 and the reading scanning direction in the pixel data reading areas 421 to 423. In this example, a part of the image data (horizontal 1440 pixels × vertical 1440 pixels) is read from the imaging device 134 of horizontal 1440 pixels × vertical 1920 pixels. A part of the image data (240 horizontal pixels × 1440 vertical pixels) is read from the imaging elements 135 and 136 of 1920 horizontal × 1440 vertical pixels. When the image data read in this way are combined, an image of approximately 2.76 million pixels (1920 pixels × 1440 pixels) having a horizontal to vertical ratio of 4: 3 is generated. A display example of the composite image generated in this way is shown in FIG.

  As described above, the pixel data reading areas 421 to 423 are an example in which the size of the reading area in the imaging elements 135 and 136 is smaller than that in the first reading method. The combined image generated by combining the image data read by the second reading method has a horizontal to vertical ratio of 4: 3 (16: 9 or the like may be used). For this reason, it is possible to generate and display a composite image having the same aspect ratio as the aspect ratio of the recording image in a currently available imaging apparatus.

  FIG. 31 is a diagram showing a display example of an image on the display unit 140 according to the first embodiment of the present invention. The example shown in FIG. 31 is a display example for displaying the image data read by the second reading method shown in FIG.

  FIG. 31A shows a subject 500 that is an imaging target of imaging processing by the mobile phone device 100. The example shown in FIG. 31A is substantially the same as the example shown in FIG. 29A except that the pixel data readout region is changed. For this reason, the same code | symbol is attached | subjected to the part which is common in Fig.29 (a), and description about the part is abbreviate | omitted. Of the imaging ranges 511 to 513, the outlines of the areas corresponding to the pixel data reading areas 421 to 423 shown in FIG.

  FIG. 31B shows a display example in which image data read out by the second reading method shown in FIG. 30 is displayed on the display unit 140 when the subject 500 is set as an imaging target.

  As described above, since the horizontal to vertical ratio of the display unit 140 is 4: 3 and the horizontal to vertical ratio is 4: 3, the composite image can be displayed on the entire screen of the display unit 140. For example, the resolution is converted so that the size of the composite image is the same as the number of pixels of the display unit 140 and the display unit 140 displays the composite image. In this case, the blank image display areas 522 and 523 shown in FIG.

  If the horizontal to vertical ratio of the composite image read and generated by the second reading method shown in FIG. 30 is not the same as the horizontal to vertical ratio of the display device, the first reading method and It can be displayed similarly. In this case, for example, a single color is displayed in the upper and lower margin portions (margin image display area) of the displayed composite image, for example.

  FIG. 31C shows a scanning direction in the display unit 140 when displaying pixel data read from the image sensors 134 to 136. Since this scanning direction is the same as the example shown in FIG. 6C, description thereof is omitted here.

[Control example of monocular landscape image capture mode]
FIG. 32 is a diagram schematically illustrating an example (third reading method) of reading image data in the imaging devices 134 to 136 according to the first embodiment of the present invention.

  FIG. 32A shows a pixel data reading area 431 from which pixel data is read in the imaging elements 134 to 136. The third reading method is a method of reading a horizontally long image that is widely used by users of image pickup apparatuses such as a digital still camera. For example, substantially all the region in the horizontal direction of the image sensor 134 is read, and in the vertical direction, An area having a smaller number of pixels than in the horizontal direction is read out. Note that reading from the image sensors 135 and 136 is not performed.

  The pixel data reading area 431 has, for example, a horizontal length H30 that is the same as the horizontal length of the pixel data readable area 400, and a vertical length V30 that is the vertical length of the pixel data readable area 400. About half of the length. The pixel data reading area 431 is, for example, an area of horizontal 1440 pixels × vertical 1080 pixels. Further, the pixel data reading area 431 is an area determined based on setting contents held in the registers 370 and 380 shown in FIG. In this example, an example in which all the pixels are read out from each pixel included in the pixel data reading area 431 will be described.

Each set value relating to the reading of the pixel data of the image sensor 134 when performing this reading operation is held in the registers 370 and 380 shown in FIG. Specifically, the following setting values (31) to (35) are held in the vertical direction imaging region setting register 371 during monocular all-pixel readout imaging and the horizontal direction imaging region setting register 381 during monocular all-pixel readout imaging. Has been.
(31) The number H30 of pixels in the horizontal direction of the area (pixel data reading area 431) to be read by the image sensor 134
(32) The number of pixels V30 in the vertical direction of the area (pixel data reading area 431) to be read by the image sensor 134
(33) In the image sensor 134, the number of pixels in the horizontal direction and the number of pixels in the vertical direction from the array end where the pixels are arranged to the reading start position. (34) In the image sensor 134, the pixels in the vertical direction from the vertical synchronization signal. Vertical back porch period until start of reading (35) In image sensor 134, horizontal back porch period from horizontal synchronizing signal to start of pixel reading in horizontal direction

  These set values may be set in advance, and by a user operation, the vertical direction imaging region setting register 371 at the time of monocular all-pixel readout imaging and the monocular all-pixel readout imaging via the signal line L2. It may be set in the horizontal imaging area setting register 381.

  FIG. 32B shows the reading start position 432 and the reading scanning direction in the pixel data reading area 431. In this example, a part of the image data (horizontal 1440 pixels × vertical 1080 pixels) is read out from the imaging device 134 of horizontal 1440 pixels × vertical 1920 pixels. Further, reading from the image sensors 135 and 136 is not performed. Using the image data read out in this way, an image of about 1560,000 pixels (horizontal 1440 pixels × vertical 1080 pixels) having a horizontal to vertical ratio of 4: 3 is generated. A display example of the image generated in this way is shown in FIG.

  As described above, the pixel data reading area 431 is an example in which reading is performed only from the image sensor 134 and reading is not performed from the image sensors 135 and 136. Then, the image generated based on the image data read by the third reading method is set to have a horizontal to vertical ratio of 4: 3 (16: 9, etc.) as in the second reading method. May be). For this reason, it is possible to generate and display an image having the same aspect ratio as the aspect ratio of the recording image in a currently available imaging apparatus. Further, for example, it is possible to generate a higher definition image than a VGA (Video Graphics Array) image (horizontal 640 pixels × vertical 480 pixels) that has been widely spread without operating the image sensors 135 and 136. Thereby, power consumption can be reduced.

  FIG. 33 is a diagram showing a display example of an image on the display unit 140 according to the first embodiment of the present invention. The example shown in FIG. 33 is a display example for displaying the image data read by the third reading method shown in FIG.

  FIG. 33A shows a subject 500 that is an imaging target of an imaging process performed by the mobile phone device 100. The example shown in FIG. 33A is substantially the same as the example shown in FIG. 29A except that the pixel data readout area is changed. For this reason, the same code | symbol is attached | subjected to the part which is common in Fig.29 (a), and description about the part is abbreviate | omitted. Of the imaging ranges 511 to 513, the outline of the area corresponding to the pixel data reading area 431 shown in FIG.

  FIG. 33B shows a display example in which image data read out by the second reading method shown in FIG. 30 is displayed on the display unit 140 when the subject 500 is set as an imaging target.

  As described above, since the horizontal to vertical ratio of the display unit 140 is 4: 3 and the horizontal to vertical ratio of the generated image is 4: 3, the image is displayed on the entire screen of the display unit 140. be able to. For example, the resolution is converted so that the image size is the same as the number of pixels of the display unit 140, and the image is displayed on the display unit 140. In this case, the blank image display areas 522 and 523 shown in FIG.

  Note that, when the horizontal to vertical ratio of the image read and generated by the third reading method shown in FIG. 32 is not the same as the horizontal to vertical ratio of the display device, it is the same as the first reading method. Can be displayed. In this case, for example, a single color is displayed in the upper and lower margin portions (margin image display area) of the displayed composite image, for example.

  FIG. 33C shows the scanning direction in the display unit 140 when displaying pixel data read from the image sensors 134 to 136. Since this scanning direction is the same as the example shown in FIG. 6C, description thereof is omitted here.

[Control example of monocular portrait image capture mode]
FIG. 34 is a diagram schematically illustrating an example (fourth reading method) of reading image data in the imaging devices 134 to 136 according to the first embodiment of the present invention.

  FIG. 34A shows a pixel data reading area 435 from which pixel data is read in the image sensors 134 to 136. The fourth reading method is a method of reading a vertically long image that is widely used by users of mobile phone devices. For example, all regions of the image sensor 134 are read. Note that reading from the image sensors 135 and 136 is not performed.

  The pixel data reading area 435 has, for example, a horizontal length H40 that is the same as the horizontal length of the pixel data readable area 400 and a vertical length V40 that is the vertical length of the pixel data readable area 400. Same length. The pixel data reading area 435 is, for example, an area of 1440 horizontal pixels × 1920 vertical pixels. Further, the pixel data reading area 435 is an area determined based on setting contents held in the registers 370 and 380 shown in FIG. In this example, an example in which all the pixels are read out from each pixel included in the pixel data reading area 435 will be described.

Each set value relating to the reading of the pixel data of the image sensor 134 when performing this reading operation is held in the registers 370 and 380 shown in FIG. Specifically, the following set values (41) to (45) are held in the vertical imaging area setting register 371 during monocular all-pixel readout imaging and the horizontal imaging area setting register 381 during monocular all-pixel readout imaging. Has been.
(41) The number of pixels H40 in the horizontal direction of the area (pixel data reading area 435) to be read by the image sensor 134
(42) The number of pixels V40 in the vertical direction of the area (pixel data reading area 435) to be read by the image sensor 134
(43) In the image sensor 134, the number of pixels in the horizontal direction and the number of pixels in the vertical direction from the array end where the pixels are arranged to the reading start position. Vertical back porch period until start of reading (45) Horizontal back porch period from horizontal synchronizing signal to start of horizontal pixel reading in image sensor 134

  These set values may be set in advance, and by a user operation, the vertical direction imaging region setting register 371 at the time of monocular all-pixel readout imaging and the monocular all-pixel readout imaging via the signal line L2. It may be set in the horizontal imaging area setting register 381.

  FIG. 34B shows the reading start position 436 and the reading scanning direction in the pixel data reading area 435. In this example, all of the image data (horizontal 1440 pixels × vertical 1920 pixels) is read from the imaging device 134 of horizontal 1440 pixels × vertical 1920 pixels. Further, reading from the image sensors 135 and 136 is not performed. Using the image data read out in this way, an image of approximately 2.76 million pixels (1440 pixels × 1920 pixels) having a horizontal to vertical ratio of 3: 4 is generated. A display example of the image generated in this way is shown in FIG.

  As described above, the pixel data reading area 435 is an example in which reading from the image sensor 134 is performed and reading from the image sensors 135 and 136 is not performed. An image generated based on the image data read by the fourth reading method has a horizontal to vertical ratio of 3: 4. For this reason, for example, a higher-definition image can be generated without operating the image sensors 135 and 136 than a vertically long VGA image (horizontal 640 pixels × vertical 480 pixels) that is widely spread. Thereby, power consumption can be reduced.

  FIG. 35 is a diagram showing a display example of an image on the display unit 140 according to the first embodiment of the present invention. The example shown in FIG. 35 is a display example that displays the image data read by the fourth reading method shown in FIG.

  FIG. 35A shows a subject 500 that is an imaging target of imaging processing by the mobile phone device 100. The example shown in FIG. 35A is substantially the same as the example shown in FIG. 29A except that the pixel data readout region is changed. For this reason, the same code | symbol is attached | subjected to the part which is common in Fig.29 (a), and description about the part is abbreviate | omitted. Of the imaging ranges 511 to 513, the outline of the area corresponding to the pixel data reading area 435 shown in FIG. 34 is indicated by a bold line.

  FIG. 35B shows a display example in which the image data read out by the fourth reading method shown in FIG. 34 is displayed on the display unit 140 when the subject 500 is the imaging target.

  As described above, since the horizontal to vertical ratio of the display unit 140 is 4: 3, an image in which the horizontal to vertical ratio of the generated image is 3: 4 is displayed on the entire screen of the display unit 140. I can't. Therefore, for example, as in the first reading method, an image with a horizontal to vertical ratio of 3: 4 is converted in resolution and displayed. For example, the resolution of the image is converted in the horizontal direction and the vertical direction so that the number of pixels in the vertical direction of the image is substantially the same as the number of vertical pixels of the display unit 140 and the ratio of horizontal to vertical is 3: 4. .

  As shown in FIG. 35B, the resolution-converted image is displayed in the horizontal central portion (captured image display area 527) of the display unit 140. Here, for example, a single color image can be displayed in the left and right margin portions (margin image display areas 528 and 529) of the displayed image.

  If the horizontal to vertical ratio of the image read and generated by the fourth reading method is not the same as the horizontal to vertical ratio of the display device, the image is displayed in the same manner as the first reading method. Can do.

  FIG. 35C shows a scanning direction in the display unit 140 when displaying pixel data read from the image sensors 134 to 136. Since this scanning direction is the same as the example shown in FIG. 6C, description thereof is omitted here.

[Control example of monocular vertically long area imaging mode]
FIG. 36 is a diagram schematically showing an example (fifth reading method) of the image data reading method in the imaging devices 134 to 136 according to the first embodiment of the present invention.

  FIG. 36A shows a pixel data reading area 437 from which pixel data is read in the imaging elements 134 to 136. The fifth reading method is a method of reading a vertically long image that is widely used by users of mobile phone devices. For example, a partial region of the image sensor 134 is read. Note that reading from the image sensors 135 and 136 is not performed.

  For example, the pixel data reading area 437 has a horizontal length H50 shorter than a horizontal length of the pixel data readable area 400, and a vertical length V50 of the pixel data readable area 400 in the vertical direction. Shorter than the length of The pixel data reading area 437 is, for example, an area of horizontal 480 pixels × vertical 640 pixels. Further, the pixel data reading area 437 is an area determined based on setting contents held in the registers 370 and 380 shown in FIG. In this example, an example in which all the pixels are read out from each pixel included in the pixel data reading area 437 will be described.

Each set value relating to the reading of the pixel data of the image sensor 134 when performing this reading operation is held in the registers 370 and 380 shown in FIG. Specifically, the following setting values (51) to (55) are held in the vertical direction imaging region setting register 371 at the time of monocular all pixel readout imaging and the horizontal direction imaging region setting register 381 at the time of monocular all pixel readout imaging. Has been.
(51) The number of pixels H50 in the horizontal direction of the area (pixel data reading area 437) to be read by the image sensor 134
(52) Number of pixels V50 in the vertical direction of the area (pixel data reading area 437) to be read out by the image sensor 134
(53) In the image sensor 134, the number of pixels in the horizontal direction and the number of pixels in the vertical direction from the array end where the pixels are arranged to the reading start position. Vertical back porch period until start of readout (55) Horizontal back porch period from horizontal synchronization signal to start of pixel readout in horizontal direction in image sensor 134

  These set values may be set in advance, and by a user operation, the vertical direction imaging region setting register 371 at the time of monocular all-pixel readout imaging and the monocular all-pixel readout imaging via the signal line L2. It may be set in the horizontal imaging area setting register 381.

  FIG. 36B shows the reading start position 438 and the reading scanning direction in the pixel data reading area 437. In this example, a part of the image data (horizontal 480 pixels × vertical 640 pixels) is read from the imaging device 134 of horizontal 1440 pixels × vertical 1920 pixels. Further, reading from the image sensors 135 and 136 is not performed. Using the image data read out in this way, an image of about 310,000 pixels (480 pixels × 640 pixels) having a horizontal to vertical ratio of 3: 4 is generated. A display example of the image generated in this way is shown in FIG.

  Thus, the pixel data reading area 437 is an example in which reading is performed from a part of the image sensor 134 and reading from the image sensors 135 and 136 is not performed. Further, an image generated based on the image data read by the fifth reading method is, for example, an image suitable for wireless transmission (that is, an image having a small amount of data) than the image generated by the fourth reading method. ).

  FIG. 37 is a diagram showing a display example of an image on the display unit 140 according to the first embodiment of the present invention. The example shown in FIG. 37 is a display example for displaying the image data read by the fifth reading method shown in FIG.

  FIG. 37A shows a subject 500 that is an imaging target of imaging processing by the mobile phone device 100. The example shown in FIG. 37A is substantially the same as the example shown in FIG. 29A except that the pixel data readout region is changed. For this reason, the same code | symbol is attached | subjected to the part which is common in Fig.29 (a), and description about the part is abbreviate | omitted. Of the imaging ranges 511 to 513, the outline of the area corresponding to the pixel data reading area 437 shown in FIG.

  FIG. 37B shows a display example in which image data read out by the fifth reading method shown in FIG. 36 is displayed on the display unit 140 when the subject 500 is set as an imaging target.

  As described above, since the horizontal to vertical ratio of the display unit 140 is 4: 3, an image having a horizontal to vertical ratio of 3: 4 cannot be displayed on the entire screen of the display unit 140. Therefore, for example, as in the fourth reading method, an image having a horizontal to vertical ratio of 3: 4 is converted in resolution and displayed. For example, the resolution of the image is converted in the horizontal direction and the vertical direction so that the number of pixels in the vertical direction of the image is substantially the same as the number of vertical pixels of the display unit 140 and the ratio of horizontal to vertical is 3: 4. .

  As shown in FIG. 37B, the resolution-converted image is displayed in the horizontal central portion (captured image display area 531) of the display unit 140. Here, for example, a single color image can be displayed in the left and right margin portions (margin image display areas 532 and 533) of the displayed image.

  If the horizontal to vertical ratio of the image read and generated by the fifth reading method is not the same as the horizontal to vertical ratio of the display device, the image is displayed in the same manner as the first reading method. Can do.

  FIG. 37C shows a scanning direction in the display unit 140 when displaying pixel data read from the image sensors 134 to 136. Since this scanning direction is the same as the example shown in FIG. 6C, description thereof is omitted here.

  Here, as described above, each set value relating to reading of pixel data of the imaging elements 134 to 136 is held in the registers 370 and 380 shown in FIG. When the second casing 120 is in the horizontally long state, the first reading method to the fifth reading method are switched every time the user presses the imaging range switch 111 shown in FIG. . In this case, every time the imaging range changeover switch 111 is pressed, the imaging control unit 201 detects the pressing, and sequentially switches the first reading method to the fifth reading method. In addition, for example, when the second casing 120 is in the horizontally long state immediately after the mobile phone device 100 is activated, the first reading method can be set.

[Example of pixel decimation and pixel addition]
In the above, as the first reading method to the fifth reading method, the example of reading all the pixels included in the pixel data reading region has been described. However, depending on the purpose of use, there may be a case where a high-definition image is not necessary. Therefore, in the following, an example in which a part of each pixel included in the pixel data reading area is read to reduce power consumption will be described.

  The following sixth to tenth readout methods are examples in which a part of each pixel included in the pixel data readout region is read out by performing pixel thinning processing in the imaging elements 134 to 136. Although not described below, a part of each pixel included in the pixel data reading area may be read out by performing pixel addition processing in the imaging elements 134 to 136.

[Thinning out the three-lens landscape image mode]
First, the sixth reading method will be described with reference to FIGS. The sixth reading method corresponds to the first reading method. In the pixel data reading regions 403 to 405 shown in FIG. 28, 1/2 pixel thinning readout is performed in the vertical direction, and 1 / This is an example in which two thinning-out reading is performed. That is, image data (horizontal 720 pixels × vertical 720 pixels) thinned out in half in the vertical direction and the horizontal direction is read from the pixel data reading region 403 in the image sensor 134 of horizontal 1440 pixels × vertical 1920 pixels. In addition, image data (960 pixels × 720 pixels) thinned in half in the vertical direction and the horizontal direction from the pixel data reading areas 404 and 405 in the imaging elements 135 and 136 of 1920 × 1440 pixels. Read out. When the image data read out in this way are combined, an image of about 1.9 million pixels (2640 pixels × 720 pixels) having a horizontal to vertical ratio of 11: 3 is generated. Similar to the example shown in FIG. 29, this image has a relatively wide angle of view in the horizontal direction and is a higher definition image than the VGA image. Further, it can be generated with less power consumption than the first reading method.

  Each set value relating to the reading of the pixel data of the image sensors 134 to 136 when performing this reading operation is held in the registers 370 and 380 shown in FIG. Specifically, the setting values (11) to (17) described above are held in the vertical direction imaging region setting register 375 at the time of compound eye pixel thinning readout imaging and the horizontal direction imaging region setting register 385 at the time of compound eye pixel thinning readout imaging. Has been. Further, the horizontal direction and vertical direction thinning intervals are held in the vertical pixel thinning interval setting register 376 during compound eye pixel thinning readout imaging and the horizontal direction pixel thinning interval setting register 386 during compound eye pixel thinning readout imaging.

  These set values may be set in advance, or may be set via the signal line L2 by a user operation.

[Thinning out the three-lens landscape image mode]
Next, a seventh reading method will be described with reference to FIGS. 30 and 31. FIG. The seventh readout method corresponds to the second readout method, and in the pixel data readout regions 421 to 423 shown in FIG. 30, half pixel readout is performed in the vertical direction, and 1 / This is an example in which two thinning-out reading is performed. That is, image data (horizontal 720 pixels × vertical 720 pixels) thinned out in half in the vertical direction and the horizontal direction is read out from the pixel data reading area 421 in the image sensor 134 of horizontal 1440 pixels × vertical 1920 pixels. Also, image data (240 pixels wide × 720 pixels high) thinned out in half in the vertical direction and the horizontal direction from the pixel data reading areas 422 and 423 in the imaging elements 135 and 136 of horizontal 1920 pixels × vertical 1440 pixels. Read out. When the image data read in this way are combined, an image of about 690,000 pixels (960 pixels × 720 pixels) having a horizontal to vertical ratio of 4: 3 is generated. Similar to the example shown in FIG. 31, this image has a relatively wide angle of view in the horizontal direction and is a higher definition image than the VGA image. Further, it can be generated with less power consumption than the second reading method.

  Each set value relating to the reading of the pixel data of the image sensors 134 to 136 when performing this reading operation is held in the registers 370 and 380 shown in FIG. Specifically, the setting values (11) to (27) described above are held in the vertical direction imaging region setting register 375 at the time of compound eye pixel thinning readout imaging and the horizontal direction imaging region setting register 385 at the time of compound eye pixel thinning readout imaging. Has been. Further, the horizontal direction and vertical direction thinning intervals are held in the vertical pixel thinning interval setting register 376 during compound eye pixel thinning readout imaging and the horizontal direction pixel thinning interval setting register 386 during compound eye pixel thinning readout imaging.

  These set values may be set in advance, or may be set via the signal line L2 by a user operation.

[Example of thinning out monocular image capture mode]
Next, an eighth reading method will be described with reference to FIGS. 32 and 33. FIG. The eighth readout method corresponds to the third readout method. In the pixel data readout region 431 shown in FIG. 32, 1/2 pixel skipping readout is performed in the vertical direction, and 1/2 readout is performed in the horizontal direction. This is an example of performing thinning readout. That is, the image data (720 pixels × 540 pixels) thinned out in half in the vertical direction and the horizontal direction is read from the pixel data reading region 431 in the image sensor 134 of horizontal 1440 pixels × vertical 1920 pixels. When the image data read out in this way are combined, an image of about 390,000 pixels (720 × 540 pixels) having a horizontal to vertical ratio of 4: 3 is generated. This image has the same angle of view as the example shown in FIG. 33, and is a higher definition image than the VGA image. Further, it can be generated with less power consumption than the third reading method.

  Each set value relating to the reading of the pixel data of the image sensor 134 when performing this reading operation is held in the registers 370 and 380 shown in FIG. Specifically, the setting values (31) to (35) described above are held in the vertical direction imaging region setting register 372 during monocular pixel thinning readout imaging and the horizontal direction imaging region setting register 382 during monocular pixel thinning readout imaging. Has been. The horizontal and vertical thinning intervals are held in the vertical pixel thinning interval setting register 373 during monocular pixel thinning readout imaging and the horizontal pixel thinning interval setting register 383 during monocular pixel thinning readout imaging.

  These set values may be set in advance, or may be set via the signal line L2 by a user operation.

[Example of thinning out monocular portrait image capture mode]
Next, a ninth reading method will be described with reference to FIGS. The ninth readout method corresponds to the fourth readout method. In the pixel data readout region 435 shown in FIG. 34, 1/2 pixel skipping readout is performed in the vertical direction, and 1/2 readout is performed in the horizontal direction. This is an example of performing thinning readout. That is, image data (720 pixels × 960 pixels) thinned out in half in the vertical direction and the horizontal direction is read out from the pixel data reading area 435 in the image sensor 134 of horizontal 1440 pixels × vertical 1920 pixels. When the image data read out in this way are combined, an image of about 690,000 pixels (720 pixels × 960 pixels) having a horizontal to vertical ratio of 3: 4 is generated. This image has the same angle of view as the example shown in FIG. 35, and is a higher definition image than the VGA image. Further, it can be generated with less power consumption than the fourth reading method.

  Each set value relating to the reading of the pixel data of the image sensor 134 when performing this reading operation is held in the registers 370 and 380 shown in FIG. Specifically, the setting values (31) to (35) described above are held in the vertical direction imaging region setting register 372 during monocular pixel thinning readout imaging and the horizontal direction imaging region setting register 382 during monocular pixel thinning readout imaging. Has been. The horizontal and vertical thinning intervals are held in the vertical pixel thinning interval setting register 373 during monocular pixel thinning readout imaging and the horizontal pixel thinning interval setting register 383 during monocular pixel thinning readout imaging.

  These set values may be set in advance, or may be set via the signal line L2 by a user operation.

[Example of thinning out monocular portrait image mode]
Next, a tenth reading method will be described with reference to FIGS. The tenth readout method corresponds to the fifth readout method, and in the pixel data readout region 437 shown in FIG. 36, 1/2 pixel thinning readout is performed in the vertical direction and 1/2 in the horizontal direction. This is an example of performing thinning readout. That is, image data (240 horizontal pixels × 320 vertical pixels) thinned out in half in the vertical direction and the horizontal direction is read out from the pixel data reading region 437 in the image sensor 134 of horizontal 1440 pixels × vertical 1920 pixels. When the image data read out in this way are combined, an image of about 80,000 pixels (240 horizontal pixels × 320 vertical pixels) having a horizontal to vertical ratio of 3: 4 is generated. This image has the same angle of view as the example shown in FIG. Further, it can be generated with less power consumption than the fifth reading method.

  Each set value relating to the reading of the pixel data of the image sensor 134 when performing this reading operation is held in the registers 370 and 380 shown in FIG. Specifically, the setting values (51) to (55) described above are held in the vertical direction imaging region setting register 372 during monocular pixel decimation readout imaging and the horizontal direction imaging region setting register 382 during monocular pixel decimation readout imaging. Has been. The horizontal and vertical thinning intervals are held in the vertical pixel thinning interval setting register 373 during monocular pixel thinning readout imaging and the horizontal pixel thinning interval setting register 383 during monocular pixel thinning readout imaging.

  These set values may be set in advance, or may be set via the signal line L2 by a user operation.

[Example of imaging mode control in the vertically long state of the second casing]
Next, a method for reading image data from each image sensor and a method for displaying the read image data when the second casing 120 is in the vertically long state will be described. In the first embodiment of the present invention, four types of imaging modes are shown as imaging modes when the second casing 120 is in the vertically long state. Here, for example, when shooting is performed with the second casing 120 in the vertically long state, it is assumed that the user does not intend to shoot a horizontally wide wide-angle image. Therefore, in the first embodiment of the present invention, an example of generating an image in a relatively narrow range in the horizontal direction when the second casing 120 is in the vertically long state is shown.

[Example of control of three-lens horizontal long-angle imaging]
The eleventh reading method is a method of reading a horizontally long image that is widely used by users of imaging devices such as digital still cameras, as in the second reading method described above. For example, a partial area of the image sensor 134 is read, and a partial area of the image sensors 135 and 136 is read. Therefore, as an eleventh readout method, an example in which all pixels are read out in the pixel data readout regions 421 to 423 (shown in FIG. 30) similar to the above-described second readout method will be described.

  Note that the pixel data read areas 421 to 423 and the contents stored in the registers 370 and 380 of the setting values related thereto are the same as in the example shown in FIG. 30A, and thus description thereof is omitted here. Further, the readout start positions 424 to 426 and the readout scanning direction in the pixel data readout areas 421 to 423 are the same as those in the example shown in FIG. 30B, and thus description thereof is omitted here. A display example of the generated composite image is shown in FIG.

  FIG. 38 is a diagram showing a display example of an image on the display unit 140 according to the first embodiment of the present invention. The example shown in FIG. 38 is a display example in which image data read by the eleventh reading method is displayed.

  FIG. 38A shows a subject 500 that is an imaging target of imaging processing by the mobile phone device 100. The example shown in FIG. 38A is substantially the same as the example shown in FIG. 29A except that the pixel data readout region is changed. For this reason, the same code | symbol is attached | subjected to the part which is common in Fig.29 (a), and description about the part is abbreviate | omitted. Of the imaging ranges 511 to 513, the outlines of the areas corresponding to the pixel data reading areas 421 to 423 shown in FIG.

  FIG. 38B shows a display example in which image data read by the eleventh reading method is displayed on the display unit 140 when the subject 500 is an imaging target. FIG. 38B shows a case where the first casing 110 is in a vertically long state. Further, when the first casing 110 is in the vertically long state, the horizontal to vertical ratio of the display unit 140 is 3: 4.

  As described above, since the horizontal to vertical ratio of the display unit 140 is 3: 4, an image in which the horizontal to vertical ratio of the generated image is 4: 3 is displayed on the entire screen of the display unit 140. I can't. Therefore, for example, as in the first reading method, an image having a horizontal to vertical ratio of 4: 3 is converted in resolution and displayed. For example, the resolution of the image is converted in the horizontal direction and the vertical direction so that the number of pixels in the horizontal direction of the image is substantially the same as the number of pixels in the horizontal direction of the display unit 140 and the horizontal to vertical ratio is 4: 3. .

  As shown in FIG. 38B, the resolution-converted image is displayed in the vertical central portion (captured image display area 541) of the display unit 140. Here, for example, a single color image can be displayed in the upper and lower margin portions (margin image display areas 542 and 543) of the displayed image.

  FIG. 38C schematically shows the scanning direction of the display unit 140 when the captured image and the blank image written in the image memory 170 are displayed.

[Example of image rotation processing]
Here, in the case where the second casing 120 is in the vertically long state, it is assumed that the image read by the image sensor is displayed on the display unit 140 in the same manner as the horizontally long state of the second casing 120 described above. To do. In this case, since the second casing 120 is rotated by 90 degrees, an image rotated by 90 degrees is displayed on the display unit 140 when the first casing 110 is used as a reference. That is, for the user, the subject included in the image displayed on the display unit 140 is rotated 90 degrees, so that the user feels uncomfortable. Therefore, in the following, an example in which an image is displayed by being rotated 90 degrees in the direction opposite to the rotation direction of the second casing 120 will be described.

  FIG. 39 is a diagram schematically showing a rotation process for rotating the image displayed on the display unit 140 according to the first embodiment of the present invention. In this example, the direction of the subject at the time of imaging, the scanning direction of the imaging device, the direction at the time of writing and reading image data to the image memory 170, the scanning direction of display on the display unit 140, and the display unit 140 The relationship with the direction of the displayed subject is shown. Note that the DSP 200 writes and reads image data to and from the image memory 170.

  FIG. 39A schematically shows a state in which the picked-up images 545 read in the scanning direction in the image pickup devices 134 to 136 are written in the image memory 170 by the DSP 200 in the scanning direction. In FIG. 39A, the scanning direction of the image sensors 134 to 136 is indicated by arrows.

  FIG. 39B schematically shows a state in which the captured image 545 written in the image memory 170 is read while scanning in a direction orthogonal to the scanning direction of the imaging elements 134 to 136. In FIG. 39B, the scanning direction of the captured image 545 read from the image memory 170 is indicated by an arrow.

  FIG. 39C shows the image memory 170 while scanning the captured image 545 read from the image memory 170 in the scanning direction shown in FIG. 39B in the same direction as the direction shown in FIG. The state of writing back to is schematically shown. In FIG. 39C, the scanning direction of the captured image 545 written to the image memory 170 is indicated by an arrow. In this way, the captured image 545 read from the image memory 170 while scanning in the direction orthogonal to the scanning direction of the image sensors 134 to 136 is written back to the image memory 170 while scanning in the read direction. Accordingly, the captured image 545 read from the imaging elements 134 to 136 can be held in the image memory 170 in a state where the captured image 545 is rotated by 90 °.

  FIG. 39D schematically shows a state in which blank images 546 and 547 are added to the captured image 545 written in the image memory 170 in the scanning direction shown in FIG. The margin images 546 and 547 are images displayed in the margin image display areas 542 and 543 shown in FIG.

  FIG. 39 (e) schematically shows a state in which the captured image 545 and blank images 546 and 547 written in the image memory 170 are read while scanning in the same direction as the first writing. By displaying the captured image 545 and the blank images 546 and 547 read in the state shown in FIG. 39E on the display unit 140, an image that does not give the user a sense of incongruity can be displayed. A display example in which the display unit 140 displays the captured image 545 and the blank images 546 and 547 read in the state illustrated in FIG. 39E is illustrated in FIG.

[Control example of monocular landscape image capture mode]
The twelfth reading method is a method of reading a horizontally long image that is widely used by users of imaging devices such as digital still cameras, as in the eleventh reading method described above. For example, substantially all the area in the horizontal direction of the image sensor 134 is read, and an area having a smaller number of pixels in the vertical direction than in the horizontal direction is read. Note that reading from the image sensors 135 and 136 is not performed. Therefore, as an twelfth readout method, an example in which all pixels are read out in the pixel data readout region 431 (shown in FIG. 32) similar to the above-described third readout method will be described.

  Note that the pixel data read-out area 431 and the contents stored in the registers 370 and 380 associated with the setting values are the same as in the example shown in FIG. 32A, and thus the description thereof is omitted here. Further, the reading start position 432 and the reading scanning direction in the pixel data reading area 431 are the same as those in the example shown in FIG. 32B, and thus the description thereof is omitted here. A display example of the generated composite image is shown in FIG.

  FIG. 40 is a diagram illustrating a display example of an image on the display unit 140 according to the first embodiment of the present invention. The example shown in FIG. 40 is a display example in which image data read by the twelfth reading method is displayed.

  FIG. 40A shows a subject 500 that is an imaging target of imaging processing by the mobile phone device 100. The example shown in FIG. 40A is substantially the same as the example shown in FIG. 29A except that the pixel data readout region is changed. For this reason, the same code | symbol is attached | subjected to the part which is common in Fig.29 (a), and description about the part is abbreviate | omitted. Of the imaging ranges 511 to 513, the outline of the area corresponding to the pixel data reading area 431 shown in FIG.

  FIG. 40B shows a display example in which image data read by the twelfth reading method is displayed on the display unit 140 when the subject 500 is the imaging target. Note that the display example in FIG. 40B is a display example of the captured image rotated by the rotation process shown in FIG.

  As described above, since the horizontal to vertical ratio of the display unit 140 is 3: 4, an image in which the horizontal to vertical ratio of the generated image is 4: 3 is displayed on the entire screen of the display unit 140. I can't. Therefore, for example, as in the eleventh reading method, an image having a horizontal to vertical ratio of 4: 3 is converted in resolution and displayed. For example, the resolution of the image is converted in the horizontal direction and the vertical direction so that the number of pixels in the horizontal direction of the image is substantially the same as the number of pixels in the horizontal direction of the display unit 140 and the horizontal to vertical ratio is 4: 3. .

  As shown in FIG. 40B, the image whose resolution has been converted is displayed on the central portion (captured image display area 551) of the display unit 140 in the vertical direction. Here, for example, a single color image can be displayed in the upper and lower margin portions (margin image display areas 552 and 553) of the displayed image.

  FIG. 40C shows a scanning direction in the display unit 140 when displaying pixel data read from the image sensor 134.

[Control example of monocular portrait image capture mode]
The thirteenth reading method is a method of reading a vertically long image that is widely used by users of mobile phone devices. For example, all regions of the image sensor 134 are read. Note that reading from the image sensors 135 and 136 is not performed. For this reason, as a thirteenth readout method, an example in which all pixels are read out in the pixel data readout region 435 (shown in FIG. 34) similar to the above-described fourth readout method will be described.

  Note that the pixel data read-out area 435 and the contents stored in the registers 370 and 380 for the set values related thereto are the same as those in the example shown in FIG. 34A, and thus the description thereof is omitted here. Further, the reading start position 436 and the reading scanning direction in the pixel data reading area 435 are also the same as the example shown in FIG. 34B, and thus the description thereof is omitted here. A display example of the generated composite image is shown in FIG.

  FIG. 41 is a diagram showing a display example of an image on the display unit 140 according to the first embodiment of the present invention. The example shown in FIG. 41 is a display example that displays image data read by the thirteenth reading method.

  FIG. 41A shows a subject 500 that is an imaging target of imaging processing by the mobile phone device 100. The example shown in FIG. 41A is substantially the same as the example shown in FIG. 29A except that the pixel data readout area is changed. For this reason, the same code | symbol is attached | subjected to the part which is common in Fig.29 (a), and description about the part is abbreviate | omitted. Of the imaging ranges 511 to 513, the outline of the area corresponding to the pixel data reading area 435 shown in FIG.

  FIG. 41B shows a display example in which the image data read out by the thirteenth reading method is displayed on the display unit 140 when the subject 500 is the imaging target. Note that the display example in FIG. 41B is a display example of the captured image rotated by the rotation process shown in FIG.

  As described above, since the horizontal to vertical ratio of the display unit 140 is 3: 4 and the horizontal to vertical ratio of the generated image is 3: 4, the image is displayed on the entire screen of the display unit 140. Can be made. For example, the resolution is converted so that the image size is the same as the number of pixels of the display unit 140, and the image is displayed on the display unit 140. In this case, a blank image display area is not necessary.

  When the horizontal to vertical ratio of the image read and generated by the thirteenth reading method is not the same as the horizontal to vertical ratio of the display device, the image is displayed in the same manner as the twelfth reading method. Can do. In this case, for example, a single color is displayed in the upper and lower margin portions (margin image display area) of the displayed composite image, for example.

  FIG. 41C shows the scanning direction of the display unit 140 when displaying pixel data read from the image sensor 134.

[Control example of monocular vertically long area imaging mode]
The fourteenth reading method is a method of reading a vertically long image that is widely used by users of mobile phone devices. For example, a partial region of the image sensor 134 is read. Note that reading from the image sensors 135 and 136 is not performed. Therefore, as the fourteenth readout method, an example in which all pixels are read out in the pixel data readout region 437 (shown in FIG. 36) similar to the fifth readout method described above will be described.

  Note that the pixel data read area 437 and the contents stored in the registers 370 and 380 for the set values related thereto are the same as in the example shown in FIG. 36A, and thus the description thereof is omitted here. Further, the reading start position 438 and the reading scanning direction in the pixel data reading area 437 are the same as those in the example shown in FIG. A display example of the generated composite image is shown in FIG.

  FIG. 42 is a diagram illustrating a display example of an image on the display unit 140 according to the first embodiment of the present invention. The example shown in FIG. 42 is a display example in which image data read by the fourteenth reading method is displayed.

  FIG. 42A shows a subject 500 that is an imaging target of imaging processing by the mobile phone device 100. The example shown in FIG. 42A is substantially the same as the example shown in FIG. 29A except that the pixel data readout area is changed. For this reason, the same code | symbol is attached | subjected to the part which is common in Fig.29 (a), and description about the part is abbreviate | omitted. Of the imaging ranges 511 to 513, the outline of the area corresponding to the pixel data reading area 437 shown in FIG.

  FIG. 42B shows a display example in which image data read out by the fourteenth reading method is displayed on the display unit 140 when the subject 500 is set as an imaging target. Note that the display example in FIG. 42B is a display example of the captured image rotated by the rotation process shown in FIG.

  As described above, since the horizontal to vertical ratio of the display unit 140 is 3: 4 and the horizontal to vertical ratio of the generated image is 3: 4, the image is displayed on the entire screen of the display unit 140. Can be made. For example, the resolution is converted so that the image size is the same as the number of pixels of the display unit 140, and the image is displayed on the display unit 140. In this case, a blank image display area is not necessary.

  When the horizontal to vertical ratio of the image read and generated by the fourteenth reading method is not the same as the horizontal to vertical ratio of the display device, the image is displayed in the same manner as the twelfth reading method. Can do. In this case, for example, a single color is displayed in the upper and lower margin portions (margin image display area) of the displayed composite image, for example.

  FIG. 42C shows a scanning direction in the display unit 140 when displaying pixel data read from the image sensor 134.

  Here, as described above, each set value relating to reading of pixel data of the imaging elements 134 to 136 is held in the registers 370 and 380 shown in FIG. When the second casing 120 is in the vertically long state, the eleventh reading method to the fourteenth reading method are switched every time the user presses the imaging range switch 111 shown in FIG. . In this case, every time the imaging range switching switch 111 is pressed, the imaging control unit 201 detects the pressing and sequentially switches the eleventh reading method to the fourteenth reading method. In addition, for example, if the second casing 120 is in the vertically long state immediately after the mobile phone device 100 is activated, the fourteenth reading method can be set.

[Example of pixel decimation and pixel addition]
In the above, as the eleventh reading method to the fourteenth reading method, the example of reading all the pixels included in the pixel data reading region has been described. However, depending on the purpose of use, there may be a case where a high-definition image is not necessary. Therefore, in the following, an example in which a part of each pixel included in the pixel data reading area is read to reduce power consumption will be described.

  The fifteenth to eighteenth reading methods described below are examples in which a part of each pixel included in the pixel data reading region is read by performing pixel thinning processing in the imaging elements 134 to 136. Although not described below, a part of each pixel included in the pixel data reading area may be read out by performing pixel addition processing in the imaging elements 134 to 136.

[Thinning out the three-lens landscape image mode]
First, the fifteenth reading method will be described with reference to FIG. The fifteenth readout method corresponds to the eleventh readout method. In the pixel data readout regions 421 to 423 shown in FIG. 30, half-pixel skip readout is performed in the vertical direction and 1 / th in the horizontal direction. This is an example in which two thinning-out reading is performed. Note that these thinning-out processes are the same as the example shown in the seventh reading method, and thus detailed description thereof is omitted here. In addition, the contents stored in the registers 370 and 380 for the setting values relating to the reading of the pixel data of the image sensors 134 to 136 in these thinning-out processes are the same as the example shown in the seventh reading method. The detailed explanation is omitted.

[Example of thinning out monocular image capture mode]
Next, a sixteenth reading method will be described with reference to FIG. The sixteenth readout method corresponds to the twelfth readout method. In the pixel data readout region 431 shown in FIG. 32, half pixel readout is performed in the vertical direction and half in the horizontal direction. This is an example of performing thinning readout. Note that these thinning-out processes are the same as in the example shown in the eighth reading method, and thus detailed description thereof is omitted here. In addition, since the contents held in the registers 370 and 380 for the setting values relating to the reading of the pixel data of the image sensors 134 to 136 in these thinning-out processes are the same as the example shown in the eighth reading method, the details here The detailed explanation is omitted.

[Example of thinning out monocular portrait image capture mode]
Next, a seventeenth reading method will be described with reference to FIG. The seventeenth readout method corresponds to the thirteenth readout method, and in the pixel data readout region 435 shown in FIG. 34, half pixel readout is performed in the vertical direction and half in the horizontal direction. This is an example of performing thinning readout. Since these thinning-out processes are the same as the example shown in the ninth reading method, detailed description thereof is omitted here. In addition, the contents stored in the registers 370 and 380 for the setting values relating to the reading of the pixel data of the image sensors 134 to 136 in these thinning-out processes are the same as the example shown in the ninth reading method. The detailed explanation is omitted.

[Example of thinning out monocular portrait image mode]
Next, the eighteenth reading method will be described with reference to FIG. The eighteenth readout method corresponds to the fourteenth readout method, and in the pixel data readout region 437 shown in FIG. 36, half-pixel thinning readout is performed in the vertical direction and half in the horizontal direction. This is an example of performing thinning readout. Note that these thinning-out processes are the same as the example shown in the tenth reading method, and thus detailed description thereof is omitted here. Further, the contents held in the registers 370 and 380 for the setting values relating to the reading of the pixel data of the image sensors 134 to 136 in these thinning-out processes are the same as the example shown in the tenth reading method. The detailed explanation is omitted.

[Reading example during monitoring operation]
Next, a method for reading pixel data when the monitoring operation is performed will be described. For example, when a still image recording mode for recording a still image is set, a monitoring operation is performed until a still image recording instruction operation is performed. In this monitoring operation, for example, before a still image recording instruction operation is performed, a monitoring image for confirming whether the orientation of the mobile phone device 100 and the size of the subject are appropriate is displayed on the display unit 140. Is the action. This monitoring image is an image for the user to confirm whether the orientation of the mobile phone device 100 and the size of the subject are appropriate, and is not an image for recording. For this reason, it is not necessary to make a high-definition image compared with the image for recording.

  In general, the number of pixels of the display device included in the imaging device is often one tenth of the number of pixels of the imaging element included in the imaging device. For this reason, when the monitoring operation is performed, the number of pixels read from the image sensor can be reduced as compared with the image recording operation. On the other hand, since the monitoring image is an image for confirming the orientation of the mobile phone device 100 and the size of the subject, the angle of view of the monitoring image is preferably the same as that in the recording operation. Therefore, in the following, a monitoring image reading method in which the angle of view is the same as that in the recording operation and the number of pixels to be read is reduced when the monitoring operation is performed will be described.

  For example, when any of the first reading method to the fifth reading method is set and the monitoring operation is performed, the same as the sixth reading method to the tenth reading method described above. In addition, the number of pixels to be read is reduced by performing pixel thinning processing. In this case, a thinning rate larger than the thinning rate (1/2) shown in the sixth to tenth reading methods described above can be obtained.

  Further, for example, when any of the sixth reading method to the tenth reading method is set, the number of pixels to be read out by performing the pixel thinning process similarly when the monitoring operation is performed. To reduce. In this case, a thinning rate (for example, a value obtained by multiplying 1/2 by 1 / M (: M> 1 (M is an integer)) can be set to be larger than the thinning rate (1/2) described above. . Note that the number of pixels to be read can be reduced in the same manner when any of the eleventh reading method to the eighteenth reading method is set and the monitoring operation is performed. Note that the number of pixels to be read may be reduced by performing pixel addition processing instead of pixel thinning processing. The display of the monitoring image is the same as the display examples of the first reading method to the fifth reading method except that the number of read pixels is different, and thus the description thereof is omitted here.

[Modified example of pixel decimation and pixel addition]
The example in which the number of pixels of the image data is reduced by performing pixel thinning and pixel addition in the imaging elements 134 to 136 has been described above. In this example, the DSP 200 performs pixel thinning and pixel addition.

  The pixel addition processing unit 221 illustrated in FIG. 8 performs pixel addition processing and pixel thinning processing, and is disposed in the DSP 200 at a position after the image buffers 211 to 219 and a position before the demosaic processing unit 222. Has been. Note that the configuration of the pixel addition processing unit 221 can be substantially the same as that of the adders 354 to 357 and 366 of the imaging elements 134 to 136.

  For example, the pixel addition processing unit 221 includes a data memory that stores a predetermined amount of image data included in at least two horizontal lines of image data in order to perform pixel addition in the vertical direction. Also, the pixel addition processing unit 221 reads out image data located in the same column on the image data from the image memory 170, and adds the read image data with an adder (that is, vertical pixel addition processing). I do.

  Further, for example, when performing horizontal pixel addition processing, the pixel addition processing unit 221 reads out image data of a plurality of columns, and adds the read image data with an adder (that is, horizontal pixel). Addition process).

  Also, for example, when both vertical pixel addition processing and horizontal pixel addition processing are performed, vertical pixel addition processing is performed in a plurality of columns, and horizontal pixel addition processing is performed on the image data obtained by this addition processing. Do more.

  After these addition processing, the pixel addition processing unit 221 outputs the image data after the addition processing to the demosaic processing unit 222 at the subsequent stage. Alternatively, the pixel addition processing unit 221 writes the image data after the addition processing in the image memory 170.

  Further, for example, the pixel addition processing unit 221 includes a selection signal line as a data line for inputting one image data of two image data input in the addition process to the adder, and the data line and the selection signal The AND result with the line is input to the adder. The pixel addition processing unit 221 holds another piece of image data. For example, when the selection signal line is selected, the value of the data line is input to the adder, and pixel addition processing is performed. On the other hand, when the selection signal line is not selected, the value input from the data line to the adder is 0, and the pixel addition process is not performed. In this case, the pixel thinning process is performed, and the image data input from the data line on the side not provided with the selection signal line of the adder is output.

[Example of the flow of image data for pixel decimation and pixel addition]
FIG. 43 is a diagram schematically illustrating the flow of image data when pixel thinning and pixel addition are performed by the image sensor 134 according to the first embodiment of the present invention. A subscript arranged in the vicinity of the thick arrow shown in FIG. 43 represents the number of pixels of the image data. In the example shown in FIG. 43, an example in which the number of pixel data is thinned by 1/2 in the horizontal direction and 1/2 is thinned in the vertical direction by using the image sensor 134 having 1920 × 1440 pixels. Show. Note that the case where the image sensor 134 is used to add ½ pixel in the horizontal direction and ½ pixel in the vertical direction can be similarly expressed. As shown in FIG. 43, the number of pixels of image data output from the image sensor 134 (horizontal 960 pixels × vertical 720 pixels) is smaller than the number of pixels of the image sensor 134 (horizontal 1920 pixels × vertical 1440 pixels). It has become.

  FIG. 44 is a diagram schematically showing the flow of image data when the number of pixels is changed by the pixel addition processing unit 221 according to the first embodiment of the present invention. A subscript arranged in the vicinity of the thick arrow shown in FIG. 44 represents the number of pixels of the image data. In the example shown in FIG. 44, the pixel addition processing unit 221 sets the number of pixel data in the horizontal direction to ½ for the image data output from the image sensor 134 having the number of pixels of 1920 × 1440 pixels in the vertical direction. An example in which the number of pixel data in is ½ is shown. As shown in FIG. 44, the number of pixels (960 pixels × 720 pixels) of the image data output from the pixel addition processing unit 221 is the number of pixels of the image sensor 134 and the image input to the pixel addition processing unit 221. It is smaller than the number of pixels of data.

  FIG. 45 is a diagram schematically showing the flow of image data when the number of pixels is changed when image data is read from the image memory 170 according to the first embodiment of the present invention. Subscripts arranged in the vicinity of the thick arrow shown in FIG. 45 represent the number of pixels of the image data. In the example shown in FIG. 45, when image data is read from the image memory 170, an example is shown in which the number of pixels of the image data is reduced by reading the image data at intervals of a certain number of pixels on the image data. . That is, the image data output from the image sensor 134 having the number of pixels of 1920 × 1440 pixels is held in the image memory 170. When the demosaic processing unit 222 reads the image data held in the image memory 170, the image data is read by setting the number of pixel data in the horizontal direction to ½ and the number of pixel data in the vertical direction to ½. As shown in FIG. 45, the number of pixels of image data read from the image memory 170 and input to the demosaic processing unit 222 is the number of pixels of the image sensor 134 and the image written to the image memory 170 from the image buffers 211 to 213. It is smaller than the number of pixels of data.

[Modified example of changing the image data area]
In this example, an area in which pixel data is read using the DSP 200 is changed to reduce the number of pixels of the image data.

  FIG. 46 is a diagram schematically showing the flow of image data when the reading area is changed by the image sensor 134 according to the first embodiment of the present invention. A subscript arranged in the vicinity of the thick arrow shown in FIG. 46 represents the number of pixels of the image data. The example illustrated in FIG. 46 illustrates an example in which image data having a pixel number of 480 pixels by 640 pixels is read using the imaging device 134 having a pixel number of 1920 pixels × 1440 pixels. As shown in FIG. 46, the number of pixels of the image sensor 134 is 1920 pixels × 1440 pixels, whereas the number of pixels of image data output from the image sensor 134 is 480 pixels × 640 pixels. ing.

  FIG. 47 is a diagram schematically showing the flow of image data when the read area is changed when image data is read from the image memory 170 according to the first embodiment of the present invention. A subscript arranged in the vicinity of the thick arrow shown in FIG. 47 represents the number of pixels of the image data. In the example shown in FIG. 47, when image data is read from the image memory 170, the image data in a part of the area is read out on the image data, thereby changing the area of the image data and reducing the number of pixels of the image data. An example is shown. That is, the image data output from the image sensor 134 having the number of pixels of 1920 × 1440 pixels is held in the image memory 170. When the demosaic processing unit 222 reads out the image data held in the image memory 170, the image data corresponding to the area of 480 pixels wide × 640 pixels high is read out. As shown in FIG. 47, the number of pixels of the image sensor 134 and the number of pixels of image data written from the image buffers 211 to 213 to the image memory 170 are 1920 pixels × 1440 pixels. On the other hand, the number of pixels of the image data read from the image memory 170 and input to the demosaic processing unit 222 is horizontal 480 pixels × vertical 640 pixels.

[Example of stopping two imaging systems during monocular imaging]
The example in which the captured image is generated using at least one of the three imaging systems has been described above. Here, for example, in order to reduce power consumption, it is preferable to stop the operation of an imaging system that does not generate a captured image. Therefore, in the following, an example of stopping the operation of an imaging system that does not generate a captured image when generating a captured image will be described.

[Configuration example of mobile phone device]
FIG. 48 is a block diagram illustrating a functional configuration example of the mobile phone device 100 according to the first embodiment of the present invention. In the configuration example shown in FIG. 48, the configuration other than the configuration related to power supply to the first imaging system 191 to the third imaging system 193 is omitted from the configuration shown in FIG. 3, and the imaging control unit 201 is added. This is an example. In this example, the operation of the imaging system is stopped by shutting off the power supply to the imaging system (control method 1 shown in FIG. 50). The power control unit 207, the power supply unit 208, and the power supply unit 209 are the same as those shown in FIG. For this reason, the following description will focus on the part related to stopping the operation of the imaging system, and the description of the part common to FIG. 3 will be omitted.

  For example, the imaging control unit 201 selects the second imaging system 192 for the power supply unit 209 when an imaging operation (monocular imaging operation) using only image data generated by the imaging element 134 is selected. And instructing the power supply to the third imaging system 193 to be cut off. Thereby, when the monocular imaging operation is performed, the operations of the second imaging system 192 and the third imaging system 193 that are not used for the imaging operation can be stopped, and power consumption can be reduced. Can do. Note that when the power supply to the second imaging system 192 and the third imaging system 193 is cut off, a clock signal, a vertical synchronization signal from the DSP 200 to the second imaging system 192 and the third imaging system 193, It is preferable to stop the output of the horizontal synchronization signal or the like. Thereby, power consumption can be further reduced.

  FIG. 49 is a block diagram illustrating a functional configuration example of the mobile phone device 100 according to the first embodiment of the present invention. Note that each configuration example illustrated in FIG. 49 shows only a configuration related to the stop processing of the imaging operation in a simplified manner. That is, only AND operation circuits 801 to 807 and each signal line are shown. Here, two input signal lines are connected to the AND circuits 801 to 807, and when an “H” signal is input from each of these input signal lines, an “H” signal is output to the output signal line. This is a circuit that performs an AND operation. In this example, when the operation of the imaging system is stopped, the power supply to the imaging system to be stopped is not shut off, and the clock supply to the imaging system to be stopped is shut off.

  FIG. 49A shows a circuit configuration example in the DSP 200 for stopping the clock supply to the imaging system. Here, the signal line 809 is a signal line for a clock signal to the image sensor 134, the signal line 810 is a signal line for a clock signal to the image sensor 135, and the signal line 811 is a clock line to the image sensor 136. It is a signal line of a signal. The signal line 812 is a signal line for an on / off control signal to the image sensor 134, the signal line 813 is a signal line for an on / off control signal to the image sensor 135, and the signal line 814 is an image sensor. This is a signal line for an on / off control signal to the element 136. The signal line 815 is a signal line for a clock signal output to the image sensor 134, the signal line 816 is a signal line for a clock signal output to the image sensor 135, and the signal line 817 is an image sensor 136. This is a signal line of a clock signal output to. For example, when the monocular imaging operation is selected, the imaging control unit 201 sets the signals of the signal lines 813 and 814 out of the signal lines 812 to 814 to the “L” signal. Thereby, the clock supply to the image sensors 135 and 136 is stopped (control method 2 shown in FIG. 50).

  FIG. 49B shows a circuit configuration example in the case of stopping the operation of the circuit for generating the clock in the imaging elements 134 to 136 without interrupting the power supply and clock supply to the imaging system. In this example, the multiplier 818 of the multiplier / dividers 391 and 392 shown in FIG. 12 is shown as an example. Note that the multiplier 818 shown in this example is a multiplier of the imaging elements 135 and 136. Here, the signal line 819 is a signal line for a clock signal from the DSP 200, and the signal line 820 is a signal line for an imaging on / off signal from the DSP 200. The signal line 821 is a signal line for the clock signal after multiplication. For example, when the monocular imaging operation is selected, the DSP 200 sets the signal on the signal line 820 as the “L” signal. As a result, the multipliers in the image sensors 135 and 136 are stopped (control method 3 shown in FIG. 50).

  In FIG. 49C, the power supply and clock supply to the image pickup system and the clock multiplier in the image pickup device are not cut off, and the vertical scanning circuit and the horizontal scanning circuit in the image pickup devices 134 to 136 are disconnected. An example of a circuit configuration in the case of stopping the operation is shown. Here, the signal line 823 is a signal line for a vertical control signal in the first line inside the vertical scanning circuit 822, and the signal line 824 is a signal line for a vertical control signal in the second line inside the vertical scanning circuit 822. . Signal lines 825 and 831 are signal lines for imaging on / off signals from the DSP 200, and signal lines 828 and 830 are signal lines for clock signals from the multiplier / frequency divider in the image sensor. A signal line 826 is a signal line for a vertical control signal in the first line output to the outside of the vertical scanning circuit 822, and a signal line 827 is a vertical line in the second line output to the outside of the vertical scanning circuit 822. It is a signal line of a control signal. A signal line 832 is a signal line for a clock signal into the horizontal scanning circuit 829. The signal line 833 is a signal line for the horizontal control signal in the first line output to the outside of the horizontal scanning circuit 829, and the signal line 834 is the horizontal line in the second line output to the outside of the horizontal scanning circuit 829. It is a signal line of a control signal. In this example, only the signal lines of the first line and the second line are shown, and the signal lines of the other lines are omitted. For example, when the monocular imaging operation is selected, the DSP 200 sets the signals on the signal lines 825 and 831 to the “L” signal. Thereby, the output of the vertical scanning signal and the horizontal scanning signal is stopped in the vertical scanning circuit and the horizontal scanning circuit in the imaging devices 135 and 136 (control method 4 shown in FIG. 50). Here, in the example shown in FIG. 49C, since there are many circuits operating inside the second imaging system 192 and the third imaging system 193 as compared with the above-described example, In comparison, the effect of reducing power consumption is small. However, since the clock is supplied to the image sensor and the multiplier inside the image sensor is also operating, when the switching operation from the monocular imaging operation to the compound eye imaging operation is performed, the second imaging system 192 and the second imaging system 192 The operation of the third imaging system 193 can be resumed quickly.

  Further, the operation of the imaging system may be stopped by fixing the vertical synchronization signal and the horizontal synchronization signal supplied from the DSP 200 to the imaging system (control method 5 shown in FIG. 50). In this case, since each synchronization signal is not input, the image pickup device cannot read image data.

  FIG. 50 is a diagram illustrating a relationship between the control method for stopping the operation of the imaging system and the signal lines in the first embodiment of the present invention. The list shown in FIG. 50 shows the relationship between each control method described above and each signal line. By performing each control method shown in FIG. 50, the operation of the imaging system can be stopped. Note that it is preferable that the signal lines in the shaded columns in the columns corresponding to the control method 1 are not output. Further, it is preferable that the signal lines in the hatched columns among the columns corresponding to the control methods 2 to 5 are fixed to “L” or “H”. Further, signal lines (signal lines from the DSP 200 to the imaging system) that are not listed in the list shown in FIG. 50 are set to any state of “H” output, “L” output, insufficient output, and no output. It may be. However, in order to reduce power consumption, it is preferable to leave no output.

[Example of capturing image data from imaging system to DSP]
FIG. 51 is a timing chart schematically showing the output timing from the image sensor and the state of writing to the image buffer in the first embodiment of the present invention. The horizontal axis shown in FIG. 51 indicates the time axis. A waveform 700 indicates a horizontal synchronization signal from the DSP 200 to each imaging system. In addition, horizontal axes 701 to 703 indicate time transitions of image data output from each imaging system to the DSP 200. In addition, horizontal axes 704 to 706 indicate time transitions in the writing state of the image buffers 211 to 219. In the example shown in FIG. 51, the writing state of a set of three image buffers corresponding to each imaging system is shown on the same line.

  For example, the image buffers 1A, 1B, and 1C facing the first imaging system 191 will be described as an example. For example, when the writing of the image data from the image sensor 134 to the first image buffer 1A is completed, the image buffer 1A enters a standby state waiting for reading into the DSP 200. During the standby state of the image buffer 1A, image data from the image sensor 134 is written into the second image buffer 1B. When the writing of the image data from the image sensor 134 to the second image buffer 1B is completed, the image buffer 1B enters a standby state waiting for reading into the DSP 200. When the writing of the image data to the second image buffer 1B is completed, the image data is read from the first image buffer 1A into the DSP 200. Then, before the writing of the image buffer 1C is completed, the reading of the image data from the image buffer 1A is completed, and the image buffer 1A can be written with the image data. Subsequently, a series of these operations are repeated. The same applies to the image buffers 2A to 2C and the image buffers 3A to 3C.

  Here, when data written in the three image buffers is read out using one data bus 204, the time in the image buffer is less than 1/3 of the time that the image sensor has written to the image buffer. It is necessary to read the image data.

  Hereinafter, the relationship between the clock frequency at which the image sensor reads out image data of each pixel and writes the image data into the image buffers 211 to 219 and the clock frequency at which the DSP 200 reads out the image data of the image buffers 211 to 219 will be described. Also, a clock frequency for the DSP 200 to read the image data of the image buffers 211 to 219 during the compound eye imaging operation and a clock frequency for the DSP 200 to read the image data of the image buffers 211 to 219 during the monocular imaging recording. The relationship will be described.

  52 to 54 show the relationship between the clock frequency for reading each pixel and writing to the image buffer in the image sensor according to the first embodiment of the present invention, and the clock frequency for reading image data from the image buffer. FIG. Here, the horizontal axis relationship shown in FIG. 52 to FIG. 54 represents the passage of time in one line, and the vertical axis relationship represents the passage of time for each line in one frame. Note that the example shown in FIG. 52 is an example corresponding to the first readout method shown in FIG. 28 (three-lens landscape image mode). Further, the example shown in FIG. 53 is an example (monocular landscape image capturing mode) corresponding to the third readout method shown in FIG. The example shown in FIG. 54 is an example corresponding to the fifth reading method shown in FIG. 36 (monocular vertically long small region image capturing mode).

  FIG. 52A shows the relationship between the clock frequency in the image sensor and the output timing of image data from the image sensor when the three-lens landscape wide-angle image capturing mode is set. A waveform 710 indicates a clock used when reading out all pixels from the entire area of the imaging elements 134 to 136 when the three-lens landscape wide-angle image capturing mode is set. In addition, horizontal axes 713 to 718 schematically show, with rectangles, time transitions of image data read from the image sensors 134 to 136 and output to the image buffers 211 to 219. Note that line 719 indicates image data output to the image buffer 211, line 720 indicates image data output to the image buffer 212, and line 721 indicates image data output to the image buffer 213. Indicates. In this example, different symbols (D11 to D1C, etc.) are attached to the rectangle only on the horizontal axis 715, and the symbols in the rectangle are omitted on the other horizontal axes. Further, a large broken circle shown on the horizontal axis 713 indicates the timing of the vertical synchronization signal, and a small dotted circle shown on the horizontal axes 713 to 718 indicates the timing of the horizontal synchronization signal. Similarly, the horizontal axis shown in FIGS. 52 to 54 below also indicates the timing of the vertical synchronization signal and the timing of the horizontal synchronization signal by attaching a large dotted circle and a small dotted circle.

  FIG. 52B shows the relationship between the clock frequency in the DSP 200 and the timing for reading image data from the image buffer when the three-lens landscape image mode is set. A waveform 722 indicates a clock used when the DSP 200 reads out image data from the image buffers 211 to 219 in the case where the three-lens landscape wide-angle image capturing mode is set. In addition, the horizontal axes 725 to 732 schematically show the time transition of the image data read from the image buffers 211 to 219 in a rectangle. In this example, different symbols (D11 to D1C, etc.) are associated with rectangles only on the horizontal axis 729 by arrows, and symbols on the rectangles are omitted on the other horizontal axes. Each of these codes corresponds to the code shown in FIG. A section 733 is a reading section of the image sensor 134, a section 734 is a reading section of the image sensor 135, and a section 735 is a reading section of the image sensor 136.

  Here, when three pieces of image data are input from the three image sensors 134 to 136 to the DSP 200, the input image data is written to the image buffers 211 to 219. The image data written in the image buffers 211 to 219 is read through one data bus 204 in the DSP 200 and written to the image memory 170 via the I / F 206 with the image memory. For example, a case is assumed where the three-lens landscape image mode is set. In this case, as shown in FIGS. 52A and 52B, each of the image buffers 211 to 136 takes less than 1/3 of the time when the image sensors 134 to 136 write the image data into the image buffers 211 to 219. It is necessary to read out 219 image data.

  FIG. 53A shows the relationship between the clock frequency in the image sensor and the output timing of the image data from the image sensor when the monocular landscape image capturing mode is set. FIG. 53B shows the relationship between the clock frequency in the DSP 200 and the timing for reading image data from the image buffer when the monocular landscape image capturing mode is set. A waveform 736 indicates a clock used when the DSP 200 reads image data from the image buffers 211 to 219 when the monocular landscape image capturing mode is set. A section 737 is a reading section of the image sensor 134. Also, parts common to the example shown in FIG.

  Here, comparing FIG. 52 and FIG. 53, when the three-lens landscape image capturing mode is set, the amount of image data is three times that when the monocular landscape image capturing mode is set. I can understand that there is. For this reason, when the three-lens horizontal long-angle image capturing mode is set, a clock frequency three times that of the three-lens horizontal long imaging operation is required as a clock frequency for reading the image data of the image buffers 211 to 219. . Similarly, when the image signal processing in the DSP 200 is performed when the three-lens landscape image mode is set, the image size is three times that when the monocular landscape image capture mode is set. A clock frequency is required.

  FIG. 54A shows the relationship between the clock frequency in the image sensor and the output timing of the image data from the image sensor when the monocular vertically small region image capturing mode is set. A waveform 738 indicates a clock used when reading out all pixels only from a specific area in the imaging elements 134 to 136 in the case where the monocular vertically long small area imaging mode is set. FIG. 54B shows the relationship between the clock frequency in the DSP 200 and the timing for reading image data from the image buffer when the monocular vertically long small region imaging mode is set. A waveform 739 indicates a clock used when the DSP 200 reads out image data from the image buffers 211 to 219 in the case where the monocular vertically long small region image capturing mode is set. A section 740 is a reading section of the image sensor 134. Parts common to the example shown in FIG. 52 are denoted by the same reference numerals.

  Here, comparing FIG. 53 and FIG. 54, it is understood that when the monocular vertically long small area image capturing mode is set, the data amount of the image data is smaller than when all the pixels on the light receiving surface are read. it can. For this reason, when the monocular vertically long small-area image capturing mode is set, a clock frequency for reading image data in the image buffer 211 is required to be smaller than that for reading all pixels. Similarly, when image signal processing inside the DSP 200 is performed, the clock frequency can be made smaller than when all pixels are read out.

[Example of operating frequency required for DSP data bus]
FIG. 55 is a diagram schematically showing a flow of image data generated by the image sensor 134 according to the first embodiment of the present invention. The configuration shown in FIG. 55 is the same as the configuration example shown in FIG. 43, except that the data format of the image data transmitted by each unit via the data bus 204 is added. In this example, it is assumed that the pixel addition processing unit 221 in the DSP 200 is not used, the image output is only the output to the display unit 140, and the output to the external display device is not performed.

  Image data written from the image sensor 134 to the image buffers 211 to 213 is Bayer data 750. Further, the image data input to the DSP 200 is up to the preceding stage of the demosaic processing unit 222 that interpolates Bayer data into RGB (R (Red: Red), G (Green: Green), B (Blue: Blue)) data, It is transmitted in the Bayer data format. That is, for writing image data from the image buffers 211 to 213 to the image memory 170 and reading image data to the demosaic processing unit 222, the image data is transmitted in the Bayer data format.

  Here, the image data demosaiced by the demosaic processing unit 222 is directly transferred between the processing units until the resolution conversion processing is performed. When performing resolution conversion with a large amount of data handled during signal processing, the image data is written to the image memory 170 via the data bus 204 so that the desired image data can be easily input at a desired timing. Then, the resolution conversion units 231 and 251 read image data necessary for resolution conversion processing from the image memory 170.

  Thereby, the image data in the RGB data format is written into the image memory 170 via the data bus 204 before the resolution conversion of the recording image. Also, image data in the RGB data format is read from the image memory 170 via the data bus 204 at the time of resolution conversion for image recording.

  When displaying an image on the display unit 140, the display unit 140 generally requests image data in the YCbCr format. Therefore, image data in the YCbCr data format is written into the image memory 170 via the data bus 204 before resolution conversion for the display device. Also, image data in the YCbCr data format is read from the image memory 170 via the data bus 204 at the time of resolution conversion for image display.

  Here, the data amount of one piece of image data transmitted via the data bus 204 will be described. For example, assuming that one piece of image data is input from the image sensor 134 to the DSP 200, the input image data is recorded as one image file on the recording medium 180 and displayed on the display unit 140. I will explain. In this case, two pieces of image data of Bayer data, two pieces of image data of YCbCr data, and two pieces of image data of RGB data are transmitted via the data bus 204. That is, two pieces of image data between the image buffers 211 to 213 and the demosaic processing unit 222 are transmitted in the Bayer data format (Bayer 751 and 752). Also, two pieces of image data between the color adjustment processing unit 226 and the resolution conversion unit 231 are transmitted in the YCbCr data format (YCbCr 753 and 756). Also, two pieces of image data between the RGB conversion processing unit 227 and the resolution conversion unit 251 are transmitted in the RGB data format (RGB 754 and 755).

  Here, the data amount of one piece of image data in the YCbCr data format is approximately twice that of image data in the Bayer format having the same image size. Further, the data amount of one piece of image data in the RGB data format is approximately three times that of the image data in the Bayer format having the same image size. For example, when image data for one sheet is recorded as one image file on the recording medium 180 and displayed on the display unit 140, approximately 12 image data in terms of Bayer data format are converted via the data bus 204. Need to be transmitted. The processing relating to the transmission of the image data is shown in FIG.

  FIG. 56 is a diagram schematically showing the relationship between the processing for occupying the data bus 204 and time in the first embodiment of this invention. FIG. 56A shows a data bus when one image data input from the image sensor 134 to the DSP 200 is recorded as one image file on the recording medium 180 and displayed on the display unit 140. The relationship between the process occupying 204 and time is shown. FIG. 56 shows the relationship when the horizontal axis is a time axis and one frame is processed. In the example shown in FIG. 55, the corresponding processes are denoted by the same reference numerals as those shown in FIG.

  FIG. 56B shows a process of occupying the data bus 204 when displaying one piece of image data input from the image sensor 134 to the DSP 200 on the display unit 140 without recording it on the recording medium 180. Shows the relationship with time. This example is an example where a so-called monitoring image display operation is performed. Here, when the monitoring image display operation is performed, the image is displayed on the display unit 140, but the image data for recording is not generated. That is, the process 754 for writing RGB data to the image memory 170 and the process 755 for reading the RGB data from the image memory 170 at the resolution conversion of the recorded image are not required before the resolution conversion of the recorded image. Therefore, in the display operation of the monitoring image, when displaying one piece of image data input from the image sensor 134 to the DSP 200 on the display unit 140, approximately six pieces of image data in terms of Bayer data format are converted into data. It only needs to be transmitted via the bus 204.

  As described above, the data amount of image data transmitted via the data bus 204 can be calculated. Hereinafter, the operating frequency required for the data bus 204 will be described.

Here, variables used for calculating the operating frequency required for the data bus 204 are shown.
H: The number of pixels in the horizontal direction of the area where pixel data is read in the image sensor.
V: The number of pixels in the vertical direction of the area where pixel data is read in the image sensor.
R: A thinning rate when performing pixel thinning readout when reading pixel data in the image sensor.
B: Bit width of each pixel data of the image.
F: The number of images that can be subjected to image signal processing per second by the DSP 200 on the image input from the image sensor.
Kb: The number of images that need to be transmitted on the data bus 204 of the DSP 200 in the Bayer data format before one image input from the image sensor is recorded on the recording medium 180 and displayed on the display unit 140.
Ky: The number of images that need to be transmitted in the YCbCr data format on the data bus 204 of the DSP 200 before one image input from the image sensor is recorded on the recording medium 180 and displayed on the display unit 140.
Kr: The number of images that need to be transmitted in the RGB data format on the data bus 204 of the DSP 200 before one image input from the image sensor is recorded on the recording medium 180 and displayed on the display unit 140.
K: A value obtained by converting the number of images to be transmitted on the data bus 204 of the DSP 200 before Bayer image data conversion until one image input from the image sensor is recorded on the recording medium 180 and displayed on the display unit 140.
Note that K is obtained by the following equation.
K = Kb × 1 + Ky × 2 + Kr × 3
D: Bit width of the data bus 204 of the DSP 200

Using these variables, the amount of data (unit: bit) DD1 input to the DSP 200 per second is obtained by Equation 13.
DD1 = H × V × R × B × F Equation 13

Further, the amount of data (unit: bit) DD2 to be transmitted on the data bus 204 per second is obtained by Expression 14.
DD2 = H × V × R × B × F × K (14)

Further, the clock frequency (unit: Hz) Fclk of the data bus 204 necessary for transmitting the image data is obtained by Expression 15.
Fclk = H × V × R × B × F × K / (D-DmodB) Equation 15

Here, the lower limit value Fclk min of the preferable range of the clock frequency Fclk can be obtained by Expression 16.
Fclk min = (H × V × R × B × F × K × 1.0) / (D-DmodB) Equation 16

  Here, there is no gap between the data on the data bus 204, and the data is not constantly transmitted. For this reason, when determining the operating frequency of the data bus 204 based on the amount of data that needs to be transmitted using the data bus 204, allow a transfer capacity of the data bus 204 of several tens of percent. Is well known to those skilled in the art.

  For example, when a margin of several percent is provided, there is a possibility that the amount of the margin provided is too small. On the other hand, when a margin of one hundred and ten percent is given, the margin is too much. These are also well known among those skilled in the art. For example, the data transmission interval on the data bus 204 becomes vacant and the transmission amount of the data bus 204 decreases by several tens of percent. For this reason, it is sufficient if the transmission capacity of the data bus 204 has a maximum margin of 100%. For this reason, the upper limit value of the preferable range of the clock frequency Fclk is as follows.

Here, the upper limit value Fclk max of the preferable range of the clock frequency Fclk can be obtained by Expression 17.
Fclk max = (H × V × R × B × F × K × 2.0) / (D-DmodB) Equation 17

Thus, since the lower limit value Fclk min and the upper limit value Fclk max of the preferable range of the clock frequency Fclk can be obtained, the preferable range of the clock frequency Fclk of the data bus 204 can be defined by Equation 18.
(H × V × R × B × F × K × 1.0) / (D-DmodB) ≦ Fclk ≦ (H × V × R × B × F × K × 2.0) / (D-DmodB) Equation 18

  Here, “D-DmodB” shown in Expressions 16 to 18 will be described.

  When the bit width of the data bus 204 is not an integral multiple of the bit width of each pixel of the image sensor, the writing operation to the image memory 170 is wasted. For example, if the bit width of the data bus 204 is 128 bits and the bit width of each pixel of the image sensor is 12 bits, 12-bit pixel data cannot be transferred for 11 pixels, and 10 pixels Can only transfer. For this reason, a waste of 8 bits occurs as a write operation. The amount indicating this waste is DmodB. That is, a value (effective data bus width) obtained by subtracting this waste amount from the bit width of the data bus 204 is “D-DmodB”.

  As shown in Equations 16 to 18, the preferable operating frequency range of the data bus 204 can be calculated using H, V, R, B, F, K, and D. However, the values of H, V, R, B, F, K, and D change according to the content of the imaging operation performed by the mobile phone device 100. For this reason, it is preferable to change the operating frequency of the data bus 204 according to each value of H, V, R, B, F, K, and D set in each imaging operation. In this manner, by changing the operating frequency of the data bus 204, the DSP 200 can reduce the operating frequency of the data bus 204 to a necessary and sufficient value, thereby reducing power consumption.

  FIG. 57 is a diagram showing parameters for determining the operating frequency of the data bus 204 for each imaging operation of the mobile phone device 100 according to the first embodiment of the present invention. By setting the parameters (H1sr, V1sr, etc.) shown in FIG. 57, the operating frequency of the data bus 204 can be determined for each imaging operation of the mobile phone device 100. An example of each of these parameters is shown in FIG.

  FIG. 58 is a diagram illustrating an example of parameters for determining the operating frequency of the data bus 204 for each imaging operation of the mobile phone device 100 according to the first embodiment of the present invention. In this example, a case where a still image recording operation is performed is shown as an example. Further, based on each parameter, the lower limit value Fclk_min and the upper limit value Fclk_max of the operating frequency of the data bus 204 can be calculated in advance. The example shown in FIG. 58 shows a case where the number of images for which still image recording processing is performed per second and the number of images for which monitoring operation is performed per second is 30.

[Example of time-sharing processing for still image recording]
Next, a time division process related to the still image recording operation will be described. For example, the time-division processing does not end within the period of one frame (that is, within one period of the vertical synchronization signal supplied to the image signal processing unit 220) in the imaging and recording process of a still image, but within a period of several frames It shall mean that each process is performed across.

  For example, even when a user takes a relatively large number of still images using a mobile phone device with a camera that has become widespread in recent years, the number is often about one in a few seconds. For this reason, when a still image recording operation is performed, for example, even if image signal processing and recording processing of a single still image is performed for about one second, there is a possibility that the convenience of the user may be impaired. It is assumed that there are few. In addition, by performing image signal processing and recording processing of one still image in this way, the operating frequency of the data bus 204 of the DSP 200 can be reduced, and power consumption can be reduced.

  In addition, after recording a single still image, until the image signal processing and recording processing are completed, a message such as “Please wait” is displayed on the display unit 140, and the still image capturing process is performed. Can be notified that it is running. In addition to the message, an icon or a sign indicating that the still image capturing process is being executed may be displayed, or a monitoring operation may be performed in parallel to display a monitoring image.

  FIG. 59 and FIG. 60 are diagrams schematically showing the time division processing performed by the image signal processing unit 220 in the first embodiment of the present invention. The examples shown in FIGS. 59 and 60 show time-division processing when a still image recording instruction operation is performed while the monitoring operation is being performed. The horizontal axis shown in FIG. 59 represents the time axis. A waveform 760 represents a vertical synchronization signal. Also, the horizontal axis 795 shows the relationship between processing that occupies the data bus 204 and time. A white rectangle shown on the horizontal axis 795 represents a process of occupying the data bus 204 necessary for the monitoring process of each frame. In addition, a rectangle with an oblique line inside the abscissa 795 represents processing (time division processing) that occupies the data bus 204 necessary for still image recording processing.

  As described above, when the monitoring operation is performed, the amount of data that needs to be transmitted via the data bus 204 is smaller than when the still image capturing operation is performed. Here, for example, a case is assumed where a still image recording instruction operation is performed at any time in the section 796. In this case, a still image recording operation is performed for frame 3 indicated by arrow 797. For example, when a recording instruction operation is received from the user, the DSP 200 straddles a frame (for example, frame 3) corresponding to the timing of the recording instruction operation and a plurality of subsequent frames (for example, frames 4 to 8). Perform image signal processing and recording processing. While the image signal processing and the recording processing are being performed, the image sensor and the DSP 200 perform the monitoring operation in parallel. When the recording operation ends (for example, the time indicated by the arrow 798), the image sensor and the DSP 200 perform only the monitoring operation.

  When performing time-division processing of still image recording processing, a monitoring operation is not performed during the time-division processing, and a single color or a prepared image is displayed on the display unit 140. You may do it. Further, these images and a message for informing the user that the image recording process is being performed may be displayed.

  FIG. 60 shows the amount of data necessary to be transmitted via the data bus 204 of the DSP 200 in order to perform the time division processing of the monitoring operation and the recording processing within the period of the vertical synchronization signal in the time division processing described above. Indicates. The example shown in FIG. 60 is obtained by modifying a part of the example shown in FIG. 56, and is different in that a recording operation time division process 759 is added in FIG. In addition, the same code | symbol is attached | subjected to the part which is common in the example shown in FIG.

  For example, the recording process shown in FIG. Then, as shown in FIG. 60B, in the monitoring operation, each signal processing (for example, recording operation time division processing 759) is performed by an amount accompanying data transmission for two Bayer images for one frame. Then, after the frame in which the recording process is started, signal processing for the recording process is performed while performing the monitoring operation after the start of the recording operation. However, in the first process, in order to complete the writing of the image data to the image memory 170, the first process is always equal to or more than one Bayer image. Note that the rectangles 751 to 753 and 756 shown in FIG. 60B correspond to the white rectangles on the horizontal axis 795 shown in FIG. 59, and the rectangle 759 shown in FIG. 60B is the horizontal axis shown in FIG. 795 corresponds to a rectangle with a diagonal line inside.

  FIG. 61 is a diagram showing an example of parameters for determining the operating frequency of the data bus 204 for the still image recording operation of the mobile phone device 100 according to the first embodiment of the present invention. The example shown in FIG. 61 is the same as FIG. 58 except that the number of images to be subjected to the still image recording process per second is one and the value related thereto is changed. Thus, by setting the number of images to be subjected to still image recording processing per second to one, the lower limit value Fclk_min and the upper limit value Fclk_max of the operating frequency of the data bus 204 can be reduced. Further, the power consumption of the DSP 200 can be reduced.

  Here, as described above, since the image generated in the monitoring operation is not an ornamental image, it is preferable to thin out pixel data to reduce power consumption. Therefore, also in the example shown in FIGS. 58 and 61, a value smaller than 1 is used as the thinning rate of the pixel data in the monitoring operation. For this reason, the operating frequency required for the data bus 204 of the DSP 200 can be made lower than the operating frequency required for the data bus 204 in the still image recording operation shown in FIGS. Can be reduced.

  FIG. 62 is a diagram showing an example of parameters for determining the operating frequency of the data bus 204 for the moving image recording operation of the mobile phone device 100 according to the first embodiment of the present invention. The example shown in FIG. 62 shows an example of a moving image recording operation as each parameter shown in FIG.

  Here, when the three-lens landscape image capturing mode is set, the image data amount is about three times as compared with the case where the monocular landscape image capture mode is set. For this reason, for example, when a still image recording operation is performed, it is possible to perform signal processing of a large image by taking several minutes of vertical synchronization signals (that is, taking several frames). . Thereby, it is possible to prevent the operating frequency required for the data bus 204 from becoming too large. However, when a moving image recording operation is performed, since a new image is input for each frame, it is necessary to perform signal processing on image data for three eyes within a period of one frame. Therefore, in order to prevent the operating frequency required for the data bus 204 from becoming too large, it is preferable to reduce the amount of data by performing image thinning processing as shown in FIG.

[Example of countermeasures for image data capture delay]
In the first embodiment of the present invention, three pieces of image data input simultaneously from three image sensors to the DSP 200 are temporarily stored in an image buffer having a ring buffer structure. By using the image buffer having the ring buffer structure, three pieces of image data input simultaneously from the three image pickup devices to the DSP 200 can be transmitted through the single data bus 204 and the I / F 205 with the image memory. A single image memory 170 can be written. However, in the case where the image buffers 211 to 219 having the ring buffer structure are used, a delay of image data occurs until the image data is read from the image buffers 211 to 219 and taken into the DSP 200 as described below. Therefore, it is necessary to perform processing for the delay generated in this way.

  FIG. 63 is a timing chart schematically showing the timing of writing to the image buffers 211 to 219 and the timing of fetching into the DSP 200 in the first embodiment of the present invention. The horizontal axis shown in FIG. 63 represents the time axis. A waveform 761 indicates a vertical synchronization signal from the DSP 200 to each imaging system, and a waveform 762 indicates a horizontal synchronization signal from the DSP 200 to each imaging system. Note that BP1 in the waveform 762 indicates a vertical back porch for two lines. In the example shown in FIG. 63, it is assumed that the capacity of one of the image buffers 211 to 219 is the same as the data amount for one line of image data input from the image sensors 134 to 136 to the DSP 200.

  The horizontal axis 763 shows the time transition of image data input from the first imaging system 191 to the DSP 200, and the horizontal axis 764 shows the time transition of image data input from the second imaging system 192 to the DSP 200. Show. A horizontal axis 765 shows time transition of image data input from the third imaging system 193 to the DSP 200.

  The horizontal axis 766 shows the time transition of the writing state of the image buffers 211 to 213, the horizontal axis 767 shows the time transition of the writing state of the image buffers 214 to 216, and the horizontal axis 768 shows the image buffer. The time transition of the write state of 217 to 219 is shown. In the example shown in FIG. 63, the writing state of a set of three image buffers corresponding to each imaging system is shown on the same line.

  A horizontal axis 769 indicates image data to be read from the image buffers 211 to 219 of the DSP 200 and sent to the DSP 200.

  A waveform 770 indicates a vertical synchronization signal having the same timing as the vertical synchronization signal input from the DSP 200 to each imaging system, and a waveform 771 indicates a vertical synchronization signal delayed for supply to the DSP 200. In addition, BP2 in the waveform 770 indicates a vertical back porch for four lines, and BP3 in the waveform 771 indicates a vertical back porch for two lines.

  As shown in FIG. 63, there is a delay of two lines from the start of the writing of valid image data to the image buffers 211 to 219 until this data is read from the image buffer and taken into the DSP 200. Occur.

  FIG. 64 is a timing chart schematically showing the timing of writing to the image buffers 211 to 219 and the timing of fetching into the DSP 200 in the first embodiment of the present invention. The horizontal axis shown in FIG. 64 represents the time axis. A waveform 781 indicates a horizontal synchronization signal from the DSP 200 to each imaging system. Note that BP4 in the waveform 781 indicates a horizontal back porch in an image generated by the imaging elements 134 to 136. In the example illustrated in FIG. 63, it is assumed that one capacity of the image buffers 211 to 219 is the same as one third of the data amount of one line of image data input from the imaging elements 134 to 136 to the DSP 200. .

  The horizontal axis 782 shows the time transition of the image data input from the first imaging system 191 to the DSP 200, and the horizontal axis 783 shows the time transition of the image data input from the second imaging system 192 to the DSP 200. Show. A horizontal axis 784 indicates time transition of image data input from the third imaging system 193 to the DSP 200.

  Also, the horizontal axis 785 shows the time transition of the writing state of the image buffers 211 to 213, the horizontal axis 786 shows the time transition of the writing state of the image buffers 214 to 216, and the horizontal axis 787 shows the image buffer. The time transition of the write state of 217 to 219 is shown. In the example shown in FIG. 64, the writing state of a set of three image buffers corresponding to each imaging system is shown on the same line.

  Further, the horizontal axis 789 shows image data to be read from the image buffers 211 to 219 of the DSP 200 and transmitted to the DSP 200.

  A waveform 790 indicates a horizontal synchronization signal having the same timing as the horizontal synchronization signal input from the DSP 200 to each imaging system, and a waveform 791 indicates a horizontal synchronization signal delayed for supply to the DSP 200. In addition, BP5 in the waveform 790 indicates a horizontal back porch, and BP6 in the waveform 791 indicates a horizontal back porch.

  As shown in FIG. 64, after the writing of the effective data of the image to the image buffers 211 to 219 is started, the effective data area of the image is read from the start of the image buffer until the data is read into the DSP 200. A delay of 2/3 lines occurs.

  By removing these delays, the interval between the vertical synchronization signal and the effective data of the image and the interval between the horizontal synchronization signal and the effective data of the image are the same as the intervals when the image data is generated by the image sensors 134 to 136. It is necessary to make it. For this reason, in the first embodiment of the present invention, at least a part of the image signal processing unit 220 in the DSP 200 has the same period and a delayed phase as the synchronization signal input from the DSP 200 to the imaging elements 134 to 136. Use the correct signal. In addition, the imaging control unit 201 of the DSP 200 generates and supplies a synchronization signal input from the DSP 200 to the imaging elements 134 to 136, and a signal whose phase is the same as that of the synchronization signal input to the imaging elements 134 to 136 and whose phase is delayed. Also generate. The generated synchronization signal is supplied to each part in the DSP 200.

[Operation example of imaging device]
FIG. 65 is a flowchart showing a processing procedure of imaging control processing by the mobile phone device 100 according to the first embodiment of the present invention.

  First, power is turned on in the mobile phone device 100 (step S901). Subsequently, it is determined whether or not an imaging operation start instruction operation has been accepted (step S902). For example, the user can start the imaging operation by performing an instruction operation for instructing the start of the imaging operation on the menu screen of the display unit 140. When the imaging operation start instruction operation is not accepted (step S902), the monitoring is continued until the imaging operation start instruction operation is accepted.

  When the imaging operation start instruction operation is accepted (step S902), the rotation state detection unit 150 detects the rotation state of the second casing 120 (step S903). Note that step S903 is an example of a detection procedure. Subsequently, the imaging control unit 201 determines whether or not the rotation state of the second housing 120 is the horizontally long state (step S904), and the rotation state of the second housing 120 is the horizontally long state. In such a case, the horizontally long state imaging process is performed (step S910). The landscape state imaging process will be described in detail with reference to FIG. On the other hand, when the rotation state of the second casing 120 is the vertically long state, the vertically long state imaging process is performed (step S950). The landscape state imaging process will be described in detail with reference to FIG. Steps S904, S910, and S950 are an example of a control procedure.

  Subsequently, it is determined whether or not an imaging operation end instruction operation has been accepted (step S905). When the imaging operation end instruction operation is not accepted (step S905), the process returns to step S903. On the other hand, when an instruction to end the imaging operation is accepted (step S905), the operation of the imaging control process is ended.

  FIG. 66 is a flowchart showing the horizontally long state imaging process (the process procedure of step S910 shown in FIG. 65) in the process procedure of the imaging control process by the mobile phone device 100 according to the first embodiment of the present invention.

  First, the imaging control unit 201 sets a three-lens landscape image mode as an initial setting (step S911). Subsequently, the imaging control unit 201 determines which one of the still image / moving image imaging modes is set (step S912). When the still image capturing mode is set (step S912), still image capturing processing is performed (step S920). The still image capturing process will be described in detail with reference to FIG. On the other hand, when the moving image capturing mode is set (step S912), a moving image capturing process is performed (step S930). Note that the moving image capturing process will be described in detail with reference to FIG.

  Subsequently, it is determined whether or not an imaging range switching instruction operation has been accepted (step S913). For example, it is determined whether or not the imaging range changeover switch 111 shown in FIG. When the imaging range switching instruction operation has not been received (step S913), the process proceeds to step S915. On the other hand, when the imaging range switching instruction operation is accepted (step S913), the imaging control unit 201 switches the imaging mode according to the accepted imaging range switching instruction operation (step S914). For example, every time the imaging range changeover switch 111 is pressed, (1) three-lens landscape image mode, (2) three-lens image image mode, (3) monocular image mode, (4) monocular image Switching is performed in the order of the vertically long image capturing mode, and (5) the monocular vertically long small region image capturing mode.

  Subsequently, it is determined whether or not the rotation state of the second housing 120 has changed (step S915). If there is no change in the rotation state of the second housing 120, the process returns to step S912. On the other hand, when there is a change in the rotation state of the second casing 120 (step S915), the operation of the horizontally long state imaging process is ended.

  FIG. 67 is a flowchart showing still image imaging processing (processing procedure in step S920 shown in FIG. 66) in the imaging control processing performed by the mobile phone device 100 according to the first embodiment of the present invention.

  First, a monitoring process is performed according to the set imaging mode (step S921). For example, when the three-lens horizontal wide-angle image capturing mode is set, the image within the thick line range shown in FIG. 29A is displayed on the display unit 140 as a monitoring image as shown in FIG. Is displayed. Further, as described above, during the monitoring process, a pixel thinning process or a pixel addition process may be performed.

  Subsequently, it is determined whether or not a still image recording instruction operation has been accepted (step S922). For example, it is determined whether or not the enter key 114 is pressed while the monitoring image is displayed on the display unit 140. If the still image recording instruction operation is not accepted (step S922), the process proceeds to step S924. On the other hand, when a still image recording instruction operation is accepted (step S922), still image recording processing is performed according to the set imaging mode (step S923). For example, when the three-lens landscape image mode is set, an image within the bold line range shown in FIG. 29A is recorded on the recording medium 180.

  Subsequently, it is determined whether or not an imaging range switching instruction operation has been accepted (step S924). If the imaging range switching instruction operation has not been received (step S924), it is determined whether or not the rotation state of the second casing 120 has changed (step S925). If there is no change in the rotation state of the second housing 120 (step S925), it is determined whether or not a switching operation from the still image capturing mode to the moving image capturing mode is accepted (step S926). For example, it is determined whether or not the still image / moving image switch 112 is pressed in a state where the still image capturing mode is set. When there is any one of reception of an imaging range switching instruction operation, a change in the rotation state of the second housing 120, and a switching operation from the still image capturing mode to the moving image capturing mode (steps S924 to S926), The operation of the still image capturing process is terminated. On the other hand, if none of the acceptance of the imaging range switching instruction operation, the change of the rotation state of the second casing 120, and the acceptance of the switching operation from the still image capturing mode to the moving image capturing mode (steps S924 to S926). Return to step S921.

  FIG. 68 is a flowchart showing moving image imaging processing (processing procedure of step S930 shown in FIG. 66) in the processing procedure of imaging control processing by the mobile phone device 100 according to the first embodiment of the present invention.

  First, a monitoring process is performed according to the set imaging mode (step S931).

  Subsequently, it is determined whether or not a moving image recording start instruction operation has been accepted (step S932). For example, it is determined whether or not the enter key 114 is pressed while the monitoring image is displayed on the display unit 140. If the moving image recording start instruction operation is not accepted (step S932), the process proceeds to step S935. On the other hand, when a moving image recording start instruction operation is accepted (step S932), a moving image recording process is performed according to the set imaging mode (step S933). Subsequently, it is determined whether or not a moving image recording end instruction operation has been accepted (step S934). For example, it is determined whether or not the enter key 114 is pressed again while the moving image recording process is being performed. If the moving image recording end instruction operation has not been accepted (step S934), the moving image recording process is continued (step S933). On the other hand, when the moving image recording end instruction operation is accepted (step S934), the moving image recording process is ended and the process proceeds to step S935.

  Subsequently, it is determined whether or not an imaging range switching instruction operation has been accepted (step S935). If the imaging range switching instruction operation has not been received (step S935), it is determined whether or not the rotation state of the second casing 120 has changed (step S936). If there is no change in the rotation state of the second housing 120 (step S936), it is determined whether or not a switching operation from the moving image capturing mode to the still image capturing mode is accepted (step S937). When there is any one of acceptance of an imaging range switching instruction operation, a change in the rotation state of the second housing 120, and acceptance of a switching operation from the moving image capturing mode to the still image capturing mode (steps S935 to S937), The moving image capturing process is terminated. On the other hand, when none of the acceptance of the imaging range switching instruction operation, the change of the rotation state of the second housing 120, and the acceptance of the switching operation from the moving image capturing mode to the still image capturing mode (steps S935 to S937). Return to step S931. When the moving image capturing mode is set, the moving image recording process may be performed with a thinning rate higher than that in the still image capturing mode.

  FIG. 69 is a flowchart showing the vertically long imaging process (the process procedure of step S950 shown in FIG. 65) in the process procedure of the imaging control process by the mobile phone device 100 according to the first embodiment of the present invention.

  First, the imaging control unit 201 sets a monocular vertically long image imaging mode as an initial setting (step S951). Subsequently, the imaging control unit 201 determines which imaging mode of the still image / moving image imaging mode is set (step S952). When the still image capturing mode is set (step S952), still image capturing processing is performed (step S960). The still image capturing process is substantially the same as the process procedure shown in FIG. 67 except that the image displayed on the display unit 140 is different. For this reason, the description below is omitted. On the other hand, when the moving image capturing mode is set (step S952), the moving image capturing process is performed (step S970). Note that the moving image imaging process is substantially the same as the processing procedure shown in FIG. 68 except that the image displayed on the display unit 140 is different. For this reason, the description below is omitted.

  Subsequently, it is determined whether or not an imaging range switching instruction operation has been accepted (step S953). If the imaging range switching instruction operation has not been received (step S953), the process proceeds to step S955. On the other hand, when the imaging range switching instruction operation is accepted (step S953), the imaging control unit 201 switches the imaging mode according to the accepted imaging range switching instruction operation (step S954). For example, every time the imaging range changeover switch 111 is pressed, (1) a monocular portrait image imaging mode, (2) a monocular portrait image area mode, (3) a monocular landscape image imaging mode, and (4) a trinocular image area narrowly Switching is performed in the order of the corner image capturing mode.

  Subsequently, it is determined whether or not the rotation state of the second housing 120 has changed (step S955). If there is no change in the rotation state of the second housing 120, the process returns to step S952. On the other hand, when there is a change in the rotation state of the second casing 120 (step S955), the operation of the vertically long state imaging process is ended.

  As described above, in the first embodiment of the present invention, it is possible to easily switch between a plurality of imaging modes. Here, for example, when performing operations such as a call operation of a mobile phone device, creation of e-mail text, and text reading, the casing is often used in a vertically long shape. For this reason, for example, even when shooting is performed using a camera-equipped mobile phone device, shooting is often performed with the casing and the display device in a vertically long state, and a vertically long image is recorded. In addition, the vertically long image thus recorded is often played back or transmitted, for example, between users of the mobile phone device.

  However, the human field of view is wider in the lateral direction. For this reason, it is assumed that by recording an image having a large aspect ratio, an image in an imaging range close to an area that falls within the human visual field can be recorded, and the uncomfortable feeling given to the user can be reduced. Therefore, in the first embodiment of the present invention, switching between a vertically long image widely used by users of camera-equipped mobile phone devices and an image with a large angle of view (for example, a panoramic image) that is relatively close to the human visual field. Can be performed only by rotating the second casing 120. As described above, since it is possible to switch between the vertically long image and the image having a large angle of view by a relatively easy operation, it is possible to prevent the shooting timing from being missed. In addition, it is possible to easily take a user-preferred image.

  In addition, a portrait image that is widely used by users of camera-equipped mobile phone devices can be displayed or recorded in the monocular portrait image capturing mode or the monocular portrait small region image capturing mode. In addition, an image having a large angle of view (for example, a panoramic image) that is relatively close to the human visual field can be displayed or recorded in the three-lens horizontal long wide-angle imaging mode, the three-lens horizontal long narrow-angle imaging mode, and the single-lens vertical image imaging mode. it can. Also, these switching operations can be easily performed by the imaging range switching switch 111. As described above, since a plurality of types of images can be switched by a relatively easy operation, it is possible to prevent the shooting timing from being missed. In addition, it is possible to easily take a user-preferred image.

[Modification of mobile phone device]
70 and 71 are diagrams showing a modification of the mobile phone device 100 according to the first embodiment of the present invention.

  FIG. 70 shows a mobile phone device 1040 in which a display unit and an imaging unit are provided in the same housing. FIG. 70 (a) shows a front side of one mode when the mobile phone device 1040 is used, and FIG. 70 (b) shows a back side of the mode. FIG. 70 (c) shows the front side of another mode when the mobile phone device 1040 is used, and FIG. 70 (d) shows the back side of the mode.

  The cellular phone device 1040 includes a first housing 110 and a second housing 120. The second housing 120 includes a display unit 140 and an imaging unit 1041. The mobile phone device 1040 is substantially the same as the mobile phone device 100 except that the display unit and the imaging unit are provided in the same housing. For this reason, the same code | symbol is attached | subjected to the part which is common in the mobile telephone apparatus 100, and the description is abbreviate | omitted.

  The imaging unit 1041 is substantially the same as the imaging unit 130 provided in the mobile phone device 100 except that the arranged position is different. That is, in the imaging unit 1041, three imaging systems are arranged side by side according to a predetermined rule, the center imaging element is in a vertically long state, and the left and right imaging elements are in a horizontally long state.

  FIG. 71 shows a cellular phone device 1050 including a single housing. FIG. 71A shows a front side of one mode when the mobile phone device 1050 is used, and FIG. 71B shows a back side of the mode. FIG. 71 (c) shows the front side of another aspect when using the cellular phone device 1050.

  The cellular phone device 1050 includes a single housing, and includes a display unit 140, an imaging unit 1051, and an attitude detection unit 1052. Note that the cellular phone device 1050 is configured by a single housing, has a posture detection unit 1052 instead of the rotation state detection unit 150, and has a different position at which the imaging unit is disposed. Is substantially the same as the mobile phone device 100. For this reason, the same code | symbol is attached | subjected to the part which is common in the mobile telephone apparatus 100, and the description is abbreviate | omitted.

  The imaging unit 1051 is substantially the same as the imaging unit 130 provided in the mobile phone device 100 except that the position where the imaging unit 1051 is arranged is different. That is, in the imaging unit 1051, three imaging systems are arranged side by side according to a predetermined rule, the center imaging element is in a vertically long state, and the left and right imaging elements are in a horizontally long state.

  The posture detection unit 1052 is built in the mobile phone device 1050 instead of the rotation state detection unit 150, and detects acceleration, movement, inclination, and the like applied to the mobile phone device 1050. The posture detection unit is realized by, for example, a gyro sensor, a fall sensor, a gravity sensor, or the like. Then, each detected information is output to the imaging control unit 201 as posture information. The imaging control unit 201 detects whether the mobile phone device 1050 is in the horizontally long state or the vertically long state based on the detected posture information. Then, the imaging operation is controlled based on the detected state. For example, imaging control similar to the control performed depending on whether the second housing of the mobile phone device 100 is in the horizontally long state or the vertically long state can be performed. Moreover, you may make it perform the imaging control which changes an imaging range according to the operation input using a predetermined operation member. As this operation input, for example, an operation input by an operation button or a touch panel can be assumed. Note that the posture detection unit 1052 is an example of a detection unit.

  As described above, the first embodiment of the present invention can be applied to the mobile phone device of each aspect.

<2. Second Embodiment>
In the first embodiment of the present invention, the mobile phone device including a plurality of imaging systems has been described as an example. In the second embodiment of the present invention, an imaging apparatus such as a digital still camera or a digital video camera will be described.

[External configuration example of imaging device]
FIG. 72 is a diagram showing an external configuration of the imaging apparatus 1000 according to the second embodiment of the present invention. 72A shows a perspective view of the front side (subject side) of the imaging apparatus 1000, and FIGS. 1B and 1C are perspective views of the back side (photographer side) of the imaging apparatus 1000. FIG. Indicates. The imaging apparatus 1000 includes a first housing 1010 and a second housing 1020 that are rotatably connected by a rotating member 1001. The imaging apparatus 1000 is realized by a digital still camera including three imaging systems, for example. In FIG. 72, for ease of explanation, the imaging apparatus 1000 is illustrated in a simplified manner, and illustration of a power switch and the like provided on the outer surface of the imaging apparatus 1000 is omitted.

  The first housing 1010 includes an imaging range changeover switch 1011, a still image / moving picture changeover switch 1012, a shutter button 1013, and an imaging unit 1030. The imaging range switch 1011, the still image / video switching switch 1012, and the imaging unit 1030 are substantially the same as the imaging range switching switch 111, the still image / video switching switch 112, and the imaging unit 130 shown in FIG. The description in is omitted. The imaging range changeover switch 1011 is an example of an operation reception unit.

  The shutter button 1013 is an operation member that instructs the start of image recording. For example, when the still image capturing mode is set, the button is pressed when the image data generated by the image capturing unit 1030 is recorded on a recording medium as a still image file.

  The second housing 1020 includes a display portion 1021. The display unit 1021 is substantially the same as the display unit 140 shown in FIG. In addition, in the imaging apparatus 1000, a first housing 1010 and a second housing 1020 are connected to be rotatable. That is, the second casing 1020 can be rotated with respect to the first casing 1010 using the rotating member 1001 (indicated by a dotted line) as a reference for rotation. Thereby, the relative positional relationship of the second housing 1020 with respect to the first housing 1010 can be changed. For example, FIG. 72 (c) shows a configuration in which the second housing 1020 is rotated 90 degrees in the direction of the arrow 1002 shown in FIG. 72 (b). Then, similarly to the first embodiment of the present invention, the imaging operation is controlled based on whether the second housing 1020 is in a horizontally long state or a vertically long state.

  In the embodiment of the present invention, the cellular phone device and the imaging device including three imaging systems have been described as an example. However, for example, the cellular phone device and the imaging device including five or more odd number imaging systems The embodiment of the present invention can be applied to. That is, an embodiment of the present invention is configured by arranging an odd number of five or more imaging systems side by side with a certain rule, and configuring the center imaging device in a vertically long state and the other left and right imaging devices in a horizontally long state. Can be applied.

  The embodiment of the present invention shows an example for embodying the present invention. As clearly shown in the embodiment of the present invention, the matters in the embodiment of the present invention and the claims Each invention-specific matter in the scope has a corresponding relationship. Similarly, the matters specifying the invention in the claims and the matters in the embodiment of the present invention having the same names as the claims have a corresponding relationship. However, the present invention is not limited to the embodiments, and can be embodied by making various modifications to the embodiments without departing from the gist of the present invention.

  The processing procedure described in the embodiment of the present invention may be regarded as a method having a series of these procedures, and a program for causing a computer to execute the series of procedures or a recording medium storing the program May be taken as As this recording medium, for example, a CD (Compact Disc), an MD (MiniDisc), a DVD (Digital Versatile Disk), a memory card, a Blu-ray (registered trademark) disc, or the like can be used.

DESCRIPTION OF SYMBOLS 11 Application processor 12 Digital baseband process part 13 Analog baseband process part 14 RF process part 15 Battery 16 Microphone 17 Speaker 18 Antenna 100, 1040, 1050 Cell-phone apparatus 101, 1001 Rotating member 110, 1010 1st housing | casing 111 1011 Imaging range switch 112, 1012 Still image / moving image switch 113 Numeric keypad 114 Enter key 115 Cross key 120, 1020 Second casing 130, 1030, 1041, 1051 Imaging unit 131-133 Optical system 134-136 Imaging element 137-139 I / F with DSP
140, 1021 Display unit 150 Rotation state detection unit 160 Program memory 170 Image memory 180 Recording medium 191 First imaging system 192 Second imaging system 193 Third imaging system 201 Imaging control unit 202 CPU
203 DMA controller 204 Data bus 205 I / F with program memory
206 I / F with image memory
207 Power Control Unit 208 Power Supply Unit 209 Power Supply Unit 210 I / F with Image Sensor
211 to 219 Image buffer 220 Image signal processing unit 221 Pixel addition processing unit 222 Demosaicing processing unit 223 YC conversion processing unit 224 Image composition processing unit 225 Sharpness processing unit 226 Color adjustment processing unit 227 RGB conversion processing unit 231 241 251 Resolution conversion Unit 232, 242 Image rotation processing unit 233 I / F with display unit
243 I / F with external display
245 External display device 252 Encoding unit 253 I / F with recording medium
261 to 263 Oscillators 264 to 266 Oscillation circuit 270 Clock generation circuit 340 Vertical scanning circuit 345 Horizontal scanning circuit 1000 Imaging device 1013 Shutter button 1052 Posture detection unit

The present invention has been made to solve the above problems, a first aspect of the imaging unit of the multiple is on one face of the housing are arranged to the short side of the housing An image pickup unit group, and a control unit that controls the image range. The image range includes an image range captured using at least a part of the image pickup unit group and an image to be displayed. , A range of images to be recorded, or a range of images to be transmitted, and the imaging unit group includes a first imaging unit whose optical axis is orthogonal to the display surface of the display unit. comprising a part, the optical axis, the are placed in the symmetry axis coincides with the optical axis of the at a predetermined angle and the first imaging unit with respect to the optical axis of the first imaging unit, two and a second imaging unit The control unit has a substantially rectangular range as the range of the image and has a length of a short side of the substantially rectangular shape. The ratio of the length of the long side selects one of the plurality of different ranges of, the first range which constitutes the plurality of ranges is in the range of an image obtained from the first imaging unit, It ranges direction of a long substantially rectangular perpendicular to the short side direction of the housing, a second range which constitutes the multiple ranges, obtained from the first imaging unit and the two second imaging unit An imaging apparatus, an electronic device, a control method thereof, and a program for causing a computer to execute the method are within a range of an image and within a substantially rectangular range extending in a short side direction of the casing. Thus, as the range of the image, an effect of the ratio of the length of the long side you 1 Tsuosen-option from among a plurality of different ranges for the length of the short side of the substantially rectangular.

Claims (15)

An imaging unit group in which a plurality of imaging units are arranged in the short side direction of the casing on one surface of the casing;
A control unit for controlling the range of the image,
The range of the image is a range of an image captured by using at least a part of the imaging unit group, a range of an image to be displayed, a range of an image to be recorded, or a transmission target. The range of the image,
The imaging unit group includes:
A first imaging unit whose optical axis is orthogonal to the display surface of the display unit;
An optical axis having a predetermined angle with respect to the optical axis of the first imaging unit and two second imaging units arranged in line symmetry with the optical axis of the first imaging unit as an axis of symmetry;
The control unit selects one of a plurality of ranges as a range of the image, the range being a substantially rectangular range, wherein the ratio of the long side length to the short side length of the substantially rectangular shape is different. Set,
The first range constituting the plurality of ranges is a substantially rectangular range that is within a range of an image obtained from the first imaging unit and that is long in a direction orthogonal to a short side direction of the housing,
The second range constituting the plurality of ranges is within a range of images obtained from the first imaging unit and the two second imaging units, and is a substantially rectangular range that is long in the short side direction of the housing. An imaging device.   The length of the long side of the second range relative to the length of the short side of the second range than the ratio of the length of the long side of the first range to the length of the short side of the first range The imaging device according to claim 1, wherein the ratio of The first range is one side of the first range and the length of the side in the short side direction of the housing, and is one side of the first range and is orthogonal to the short side direction of the housing. A ratio of the length of the side in the direction is a substantially rectangular shape of 3: 4 or 9:16;
The second range is one side of the second range and the length of the side in the short side direction of the casing, and one side of the second range and is orthogonal to the short side direction of the casing. The imaging apparatus according to claim 2, wherein the ratio of the side length is a substantially rectangular shape larger than 16: 9.   The range of the direction orthogonal to the short side direction of the said housing | casing in the said 1st range is wider than the range of the direction orthogonal to the short side direction of the said housing | casing in the said 2nd range. An imaging apparatus according to claim 1. Each of the plurality of imaging units includes an imaging element;
The imaging device has a substantially rectangular part used for imaging,
The imaging device included in the first imaging unit is arranged so that the part is a substantially rectangular shape that is long in a direction orthogonal to the short side direction of the housing,
5. The imaging device according to claim 1, wherein the two imaging elements included in the two second imaging units are arranged such that the part is a substantially rectangular shape that is long in a short side direction of the housing. . A procedure for performing an imaging operation using an imaging unit group in which a plurality of imaging units are arranged in the short side direction of the casing on one surface of the casing;
A control procedure for controlling the range of the image,
The range of the image is a range of an image captured by using at least a part of the imaging unit group, a range of an image to be displayed, a range of an image to be recorded, or a transmission target. The range of the image,
The imaging unit group includes:
A first imaging unit whose optical axis is orthogonal to the display surface of the display unit;
An optical axis having a predetermined angle with respect to the optical axis of the first imaging unit and two second imaging units arranged in line symmetry with the optical axis of the first imaging unit as an axis of symmetry;
In the control procedure, as the range of the image, one of a plurality of ranges in which the ratio of the length of the long side to the length of the short side of the substantially rectangle is different is selected. Set,
The first range constituting the plurality of ranges is a substantially rectangular range that is within a range of an image obtained from the first imaging unit and that is long in a direction orthogonal to a short side direction of the housing,
The second range constituting the plurality of ranges is within a range of images obtained from the first imaging unit and the two second imaging units, and is a substantially rectangular range that is long in the short side direction of the housing. A control method for an imaging apparatus.   The length of the long side of the second range relative to the length of the short side of the second range than the ratio of the length of the long side of the first range to the length of the short side of the first range The method for controlling an imaging apparatus according to claim 6, wherein the ratio of the imaging device is large. The first range is one side of the first range and the length of the side in the short side direction of the housing, and is one side of the first range and is orthogonal to the short side direction of the housing. A ratio of the length of the side in the direction is a substantially rectangular shape of 3: 4 or 9:16;
The second range is one side of the second range and the length of the side in the short side direction of the casing, and one side of the second range and is orthogonal to the short side direction of the casing. The method of controlling an imaging apparatus according to claim 7, wherein the ratio of the side length is a substantially rectangular shape larger than 16: 9.   The range of the direction orthogonal to the short side direction of the said housing | casing in the said 1st range is wider than the range of the direction orthogonal to the short side direction of the said housing | casing in the said 2nd range. A method for controlling the imaging apparatus according to claim 1. Each of the plurality of imaging units includes an imaging element;
The imaging device has a substantially rectangular part used for imaging,
The imaging device included in the first imaging unit is arranged so that the part is a substantially rectangular shape that is long in a direction orthogonal to the short side direction of the housing,
The image pickup apparatus according to claim 6, wherein the two image pickup elements included in the two second image pickup units are arranged so that the part is a substantially rectangular shape that is long in a short side direction of the casing. Control method. An imaging unit group in which a plurality of imaging units are arranged in the short side direction of the casing on one surface of the casing;
A control unit for controlling the range of the image,
The range of the image is a range of an image captured by using at least a part of the imaging unit group, a range of an image to be displayed, a range of an image to be recorded, or a transmission target. The range of the image,
The imaging unit group includes:
A first imaging unit whose optical axis is orthogonal to the display surface of the display unit;
An optical axis having a predetermined angle with respect to the optical axis of the first imaging unit and two second imaging units arranged in line symmetry with the optical axis of the first imaging unit as an axis of symmetry;
The control unit selects one of a plurality of ranges as a range of the image, the range being a substantially rectangular range, wherein the ratio of the long side length to the short side length of the substantially rectangular shape is different. Set,
The first range constituting the plurality of ranges is a substantially rectangular range that is within a range of an image obtained from the first imaging unit and that is long in a direction orthogonal to a short side direction of the housing,
The second range constituting the plurality of ranges is within a range of images obtained from the first imaging unit and the two second imaging units, and is a substantially rectangular range that is long in the short side direction of the housing. Is an electronic device.   The length of the long side of the second range relative to the length of the short side of the second range than the ratio of the length of the long side of the first range to the length of the short side of the first range The electronic device according to claim 11, wherein the ratio is large. The first range is one side of the first range and the length of the side in the short side direction of the housing, and is one side of the first range and is orthogonal to the short side direction of the housing. A ratio of the length of the side in the direction is a substantially rectangular shape of 3: 4 or 9:16;
The second range is one side of the second range and the length of the side in the short side direction of the casing, and one side of the second range and is orthogonal to the short side direction of the casing. The electronic device according to claim 12, wherein the ratio of the length to the side is a substantially rectangular shape larger than 16: 9.   The range in the direction orthogonal to the short side direction of the casing in the first range is wider than the range in the direction orthogonal to the short side direction of the casing in the second range. The electronic device according to Crab. Each of the plurality of imaging units includes an imaging element;
The imaging device has a substantially rectangular part used for imaging,
The imaging device included in the first imaging unit is arranged so that the part is a substantially rectangular shape that is long in a direction orthogonal to the short side direction of the housing,
The electronic device according to claim 11, wherein the two image pickup devices included in the two second image pickup units are arranged so that the portion has a substantially rectangular shape that is long in a short side direction of the housing. .

Images