JP2024051145A - Electronic apparatus - Google Patents

Abstract

To provide an electronic apparatus that is easy to use.SOLUTION: An electronic apparatus comprises: an imaging element that includes an imaging region containing a first region where an imaging operation is performed in a first imaging condition and a second region where the imaging operation is performed in a second imaging condition; and a control part that records a file including a first image data of a first subject imaged in the first region, a second image data of a second subject image imaged in the second region, first data related to the first imaging condition, second data related to the second imaging condition, third data related to the use of the first region, and fourth data related to the use of the second region to a recording part.SELECTED DRAWING: Figure 1

Description

The present invention relates to electronic devices.

There is a known imaging device that captures images with different exposure times for each imaging area (see Patent Document 1). However, there is no mention of recording the imaging conditions for each imaging area.

JP 2006-197192 A

According to a first aspect of the present invention, an electronic device includes an image sensor having an imaging area including a first area where an imaging operation is performed under a first imaging condition and a second area where an imaging operation is performed under a second imaging condition, and a control unit that records in a recording unit a file including first image data of a first subject imaged in the first area, second image data of a second subject imaged in the second area, first data related to the first imaging condition, second data related to the second imaging condition, third data related to the use of the first area, and fourth data related to the use of the second area.

1 is a block diagram showing a configuration of an imaging apparatus according to a first embodiment of the present invention; FIG. 2 is a plan view illustrating an imaging surface of an imaging element. FIG. 2 is a schematic diagram showing a configuration of an image file according to the present embodiment. FIG. 2 is an explanatory diagram of a still image capturing function A. 13 is a diagram showing a schematic configuration of an image file created when capturing an image using a still image capturing function A. FIG. FIG. 2 is an explanatory diagram of a video imaging function A. 13 is a diagram showing a schematic configuration of an image file created when capturing an image using a video capturing function A. FIG. FIG. 13 is an explanatory diagram of a still image capturing function B. FIG. 13 is a diagram showing an example of a layout of a large group. 13 is a diagram showing a schematic configuration of an image file created when capturing an image using a still image capturing function B. FIG. FIG. 13 is an explanatory diagram of a video shooting function B. FIG. 13 is an explanatory diagram of a video shooting function B. 13 is a diagram showing a schematic configuration of an image file created when capturing an image using the video capturing function B. FIG. FIG. 2 is an explanatory diagram of a mixed imaging function. 10 is a diagram showing a schematic configuration of an image file created when capturing an image using the mixed imaging function. FIG. FIG. 11 is a diagram illustrating a directory structure of a memory card according to a second embodiment. FIG. 13 is a diagram illustrating a structure of each file according to a second embodiment. FIG. 13 is a diagram illustrating a structure of each file according to a second embodiment. FIG. 11 is an explanatory diagram of a modified example 2. FIG. 13 is an explanatory diagram of a modified example 3. FIG. 13 is an explanatory diagram of a modified example 4. FIG. 13 is an explanatory diagram of modified example 7. FIG. 2 is a cross-sectional view of a stacked imaging element. FIG. 2 is a diagram illustrating a pixel array and blocks of an imaging chip. FIG. 4 is a circuit diagram corresponding to a unit of the imaging chip. FIG. 13 is a diagram illustrating a structure of an image file in the first modified example. FIG. 13 is a diagram illustrating a structure of a data section of an image file in the first modified example. FIG. 13 is an explanatory diagram of modified example 6. FIG. 13 is a block diagram illustrating a configuration of a remote image shooting system according to a third embodiment. 13A to 13C are diagrams illustrating examples of images captured in the third embodiment. 13A to 13C are diagrams illustrating examples of images captured in the third embodiment. 13A to 13C are diagrams illustrating examples of images captured in the third embodiment.

(First embodiment)
First, a stacked imaging element 22 mounted on an electronic device (e.g., imaging device 10) according to an embodiment will be described. The stacked imaging element 22 is described in WO2013/164915, which was previously filed by the applicant of the present application. FIG. 23 is a cross-sectional view of the stacked imaging element 22. The imaging element 22 includes a back-illuminated imaging chip 2111 that outputs pixel signals corresponding to incident light, a signal processing chip 2112 that processes the pixel signals, and a memory chip 2113 that stores the pixel signals. The imaging chip 2111, the signal processing chip 2112, and the memory chip 2113 are stacked and electrically connected to each other by a connecting portion 2109 having electrical conductivity such as Cu.

As shown in Figure 23, incident light mainly enters in the positive direction of the Z axis, as indicated by the white arrow. As shown by the coordinate axes, the left direction on the paper, perpendicular to the Z axis, is the positive X axis, and the front direction on the paper, perpendicular to the Z axis and the X axis, is the positive Y axis. In the following figures, the coordinate axes are displayed so that the orientation of each figure can be seen, based on the coordinate axes in Figure 23.

The imaging chip 2111 is, for example, a CMOS image sensor. Specifically, the imaging chip 2111 is a back-illuminated MOS image sensor. The imaging chip 2111 has a microlens layer 2101, a color filter layer 2102, a passivation layer 2103, a semiconductor layer 2106, and a wiring layer 108. In the imaging chip 2111, the microlens layer 2101, the color filter layer 2102, the passivation layer 2103, the semiconductor layer 2106, and the wiring layer 2108 are arranged in this order in the positive direction of the Z axis.

The microlens layer 2101 has a plurality of microlenses L. The microlenses L focus the incident light on the photoelectric conversion unit 2104 described later. The color filter layer 2102 has a plurality of color filters F. The color filter layer 2102 has a plurality of types of color filters F with different spectral characteristics. Specifically, the color filter layer 2102 has a first filter (R) with a spectral characteristic that mainly transmits red component light, a second filter (Gb, Gr) with a spectral characteristic that mainly transmits green component light, and a third filter (B) with a spectral characteristic that mainly transmits blue component light. In the color filter layer 102, the first filter, the second filter, and the third filter are arranged, for example, in a Bayer array. The passivation layer 2103 is made of a nitrogen film or an oxide film, and protects the semiconductor layer 2106.

The semiconductor layer 2106 has a photoelectric conversion unit 2104 and a readout circuit 2105. The semiconductor layer 2106 has a plurality of photoelectric conversion units 2104 between a first surface 2106a, which is a light incidence surface, and a second surface 2106b opposite to the first surface 2106a. The semiconductor layer 2106 has a plurality of photoelectric conversion units 2104 arranged in the X-axis direction and the Y-axis direction. The photoelectric conversion unit 2104 has a photoelectric conversion function of converting light into electric charge. The photoelectric conversion unit 2104 also accumulates electric charge due to a photoelectric conversion signal. The photoelectric conversion unit 2104 is, for example, a photodiode. The semiconductor layer 2106 has a readout circuit 105 on the second surface 2106b side of the photoelectric conversion unit 2104. The semiconductor layer 2106 has a plurality of readout circuits 2105 arranged in the X-axis direction and the Y-axis direction. The readout circuit 2105 is composed of multiple transistors, and reads out the image data generated by the charge photoelectrically converted by the photoelectric conversion unit 2104, and outputs it to the wiring layer 2108.

The wiring layer 2108 has multiple metal layers. The metal layers are, for example, Al wiring, Cu wiring, etc. The wiring layer 2108 outputs image data read by the readout circuit 2105. The image data is output from the wiring layer 2108 to the signal processing chip 2112 via the connection portion 2109.

The connection portion 2109 may be provided for each photoelectric conversion portion 2104. The connection portion 2109 may be provided for each of a plurality of photoelectric conversion portions 2104. When the connection portion 2109 is provided for each of a plurality of photoelectric conversion portions 2104, the pitch of the connection portion 2109 may be larger than the pitch of the photoelectric conversion portions 2104. The connection portion 2109 may be provided in a peripheral region of the region in which the photoelectric conversion portions 2104 are arranged.

The signal processing chip 2112 has multiple signal processing circuits. The signal processing circuits perform signal processing on the image data output from the imaging chip 2111. The signal processing circuits are, for example, an amplifier circuit that amplifies the signal value of the image data, a correlated double sampling circuit that performs noise reduction processing on the image data, and an analog/digital (A/D) conversion circuit that converts an analog signal into a digital signal. A signal processing circuit may be provided for each photoelectric conversion unit 2104.

The signal processing circuit may be provided for each of the photoelectric conversion units 2104. The signal processing chip 2112 has a plurality of through electrodes 2110. The through electrodes 2110 are, for example, silicon through electrodes. The through electrodes 2110 connect the circuits provided in the signal processing chip 2112 to each other. The through electrodes 2110 may also be provided in the peripheral region of the imaging chip 2111 and in the memory chip 2113. Note that some of the elements constituting the signal processing circuit may be provided in the imaging chip 2111. For example, in the case of an analog/digital conversion circuit, a comparator that compares an input voltage with a reference voltage may be provided in the imaging chip 2111, and circuits such as a counter circuit and a latch circuit may be provided in the signal processing chip 2112.

The memory chip 2113 has multiple storage units. The storage units store image data that has been subjected to signal processing by the signal processing chip 2112. The storage units are, for example, volatile memories such as DRAMs. A storage unit may be provided for each photoelectric conversion unit 2104. Also, a storage unit may be provided for each of the multiple photoelectric conversion units 2104. The image data stored in the storage units is output to a downstream image processing unit.

Figure 24 is a diagram illustrating the pixel array of the imaging chip 2111 and the unit area 2131. In particular, it shows the imaging chip 2111 observed from the back side (imaging surface) side. For example, more than 20 million pixels are arranged in a matrix in the pixel area. In the example of Figure 24, four pixels (2 pixels x 2 pixels) adjacent to each other form one unit area 2131. The grid lines in the figure show the concept of adjacent pixels being grouped to form the unit area 2131. The number of pixels forming the unit area 2131 is not limited to this, and may be around 1000, for example 32 pixels x 32 pixels, or it may be more or less than that, or even 1 pixel.

As shown in a partially enlarged view of the pixel region, the unit region 2131 in FIG. 24 contains a so-called Bayer array consisting of four pixels: green pixels Gb, Gr, blue pixel B, and red pixel R. The green pixels Gb and Gr are pixels that have a green filter as the color filter F and receive light in the green wavelength band of the incident light. Similarly, the blue pixel B is a pixel that has a blue filter as the color filter F and receives light in the blue wavelength band, and the red pixel R is a pixel that has a red filter as the color filter F and receives light in the red wavelength band.

In this embodiment, a plurality of blocks are defined so that each block includes at least one unit area 2131. That is, the minimum unit of one block is one unit area 2131. As described above, the smallest possible value for the number of pixels forming one unit area 2131 is one pixel. Therefore, when one block is defined in pixel units, the smallest possible number of pixels that can define one block is one pixel. Each block can control the pixels contained in each block with different control parameters. In each block, all unit areas 2131 in that block, i.e., all pixels in that block, are controlled under the same imaging conditions. That is, photoelectric conversion signals with different imaging conditions can be obtained between a pixel group included in one block and a pixel group included in another block. Examples of the control parameters are frame rate, gain, thinning rate, number of summation rows or number of summation columns for adding photoelectric conversion signals, charge accumulation time or number of accumulations, number of digitization bits (word length), etc. The imaging element 22 can freely perform thinning not only in the row direction (the X-axis direction of the imaging chip 2111) but also in the column direction (the Y-axis direction of the imaging chip 2111). Furthermore, the control parameters may be parameters for image processing.

Figure 25 is a diagram explaining the circuit in unit area 2131. In the example of Figure 25, one unit area 2131 is formed by four pixels, 2 pixels x 2 adjacent pixels. As mentioned above, the number of pixels included in unit area 2131 is not limited to this, and may be 1000 pixels or more, or may be as small as 1 pixel. The two-dimensional position of unit area 2131 is indicated by symbols A to D.

The reset transistors (RST) of the pixels included in unit region 2131 are configured to be able to be turned on and off individually for each pixel. In FIG. 25, a reset wiring 2300 is provided to turn on and off the reset transistor of pixel A, and a reset wiring 2310 to turn on and off the reset transistor of pixel B is provided separately from the reset wiring 2300. Similarly, a reset wiring 2320 to turn on and off the reset transistor of pixel C is provided separately from the reset wirings 2300 and 2310. A dedicated reset wiring 2330 is also provided for another pixel D to turn on and off the reset transistor.

The transfer transistors (TX) of the pixels included in unit region 2131 are also configured to be able to be turned on and off individually for each pixel. In FIG. 25, transfer wiring 2302 that turns on and off the transfer transistor of pixel A, transfer wiring 2312 that turns on and off the transfer transistor of pixel B, and transfer wiring 2322 that turns on and off the transfer transistor of pixel C are provided separately. For the other pixel D, a dedicated transfer wiring 2332 is also provided to turn on and off the transfer transistor.

Furthermore, the selection transistors (SEL) of the pixels included in unit region 2131 are configured to be able to be turned on and off individually for each pixel. In FIG. 25, selection wiring 2306 for turning on and off the selection transistor of pixel A, selection wiring 2316 for turning on and off the selection transistor of pixel B, and selection wiring 2326 for turning on and off the selection transistor of pixel C are provided separately. For another pixel D, a dedicated selection wiring 2336 is also provided for turning on and off the selection transistor.

The power supply wiring 2304 is commonly connected to pixels A to D included in the unit area 2131. Similarly, the output wiring 2308 is commonly connected to pixels A to D included in the unit area 2131. The power supply wiring 2304 is commonly connected between multiple unit areas, but the output wiring 2308 is provided individually for each unit area 2131. The load current source 2309 supplies a current to the output wiring 2308. The load current source 2309 may be provided on the imaging chip 2111 side or on the signal processing chip 2112 side.

By individually turning on and off the reset transistors and transfer transistors of the unit area 2131, charge accumulation including the charge accumulation start time, accumulation end time, and transfer timing can be controlled for pixels A to D included in the unit area 2131. In addition, by individually turning on and off the selection transistors of the unit area 2131, the photoelectric conversion signals of each of pixels A to D can be output via a common output wiring 2308.

Here, the so-called rolling shutter method is known, which controls charge accumulation in a regular order for rows and columns for pixels A to D included in unit area 2131. When pixels are selected for each row using the rolling shutter method and then columns are specified, photoelectric conversion signals are output in the order of "ABCD" in the example of Figure 25.

In this way, by configuring the circuit based on the unit area 2131, the charge accumulation time can be controlled for each unit area 2131. In other words, photoelectric conversion signals with different frame rates can be output between the unit areas 2131. Also, by having the unit areas 2131 included in some blocks of the imaging chip 2111 perform charge accumulation (imaging) while the unit areas 2131 included in other blocks are rested, imaging can be performed only in a specified block of the imaging chip 2111 and the photoelectric conversion signal can be output. Furthermore, by switching the block that performs charge accumulation (imaging) between frames (the target block for accumulation control), imaging can be performed sequentially in different blocks of the imaging chip 2111 and the photoelectric conversion signal can be output.

As described above, output wiring 2308 is provided corresponding to each unit area 2131. Since the imaging element 22 is formed by stacking the imaging chip 2111, the signal processing chip 2112, and the memory chip 2113, by using the output wiring 2308 for electrical connection between the chips using the connection portion 2109, it is possible to route the wiring without increasing the size of each chip in the planar direction.

Fig. 1 is a block diagram showing the configuration of an imaging device according to a first embodiment. The imaging device 10 is an integrated lens camera. The imaging device 10 includes an imaging optical system 21, an imaging element 22, a control unit 23, an LCD monitor 24, a memory card 25, an operation unit 26, a DRAM 27, a flash memory 28, and an audio recording unit 29.

The imaging optical system 21 is composed of multiple lenses and forms a subject image on the imaging surface of the imaging element 22. Note that in FIG. 1, the imaging optical system 21 is illustrated as a single lens.

The imaging element 22 is an imaging element such as a CMOS or CCD, which captures the subject image formed by the imaging optical system 21 and outputs an imaging signal. The control unit 23 is an electronic circuit that controls each part of the imaging device 10, and is composed of a CPU and its peripheral circuits. A predetermined control program is written in advance in the flash memory 28, which is a non-volatile storage medium. The control unit 23 controls each part by reading and executing the control program from the flash memory 28. This control program uses the DRAM 27, which is a volatile storage medium, as a working area.

The liquid crystal monitor 24 is a display device that uses a liquid crystal panel. The control unit 23 causes the image sensor 22 to repeatedly capture an image of the subject at a predetermined cycle (for example, 1/60th of a second). Then, various types of image processing are performed on the image signal output from the image sensor 22 to create a so-called through image, which is displayed on the liquid crystal monitor 24. In addition to the through image, the liquid crystal monitor 24 also displays, for example, a setting screen for setting imaging parameters (imaging conditions).

The control unit 23 creates an image file (described later) based on the imaging signal output from the imaging element 22, and records the image file on a memory card 25, which is a portable recording medium. The operation unit 26 has various operation members such as push buttons, and outputs operation signals to the control unit 23 in response to the operation of these operation members. The sound recording unit 29 is composed of, for example, a microphone, and converts environmental sounds into audio signals and inputs them to the control unit 23. Note that instead of recording the image file 40 on the memory card 25, which is a portable recording medium, it may be recorded on a hard disk or other recording medium (not shown) built into the imaging device 10.

2(a) is a schematic plan view of the imaging surface 30 of the imaging element 22, and FIG. 2(b) is an enlarged plan view of a partial region 30a of the imaging surface 30. As shown in FIG. 2(b), a large number of imaging pixels 31 are arranged two-dimensionally on the imaging surface 30. Each imaging pixel 31 has a color filter (not shown). The color filters are of three types: red (R), green (G), and blue (B), and the notations "R", "G", and "B" in FIG. 2(b) indicate the type of color filter that the imaging pixel 31 has. As shown in FIG. 2(b), the imaging pixels 31 equipped with such color filters are arranged in a so-called Bayer array on the imaging surface 30 of the imaging element 22.

The imaging pixel 31 with a red filter photoelectrically converts light in the red wavelength band of the incident light and outputs a light reception signal (photoelectric conversion signal). Similarly, the imaging pixel 31 with a green filter photoelectrically converts light in the green wavelength band of the incident light and outputs a light reception signal. Furthermore, the imaging pixel 31 with a blue filter photoelectrically converts light in the blue wavelength band of the incident light and outputs a light reception signal.

The imaging element 22 of this embodiment is configured so that each unit group 32 consisting of a total of four imaging pixels 31 (2 pixels x 2 pixels adjacent to each other) can be individually controlled by the control unit 23. For example, when charge accumulation is started simultaneously for two different unit groups 32, charge reading, i.e., light reception signal reading, can be performed 1/30 seconds after the start of charge accumulation in one unit group 32, and charge reading can be performed 1/15 seconds after the start of charge accumulation in the other unit group 32. In other words, the imaging element 22 can set a different exposure time (charge accumulation time, so-called shutter speed) for each unit group 32 in one imaging session.

In addition to the exposure time described above, the image sensor 22 can vary the amplification rate of the image signal (so-called ISO sensitivity) for each unit group 32. The image sensor 22 can change the timing for starting charge accumulation and the timing for reading out the light receiving signal for each unit group 32. In other words, the image sensor 22 can change the frame rate during video capture for each unit group 32.

In summary, the imaging element 22 is configured to be able to vary imaging conditions such as exposure time, amplification rate, and frame rate for each unit group 32. For example, if a readout line (not shown) for reading out an imaging signal from a photoelectric conversion unit (not shown) of the imaging pixel 31 is provided for each unit group 32 and the imaging signal can be read out independently for each unit group 32, the exposure time (shutter speed) can be made different for each unit group 32. In addition, if an amplifier circuit (not shown) for amplifying an imaging signal generated by photoelectrically converted charges is provided independently for each unit group 32 and the amplification rate of the amplifier circuit can be controlled independently for each amplifier circuit, the signal amplification rate (ISO sensitivity) can be made different for each unit group 32.

The number of imaging pixels 31 constituting the unit group 32 does not have to be 4 pixels, or 2×2, as described above. The unit group 32 only needs to have at least one imaging pixel 31, and may have more than four imaging pixels 31. The imaging conditions that can be made different for each unit group 32 may be other than those described above. For example, if a liquid crystal panel having sections (each section corresponds to one unit group 32) that can be controlled independently for each unit group 32 is provided in the image sensor 22 and used as an on-off neutral density filter, it becomes possible to control the brightness (aperture value) for each unit group 32.

Next, the image file 40 that the control unit 23 creates and records on the memory card 25 will be described. FIG. 3 is a schematic diagram showing the structure of an image file according to this embodiment. The image file 40 consists of two blocks: a header section 41 and a data section 42.

The header section 41 is a block located at the beginning of the image file 40, and stores the file basic information section 43, mask section 44, and imaging information section 45 in the order described above. The file basic information section 43 records, for example, the size and offset of each section in the image file 40 (header section 41, data section 42, mask section 44, imaging information section 45, etc.). The mask section 44 records imaging condition information and mask information, etc., which will be described later. The imaging information section 45 records information related to imaging, such as the model name of the imaging device 10 and information about the imaging optical system 21 (for example, information about optical characteristics such as aberration). The data section 42 is a block located after the header section 41, and records image information, audio information, etc.

Next, the imaging functions of the imaging device 10 and the image files 40 created (recorded) by each imaging function will be described. The user can perform a predetermined operation on the operating members of the operation unit 26 to switch (select) each imaging function described below. The control unit 23 captures an image based on the selected imaging function, creates an image file 40, and records it on the memory card 25.

(1) Still image capture function A (single still image)
The still image capturing function A is a function for capturing still images by dividing the captured image screen into a plurality of partial regions and setting imaging conditions for each of the plurality of partial regions.

Figure 4(a) shows a schematic diagram of an imaging screen 50 (imaging range) of the imaging element 22 and a subject 51. The procedure for imaging the subject 51 shown in Figure 4(a) using the still image imaging function A will be described. The control unit 23 images the subject 51 once before the main imaging. Hereinafter, the imaging performed prior to the main imaging is referred to as a preliminary imaging. Note that the preliminary imaging may also serve as an imaging performed to create a live view image (a so-called through image), for example.

The control unit 23 executes a predetermined image analysis process on the image of the subject 51 obtained by the preliminary image capture (an image in which the subject 51 is captured). The image analysis process is a process of detecting the main subject portion and the background portion, for example, by a well-known subject detection technology (a technology that calculates features to detect the area in which a predetermined subject exists). The image analysis process divides the captured image screen 50 into a main subject region 52 where the main subject portion exists, and a background region 53 where the background portion exists.

In FIG. 4(a), the main subject region 52 is shown as an area that roughly includes the subject 51, but the main subject region 52 may have a shape that follows the outline of the subject 51. In other words, the main subject region 52 may be set so as to include as little as possible of anything other than the subject 51.

The control unit 23 sets different imaging conditions for each unit group 32 in the main subject region 52 and each unit group 32 in the background region 53. For example, a faster shutter speed is set for each of the former unit groups 32 compared to each of the latter unit groups 32. In this way, image blurring is less likely to occur in the main subject region 52 during actual imaging.

Furthermore, when the main subject region 52 is backlit due to the influence of a light source such as the sun in the background region 53, a relatively high ISO sensitivity and a slow shutter speed are set for each of the former unit groups 32. Also, a relatively low ISO sensitivity and a fast shutter speed are set for each of the latter unit groups 32. In this way, it is possible to prevent crushed shadows in the main subject region 52 that is backlit and blown highlights in the background region 53 where there is a large amount of light in the actual image capture.

The image analysis process may be a process different from the process of detecting the main subject portion and the background portion described above. For example, the image analysis process may be a process of detecting portions of the entire image capture screen 50 that are brighter than a certain level (portions that are too bright) and portions that are less than a certain level (portions that are too dark). When the image analysis process is of this type, the control unit 23 sets the shutter speed and ISO sensitivity for the unit groups 32 included in the former region so that the exposure value (Ev value) is lower than that of the unit groups 32 included in the other regions. Also, for the unit groups 32 included in the latter region, the shutter speed and ISO sensitivity are set so that the exposure value (Ev value) is higher than that of the unit groups 32 included in the other regions. In this way, the dynamic range of the image obtained by this imaging can be expanded beyond the original dynamic range of the image sensor 22.

Figure 5 is a diagram showing a schematic configuration of an image file 40 created when capturing an image using still image capturing function A. In the mask section 44, identification information 60, capturing condition information 61, and mask information 62a are recorded in the order described above. Identification information 60 is information indicating that this image file 40 was created by still image capturing function A.

The imaging condition information 61 is information that indicates what kind of use (purpose, role) exists in the unit group 32. For example, as described above, when the imaging screen 50 (FIG. 4(a)) is divided into the main subject region 52 and the background region 53, each unit group 32 belongs to either the main subject region 52 or the background region 53. In other words, the unit group 32 has either the use of "taking a still image of the main subject part" or the use of "taking a still image of the background part". The imaging condition information 61 is information that indicates that when this image file 40 was created, the unit group 32 had two types of uses, "taking a still image of the main subject part" and "taking a still image of the background part", and a unique number assigned to each of these uses. For example, the number 1 is assigned to the use of "taking a still image of the main subject part", and the number 2 is assigned to the use of "taking a still image of the background part".

Mask information 62a is information that indicates the use (purpose, role) of each unit group 32. In this embodiment, mask information 62a is "information that expresses the numbers assigned to the imaging condition information 61 in the form of a two-dimensional map according to the positions of unit groups 32." In other words, when unit groups 32 arranged two-dimensionally are specified by two-dimensional coordinates (x, y) using two integers x and y, the use of unit group 32 at position (x, y) is expressed by the number at position (x, y) in mask information 62a. For example, if the number "1" is entered at coordinate (3, 5) in mask information 62a, it can be seen that the unit group 32 at coordinate (3, 5) has been given the use of "capturing a still image of the main subject portion." In other words, it can be seen that unit group 32 at coordinate (3, 5) belongs to main subject region 52.

An example of mask information 62a corresponding to the captured image 50 shown in FIG. 4(a) is shown in FIG. 4(b). A "1" is stored in the position of the unit group 32 that belongs to the main subject region 52, and a "2" is stored in the position of the unit group 32 that belongs to the background region 53.

The data section 42 stores mask information 62b, image information 64, Tv value map 65, Sv value map 66, Bv value map 67, and Av value information 68 in the order described above. Mask information 62b is the same as mask information 62a stored in mask section 44. The same mask information 62a, 62b is stored in both mask section 44 and data section 42 in order to make it easier to handle the image file 40.

Although the details will be described later, in the image file 40 created by the other functions, different mask information 62a, 62b may be stored in the mask section 44 and the data section 42. In the still image capturing function A, for example, if the mask information 62b is stored in the data section 42 and the mask information 62a is not stored in the mask section 44, the structure of the image file 40 will change for each function. In this way, the handling of the image file 40 becomes complicated, so in this embodiment, the same mask information 62a, 62b is intentionally placed in both the mask section 44 and the data section 42 to minimize the difference in the structure of the image file 40 for each function. Note that either one of the mask information 62a, 62b may be omitted, in which case it is possible to reduce the size of the storage area occupied by the image file 40. In addition, even if both the mask information 62a, 62b are recorded, it is possible to know whether it is necessary to read both the mask information 62a, 62b from the identification information, so that if it is determined that one is not necessary for playback processing, the reading of that one can be skipped, thereby shortening the file reading time.

In the following description, the mask information 62a stored in the mask section 44 and the mask information 62b stored in the data section 42 are collectively referred to as mask information 62.

Image information 64 is information in which the imaging signal output from the imaging element 22 during actual imaging is recorded in a form before various image processing is applied, and is so-called RAW image data. Tv value map 65 is information in which the Tv value indicating the shutter speed set for each unit group 32 is expressed in the form of a two-dimensional map according to the position of the unit group 32. For example, the shutter speed set for the unit group 32 located at the coordinates (x, y) can be determined by checking the Tv value stored at the coordinates (x, y) of the Tv value map 65.

The Sv value map 66 is information in which the Sv value, which indicates the ISO sensitivity set for each unit group 32, is expressed in the form of a two-dimensional map, similar to the Tv value map 65. The Bv value map 67 is information in which the Bv value, which indicates the subject brightness measured for each unit group 32 during actual imaging, i.e., the brightness of the subject light incident on each unit group 32, is expressed in the form of a two-dimensional map, similar to the Tv value map 65. The Av value information 68 is information that indicates the aperture value during actual imaging. In this embodiment, unlike the Tv value, Sv value, and Bv value, the Av value is not a value that exists for each unit group 32. Therefore, unlike the Tv value, Sv value, and Bv value, only a single value is stored for the Av value, and it is not information in which multiple values are mapped two-dimensionally.

As described above, the control unit 23 performs imaging using the still image imaging function A, and records on the memory card 25 an image file 40 in which image information 64 generated by the imaging element 22, in which imaging conditions can be set for each unit group 32, and data related to the imaging conditions for each unit group 32 (imaging condition information 61, mask information 62, Tv value map 65, Sv value map 66, Bv value map 67, etc.) are associated. This type of image file storage format is referred to as a batch storage format (chronological type) in this specification.

In the above explanation, the image information 64 is described as RAW image data, but it may be compressed (developed) image data instead of RAW image data.

(2) Video capture function A (single video)
The video imaging function A is a function for dividing the imaging screen into a plurality of partial regions, setting imaging conditions for each of the plurality of partial regions, and imaging a video. The difference from the still image imaging function A is that it captures a video instead of a still image. Since it captures a video instead of a still image, the "use of each unit group 32" described in the still image imaging function A may change for each frame.

6(a) shows a schematic diagram of the imaging screen 50 (imaging range) of the imaging element 22 and the subject 51. The control unit 23 performs a preliminary imaging before the actual imaging. Then, a predetermined image analysis process is performed on the image of the subject 51 obtained by the preliminary imaging (an image in which the subject 51 is captured). The image analysis process divides the imaging screen 50 into a main subject region 52 in which the main subject portion exists and a background region 53 in which the background portion exists. The control unit 23 sets different imaging conditions for each unit group 32 in the main subject region 52 and each unit group 32 in the background region 53, performs the actual imaging of the first frame, and creates image data. An example of the mask information 62 at this time is shown in FIG. 6(b). As an example, in the mask information 62 shown in FIG. 6(b), the unit group 32 belonging to the main subject region 52 is assigned the number "1", and the unit group 32 belonging to the background region 53 is assigned the number "2".

Next, the control unit 23 performs image analysis processing on the image data of the first frame to detect the main subject portion and the background portion. As a result, the image data of the first frame is divided into a main subject region 52 and a background region 53, as shown in FIG. 6(c). The control unit 23 sets different imaging conditions for each unit group 32 in the main subject region 52 and each unit group 32 in the background region 53, performs actual imaging of the second frame, and creates image data. An example of the mask information 62 at this time is shown in FIG. 6(d).

Mask information 62 (Figure 6(b)) corresponding to the result of preliminary imaging and mask information 62 (Figure 6(d)) corresponding to the result of actual imaging of the first frame are captured at different times (there is a time difference), so if, for example, subject 51 is moving or the user moves imaging device 10, these two pieces of mask information 62 will have different contents. In other words, mask information 62 is dynamic information that changes over time. Therefore, in a certain unit group 32, different imaging conditions are set for actual imaging of the first frame and actual imaging of the second frame.

The control unit 23 records the mask information 62b, Tv value map 65, Sv value map 66, Bv value map 67, and Av value information 68 for each frame in the image file 40 together with the image information 64 for each frame. Therefore, after capturing an image, all the information at the time of capturing the image can be obtained from the image file 40, and this information can be effectively used for video playback, etc.

The process for capturing images from the third frame onwards is similar to the process for the second frame described above, and therefore will not be described here. The control unit 23 repeatedly executes the process described above until capturing images is complete (for example, until a predetermined time has elapsed or the user performs a predetermined operation to end capturing images).

Figure 7 is a diagram showing a schematic of the structure of an image file 40 created when capturing an image using the video capture function A. The differences from the still image capture function A shown in Figure 5 are described in detail below.

The identification information 60 indicates that this image file 40 was created by the video capture function A. The capture condition information 61 is the capture condition information 61 of the still image capture function A plus the frame rate. In other words, the capture condition information 61 is information indicating that when this image file 40 was created, the unit group 32 had two uses, for example, "capturing video of the main subject part at 60 fps" and "capturing video of the background part at 30 fps," and indicating a unique number assigned to each of these uses. For example, the number 1 is assigned to the use of "capturing video of the main subject part at 60 fps," and the number 2 is assigned to the use of "capturing video of the background part at 30 fps."

The mask information 62a is the same information as in the still image capture function A described above. However, in the case of video capture, as described above, the mask information 62 is dynamic information that changes for each frame, so it is necessary to determine which frame's mask information 62 is to be recorded in the header section 41. In this embodiment, the mask information 62a representing the imaging conditions set for each unit group 32 when capturing the first frame, i.e., the mask information 62 exemplified in FIG. 6(b), is recorded in the header section 41. This is done to prevent the handling of the image file 40 from becoming cumbersome, as described in the explanation of the still image capture function A.

In the data section 42, blocks 70 for one frame are stored in the order of capture for each frame. One block 70 consists of mask information 62, image information 64, Tv value map 65, Sv value map 66, Bv value map 67, and Av value information 68. In addition to the blocks 70 for each frame, audio information 71 is stored in the data section 42. To facilitate video playback, the audio information 71 is divided into information for one frame, multiplexed with the blocks 70, and stored in the data section 42. Note that the audio information 71 may be multiplexed for a predetermined number of frames, rather than for one frame. The information in the blocks 70 is the same as that in the case of the still image capture function A, except that it is recorded for each frame, and so a description is omitted.

The image file 40 is not limited to the one in which the mask information 62b of each frame is recorded together with the image information of each frame. For example, when the same captured scene is consecutive, such as in the case of video capture by a surveillance camera, the mask information 62 of the initial frame is used until the captured scene changes, i.e., the mask information 62 is changed. In this case, the control unit 23 records the mask information 62a set when the first frame is captured in the header unit 41. If the mask information 62 set when the second frame is captured is the same as the mask information 62a of the first frame, the control unit 23 does not record the mask information 62b in the block 70 of the second frame in the data unit 42, and gives information indicating that the mask information 62b does not exist (mask presence/absence information). For example, if the number "1" is given as the mask presence/absence information, it indicates that the mask information 62b is recorded in the data unit 42, and if the number "0" is given, it indicates that the mask information 62b is not given to the data unit 42. That is, "0" is given as the mask presence/absence information to the block 70 of the second frame.

For subsequent frames, if the captured scene is the same as the first frame, mask information 62b is not recorded in each block 70 of the data section 42, and "0" is assigned as the mask presence/absence information. If the captured scene changes from the scene captured in the first frame, the control section 23 assigns "1" as mask presence/absence information to the block 70 of that frame in the data section 42, and records mask information 62b that is set according to the change in scene. This makes it possible to reduce the size of the storage area by reducing the mask information 62b in the data section 42 when the same captured scene is repeated.

Furthermore, if the imaging scene is the same as the immediately preceding frame, the immediately preceding mask information 62 may be used. In this case, the control unit 23 records the mask information 62a set when imaging the first frame in the header section 41. If the mask information 62 set when imaging the second frame is the same as the mask information 62a of the first frame, the control unit 23 does not record the mask information 62b in the block 70 of the second frame in the data section 42, and gives information indicating that the mask information 62b is the same as the mask information 62a of the immediately preceding frame (mask identical information). For example, if the number "1" is given as this mask identical information, it indicates that the imaging scene is different from the immediately preceding frame and therefore the mask information 62b is recorded in the data section 42. If the number "0" is given, it indicates that the imaging scene is the same as the immediately preceding frame and therefore the mask information 62b is not given to the data section 42. That is, "0" is given as the mask identical information to the block 70 of the second frame.

When the imaging scene of the third frame is different from the imaging scene of the second frame, the control unit 23 sets the mask information 62 of the third frame, assigns "1" as the mask identity information to the block 70 of the third frame in the data unit 42, and records the mask information 62b. For subsequent frames, when an imaging scene different from the imaging scene of the immediately preceding frame is obtained, the control unit 23 assigns "1" as the mask identity information to the block 70 of the frame in the data unit 42 and records the mask information 62b, and when the same imaging scene is obtained, the control unit 23 assigns "0" as the mask identity information and does not record the mask information 62b. As a result, when the same imaging scene is consecutive, it is possible to reduce the mask information 62b in the data unit 42 and reduce the size of the storage area.

As described above, the control unit 23 performs imaging using the video imaging function A, and records on the memory card 25 an image file 40 in which image information 64 generated by the imaging element 22, in which imaging conditions can be set for each unit group 32, and data related to the imaging conditions for each unit group 32 (imaging condition information 61, mask information 62, Tv value map 65, Sv value map 66, Bv value map 67, etc.) are associated. This type of image file storage format is referred to as a batch storage format (time series type) in this specification.

(3) Still image capture function B (multiple still images)
The still image capturing function B is a function for simultaneously capturing a plurality of still images of the same subject under different capturing conditions in one capture.

Figure 8(a) shows a schematic diagram of the imaging surface 30 of the imaging element 22. Figure 8(b) shows a schematic diagram of an enlarged partial area 30b of the imaging surface 30. In the still image imaging function B, the multiple unit groups 32 arranged in a two-dimensional shape are further classified into multiple large groups 81. At this time, the unit groups 32 are classified so that the unit groups 32 belonging to a certain large group 81 are uniformly arranged on the entire imaging surface 80. For example, in Figure 8(b), all the unit groups 32 are divided into blocks 82 consisting of four unit groups 32 of 2 x 2, and the upper left unit group 32 of each block 82 is classified into the first large group 811, the lower left unit group 32 into the second large group 812, the upper right unit group 32 into the third large group 813, and the lower right unit group 32 into the fourth large group 814. In FIG. 8(b), each schematic rectangle represents one unit group 32, and the number written inside the rectangle indicates the type of large group 81 to which the unit group 32 belongs.

When capturing the actual image, the control unit 23 sets different imaging conditions for the unit groups 32 belonging to the first large group 811, the unit groups 32 belonging to the second large group 812, the unit groups 32 belonging to the third large group 813, and the unit groups 32 belonging to the fourth large group 814. For example, the control unit 23 sets the shutter speed and ISO sensitivity to different values to capture the actual image. The control unit 23 records the image information captured in this manner in the image file 40. The image information recorded here is intended to be used by aggregating each pixel value for each large group 81, as shown diagrammatically in FIG. 8(c).

For example, as shown in FIG. 8C, when only pixel values corresponding to the unit groups 32 belonging to the first large group 811 are extracted from the image information 64 and arranged two-dimensionally, a first image information 641 consisting of pixel values of 1/4 of the number of pixels of the image sensor 22 is obtained. Similarly, when only pixel values corresponding to the unit groups 32 belonging to the second large group 81 are extracted from the image information 64 and arranged two-dimensionally, a second image information 642 consisting of pixel values of 1/4 of the number of pixels of the image sensor 22 and captured under different imaging conditions from the first image information 641 described above is obtained, which shows the same subject 51 as the first image information 641. Similarly, a third image information 643 and a fourth image information 644 are obtained. These four image information 641, 642, 643, and 644 are all images of the same subject 51, and are images with different imaging conditions. In other words, as mentioned at the beginning, it can be said that four still images of the same subject 51, each of which is captured under different imaging conditions, are captured simultaneously in one imaging session.

The image information 64 in the image file 40 is an image in which pixel outputs from each imaging pixel 31 of the imaging element 22 are arranged according to the position of the imaging pixel 31. In other words, the process of creating the above-mentioned four pieces of image information 641, 642, 643, and 644 is performed during playback or development when the image file 40 is read from the memory card 25. In addition, the image information 64 is not necessarily used only for creating the four pieces of image information 641, 642, 643, and 644. If the image information 64 is used as is (playback, etc.), the imaging conditions for each adjacent unit group 32 are different, resulting in an unnatural image, for example, with a checkered pattern. However, since the imaging conditions (for example, Tv value, Sv value, etc.) for each unit group 32 are recorded in the image file 40, such unnatural images can be prevented by combining these imaging conditions and the image information 64 and developing them. For example, in a unit group 32 with a higher exposure value (Ev value) than the other unit groups 32, development can be performed with a lower brightness than the other unit groups 32.

The above describes an example in which the unit groups 32 are classified into four large groups 811, 812, 813, and 814, but the number is not limited to four, and any number of large groups 81 may be used to capture any number of still images simultaneously. Furthermore, the layout of the large groups 81 (the method of classifying the unit groups 32) is not limited to classifying each of the 2 x 2 unit groups 32 into a different large group 81.

9(a) and (b) show examples of this point. In FIG. 9(a), all unit groups 32 are divided into a total of nine 3×3 sets, and in each set, the nine unit groups 32 included in the set are assigned to the first to ninth large groups 81, respectively. By adopting such a layout, nine images 641 to 649 with different imaging conditions can be simultaneously captured in one imaging. Also, in FIG. 9(b), all unit groups 32 are divided into a total of nine 3×3 sets, and in each set, the unit group 32 in the upper left corner is assigned to the first large group 81, and the four unit groups 32 in the lower right 2×2 are assigned to the second large group 81. The remaining four unit groups 32 are not used for imaging. In this way, two images 641, 642 with different imaging conditions can be captured simultaneously in one imaging session, but the image 642 corresponding to the second large group 81 has four times the number of pixels as the image 641 corresponding to the first large group 81. In other words, two images 641, 642 with different imaging conditions can be captured simultaneously in one imaging session, and the two images 641, 642 have different numbers of pixels.

Figure 10 is a diagram showing a schematic of the structure of an image file 40 created when capturing an image using still image capture function B. The differences from the case of still image capture function A shown in Figure 5 will be described in detail below.

The identification information 60 indicates that this image file 40 was created by the still image capture function B. The capture condition information 61 is information indicating what use exists for the unit group 32. In the still image capture function B, each unit group 32 has, for example, one of the uses of "forming the first image information 641", "forming the second image information 642", "forming the third image information 643", and "forming the fourth image information 644". The capture condition information 61 is information indicating that the unit group 32 had these four uses when this image file 40 was created, and indicating a unique number assigned to each of these uses. For example, the numbers 1 to 4 are assigned to the uses of "forming the first to fourth image information 641 to 644", respectively.

The mask information 62a in the still image capture function B is information that indicates the purpose of each unit group 32, similar to the case of the still image capture function A. In other words, the mask information 62a is "information that expresses the numbers assigned to the capture condition information 61 in the form of a two-dimensional map in accordance with the positions of the unit groups 32." For example, if the number "1" is entered at the coordinate (3, 5) in the mask information 62a, it can be seen that the unit group 32 at the coordinate (3, 5) belongs to the first large group 811, i.e., it constitutes the first image information 641.

In this embodiment, the large group 81 numbered "0" is a special large group 81 that represents a unit group 32 that was not used in imaging. In other words, a unit group 32 to which the number "0" is assigned in the mask information 62a is not used in imaging (the imaging signal is not read out during actual imaging), and the image information 64 recorded in the data section 42 does not include information about that unit group 32 (or invalid dummy information is recorded as information about that unit group 32).

For example, if it is sufficient to capture images under three different imaging conditions at the same time and it is not necessary to capture images under four different imaging conditions, the number "0" can be assigned to the mask information 62a of the unit group 32 labeled "4" among the unit groups 32 shown in FIG. 8(b).

The configuration of the data section 42 is the same as that of the still image capture function A. That is, the data section 42 stores mask information 62b, image information 64, a Tv value map 65, an Sv value map 66, a Bv value map 67, and Av value information 68. Furthermore, the mask information 62b is the same information as the mask information 62a stored in the mask section 44.

Incidentally, the mask information 62b may not be the same as the mask information 62a in the mask section 44, but may store information indicating whether each of the unit groups 32 is valid or invalid. For example, the numerical value "0" may be assigned to unit groups 32 that are not used for imaging (the imaging signal is not read out during imaging) and the numerical value "1" may be assigned to unit groups 32 that are used for imaging (the imaging signal is read out during imaging), and a map in which these are arranged two-dimensionally according to the positions of the unit groups 32 may be stored in the data section 42 as the mask information 62b. The same applies to the video imaging function B and the mixed imaging function described below.

As described above, the control unit 23 performs imaging using the still image imaging function B, and records on the memory card 25 an image file 40 in which image information 64 generated by the imaging element 22, in which imaging conditions can be set for each unit group 32, and data related to the imaging conditions for each unit group 32 (imaging condition information 61, mask information 62, Tv value map 65, Sv value map 66, Bv value map 67, etc.) are associated. This type of image file storage format is referred to as a batch storage format (image collection type) in this specification.

(4) Video capture function B (multiple videos)
The video imaging function B is a function for simultaneously capturing videos of the same subject under different imaging conditions in one capture. The difference from the still image capturing function B is that it captures videos instead of still images. Although it captures videos, unlike the video imaging function A, the unit groups 32 classified into a certain large group 81 are not classified into different large groups 81 for each frame. However, depending on the frame rate setting, a unit group 32 included in a certain frame (valid in a certain frame) may not be included in another frame (invalid in another frame). Below, the video imaging function B will be described according to the frame rate setting.

11 is an explanatory diagram of the video imaging function B when the frame rate is the same for all the large groups 81. In this case, the imaging conditions that differ for each large group 81 are imaging conditions other than the frame rate (e.g., shutter speed, ISO sensitivity, etc.). Even if the exposure times are different, the frame rate, i.e., the signal readout period, is the same, so that in all the large groups 81, the image signal is readout at a predetermined period T1 that corresponds to the frame rate.

Since imaging is performed at the same frame rate in all unit groups 32, all unit groups 32 are used for imaging in all frames. In other words, imaging signals are read out from all unit groups 32 in all frames, and the image information 64 of all frames contains imaging signals read out from all unit groups 32. For example, the first image information 64 is created at time t1, which is a predetermined period T1 after imaging start time t0. This image information 64 includes an image of the first frame (frame indicated as #1 in FIG. 11; the same applies below) in the first large group 81, an image of the first frame in the second large group 81, an image of the first frame in the third large group 81, and an image of the first frame in the fourth large group 81. The same applies to the second and subsequent image information 64.

12 is an explanatory diagram of the video capture function B in the case where different frame rates are set for all the large groups 81. In this example, the frame rates set for the first large group 811 are 60 fps, the second large group 812 are 50 fps, the third large group 813 are 24 fps, and the fourth large group 814 are 25 fps.

In this way, when the frame rates of the large groups 81 are different, the control unit 23 records each frame based on the fastest frame rate. In other words, image information 64 is recorded every predetermined period T2 (about 16.7 milliseconds) corresponding to 60 fps. For example, at time t11, which is the predetermined period T2 after the imaging start time t0, image information 64 is created based on the imaging signal read from the unit group 32 belonging to the first large group 811 and recorded in the image file 40. At time t11, the imaging signal of the first frame has not been read from the other large groups 812, 813, and 814, so the imaging signals of those groups are not included in this image information 64. In FIG. 12, the symbol "X" indicates that the imaging signal has not been read from a specific unit group 32 and that the imaging signal is not included in the image information 64.

At time t12, a predetermined period T2 after time t11, not only the second main imaging (second frame) of the first large group 811 but also the first main imaging (first frame) of the second large group 812 (50 fps) is completed. Therefore, the control unit 23 records the imaging signals read out from the unit groups 32 belonging to the first large group 811 and the unit groups 32 belonging to the second large group 812 in the image file 40. No imaging signals are read out from the unit groups 32 belonging to the third large group 813 or the unit groups 32 belonging to the fourth large group 814, and are not recorded in the image file 40.

In this way, when the frame rates of the large groups 81 are different, a portion of the image information 64 may be missing (invalidated). The control unit 23 indicates that an imaging signal corresponding to a specific unit group 32 is not included in the image information 64 by using the mask information 62b recorded for each frame. The specific configuration of the mask information 62b will be described later.

Figure 13 is a diagram showing a schematic of the structure of an image file 40 created when capturing an image using video capture function B. Below, the differences from the video capture function A shown in Figure 7 and the still image capture function B shown in Figure 10 will be described in detail.

The identification information 60 indicates that this image file 40 was created by the video capture function B. The capture condition information 61 is information indicating what kind of use exists in the unit group 32. The capture condition information 61 in the video capture function B is the capture condition information 61 in the still image capture function B with the frame rate added. In other words, the capture condition information 61 is information indicating that when this image file 40 was created, the unit group 32 had four types of uses, for example, "forming the first image information 641, which is a video at 60 fps," "forming the second image information 642, which is a video at 50 fps," "forming the third image information 643, which is a video at 24 fps," and "forming the fourth image information 644, which is a video at 25 fps," as well as a unique number assigned to each of these uses. For example, the numbers 1 to 4 are assigned to the uses "forming the first to fourth image information 641 to 644," respectively.

The mask information 62a in the video imaging function B is information that indicates the purpose of each unit group 32, as in the case of the still image imaging function B. In other words, the mask information 62a is "information that expresses the numbers assigned to the imaging condition information 61 in the form of a two-dimensional map in accordance with the positions of the unit groups 32." For example, if the number "1" is entered at the coordinate (3, 5) in the mask information 62a, it can be seen that the unit group 32 at the coordinate (3, 5) belongs to the first large group 811, i.e., it constitutes the first image information 641.

The configuration of the data section 42 is the same as that of the video imaging function A. That is, the data section 42 stores a block 70 for each frame in the order of imaging. One block 70 consists of mask information 62b, image information 64, a Tv value map 65, an Sv value map 66, a Bv value map 67, and Av value information 68. In addition to the block 70 for each frame, the data section 42 also stores audio information 71.

In addition to the numbers (e.g., numbers 1 to 4) specified by the imaging condition information 61 described above, the number "0" may be stored in the mask information 62b. This number "0" indicates that the unit group 32 is not used for imaging in the corresponding frame (the imaging signal is not read out during imaging). As described above, when capturing multiple videos with different frame rates, the image information 64 of a certain frame may not store an imaging signal corresponding to a specific unit group 32. In such a case, the control unit 23 sets the numerical value of the mask information 62 corresponding to that unit group 32 in the frame to "0". Here, for the unit group 32 in which the numerical value of the mask information 62b is set to "0", no valid values are recorded for information other than the image information 64, i.e., the Tv value in the Tv value map 65, the Sv value in the Sv value map 66, and the Sv value in the Bv value map 67.

For a unit group 32 in which the numerical value of the mask information 62b is set to "0", the imaging signal of the previous frame of that unit group 32 may be recorded in the image information 64. Similarly, the values of the Tv value in the Tv value map 65, the Sv value in the Sv value map 66, and the Sv value in the Bv value map 67 in the previous frame may be recorded. The same applies to the mixed imaging function described below.

In addition, even if the frame rates of the large groups 81 are different as in (4-2) above, it is not necessary to record the mask information 62b in the data section 42. For example, even if the frame rates of 60 fps for the first large group 811, 50 fps for the second large group 812, 24 fps for the third large group 813, and 25 fps for the fourth large group 814 are set as described above, image signals are read out from the unit groups 32 that constitute all of the large groups 81 at a predetermined period. That is, at each predetermined period, frames in which "0" is not stored as the mask information 62 in any of the unit groups 32 appear with a regularity. The control section 23 (recording control section, generation section) generates information indicating this regularity and records it in the header section 41, and does not record the mask information 62b in the data section 42. In this case, the control unit 23 may record information indicating the regularity in the header section 41, for example, in the case shown in FIG. 12, indicating that in the first frame, large group 811 is valid and large groups 812 to 814 are invalid, in the second frame, large groups 811 and 812 are valid and large groups 813 and 814 are invalid, and in the third frame, large groups 811 to 814 are valid. By recording information with regularity, it becomes unnecessary to record mask information 62b for each frame, and the size of the recording area occupied by the image file 40 can be reduced.

As described above, the control unit 23 performs imaging using the video imaging function B, and records on the memory card 25 an image file 40 in which image information 64 generated by the imaging element 22, in which imaging conditions can be set for each unit group 32, and data related to the imaging conditions for each unit group 32 (imaging condition information 61, mask information 62, Tv value map 65, Sv value map 66, Bv value map 67, etc.) are associated. This type of image file storage format is referred to as a batch storage format (image collection type) in this specification.

(5) Mixed imaging function (video and still images)
The mixed imaging function is a function that combines the still image imaging function B and the video imaging function B, and is a function that simultaneously captures a still image and a video of the same subject under different imaging conditions in one imaging session.

In the mixed imaging function, the control unit 23 further classifies the multiple unit groups 32 arranged two-dimensionally into multiple large groups 81, similar to the still image imaging function B and the video imaging function B. The control unit 23 performs video imaging for some of the large groups 81, similar to the video imaging function B. During the video imaging, the control unit 23 performs still image imaging using the remaining large groups 81, similar to the still image imaging function B. This still image imaging may be performed repeatedly at a certain cycle (automatic imaging), for example, or may be performed in response to a specific operation performed by the user (manual imaging).

Figure 14 is an explanatory diagram of the mixed imaging function. Here, four large groups 811 to 814 are assumed, with the first large group 811 capturing video at 60 fps, the second large group 812 capturing video at 50 fps, and the third and fourth large groups 813 and 814 capturing still images.

As with the video capture function B, the control unit 23 records each frame based on the fastest frame rate (e.g., 60 fps). While still image capture is not being performed, image signals are not read out from the unit groups 32 belonging to the third and fourth large groups 813 and 814. In other words, the image information 64 recorded for each frame does not include image signals from the unit groups 32 belonging to the third and fourth large groups 813 and 814 that correspond to still images. When still image capture is performed, the control unit 23 causes the image information 64 corresponding to the immediately following frame to include the image signals read out by still image capture at the timing when the still image capture is completed (the timing when the image signals are read out from the unit groups 32 belonging to the third and fourth large groups 813 and 814).

Figure 15 is a diagram showing the structure of an image file 40 created when capturing an image using the mixed capture function. The differences from the video capture function B shown in Figure 13 are described in detail below.

The identification information 60 indicates that this image file 40 was created by the mixed imaging function. The imaging condition information 61 is information indicating what kind of use exists for the unit group 32. The imaging condition information 61 in the mixed imaging function is information indicating, for example, that when this image file 40 was created, the unit group 32 had four types of uses, namely, "forming first image information 641, which is a video at 60 fps," "forming second image information 642, which is a video at 30 fps," "forming third image information 643, which is a still image," and "forming fourth image information 644, which is a still image," as well as a unique number assigned to each of these uses. For example, the numbers 1 to 4 are assigned to the uses "forming first to fourth image information 641 to 644," respectively.

The mask information 62a in the mixed imaging function is information that indicates the purpose of each unit group 32, similar to the case of the video imaging function B. That is, the mask information 62a is "information in which the numbers assigned to the imaging condition information 61 are expressed in the form of a two-dimensional map in accordance with the positions of the unit groups 32." For example, if the number "1" is entered at the coordinate (3, 5) in the mask information 62a, it can be seen that the unit group 32 at the coordinate (3, 5) belongs to the first large group 811, that is, it constitutes the first image information 641.

In the mixed imaging function, an index section 73 is further added to the header section 41. Index information 74 is recorded in the index section 73, which indicates which of the multiple blocks 70 (corresponding to multiple frames) recorded in the data section 42 stores a still image. The index information 74 contains one or more pieces of information (as many as the number of still image captures) such as "a still image is contained in the third image information 643 contained in the image information 64 of the fifth frame." The index section 73 is provided to enable quick search for still images from among the multiple blocks 70.

Note that index information 74 does not have to specify the recording position of a still image based on the number of frames. For example, it is also possible to specify the recording position of a still image based on the playback time of a video. In this case, index information 74 may be information such as "a still image is included in third image information 643 included in image information 64 at 3 minutes 15 seconds."

Each time a still image is captured during imaging using the mixed imaging function, the control unit 23 adds the frame number and time at which the still image was captured to the index unit 73 as index information 74. Note that instead of directly adding the information to the index unit 73 of the image file 40 in the memory card 25, the information may be temporarily stored in the DRAM 27, and the information in the DRAM 27 may be transferred to the index unit 73 of the image file 40 in the memory card 25 when the mixed imaging function is terminated.

The configuration of the data section 42 is the same as that of the video capture function B. That is, the data section 42 stores a block 70 for each frame in the order of capture. One block 70 consists of mask information 62, image information 64, a Tv value map 65, an Sv value map 66, a Bv value map 67, and Av value information 68. The data section 42 also stores audio information 71 along with the block 70 for each frame.

As described above, the control unit 23 performs imaging using the mixed imaging function, and records on the memory card 25 an image file 40 in which image information 64 generated by the imaging element 22, in which imaging conditions can be set for each unit group 32, and data related to the imaging conditions for each unit group 32 (imaging condition information 61, mask information 62, Tv value map 65, Sv value map 66, Bv value map 67, etc.) are associated. This type of image file storage format is referred to as a batch storage format (image collection type) in this specification.

Next, the image playback process by the control unit 23 will be described. The image playback process is a process of creating an image of a subject from an image file 40 recorded on the memory card 25 by the various imaging functions described above. The control unit 23 displays the created image, for example, on the LCD monitor 24, or records it on the memory card 25 as a file separate from the image file 40 described above.

The control unit 23 opens the image file 40 (Figs. 5, 7, 10, 13, 15) and first reads the file basic information section 43. This determines the offset and size of the mask section 44 and data section 42 of the image file 40. Next, the control unit 23 reads the identification information 60 from the mask section 44 of the image file 40. This allows the control unit 23 to recognize which imaging function created this image file 40. The subsequent processing differs for each imaging function. Therefore, the image playback processing will be explained below for each of the imaging functions mentioned above.

(1) Still image capture function A (single still image)
When the control unit 23 recognizes that the image file 40 is a file created by the still image capturing function A shown in Fig. 5, it reads out the capturing condition information 61 and the mask information 62a from the mask unit 44. This allows the control unit 23 to identify which range (which unit group 32) of the entire captured image is the main subject portion or the background portion, and to change the image composition between the main subject portion and the background portion. For example, the main subject portion can be emphasized by applying edge enhancement processing to make it sharper and blurring processing to the background portion.

Next, the control unit 23 reads out the image information 64, the Tv value map 65, the Sv value map 66, the Bv value map 67, and the Av value information 68 from the data unit 42. Then, based on the Tv value map 65, the Sv value map 66, the Bv value map 67, and the Av value information 68, the control unit 23 executes a so-called development process on the image information 64. If the image information 64 is RAW data, the control unit 23 executes a well-known demosaic process on the image information 64 that does not have color information, for example, to create an image that has color information. In addition, the control unit 23 performs image processing such as adjustment of color and brightness, noise reduction, etc., based on the Sv value map 66, etc. For example, the unit group 32 with a large Sv value (high sensitivity) is more susceptible to noise than the other unit groups 32. Therefore, the control unit 23 increases (strengthens) the intensity of noise reduction as the Sv value increases. The control unit 23 displays the image created in the above manner, for example, on the liquid crystal monitor 24 or records it on the memory card 25.

As described above, when the control unit 23 plays back the image file 40 created by the still image capture function A, it reads out the image capture condition information 61 and mask information 62a recorded in the mask section 44 before the information recorded in the data section 42, such as the image information 64. Because the mask section 44 is recorded before the data section 42, it is possible to minimize the seeks that occur during the playback process.

As described above, the data section 42 stores mask information 62b that is the same as the mask information 62a stored in the header section 41. Therefore, the control section 23 may read out the mask information 62b from the data section 42 instead of the mask information 62a.

(2) Video capture function A (single video)
When the control unit 23 recognizes that the image file 40 is a file created by the video imaging function A shown in Fig. 7, it reads out the mask information 62a from the mask section 44. The control unit 23 determines which range (which unit group 32) of the entire imaging screen is the main subject part or the background part. Next, the control unit 23 reads out the imaging condition information 61 from the mask section 44. This allows the control unit 23 to recognize the frame rates of the main subject part and the background part. Next, the control unit 23 reads out the image information 64, the Tv value map 65, the Sv value map 66, the Bv value map 67, and the Av value information 68 in order from the first block 70 of the data section 42, and creates each frame that constitutes the video.

When creating each frame, the control unit 23 first reads out the mask information 62b from the block 70. Then, it identifies which area (which unit group 32) in the frame is the main subject part and which is the background part. Thereafter, the control unit 23 performs different image processing on the main subject part and the background part, as described in the explanation of the still image capturing function A. The control unit 23 displays the video made up of the frames created in the above manner, for example, on the LCD monitor 24 or records it on the memory card 25.

As described above, when the control unit 23 plays back the image file 40 created by the video capture function A, it reads out the mask information 62b before the image information 64 and other information recorded in the block 70. Because the mask information 62b is recorded before the image information 64 and other information, it is possible to minimize the seeks that occur during the playback process.

In addition, since the mask information 62b of the first block of the data section 42 and the mask information 62a recorded in the mask section 44 are the same information, the control section 23 may be configured not to read the mask information 62a from the mask section 44.

(3) Still image capture function B (multiple still images)
10, the control unit 23 reads out the imaging condition information 61 and the mask information 62a from the mask unit 44. This allows the control unit 23 to identify how many types of still images have been captured simultaneously and which unit groups 32 compose which still images. In other words, the control unit 23 identifies how many large groups 81 exist and which large group 81 each unit group 32 belongs to.

The control unit 23 then reads out the image information 64, the Tv value map 65, the Sv value map 66, the Bv value map 67, and the Av value information 68 from the data unit 42. Then, for each large group 81, the control unit 23 executes what is called a development process on the image information 64 based on the Tv value map 65, the Sv value map 66, the Bv value map 67, and the Av value information 68 to create a still image. This creates multiple still images (for example, four still images). The control unit 23 displays the images created in this manner, for example, on the LCD monitor 24 or records them on the memory card 25.

As described above, when the control unit 23 plays back the image file 40 created by the still image capture function B, it reads out the image capture condition information 61 and mask information 62a recorded in the mask section 44 before the information recorded in the data section 42, such as the image information 64. Because the mask section 44 is recorded before the data section 42, it is possible to minimize the seeks that occur during the playback process.

As described above, the data section 42 stores mask information 62b that is the same as the mask information 62a stored in the header section 41. Therefore, the mask information 62b may be read from the data section 42 instead of the mask information 62a.

(4) Video capture function B (multiple videos)
When the control unit 23 recognizes that the image file 40 is a file created by the video capture function B shown in Fig. 13, it reads out the mask information 62a and the capture condition information 61 from the mask section 44. This allows the control unit 23 to identify how many types of videos have been captured simultaneously, which unit groups 32 compose which videos, and the frame rate of each video. That is, it identifies how many large groups 81 exist, which large group 81 each unit group 32 belongs to, and at what frame rate each large group 81 was captured. Next, the control unit 23 reads out the image information 64, the Tv value map 65, the Sv value map 66, the Bv value map 67, and the Av value information 68 in order from the first block 70 of the data section 42, and creates each frame that composes each video.

When creating each frame, the control unit 23 first reads out the mask information 62b from the block 70. The control unit 23 then identifies which large group 81 contains a pixel signal corresponding to the image information 64 of the block 70. The control unit 23 then creates a frame corresponding to each large group 81, but does not create a frame for a large group 81 whose pixel signal is not contained in the image information 64 of the block 70. The control unit 23 displays the video made up of the frames created in this manner, for example, on the LCD monitor 24 or records it on the memory card 25.

As described above, when the control unit 23 plays back the image file 40 created by the video capture function B, it reads out the mask information 62a, 62b before the image information 64 and other information recorded in the block 70. Because the mask information 62a, 62b is recorded before the image information 64 and other information, it is possible to minimize the seeks that occur during the playback process.

In addition, since the mask information 62b of the first block of the data section 42 and the mask information 62a recorded in the mask section 44 are the same information, the control section 23 may be configured not to read the mask information 62a from the mask section 44.

(5) Mixed imaging function (video and still images)
When the control unit 23 recognizes that the image file 40 is a file created by the mixed image capture function shown in Fig. 15, it reads out the mask information 62a and the image capture condition information 61 from the mask unit 44. This allows the control unit 23 to identify how many types of moving images and still images have been captured at the same time, which unit groups 32 compose which still images and moving images, and the frame rate of each moving image. That is, the control unit 23 identifies how many large groups 81 exist, whether each large group 81 is a still image or a moving image, if it is a moving image, what is its frame rate, and which large group 81 each unit group 32 belongs to. Next, the control unit 23 reads out the image information 64, the Tv value map 65, the Sv value map 66, the Bv value map 67, and the Av value information 68 in order from the first block 70 of the data unit 42, and creates each frame constituting each moving image and each still image.

When creating each frame of a video or a still image, the control unit 23 first reads out the mask information 62b from the block 70. Then, it identifies which large group 81 the pixel signal corresponding to is included in the image information 64 of the block 70. The control unit 23 then creates frames or still images corresponding to each large group 81, but does not create frames or still images for large groups 81 whose pixel signals are not included in the image information 64 of the block 70. The control unit 23 displays the video or still images made up of the frames created in this manner, for example, on the LCD monitor 24 or records them on the memory card 25.

As described above, when the control unit 23 plays back the image file 40 created by the mixed imaging function, it reads out the mask information 62a, 62b before the image information 64 and other information recorded in the block 70. Because the mask information 62a, 62b is recorded before the image information 64 and other information, it is possible to minimize the seeks that occur during the playback process.

In addition, since the mask information 62b of the first block of the data section 42 and the mask information 62a recorded in the mask section 44 are the same information, the control section 23 may be configured not to read the mask information 62a from the mask section 44.

The image playback process is described as a process of creating an image of a subject from an image file 40 recorded on the memory card 25 by the various imaging functions described above, but it may also be a process of creating a still image or a video from the image file 40 before it is recorded on the memory card 25. The control unit 23 may perform a compression process after the still image or video has been created.

Note that an electronic device other than the imaging device 10 (hereinafter referred to as a playback device) may execute the above-mentioned playback process. For example, when the memory card 25 is removed from the imaging device 10 and the memory card 25 is inserted into a playback device such as a PC external to the imaging device 10, the playback device may read the image file 40 from the memory card 25, execute the above-mentioned playback process, and play back the image. In addition, image information 64 and the like may be exchanged by performing data communication such as wireless communication between the imaging device 10 and the playback device.

According to the imaging device according to the first embodiment described above, the following advantageous effects can be obtained.
(1) The imaging element 22 has a plurality of unit groups 32 (imaging regions), and imaging conditions can be set for each unit group 32. The control unit 23 records image information 64 (image data) generated by the imaging element 22 in association with data (imaging condition data) such as imaging condition information 61, mask information 62, Tv value map 65, Sv value map 66, and Bv value map 67 relating to the imaging conditions for each unit group 32. In this manner, it is possible to provide a user-friendly imaging device 10 in which it is possible to know what imaging conditions are applied to each pixel when playing back an image file 40 that is the imaging result.

(2) Information about the imaging conditions recorded in association with the image information 64 includes, for example, information about the exposure when imaging a subject with the imaging element 22 and information about the brightness of the subject imaged by the imaging element 22. Specifically, it includes a Bv value map 67, which is information about the brightness of the subject imaged by the imaging element 22, a Tv value map 65, which is the accumulation time of charge by a photoelectric conversion unit (not shown), an Sv value map 66, which is the amplification rate by an amplifier (not shown), and the like. All of this information can be said to be information about the imaging operation of the imaging element 22. This makes it possible to perform more appropriate image processing when playing back the image file 40.

(3) The control unit 23 records information about the imaging conditions that change each time an image is captured, in association with the image information 64. In this way, appropriate information is added to each image file 40, enabling more appropriate image processing to be performed during playback.

(4) The control unit 23 records multiple pieces of information about the imaging conditions corresponding to the image information 64 in chronological order in a single image file 40. In this way, for example, when a video is recorded in the image file 40, image processing based on the information can be easily performed.

(5) In an image file 40 having a header section 41 and a data section 42 (image data section) in which image information 64 is recorded, the control section 23 records information about the imaging conditions in at least one of the header section 41 and the data section 42. In this way, for example, when playing back the image file 40, it is possible to know what imaging conditions are applied to each pixel.

(6) The control unit 23 records the imaging condition information 61 and mask information 62 related to the purpose of each of the multiple unit groups 32 in association with the image information 64. In this manner, for example, when playing back the image file 40, it is possible to know what imaging conditions are applied to each pixel.

(7) The mask information 62 includes dynamic information that changes over time. Specifically, the mask information 62 includes information indicating whether or not the image information 64 includes a pixel value corresponding to a pixel signal read from an imaging pixel 31 that belongs to a unit group 32, or information indicating to which of a plurality of different groups each of the unit groups 32 is classified. In this way, image processing that utilizes the dynamic information can be performed, for example, when playing back the image file 40.

(8) The mask information 62 includes static information that does not change over time. Specifically, it includes information that indicates the roles of the multiple unit groups 32. Furthermore, the mask information 62a includes information that indicates which of multiple different groups each of the multiple unit groups 32 was classified into at the beginning of image capture. In this way, it is possible to perform image processing using the static information, for example, when playing back the image file 40.

(9) The control unit 23 records multiple pieces of mask information 62b corresponding to multiple pieces of image information 64 in chronological order within a single image file 40. In this way, for example, when playing back the image file 40, the imaging conditions can be tracked in chronological order.

(10) In an image file 40 having a header section 41 and a data section 42 (image data section) in which image information 64 is recorded, the control section 23 records mask information 62 in at least one of the header section 41 and the data section 42. By doing so, for example, when playing back the image file 40, it is possible to know what imaging conditions are applied to each pixel.

(11) The multiple unit groups 32 include unit groups 32 that are imaged at a first frame rate and unit groups 32 that are imaged at a second frame rate that is slower than the first frame rate, and the control unit 23 records multiple pieces of image information 64 based on the first frame rate. In this way, information about all frames can be reliably recorded without omission.

(12) The control unit 23 records audio information 71 (audio data) corresponding to the imaging periods of the multiple pieces of image information 64 in association with the multiple pieces of image information 64. This makes it possible to play back video including audio.

(13) The control unit 23 records at least one of the information on the imaging pattern of the image information 64, the information on the storage method of the image information 64, and the information on the imaging conditions for each unit group 32 in the header section 41 of the image file 40, which is made up of two blocks, the header section 41 and the data section 42. By doing so, for example, when playing back the image file 40, it is possible to know what imaging conditions are applied to each pixel.

Second Embodiment
The imaging device according to the second embodiment has a similar configuration to the imaging device 10 according to the first embodiment, but the recording method of the image file 40 in the still image imaging function B, the video imaging function B, and the mixed imaging function is different from that in the first embodiment. This point will be described in detail below.

As described above, the still image capturing function B, the video capturing function B, and the mixed capturing function are functions for capturing multiple still images and videos of the same subject simultaneously in one capture. In this embodiment, the control unit 23 records the multiple captured still images and videos into multiple image files 40, rather than into a single image file 40. At this time, the control unit 23 records each of the divided and recorded image files 40 in association with each other. Therefore, although the multiple image files 40 are recorded as separate files for convenience, the information that they are files obtained by a single capture is not lost, as in the first embodiment. In other words, when handling the multiple image files 40 later, they can be recognized and handled as being obtained by a single capture, as in the first embodiment.

Figure 16 is a diagram showing a schematic of the directory structure of the memory card 25. A DCIM directory 91a exists in the root directory 90 of the memory card 25. Furthermore, a subdirectory 91b for storing images exists in the DCIM directory 91a. Each time an image is captured using the still image capture function B, the video capture function B, or the mixed image capture function, the control unit 23 creates one image capture set directory 92 in this subdirectory 91b. In other words, one image capture set directory 92 corresponds to one image capture.

In the imaging set directory 92, one management data file 93 and a subdirectory 94 are created for each use of the unit group 32. For example, if there are four uses of the unit group 32, four subdirectories 94 are created. At least one image file 40 corresponding to the use of the unit group 32 is created in each subdirectory 94. For example, if the use of the unit group 32 is video capture, only one video file 401 is recorded in the subdirectory 94 corresponding to that use. Also, if the use of the unit group 32 is still image capture, still image files 402 are recorded in the subdirectory 94 corresponding to that use for the number of times imaging is performed. Note that in the case of still image capture function B, only one still image file 402 is recorded for each use in one imaging, so one still image file 402 is recorded in each subdirectory 94.

Figure 17(a) is a diagram showing a schematic structure of a management data file 93. The management data file 93 is a file in which information relating the image files 40 recorded in each subdirectory 94 is recorded, and is made up of a file basic information section 43, a mask section 44, an index section 73, and an imaging information section 45. The file basic information section 43, the mask section 44, and the imaging information section 45 are the same as the sections of the same name in the image file 40 described in Figure 15 etc. The index section 73 records layout information 96 indicating which purpose of the unit group 32 each subdirectory 94 corresponds to.

Figure 17(b) is a diagram showing a schematic structure of a still image file 402 recorded in the subdirectory 94. In the still image file 402, mask information 62b, image information 64, a Tv value map 65, an Sv value map 66, a Bv value map 67, and Av value information 68 are recorded. The Av value information 68 is the same as that described in Figure 10, so a description of it will be omitted.

Mask information 62b, image information 64, Tv value map 65, Sv value map 66, and Bv value map 67 are information in which only values corresponding to one large group 81 are extracted from the information of the same name described in FIG. 10 and arranged in a two-dimensional form. For example, in the image file 40 described in FIG. 10, mask information 62b is "information in which the numbers assigned to the imaging condition information 61 are expressed in the form of a two-dimensional map in accordance with the positions of the unit groups 32," and the number of values included in mask information 62b is the same as the number of unit groups 32. In contrast, mask information 62b in the still image file 402 is information in which only values corresponding to the large group 81 corresponding to this subdirectory 94 are extracted from all of those values and expressed in the form of a two-dimensional map. Similarly, for image information 64, Tv value map 65, Sv value map 66, and Bv value map 67, one still image file 402 contains only values corresponding to one large group 81.

Figure 18 is a diagram showing a schematic structure of a video file 401 recorded in a subdirectory 94. In the video file 401, a block 70 for one frame is stored for each frame in the order of capture. One block 70 consists of mask information 62b, image information 64, a Tv value map 65, an Sv value map 66, a Bv value map 67, and Av value information 68. In addition to the block 70 for each frame, audio information 71 is also stored in the video file 401. The Av value information 68 is the same as that described in Figure 13, so a description thereof will be omitted.

Mask information 62b, image information 64, Tv value map 65, Sv value map 66, and Bv value map 67 are information in which only values corresponding to one large group 81 are extracted from the information of the same name described in FIG. 13 and arranged two-dimensionally. This is similar to the case of the still image file 402 described above, so a description will be omitted.

As described above, the control unit 23 associates the image information 64 generated by the imaging element 22, in which imaging conditions can be set for each unit group 32, with data related to the imaging conditions for each unit group 32 (imaging condition information 61, mask information 62, Tv value map 65, Sv value map 66, Bv value map 67, etc.) and records them on the memory card 25. Unlike the first embodiment, the management data file 93, video file 401, and still image file 402 recorded on the memory card 25 are associated with each other by the layout information 96 in the management data file 93, although they are not in the form of a single image file 40. This type of image file storage format is referred to as a split storage format in this specification.

The imaging device according to the second embodiment described above provides the same effects as the first embodiment.

Third Embodiment
In this embodiment, a remote image photography system having an imaging device 10 that generates an image file 40 described in the first and second embodiments will be described. In the remote image photography system of this embodiment, a photographer takes images in response to requests from multiple clients and transmits the taken images to the clients. In the following, a case where four clients make requests to take pictures will be described as an example.

Figure 29 is a block diagram showing a schematic configuration of the main components of a remote image capturing system 700 according to a third embodiment. The remote image capturing system 700 includes an imaging device 10, an electronic device 711 of a client A, an electronic device 712 of a client B, an electronic device 713 of a client C, and an electronic device 714 of a client D. In the following description, the electronic devices 711 to 714 will be collectively referred to as electronic device 710. The imaging device 10 and the electronic device 710 are connected via a communication network 720. The communication network 720 is, for example, a communication line such as an Internet line or a mobile or fixed telephone line.

The electronic device 710 is, for example, a computer, a television, a mobile information terminal such as a smartphone or a tablet that can connect to the communication network 720 and send and receive various information. The electronic device 710 may also be a digital camera with a communication function. The electronic devices 711-714 of each of the clients A-D transmit information on various settings (hereinafter referred to as setting information) such as the shooting conditions when shooting an image, and shooting instructions to the imaging device 10 via the communication network 720. The electronic devices 711-714 of each of the clients A-D also receive images shot by the imaging device 10 in accordance with the shooting instructions via the communication network 720.

The imaging device 10 is the imaging device 10 of the first and second embodiments described above, with a communication unit 101 for transmitting and receiving information via the communication circuit network 720. The communication unit 101 connects to the communication circuit network 720 under the control of the control unit 23. The communication unit 101 connects to the communication circuit network 720 according to a predetermined protocol, and receives setting information and shooting instructions from the electronic device 710 via the communication circuit network 720. The communication unit 101 also connects to the communication circuit network 720 according to a predetermined protocol, and transmits generated images to the electronic device 710 via the communication circuit network 720.

The control unit 23 of the imaging device 10 includes an imaging control unit 231 and an image processing unit 232. The imaging control unit 231 sets imaging conditions and controls imaging for each unit group 32 of the imaging element 22 in accordance with setting information and imaging instructions received by the communication unit 101. The image processing unit 232 generates an image file 40 using at least one of the various storage formats described in the first and second embodiments based on the imaging signal generated by the imaging element 22.
Hereinafter, the setting of the shooting conditions for the image sensor 22 and the shooting control will be described with reference to examples of setting information and shooting instructions from the clients A to D.

<Example 1>
FIG. 30 shows a schematic diagram of the subject and background to be photographed in Example 1. FIG. 30(a) shows an example of a case where an airplane in flight is photographed as the subject S1. In this case, the sky and clouds are the background S2. Assume that each of the clients A to D sets the photographing conditions A1 to D1 for the subject shown in FIG. 30(a) as follows. (1) Photographing condition A1: The subject S1 is appropriately exposed, and a still image is photographed at any timing desired by the client A. (2) Photographing condition B1: The background S2 is appropriately exposed, and a still image is photographed at any timing desired by the client B. (3) Photographing condition C1: The subject S1 and the background S2 are appropriately exposed by high dynamic range (hereinafter, HDR), and a still image is photographed at any timing desired by the client C. (4) Photographing condition D1: The background S2 is appropriately exposed, and a video is photographed.

When the imaging control unit 231 receives the above shooting conditions A1 to D1 as the setting conditions, it sets the shooting conditions A1 to D1 for the multiple unit groups 32 in each block 82 on the imaging surface 30 of the imaging element 22 as shown in FIG. 8. For example, the imaging control unit 231 sets the shooting condition A1 for the upper left unit group 23 of the block 82, sets the shooting condition B1 for the lower left unit group 32, sets the shooting condition C1 for the upper right unit group 32, and sets the shooting condition D1 for the lower right unit group 32.

When the imaging control unit 231 receives a shooting instruction from the electronic device 710, it causes the imaging element 22 to perform a shooting operation and output an imaging signal. In this case, when a shooting instruction is received from the electronic device 711 of client A, the imaging control unit 231 causes each block 82 to shoot the upper left unit group 32 under shooting condition A1 and output an imaging signal. When a shooting instruction is received from the electronic device 712 of client B, the imaging control unit 231 causes each block 82 to shoot the lower left unit group 32 under shooting condition B1 and output an imaging signal. When a shooting instruction is received from the electronic device 713 of client C, the imaging control unit 231 causes each block 82 to shoot the upper right unit group 32 under shooting condition C1 and output an imaging signal. When a shooting instruction is received from the electronic device 714 of client D, the imaging control unit 231 causes each block 82 to shoot the lower right unit group 32 under shooting condition D1 and output an imaging signal at a predetermined frame rate.

Images PA to PD captured under shooting conditions A1 to D1 are shown diagrammatically in Fig. 30(b) to Fig. 30(e), respectively. Note that in Fig. 30(b) to Fig. 30(e), subject S1 and background S2 captured with proper exposure are shown with dots, and subject S1 captured as a silhouette due to underexposure is shown with diagonal lines.

When photographed under the photographing condition A1, as shown in FIG. 30(b), the subject S1 is properly exposed, and an image PA is generated in which the background S2, such as the sky and clouds, is overexposed. When photographed under the photographing condition B1, as shown in FIG. 30(c), the background S2 is properly exposed, and an image PB is generated in which the subject S1 is underexposed and becomes a silhouette. When photographed under the photographing condition C1, as shown in FIG. 30(d), an image PC is generated in which the subject S1 and background S2 are properly exposed. When photographed under the photographing condition D1, as shown in FIG. 30(e), an image PD is generated as a video in which the background S2 is properly exposed and the subject S1 becomes a silhouette. The above images PA to PD have the same angle of view and composition as when photographed using all the pixels on the imaging surface 30 of the imaging element 22, but the resolution is 1/4.

The image processing unit 232 uses the imaging signal output from the imaging element 22 to generate an image file 40, for example, in the batch storage format or the divided storage format described in the second embodiment. For example, among the image files 40 generated in the divided storage format, the still image file 402 corresponding to the image PA is transmitted to the electronic device 711 of the client A via the communication unit 101. Among the generated image files 40, the still image file 402 corresponding to the image PB is transmitted to the electronic device 712 of the client B via the communication unit 101. Among the generated image files 40, the still image file 402 corresponding to the image PC is transmitted to the electronic device 713 of the client C via the communication unit 101. Among the generated image files 40, the video file 401 corresponding to the image PD is transmitted to the electronic device 714 of the client D via the communication unit 101. Note that in the above-mentioned example 1, when a video is being shot, image data corresponding to the image PD may be transmitted sequentially to the electronic device 714 of the client D via the communication unit 101.

In the remote image capture system 700 described in Example 1, in response to different capture requests (i.e., setting conditions and capture instructions) from each of the clients A to D, the capture conditions can be set or changed, and capture can be started and ended according to each capture request. This makes it possible to generate different images captured under the capture conditions desired by each of the clients A to D using a single imaging device 10.

<Example 2>
In Example 2, the control unit 23 controls the timing of transmitting the image file 40 from the imaging device 10 to the electronic device 710. Below, a description will be given of a case where there is no instruction to shoot a video from each of the clients A to D and a case where there is an instruction to shoot a video.

- If there is no instruction to shoot a video -
As an example of a case where there is no instruction to shoot a video, it is assumed that each of clients A to D sets shooting conditions A2 to D2 for the subject shown in FIG. 30(a) as follows. (1) Shooting condition A2: The subject S1 is appropriately exposed, and a still image is shot at any timing desired by client A. (2) Shooting condition B2: The background S2 is appropriately exposed, and a still image is shot at any timing desired by client B. (3) Shooting condition C2: The subject S1 and background S2 are appropriately exposed using high dynamic range (HDR), and a still image is shot at any timing desired by client C. (4) Shooting condition D2: The background S2 is appropriately exposed, and interval still image shooting (time lapse shooting) is performed, in which still images are shot at a predetermined time interval.

In this example, similar to the above example 1, each time a shooting instruction is received from the electronic devices 711-714 of each of clients A-D, the imaging control unit 231 controls the image sensor 22 to output an imaging signal, and the image processing unit 232 generates an image file 40. The control unit 23 controls the communication unit 101 to transmit the generated image file 40 to the electronic device 710 that sent the shooting instruction.

- If you are instructed to shoot a video -
When there is an instruction to capture a moving image, the control unit 23 controls the communication unit 101 to transmit the generated image file 40 to the electronic device 710 at a timing according to the frame rate of the moving image to be generated.
First, as a first example when an instruction to shoot a video is given, it is assumed that each of clients A to D sets shooting conditions A3 to D3 for the subject shown in Fig. 30(a) as follows. (1) Shooting condition A3: shoot video at a frame rate of 120 fps. (2) Shooting condition B3: shoot video at a frame rate of 60 fps. (3) Shooting condition C3: capture still images of subject S1 with appropriate exposure, low sensitivity, and a slow shutter speed. (4) Shooting condition D3: perform interval still image shooting (time lapse shooting) in which still images are captured at predetermined time intervals.

In this case, the control unit 23 controls the communication unit 101 to transmit to the electronic device 710 the video file 401 and still image file 402 captured and generated under the other shooting conditions B3 to D3 at the same time as transmitting the video file 401 corresponding to the video with the fastest frame rate (i.e., 120 fps under shooting condition A3). That is, the communication unit 101 transmits the image file 40 to the electronic device 710 every 1/120 [s], which is the timing at which one frame of the video file 401 is generated under shooting condition A3.

In this case, since one frame of the moving image file 401 generated under the shooting condition B3 is generated every 1/60 [s], the communication unit 101 transmits the moving image file 401 under the shooting condition B3 at one of the two times when the image file 40 is transmitted. After the still image file 402 generated under the shooting condition C3 or D3 is generated, it is temporarily stored in a storage medium such as the DRAM 27 until the communication unit 101 transmits the image file 40. The communication unit 101 transmits these still image files 402 to the electronic device 710 at the same time as transmitting the image file 40. Note that if the timing of the generation of the still image file 402 and the timing of the generation of the moving image file 401 under the shooting condition A3 are aligned, the still image file 402 is not stored in the DRAM 27 and is transmitted to the electronic device 710 by the communication unit 101.

Next, as a second example of a case where a video shooting instruction is given, it is assumed that clients A to D set shooting conditions A4 to D4 as follows for the subject to be shot shown in FIG.
(1) Shooting condition A4: Shoot video at a frame rate of 60 fps.
(2) Shooting condition B4: Video is shot at a frame rate of 50 fps.
(3) Shooting condition C5: A still image is shot of the subject S1 with appropriate exposure, low sensitivity, and a slow shutter speed.
(4) Shooting condition D5: Interval still image shooting (time lapse shooting) is performed, in which still images are shot at predetermined time intervals.

In this case, the control unit 23 controls the communication unit 101 to transmit to the electronic device 710 the video file 401 and still image file 402 captured and generated under the other shooting conditions B3 to D3 at the same time as transmitting the video file 401 corresponding to the video with the fastest frame rate (i.e., 60 fps under shooting condition A4). That is, the communication unit 101 transmits the image file 40 to the electronic device 710 every 1/60 [s], which is the timing at which one frame of the video file 401 is generated under shooting condition A4.

In this case, one frame of the video file 401 generated under the shooting condition B4 is generated every 1/50 [s]. After being generated, it is temporarily stored in a storage medium such as DRAM 27 until it is time for the communication unit 101 to transmit the video file 401 generated under the shooting condition A4. The communication unit 101 transmits the video file 401 under the shooting condition B4 to the electronic device 710 at the timing of transmitting the video file 401 under the shooting condition A4. As with the first example, the still image files 402 generated under the shooting condition C4 or D4 are temporarily stored in a storage medium such as DRAM 27 after being generated until it is time for the communication unit 101 to transmit the video file 401 under the shooting condition A4. The communication unit 101 transmits these still image files 402 to the electronic device 710 at the timing of transmitting the video file 401 under the shooting condition A4. When the timing at which the still image file 402 is generated coincides with the timing at which the video file 401 is generated under shooting condition A4, the still image file 402 is not stored in the DRAM 27, but is transmitted to the electronic device 710 by the communication unit 101. Also in the above-mentioned example 2, when a video is being shot, the image data may be transmitted sequentially to the electronic device 714 of the client D via the communication unit 101.

<Example 3>
In the remote image shooting system 700, when the image pickup element 22 of the image pickup device 10 is divided into unit groups 32 according to the number of clients, the resolution of the image generated for each client is reduced. In the image pickup device 10 of the remote image shooting system 700 of Example 3, the number of clients who instruct shooting is limited to a range that can satisfy the required resolution desired by each client. For example, when the resolution of the image pickup element 22 is 8K (7680 x 4320), if the required resolution desired by the client is a full high-definition (FHD) resolution (1920 x 1080), the control unit 23 limits the number of clients to 16. Also, if the required resolution desired by the client is a 4K resolution (3840 x 2160), the control unit 23 limits the number of clients to 4. That is, when shooting at a required resolution of 4K according to shooting instructions from four clients A to D, even if a shooting instruction is given from a new client, the control unit 23 does not accept the shooting instruction from the new client.

In the remote image photographing system 700 described in Example 3, while satisfying the resolution required by each of the clients A to D, it is possible to set or change the photographing conditions and start and end photographing in response to different photographing requests from each of the clients A to D.
It should be noted that a limit may be imposed on the resolution that can be transmitted to the electronic device 710 depending on the communication line speed of the communication circuit network 720 .

<Example 4>
In Example 4, a case where setting instructions are given from more than the upper limit of the number of clients capable of capturing different images will be described. In the following description, a case where shooting conditions are assigned to the unit groups 32 of the image sensor 22 included in the imaging device 10 in accordance with the setting instructions from clients A to D, and then a setting instruction is sent from the electronic device 715 of client E will be described as an example. As an example of this case, it is assumed that clients A to E set shooting conditions A5 to E5 as follows for the subject of shooting shown in FIG. 30(a).
(1) Shooting condition A5: Shoot video at a frame rate of 60 fps.
(2) Shooting condition B5: A still image is captured of the subject S1 with appropriate exposure at any timing desired by the client B.
(3) Shooting condition C5: A still image is shot at any timing desired by client C using automatic exposure control.
(4) Shooting condition D5: Interval still image shooting (time lapse shooting) is performed in which still images are shot at predetermined time intervals with proper exposure of the background S2.
(5) Shooting condition E5: A still image is captured at any timing desired by client E using automatic exposure control.

Of the above shooting conditions A5 to E5, shooting condition C5 and shooting condition E5 are the same except for the timing of shooting still images, i.e., in terms of performing automatic exposure control, and can therefore be considered similar shooting conditions. In this case, the imaging control unit 231 integrates shooting condition E5 into shooting condition C5. The imaging control unit 231 also uses the unit group 32 used for shooting under shooting condition C5 for shooting under shooting condition E5. In other words, the imaging control unit 231 causes the same unit group 32 to perform a shooting operation when a shooting instruction is received from the electronic device 713 of client C and when a shooting instruction is received from the electronic device 715 of client E.

When photographing conditions C5 and E5 are integrated, client E may be notified that the photographing instruction from client E will be executed using unit group 32 used for photographing by client C. In this case, control unit 23 controls communication unit 101 to transmit information regarding the integration of photographing conditions to electronic device 715 of client E. When electronic device 715 receives information regarding the integration of photographing conditions, it may notify client E that the photographing conditions will be integrated, for example, by displaying a message on a monitor if a monitor is provided, or by outputting a message from a speaker if a speaker is provided.

When shooting conditions C5 and E5 are integrated, when the image processing unit 232 generates a still image file 402 from an imaging signal output in response to a shooting instruction from client C, the control unit 23 controls the communication unit 101 to transmit the still image file 402 to the electronic device 713 of client C. When the image processing unit 232 generates a still image file 402 from an imaging signal output in response to a shooting instruction from client E, the control unit 23 controls the communication unit 101 to transmit the still image file 402 to the electronic device 715 of client E.

When a setting instruction from the electronic device 713 of client C and a setting instruction from the electronic device 715 of client E are received before the shooting conditions are set for the unit group 23 of the image sensor 22, the image capture control unit 231 assigns the shooting conditions to the unit group 32 by giving priority to the setting instruction received first. Also, when a setting instruction is received from the electronic device 715 of client E while the image sensor 22 is being photographed according to the shooting conditions C or while the still image file 402 is being generated, the image capture control unit 231 integrates the shooting conditions after the communication unit 101 finishes transmitting the still image file 402 to the electronic device 713 of client C.

In the above description, the imaging control unit 231 automatically determines the imaging conditions to be integrated with the imaging conditions E5 of the client E, but the imaging conditions to be integrated may be selectable from A5 to D5 by the client E himself. In this case, the control unit 23 controls the communication unit 101 to transmit information indicating the contents of the imaging conditions A5 to D5 to the electronic device 715 of the client E. The electronic device 715 of the client E displays the received information indicating the contents of the imaging conditions A5 to D5 on, for example, a monitor. The client E selects one of the imaging conditions A5 to D5 displayed by operating an operating member such as an operating button or a touch panel. When the imaging conditions are selected by the client E, the electronic device 715 transmits the selected imaging conditions to the imaging device 10. The imaging control unit 231 integrates the received imaging conditions with the imaging conditions E5 of the client E.

In the above description, the shooting conditions for still images are integrated, but the shooting conditions for moving images may be integrated. As an example of this case, assume that the shooting conditions A6 to D6 for the subject shown in FIG. 30(a) are set by each of clients A to D as follows, and then client E sets the following shooting condition E6.
(1) Shooting condition A6: Shoot a video at a frame rate of 60 fps with an EV value of ±0. Note that a shutter speed of 1/120 is permitted.
(2) Shooting condition B6: A still image is captured of the subject S1 with appropriate exposure at any timing desired by the client B.
(3) Shooting condition C6: A still image is shot at any timing desired by client C using automatic exposure control.
(4) Shooting condition D6: Interval still image shooting (time lapse shooting) is performed in which still images are shot at predetermined time intervals with proper exposure of the background S2.
(5) Shooting condition E6: Shoot a video at an EV value of −0.3 EV and a frame rate of 60 fps.

Of the above shooting conditions A6 to E6, shooting conditions C6 and E6 have different exposure conditions when shooting a video. In this case, the imaging control unit 231 integrates shooting condition E6 into shooting condition C6, and uses the unit group 32 used for shooting under shooting condition C6 also for shooting under shooting condition E6. The imaging control unit 231 sets the frame rate when shooting a video to 120 fps, sets the EV value to ±0 in the 2nth frame (n is a positive integer) according to shooting condition C6 of client C, and sets the EV value to -0.3 EV in the (2n+1)th frame according to shooting condition E6 of client E, and shoots the video. As a result, videos shot under shooting condition C6 and videos shot under shooting condition E6 are generated alternately, that is, at a frame rate of 60 fps.

In the remote image capture system 700 described in Example 4, even if the number of clients exceeds the upper limit, similar shooting conditions among the requested shooting conditions can be integrated to perform shooting.

<Example 5>
In Example 5, a case will be described in which an aperture change is requested by at least one of the multiple clients A to D. In the following description, a case will be taken as an example in which the shooting conditions A7 of the client A include an aperture change. In this case, the processing of the imaging control unit 231 differs depending on whether or not the shooting conditions B7 to D7 of the other clients B to D include a prohibition on changing the aperture.

If the shooting conditions B7 to D7 include prohibition of changing the aperture, the imaging control unit 23 will not capture images according to the shooting condition A7 from the client A. In other words, the imaging control unit 23 will reject the request to capture images according to the shooting condition A7.

If the photographing conditions B7 to D7 do not include prohibition of changing the aperture, the imaging control unit 23 drives the aperture (not shown) according to the photographing condition A7 of the client A. In response to photographing instructions from the clients B to D, the imaging control unit 231 changes the ISO sensitivity and shutter speed according to the changed aperture so as to obtain the exposure required by the photographing conditions B7 to D7. If the change in aperture results in an image with a degree of blur different from that desired by the clients B to D, the degree of blur is corrected by applying image processing to the image generated by the image processing unit 232.
When the shooting conditions B7 to D7 include shooting of a moving image, the image capture control unit 231 may impose a restriction on the change of the aperture of the image capture condition A7 so that the shutter speed is not extremely fast for the requested frame rate. For example, when the frame rate is 60 fps, the image capture control unit 231 may limit the change of the aperture so that the shutter speed is within a range of 1/60 [s] to 1/240 [s].

In the remote image capture system 700 described in Example 5, even if the multiple capture conditions include changing the aperture of the imaging device 10, multiple images set under multiple different capture conditions can be captured with a single imaging device 10.

<Example 6>
In Example 6, when the resolutions included in the imaging conditions of the clients are different, the number of unit blocks 32 set for each imaging condition is varied according to the resolution. As an example of this case, it is assumed that clients A and B set imaging conditions A8 and B8 as follows for the subject of imaging shown in FIG. 30(a).
(1) Shooting condition A8: Shoot video at a frame rate of 60 fps.
(2) Shooting condition B8: Shoot still images at high resolution.

When the imaging conditions A8 and B8 are received as the setting conditions, the imaging control unit 231 sets the imaging conditions A8 and B8 for the four unit groups 32 in each block 82 on the imaging surface 30 of the imaging element 22. For example, the imaging control unit 231 sets the imaging condition A8 for one unit group 32 (for example, the upper left unit group 23) of the four unit groups 32 in the block 82. The imaging control unit 231 sets the imaging condition B8 for the other three unit groups 32 of the four unit groups 32 in the block 82.

When generating a still image using an imaging signal from a unit group 32 for which shooting condition B8 is set, the image processing unit 232 may generate data corresponding to the unit group 32 at the top left of the block 82 by performing an interpolation process using data corresponding to the surrounding unit groups 32. Furthermore, when the timing of shooting a video under shooting condition A8 coincides with the timing of shooting a still image under shooting condition B8, the image processing unit 232 may use the data of the unit group 32 at the top left of the block 82 used to shoot the video as data for generating a high-resolution still image.

In the remote image capture system 700 described in Example 6, the number of unit groups 32 of the image capture elements 22 for which each capture condition is set can be varied based on the capture conditions set by the client, so that multiple different images having the resolution desired by the client can be generated by a single image capture device 10.

<Example 7>
In the remote image photographing system 700 of Example 7, before the same image is photographed, photographing conditions are set by adjusting exposure and the like for multiple clients in multiple areas of the image, and the image is photographed under the set photographing conditions. In the following explanation, an example is given in which three clients A, B, and C share the task of setting photographing conditions for one image.

In Example 7, an example will be described in which image P shown in Fig. 31(a) is divided into three areas as shown in Fig. 31(b): area S3 (area corresponding to a person's face), area S4 (area corresponding to areas other than the person's face), and area S5 (area corresponding to the background). A case will be described in which client A sets shooting condition A9 for area S3 in image P, client B sets shooting condition B9 for area S4, and client C sets shooting condition C9 for area S5.

The imaging control unit 231 causes the imaging element 22 to perform imaging operations and output imaging signals without setting different imaging conditions for each unit group 32. The image processing unit 232 uses the output imaging signals to generate three identical images for clients A, B, and C, i.e., image P shown in FIG. 31(a), and the communication unit 101 transmits image P to the electronic devices 711 to 713 of clients A to C, respectively.

Each of the electronic devices 711-713 displays the received image P on a monitor or the like. While checking the image displayed on the monitor, clients A-C individually adjust the exposure for the areas S3-S5 for which they are responsible. In this case, electronic device 711 extracts area S3 from image P using subject area recognition processing such as well-known face recognition or object recognition. Electronic device 712 extracts area S4 from image P using subject area recognition processing. Electronic device 713 extracts area S5 from image P using subject recognition processing.

Each of clients A to C adjusts the exposure of the areas S3 to S5 for which they are responsible while checking the status of the entire image, such as changes in brightness. Various parameters representing the exposure of area S3 adjusted by client A are associated with area S3 and transmitted by electronic device 711 as shooting condition A. Various parameters representing the exposure of area S4 adjusted by client B are associated with area S4 and transmitted by electronic device 712 as shooting condition B. Various parameters representing the exposure of area S5 adjusted by client C are associated with area S5 and transmitted by electronic device 714 as shooting condition C.

When the communication unit 101 receives the photographing conditions A to C, the imaging control unit 231 sets the received photographing conditions A to C for the unit groups 32 at positions corresponding to the areas S3 to S5 on the imaging surface 30 of the imaging element 22. The imaging control unit 231 causes the imaging element 22 to perform a photographing operation and output an imaging signal in accordance with the received photographing instruction. The image processing unit 232 generates an image file 40 using the output imaging signal.
Note that a limit may be placed on the amount of adjustment so that the exposure adjustment for each of the regions S3 to S5 by each of the clients A to C is not excessive. For example, in the above example, the amount of adjustment for the region S3 corresponding to the person's face may be limited to +2/3 EV of the exposure for the region S4 corresponding to the portion other than the person's face.

In the example shown in FIG. 31, multiple clients adjust the exposure for an image in which one person is captured, but multiple clients may each adjust the exposure for a different person being photographed for an image in which multiple people are captured. This case will be described below.

Figure 32(a) shows an example of image P in which three people are photographed, while Figure 32(b) shows image P divided into four areas: area S6 (corresponding to the person on the left side of the figure), area S7 (corresponding to the person in the center of the figure), area S8 (corresponding to the person on the right side of the figure), and background area S9. We will explain the case where client A sets shooting condition A10 for area S6 in image P, client B sets shooting condition B10 for area S7, and client C sets shooting condition C10 for area S8.

In the same manner as described above, when the electronic devices 711-713 of each of clients A-C receive the same image P, they display the image of the received image file on a monitor or the like. While checking the image displayed on the monitor, clients A-C individually adjust the exposure for the areas S6-S8 for which they are responsible. In this case as well, electronic device 711 extracts area S6 from image P using subject area recognition processing such as well-known face recognition or object recognition. Electronic device 712 extracts area S7 from image P using subject area recognition processing. Electronic device 713 extracts area S8 from image P using subject recognition processing.

Each of clients A to C adjusts the exposure of the areas S6 to S8 for which it is responsible while checking the status of the overall image, such as changes in luminance. Various parameters representing the exposure of area S6 adjusted by client A are associated with area S6 and transmitted by electronic device 711 as shooting condition A. Various parameters representing the exposure of area S7 adjusted by client B are associated with area S7 and transmitted by electronic device 712 as shooting condition B. Various parameters representing the exposure of area S8 adjusted by client C are associated with area S8 and transmitted by electronic device 714 as shooting condition C. Thereafter, in the same manner as described above, the imaging control unit 231 sets the shooting conditions for the unit group 32, the imaging element 22 performs a shooting operation, and the image processing unit 232 generates an image file 40.

Alternatively, as shown in FIG. 32(c), the image P may be divided into areas S61-S8 corresponding to people, and areas S61-S82 corresponding to non-faces. In this case, each of the clients A-C may adjust the exposure of the areas S61-S81 and S62-82 so that the areas S61-S81, i.e., the faces of the people, are exposed as desired. By adjusting the exposure of the areas S61-S81 corresponding to the faces of the people, the exposure of the areas S62-S82 corresponding to non-faces may be automatically adjusted according to the exposure of the areas S61-S81 after adjustment. By adjusting the exposure of the areas S62-S82 corresponding to non-faces, the exposure of the areas S61-S81 corresponding to the faces of the people may be automatically adjusted according to the exposure of the areas S62-S82 after adjustment.

In the example described with reference to FIG. 32, if the image generated by the imaging device 10 includes the face of a person for whom clients A to C wish to adjust the exposure, clients A to C may be automatically associated with areas S6 to S8. In this case, electronic devices 711 to 713 on the clients A to C side have in advance information, such as a facial image, about the face of the person desired as the subject of exposure adjustment. Electronic devices 711 to 713 detect the facial image in the received image, for example, using known template matching, etc.

The above-described remote image photography system 700 can be used, for example, as a surveillance system having a surveillance camera. In this case, the multiple clients are, for example, a security company and the owner of the house in which the surveillance camera is installed. It is conceivable that the security company and the owner of the house each wish to focus on different areas of the generated image as surveillance targets. In this case, it becomes possible for the security company and the owner of the house to set different photography conditions so that the desired images can be obtained for the areas each wish to focus on.

In addition, the above-mentioned remote image photography system 700 has been described as a system in which the imaging device 10 and the electronic device 710 transmit and receive images and various information via the communication circuit network 720, but is not limited to this example. For example, the remote image photography system 700 may have a server device connected to the communication circuit network 720 and transmits and receives images and information. In this case, images and various information are transmitted and received between the imaging device 10 and the electronic device 710 via the server device. In addition, the server device may generate an image file 40 using an imaging signal output by the imaging device 10 in the same manner as the image processing unit 232.

In the remote image capture system 700 in Example 7, different clients can adjust the exposure for each of the multiple areas S6 to S8 of one image P, set the capture conditions for each area S6 to S8, and capture the images. This makes it possible to share the work required to set the capture conditions among multiple clients, which contributes to improving work efficiency.

The following modifications are also within the scope of the present invention, and one or more of the modifications may be combined with the above-described embodiment.

(Variation 1)
In the first embodiment, it has been described that the first image information 641 and the second image information 642 are created when the image file 40 is played back, but they may be recorded in advance in the image file 40. In other words, in the second embodiment, the moving images and still images recorded as separate files in separate subdirectories 94 for each large group 81 may be recorded in a single image file 40. In this case, one frame of data recorded in the image file 40 corresponds to each large group 81.

For example, consider the case where two videos (a first video and a second video) that were recorded in two separate files in the second embodiment are recorded in a single image file 40. In this case, data relating to the first frame, second frame, third frame, etc. of the first video can be recorded in chronological order starting from the top of the data section 42, and then data relating to the first frame, second frame, third frame, etc. of the second video can be recorded in chronological order. In this way, the load of the playback process can be reduced.

As a different recording method, data regarding each frame of the first video and data regarding each frame of the second video can be recorded in chronological order of each frame. That is, each frame of the two videos can be recorded in chronological order of capture, such as the first frame of the first video, the first frame of the second video, the second frame of the first video, etc. In this way, the load of the recording process can be reduced.

When recording the first image information 641, the second image information 642, etc. in the image file 40 as described above, the mask information 62b can be reduced by using the following recording method. Note that in the following explanation, an example will be given in which the image file 40 is generated by the mixed imaging function as shown in FIG. 14. Also, FIG. 26 shows a schematic diagram of the structure of the image file 40 generated at this time. Note that one frame's worth of data, that is, a set of one block 70 and one piece of audio information, is referred to as frame data 7 for convenience.

In the image file 40, frame data 7 including image information 64 corresponding to the first to fourth large groups 81 are recorded individually in chronological order. Naturally, the Tv value map 65 and the like included in each frame data 7 are also in a state corresponding to each image information. When imaging is performed using the mixed imaging function shown in FIG. 14, the image information 64 captured in the first frame (i.e., the first image information 641) is recorded in the data section 42 as the first frame data 7. The image information 64 captured in the second frame (i.e., the first image information 641 and the second image information 642) is recorded as the second and third frame data 7, respectively. The image information 64 captured in the third frame (i.e., the first image information 641, the second image information 642, and the third image information 643) is recorded as the fourth, fifth, and sixth frame data 7, respectively.

FIG. 26 shows a schematic structure of the generated image file 40. When the image information 64 is recorded as described above, instead of the mask information 62b, image identification information 621 corresponding to the unique number assigned as the identification information 60 recorded in the mask section 44 of the header section 41 and a next image address 622 indicating the recording position of the next frame data 7 are recorded in each frame data 7. That is, as in the case where only the data section 42 of the image file 40 is shown in FIG. 27, the number "1" indicating that it is the first image information 641 is recorded in the image identification information 621 of the first and second frame data 7. The number "2" indicating that it is the second image information 642 is recorded in the image identification information 621 of the third frame data 7. As a result, the size of the recording area can be reduced compared to the case where the mask information 62b is recorded in the data section 42 as in the embodiment.
Incidentally, the next image address 622 may indicate the recording position of the frame data 7 in which the next image having the same image identification information 621 is recorded, instead of indicating the recording position of the next frame data 7 .

When reproducing the image file 40 as described above, the control unit 23 searches for the image information 64 to be reproduced while referring to the image identification information 621 and the next image address 622 in the data unit 42. In FIG. 27, when searching for image information 642, the control unit 23 refers to the image identification information 621 of the first frame data 7, determines that it is not image information 642, and skips to the second frame data 7 by referring to the next image address 622. Since the image identification information 621 in the second frame data 7 also indicates that it is not image information 642, the control unit 23 skips to the third frame data 7 by referring to the next image address 622. Since the number "2" indicating that image information 642 is recorded is recorded in the image identification information 621 of the third frame data 7, the control unit 23 reads the image information 642 from this frame data 7. Therefore, when reading an image, image information 64, Tv value map 65, Sv value map 66, Bv value map 67, and Av value information 68 that are different from the desired image can be skipped, thereby shortening the time required to search for the desired image.

(Variation 2)
In the explanation of the first embodiment, it has been stated that the image information 64 and various map information are recorded in the data section 42 of the image file 40 created by the moving image capturing function B and the mixed capturing function according to the arrangement of the unit groups 32 in the imaging element 22. However, the image information 64 and various map information may be recorded in an arrangement different from the arrangement of the unit groups 32. This point will be described in detail below.

Figure 19 is an explanatory diagram of the second modification. Here, the unit groups 32 are classified into four large groups 81, as in Figure 8(b). However, the image information 64 generated by the control unit 23 after that does not have the image signals arranged according to the arrangement of the unit groups 32. Specifically, the image signals are aggregated for each large group 81, and then the image information 64 is generated by linking them. For example, when the image information 64 is divided into four 2 x 2 regions, the upper left region aggregates the image signals from the unit groups 32 belonging to the first large group 81, the lower left region aggregates the image signals from the unit groups 32 belonging to the second large group 81, the upper right region aggregates the image signals from the unit groups 32 belonging to the third large group 81, and the lower right region aggregates the image signals from the unit groups 32 belonging to the fourth large group 81.

As described above, when changing the arrangement of the imaging signals in the image information 64, the arrangements of the Tv value map 65, Sv value map 66, mask information 62, etc. must also be changed accordingly.

The arrangement of the image information 64 may also be changed in other ways. In other words, as long as the arrangement in the image information 64 corresponds to the arrangement in the other information relating to the imaging conditions (mask information 62, etc.) in the image file 40, the arrangement itself may be of any type.

(Variation 3)
In the moving image capturing function B or the mixed capturing function, the use of the unit group 32 may be changed for each frame. For example, as shown in FIG. 20, in odd-numbered frames, the unit group 32 is classified into the first to fourth large groups 81, respectively, so that image information 64 including four pieces of image information 641, 642, 643, and 644 with different capturing conditions is obtained. Then, in even-numbered frames, the unit group 32 is classified only into the fifth large group 81, so that only a single piece of image information 64 is obtained. In other words, a plurality of images with different capturing conditions and a relatively small number of pixels and a single image with a relatively large number of pixels may be captured in a time-division manner. In addition, this modified example 3 can be applied to the modified examples 1 and 2 described above.

(Variation 4)
In the moving image capturing function B or the mixed image capturing function, one unit group 32 may have multiple uses. For example, as shown in Fig. 21, the unit groups 32 may be classified into first to fourth major groups 81, and all the unit groups 32 may also be classified into a fifth major group 81. In this case, when the image file 40 is reproduced (developed, etc.) according to the former classification, image information 64 including four pieces of image information 641, 642, 643, and 644 is obtained, and when the image file 40 is reproduced (developed, etc.) according to the latter classification, a single image information 64 having a larger number of pixels is obtained.

(Variation 5)
In the explanation of the still image capturing function B, it was stated that a unit group 32 assigned the number "0" on the mask information 62 is not used for capturing images, and the image information 64 recorded in the data section 42 does not include any information relating to that unit group 32. However, in the still image capturing function A and the video capturing function A, the number "0" may also have the same meaning.

In addition, in the mask information 62 of the header section 41, the number "0" may represent that the number is not used for imaging. For example, in still image imaging function B or video imaging function B, if the entire imaging screen is divided into four 2 x 2 unit groups 32 and different uses are assigned to each of the four unit groups 32, if the vertical number (number of rows) of the unit groups 32 is odd, there will be one extra row. In such a case, the mask information 62 recorded in the header section 41 may assign the number "0" to the extra row, assuming that the extra row is not used for imaging.

Note that the number "0" mentioned above is just an example, and other numbers may be treated in the same way as the number "0" mentioned above.

(Variation 6)
The structure of the image file 40 may be different from that of each of the above-mentioned embodiments. Furthermore, the information on the imaging conditions recorded in the image file 40 may be different from that described in the first embodiment and the like. For example, the recording of some information, such as the Sv value map 66, may be omitted. Conversely, information other than that described above may be added. Furthermore, the recording form may be different from that of the above-mentioned embodiments. For example, the Av value information 68 may be recorded as an Av value map in which the Av values for each unit group 32 are arranged two-dimensionally, similar to the Tv value and Sv value. Further information different from the various information described above may be recorded in the data section 42. For example, distance information to the subject measured by a well-known distance measuring technique may be recorded in the data section 42. Furthermore, this distance information may be a so-called depth map in which the distances to the subject measured for each unit group 32 are arranged two-dimensionally. As another example, information on the state of the imaging optical system 21 (for example, focal length, etc.) may be recorded. Furthermore, taking into consideration that such information changes during shooting of a video, such information may be recorded for each frame.

Alternatively, recording of the mask information 62 may be omitted. In this case, during playback, the control unit 23 uses the Tv value map 65, Sv value map 66, and Bv value map 67 to generate information (mask equivalent information) that indicates the use (purpose, role) of each unit group 32 that corresponds to the mask information 62. For example, the control unit 23 refers to the Tv value map 65 and/or the Sv value map 66. The control unit 23 considers the coordinates (x, y) on the referenced Tv value map 65 where the same shutter speed is stored to be the same area.

For example, as shown in FIG. 28(a), when there are regions R1 and R2 in the Tv value map 65 that have different shutter speeds, the control unit 23 determines that the regions on the imaging screen 50 corresponding to the regions R1 and R2 correspond to the main subject region 52 and the background region 53, respectively. For example, when the shutter speed in region 1 is faster than the shutter speed in region 2, the control unit 23 regards the region on the imaging screen 50 corresponding to region R1 as the main subject region 52, and regards the region on the imaging screen 50 corresponding to region R2 as the background region 53. Then, the control unit 23 generates mask equivalent information 63 shown in FIG. 28(b). In FIG. 28(b), "1" is stored at the position of the unit group 32 included in the main subject 52, and "2" is stored at the position of the unit group 32 included in the background region 53.

When the Sv value map 66 is used, the control unit 23 regards the coordinates (x, y) where the same ISO sensitivity is stored as the same region, and can generate the mask equivalent information 63 in a similar manner. When both the Tv value map 65 and the Sv value map 66 are used, the control unit 23 can regard the coordinates where the same shutter speed is stored and the coordinates where the same ISO sensitivity is stored as the same region, and can perform detailed region division. Furthermore, by referring to the Bv value map 67 in addition to the Tv value map 65 and/or the Sv value map 66, the control unit 23 can use the distribution of subject brightness to divide the regions, and can further improve the accuracy of region division.

Furthermore, the control unit 23 may generate the mask equivalent information 63 by referring to the Bv value map 67. In this case, the control unit 23 regards the coordinates (x, y) in which the subject brightness within a predetermined range is stored as the same area, and generates the mask equivalent information 63 in the same manner as described above. When the above-mentioned distance information, i.e., the depth map, is recorded, the control unit 23 can generate the mask equivalent information 63 using the depth map and the Bv value map 67. In this case, the control unit 23 regards the coordinates (x, y) in which the same subject distance is stored as the same area within the area divided into the same area by referring to the Bv value map 67, and generates the mask equivalent information 63. For example, for an area with low subject brightness, it is possible to divide the area into an area corresponding to the black subject itself and an area corresponding to the shadow of the subject.

In this manner, the control unit 23 can generate mask-equivalent information based on the imaging conditions during playback even if the mask information 62 is not recorded in the image file 40. This eliminates the need to record the mask information 62 in the image file 40, and the recording area occupied by the image file 40 can be reduced.
In the above description, the Tv value map 65, the Sv value map 66, and the Bv value map 67 are used to generate the mask-equivalent information 63 equivalent to the mask information 62 generated during the still image capture function A or the video capture function A, but the present invention is not limited to this example. For example, the control unit 23 can refer to the Tv value map 65 and the Sv value map 66, and regard unit groups 32 corresponding to coordinates in which the same shutter speed is stored and coordinates in which the same ISO sensitivity is stored as constituting the same image information 64, thereby generating the mask-equivalent information 63 equivalent to the mask information 62 generated during the still image capture function B, the video capture function B, or the mixed capture function.

(Variation 7)
In each of the above-described embodiments, an imaging device that is a single electronic device having an imaging element 22 and a control unit 23 has been described, but the present invention is not limited to such embodiments. For example, the present invention can also be applied to an electronic device that controls an externally provided imaging element 22. Below, a detailed description will be given of an embodiment in which an imaging unit 1001 equipped with an imaging element 22 is controlled from an external device.

22 is a block diagram showing a schematic configuration of an imaging system according to the seventh modified example. The imaging system 1000 shown in FIG. 22 is composed of an imaging unit 1001 and an electronic device 1002. The imaging unit 1001 includes the imaging optical system 21 and the imaging element 22 described in the first embodiment, and further includes a first communication unit 1003. The electronic device 1002 includes the control unit 23, the liquid crystal monitor 24, the memory card 25, the operation unit 26, the DRAM 27, the flash memory 28, and the sound recording unit 29 described in the first embodiment, and further includes a second communication unit 1004. The first communication unit 1003 and the second communication unit 1004 can perform bidirectional data communication, for example, by well-known wireless communication technology or optical communication technology. The imaging unit 1001 and the electronic device 1002 may be connected by a wired cable or the like, and the first communication unit 1003 and the second communication unit 1004 may perform bidirectional data communication.

In the imaging system 1000 according to the seventh modification, the control unit 23 controls the imaging element 22 by data communication via the second communication unit 1004 and the first communication unit 1003. For example, by transmitting and receiving predetermined control data to and from the imaging unit 1001, different imaging conditions can be set for each unit group 32, and imaging signals can be read out from each unit group 32.

As described above, in the imaging system 1000, it is the control unit 23 that controls each unit group 32. The electronic device 1002 does not have an imaging element 22, but performs control similar to that of the first embodiment by controlling the imaging element 22 (imaging unit 1001) provided outside the electronic device 1002. In other words, the present invention can be applied to electronic devices that do not have an imaging element 22.

(Variation 8)
In order to reduce the data amount of the image information 64, the image information 64 may be compressed and recorded using a known lossless compression technique. The image information 64 may also be recorded in the form of a difference value between adjacent pixels. For example, at a position where the pixel value (image capture signal) of a certain pixel is recorded, a difference value between the pixel on the left side of the pixel may be recorded. Alternatively, a difference value between the pixel value of a certain pixel and the average value of the pixel values of all pixels in a predetermined area may be recorded, or a difference value between the pixel value of a certain pixel and the average value of all pixels may be recorded.

In addition, in the case of video, the amount of data can be further reduced by recording the difference value from the pixel value at the same position in the previous frame. Alternatively, a pixel value can be recorded only when the value is different from the pixel value at the same position in the previous frame, and if the pixel value is the same as in the previous frame, the pixel value is not recorded. This can also be applied to imaging conditions (Sv value, Tv value, etc.). For example, for a certain unit group 32, if the Sv value is the same as in the previous frame, the Sv value may not be recorded.

When recording image information 64 in the form described above, it is necessary to perform a process to restore the original pixel values from these forms during playback (development).

(Variation 9)
In the case of the video imaging function B or the mixed imaging function, when the frame rates of the large groups 81 are different, the control unit 23 records each frame based on the fastest frame rate, but this is not limited to this example. For example, the control unit 23 may record each frame at a frame rate corresponding to the least common multiple of the respective frame rates. As shown in FIG. 14, when the first large group 811 captures a video at 60 fps and the second large group 812 captures a video at 50 fps, the frames are recorded at 300 frames/second, which is the least common multiple of 60 fps and 50 fps. That is, the image information 64 based on the imaging signal from the large group 811 set to 60 fps is recorded every 5 frames, and the image information 64 based on the imaging signal from the second group 812 set to 50 fps is recorded every 6 frames.

(Variation 10)
In the above-described embodiments, examples in which the present invention is applied to a lens-integrated camera have been described, but the present invention can also be applied to, for example, a lens-interchangeable camera. Furthermore, the present invention is not limited to cameras, and can also be applied to electronic devices with cameras, such as PCs, mobile phones, smartphones, and tablets.

As long as the characteristics of the present invention are not impaired, the present invention is not limited to the above-described embodiment, and other forms that are conceivable within the scope of the technical concept of the present invention are also included within the scope of the present invention.

10...imaging device, 21...imaging optical system, 22...imaging element, 23...control unit, 24...liquid crystal monitor, 25...memory card, 26...operation unit, 27...DRAM, 28...flash memory, 29...audio recording unit, 101...communication unit, 231...imaging control unit, 232...image processing unit, 700...remote image capture system, 710...electronic device

Claims (1)

an imaging element having an imaging area including a first area in which an imaging operation is performed under a first imaging condition and a second area in which an imaging operation is performed under a second imaging condition;
an electronic device comprising: a control unit that records in a recording unit a file having first image data of a first subject captured in the first area, second image data of a second subject captured in the second area, first data relating to the first imaging conditions, second data relating to the second imaging conditions, third data relating to a use of the first area, and fourth data relating to a use of the second area.

Images