WO2019065917A1

WO2019065917A1 - Moving-image compression device, electronic apparatus, and moving-image compression program

Info

Publication number: WO2019065917A1
Application number: PCT/JP2018/036131
Authority: WO
Inventors: 大作小宮; 直樹關口
Original assignee: 株式会社ニコン
Priority date: 2017-09-29
Filing date: 2018-09-27
Publication date: 2019-04-04

Abstract

A moving-image compression device for compressing moving-image data that includes a plurality of frames generated from the output of an imaging element having a plurality of imaging regions in which different resolutions are settable. A setting unit for setting, on the basis of the resolution of an image region to be predicted in a plurality of image regions corresponding to the plurality of imaging regions in a frame to be predicted among the plurality of frames, a prediction processing unit in which unit the image region to be predicted is predicted. A prediction unit for predicting the image region to be predicted, on the basis of the prediction processing unit that was set by the setting unit. An encoding unit for encoding the frame to be predicted, using the result of prediction by the prediction unit.

Description

Video compression apparatus, electronic device, and video compression program

Capture by reference

This application claims priority to Japanese Patent Application No. 2017-192109, which is a Japanese application filed on Sep. 29, 2017, and is incorporated into the present application by reference to the contents thereof.

The present invention relates to a video compression apparatus, an electronic device, and a video compression program.

There has been proposed an electronic device provided with an imaging device (hereinafter, referred to as a stacked imaging device) in which a back side illumination type imaging chip and a signal processing chip are stacked (see Patent Document 1). The stacked imaging device is stacked such that the back side illumination type imaging chip and the signal processing chip are connected via the microbumps in each predetermined area. However, when it is possible to set a plurality of resolutions in the imaging region in the layered imaging element, since frames imaged at a plurality of resolutions are output, such moving image compression of the frames is not conventionally considered.

Unexamined-Japanese-Patent No. 2006-49361

A moving picture compression apparatus according to one aspect of the technology disclosed in the present application is a moving picture compression apparatus that compresses moving picture data including a plurality of frames generated from an output of an imaging device having a plurality of imaging areas in which different resolutions can be set. A prediction processing unit for predicting the prediction target image area based on the resolution of the prediction target image area in the plurality of image areas corresponding to the plurality of imaging areas in the prediction target frame among the plurality of frames; A setting unit configured to set, a prediction unit configured to predict the prediction target image area based on a prediction processing unit set by the setting unit, and a code encoding the prediction target frame using a prediction result by the prediction unit And a conversion unit.

An electronic device, which is one aspect of the technology disclosed in the present application, includes an imaging device having a plurality of imaging regions in which different resolutions can be set, and the plurality of frames generated from the output of the imaging device in the prediction target frame. A setting unit configured to set a prediction processing unit for predicting the prediction target image region based on the resolution of a prediction target image region among a plurality of image regions corresponding to a plurality of imaging regions; and the prediction set by the setting unit A prediction unit that predicts the prediction target image area based on a processing unit, and an encoding unit that encodes the prediction target frame using a prediction result of the prediction unit.

A moving picture compression program according to an aspect of the technology disclosed in the present application causes a processor to compress moving picture data including a plurality of frames generated from an output of an imaging element having a plurality of imaging areas in which different resolutions can be set. The moving image compression program, wherein the processor is configured to, based on a resolution of a prediction target image area in a plurality of image areas corresponding to the plurality of imaging areas in the prediction target frame among the plurality of frames, the prediction target image A setting process for setting a prediction process unit for predicting an area; a prediction process for predicting the prediction target image area based on the unit area set by the setting process; and the prediction using the prediction result by the prediction process And an encoding process of encoding the target frame.

FIG. 1 is a cross-sectional view of a stacked imaging device. FIG. 2 is a diagram for explaining the pixel array of the imaging chip. FIG. 3 is a circuit diagram of the imaging chip. FIG. 4 is a block diagram showing an example of the functional configuration of the imaging device. FIG. 5 is an explanatory view showing an example of the block configuration of the electronic device. FIG. 6 is an explanatory view showing a configuration example of a moving image file. FIG. 7 is an explanatory view showing the relationship between the imaging plane and the subject image. FIG. 8 is an explanatory view showing a specific configuration example of a moving image file. FIG. 9 is an explanatory view showing an example of imaging on an imaging plane in which different resolutions are set. FIG. 10 is an explanatory view showing a prediction example of 16 × 16 prediction. FIG. 11 is an explanatory view showing a prediction example of 4 × 4 prediction. FIG. 12 is a block diagram showing a configuration example of the control unit shown in FIG. FIG. 13 is a block diagram showing a configuration example of the compression unit. FIG. 14 is a flowchart illustrating an example of a preprocessing procedure by the preprocessing unit. FIG. 15 is a flowchart illustrating an example of an image processing procedure by the image processing unit. FIG. 16 is a flowchart of an example of the intra-frame prediction processing procedure by the intra-frame prediction processing unit.

<Configuration Example of Imaging Element>
First, a stacked imaging device mounted on an electronic device will be described. The layered imaging device is described in Japanese Patent Application No. 2012-139026 filed by the applicant of the present application. The electronic device is, for example, an imaging device such as a digital camera or a digital video camera.

FIG. 1 is a cross-sectional view of a stacked imaging device 100. FIG. A stacked imaging device (hereinafter simply referred to as “imaging device”) 100 processes a back-illuminated imaging chip (hereinafter simply referred to as “imaging chip”) 113 that outputs a pixel signal corresponding to incident light, and the pixel signal. A signal processing chip 111 and a memory chip 112 for storing pixel signals are provided. The imaging chip 113, the signal processing chip 111, and the memory chip 112 are stacked and electrically connected to each other by the bump 109 having conductivity such as Cu.

As shown in FIG. 1, incident light is mainly incident in the Z-axis plus direction indicated by a white arrow. In the present embodiment, in the imaging chip 113, the surface on which incident light is incident is referred to as the back surface. Further, as indicated by the coordinate axes, the left direction in the drawing, which is orthogonal to the Z axis, is taken as the plus direction of the X axis, and the near direction in the drawing, which is orthogonal to the Z axis and the X axis, is taken as the plus direction. In the following several figures, coordinate axes are displayed so that the orientation of each figure can be known with reference to the coordinate axes in FIG.

One example of the imaging chip 113 is a backside illuminated MOS (Metal Oxide Semiconductor) image sensor. The PD (photodiode) layer 106 is disposed on the back side of the wiring layer 108. The PD layer 106 is two-dimensionally arranged, and includes a plurality of PDs 104 which store charges corresponding to incident light, and a transistor 105 provided corresponding to the PDs 104.

A color filter 102 is provided on the incident side of incident light in the PD layer 106 via a passivation film 103. The color filter 102 has a plurality of types that transmit different wavelength regions, and has a specific arrangement corresponding to each of the PDs 104. The arrangement of the color filters 102 will be described later. The combination of the color filter 102, the PD 104, and the transistor 105 forms one pixel.

A microlens 101 is provided on the color filter 102 on the incident side of the incident light corresponding to each pixel. The microlenses 101 condense incident light toward the corresponding PDs 104.

The wiring layer 108 has a wiring 107 for transmitting the pixel signal from the PD layer 106 to the signal processing chip 111. The wiring 107 may be a multilayer, and passive elements and active elements may be provided.

A plurality of bumps 109 are disposed on the surface of the wiring layer 108. The plurality of bumps 109 are aligned with the plurality of bumps 109 provided on the opposite surface of the signal processing chip 111, and the imaging chip 113 and the signal processing chip 111 are aligned by pressure or the like. The bumps 109 are joined to be electrically connected.

Similarly, a plurality of bumps 109 are disposed on the surfaces facing each other of the signal processing chip 111 and the memory chip 112. These bumps 109 are aligned with each other, and the signal processing chip 111 and the memory chip 112 are pressurized or the like, whereby the aligned bumps 109 are joined and electrically connected.

The bonding between the bumps 109 is not limited to Cu bump bonding by solid phase diffusion, and micro bump bonding by solder melting may be employed. Also, for example, about one bump 109 may be provided for one block described later. Therefore, the size of the bumps 109 may be larger than the pitch of the PDs 104. Further, in the peripheral area other than the pixel area in which the pixels are arranged, bumps larger than the bumps 109 corresponding to the pixel area may be provided.

The signal processing chip 111 has TSVs (silicon through electrodes) 110 which mutually connect circuits respectively provided on the front and back surfaces. The TSVs 110 are preferably provided in the peripheral area. The TSV 110 may also be provided in the peripheral area of the imaging chip 113 and the memory chip 112.

FIG. 2 is a diagram for explaining the pixel arrangement of the imaging chip 113. As shown in FIG. In particular, a state in which the imaging chip 113 is observed from the back surface side is shown. (A) is a top view which shows typically the imaging surface 200 which is the back surface of the imaging chip 113, (b) is the top view which expanded the partial area 200a of the imaging surface 200. FIG. As shown in (b), a large number of pixels 201 are two-dimensionally arranged on the imaging surface 200.

Each of the pixels 201 has a color filter (not shown). The color filter consists of three types of red (R), green (G), and blue (B), and the notation “R”, “G”, and “B” in (b) is a color filter that the pixel 201 has Represents the type of As shown in (b), on the imaging surface 200 of the imaging element 100, the pixels 201 provided with such color filters are arranged according to a so-called Bayer arrangement.

The pixel 201 having a red filter photoelectrically converts light in the red wavelength band of incident light and outputs a light reception signal (photoelectric conversion signal). Similarly, the pixel 201 having a green filter photoelectrically converts light in the green wavelength band among incident light and outputs a light reception signal. Further, the pixel 201 having a blue filter photoelectrically converts light in the blue wavelength band among incident light and outputs a light reception signal.

The image sensor 100 is configured to be individually controllable for each unit group 202 including a total of four pixels 201 of adjacent 2 pixels × 2 pixels. For example, when charge storage is started simultaneously for two unit groups 202 different from each other, charge readout is performed 1/30 seconds after charge storage start in one unit group 202, that is, light reception signals are read, In the unit group 202, charge readout is performed 1/15 seconds after the start of charge accumulation. In other words, the imaging device 100 can set different exposure times (charge accumulation time, so-called shutter speed) for each unit group 202 in one imaging.

The imaging device 100 can make the amplification factor (so-called ISO sensitivity) of an imaging signal different for each unit group 202 besides the above-described exposure time. The imaging device 100 can change the timing to start the charge accumulation and the timing to read out the light reception signal for each unit group 202. That is, the imaging element 100 can change the frame rate at the time of moving image capturing for each unit group 202.

To summarize the above, the imaging device 100 is configured to be able to make the imaging conditions such as the exposure time, the amplification factor, the frame rate, and the resolution different for each unit group 202. For example, a reading line (not shown) for reading an imaging signal from a photoelectric conversion unit (not shown) of the pixel 201 is provided for each unit group 202, and the imaging signal can be read independently for each unit group 202. For example, the exposure time (shutter speed) can be made different for each unit group 202.

Further, an amplification circuit (not shown) for amplifying an imaging signal generated by the photoelectrically converted charge is provided independently for each unit group 202, and the amplification factor by the amplification circuit can be controlled independently for each amplification circuit. For example, the amplification factor (ISO sensitivity) of the signal can be made different for each unit group 202.

In addition to the imaging conditions described above, the imaging conditions that can be varied for each unit group 202 include frame rate, gain, resolution (thinning rate), number of added rows or number of added columns for adding pixel signals, charge The storage time or number of storage, the number of bits for digitization, and the like. Furthermore, the control parameter may be a parameter in image processing after acquisition of an image signal from a pixel.

Further, for the imaging condition, for example, a liquid crystal panel having sections that can be controlled independently for each unit group 202 (one section corresponds to one unit group 202) is provided in the imaging element 100, and a light reduction filter that can be turned on and off If it is used, it becomes possible to control the brightness (aperture value) for each unit group 202.

The number of pixels 201 constituting the unit group 202 may not be the 2 × 2 four pixels described above. The unit group 202 may have at least one pixel 201, and conversely, may have more than four pixels 201.

FIG. 3 is a circuit diagram of the imaging chip 113. As shown in FIG. In FIG. 3, a rectangle surrounded by a dotted line representatively represents a circuit corresponding to one pixel 201. In addition, a rectangle surrounded by an alternate long and short dash line corresponds to one unit group 202 (202-1 to 202-4). Note that at least a part of each of the transistors described below corresponds to the transistor 105 in FIG.

As described above, the reset transistor 303 of the pixel 201 is turned on / off in unit group 202 units. In addition, the transfer transistor 302 of the pixel 201 is also turned on / off in unit group 202 units. In the example shown in FIG. 3, reset wirings 300-1 for turning on / off the four reset transistors 303 corresponding to the upper left unit group 202-1 are provided, and four corresponding to the unit group 202-1 are provided. A TX wire 307-1 for supplying a transfer pulse to the transfer transistor 302 is also provided.

Similarly, a reset wiring 300-3 for turning on / off the four reset transistors 303 corresponding to the lower left unit group 202-3 is provided separately from the reset wiring 300-1. Also, a TX wiring 307-3 for supplying transfer pulses to the four transfer transistors 302 corresponding to the unit group 202-3 is provided separately from the TX wiring 307-1.

Similarly, for the upper right unit group 202-2 and the lower right unit group 202-4, the reset wiring 300-2 and TX wiring 307-2 and the reset wiring 300-4 and TX wiring 307-4 are respectively unit groups It is provided in 202.

The 16 PDs 104 corresponding to each pixel 201 are connected to the corresponding transfer transistors 302, respectively. A transfer pulse is supplied to the gate of each transfer transistor 302 via the TX wiring of each unit group 202. The drain of each transfer transistor 302 is connected to the source of the corresponding reset transistor 303, and a so-called floating diffusion FD between the drain of the transfer transistor 302 and the source of the reset transistor 303 is connected to the gate of the corresponding amplification transistor 304. Ru.

The drains of the reset transistors 303 are commonly connected to a Vdd wiring 310 to which a power supply voltage is supplied. A reset pulse is supplied to the gate of each reset transistor 303 via the reset wiring of each unit group 202.

The drains of the respective amplification transistors 304 are commonly connected to a Vdd wiring 310 to which a power supply voltage is supplied. The source of each amplification transistor 304 is connected to the drain of the corresponding selection transistor 305. The gate of each selection transistor 305 is connected to a decoder wiring 308 to which a selection pulse is supplied. The decoder wiring 308 is provided independently for each of the 16 selection transistors 305.

The source of each selection transistor 305 is connected to the common output wiring 309. The load current source 311 supplies a current to the output wiring 309. That is, the output wiring 309 for the selection transistor 305 is formed by a source follower. The load current source 311 may be provided on the imaging chip 113 side or may be provided on the signal processing chip 111 side.

Here, the flow from the charge accumulation start to the pixel output after the charge end will be described. When a reset pulse is applied to the reset transistor 303 through the reset wiring of each unit group 202 and a transfer pulse is applied to the transfer transistor 302 through the TX wiring of each of the unit groups 202 (202-1 to 202-4), The potentials of the PD 104 and the floating diffusion FD are reset for each unit group 202.

When the application of the transfer pulse is released, each PD 104 converts incident light to be received into charge and accumulates it. Thereafter, when the transfer pulse is applied again in a state where the reset pulse is not applied, the accumulated charge is transferred to the floating diffusion FD, and the potential of the floating diffusion FD becomes a signal potential after charge accumulation from the reset potential. .

Then, when a selection pulse is applied to the selection transistor 305 through the decoder wiring 308, a change in signal potential of the floating diffusion FD is transmitted to the output wiring 309 through the amplification transistor 304 and the selection transistor 305. Thus, the pixel signal corresponding to the reset potential and the signal potential is output from the unit pixel to the output wiring 309.

As described above, for the four pixels forming the unit group 202, the reset wiring and the TX wiring are common. That is, the reset pulse and the transfer pulse are simultaneously applied to four pixels in the unit group 202, respectively. Therefore, all the pixels 201 forming a certain unit group 202 start charge accumulation at the same timing, and end charge accumulation at the same timing. However, pixel signals corresponding to the accumulated charges are selectively output from the output wiring 309 by sequentially applying selection pulses to the respective selection transistors 305.

Thus, the charge accumulation start timing can be controlled for each unit group 202. In other words, different unit groups 202 can be imaged at different timings.

FIG. 4 is a block diagram showing a functional configuration example of the imaging device 100. As shown in FIG. The analog multiplexer 411 selects 16 PDs 104 forming the unit group 202 in order, and outputs the respective pixel signals to the output wiring 309 provided corresponding to the unit group 202. The multiplexer 411 is formed on the imaging chip 113 together with the PD 104.

The pixel signal output via the multiplexer 411 is subjected to CDS and A / A by the signal processing circuit 412 for performing correlated double sampling (CDS) and analog / digital (A / D) conversion, which is formed in the signal processing chip 111. D conversion is performed. The A / D converted pixel signals are delivered to the demultiplexer 413 and stored in the pixel memory 414 corresponding to each pixel. The demultiplexer 413 and the pixel memory 414 are formed in the memory chip 112.

The arithmetic circuit 415 processes the pixel signal stored in the pixel memory 414 and delivers it to the image processing unit in the subsequent stage. The arithmetic circuit 415 may be provided in the signal processing chip 111 or in the memory chip 112. Although FIG. 4 shows the connection of four unit groups 202, in reality, these units exist for each of the four unit groups 202 and operate in parallel.

However, the arithmetic circuit 415 may not be present for every four unit groups 202. For example, one arithmetic circuit 415 sequentially refers to the values of the pixel memory 414 corresponding to each of the four unit groups 202. It may be processed.

As described above, the output wirings 309 are provided corresponding to each of the unit groups 202. Since the imaging element 100 has the imaging chip 113, the signal processing chip 111, and the memory chip 112 stacked, by using the electrical connection between the chips using the bumps 109 for the output wiring 309, each chip is made in the surface direction The wiring can be routed without increasing the size.

<Example of block configuration of electronic device>
FIG. 5 is an explanatory view showing an example of the block configuration of the electronic device. The electronic device 500 is, for example, a lens-integrated camera. The electronic device 500 includes an imaging optical system 501, an imaging element 100, a control unit 502, a liquid crystal monitor 503, a memory card 504, an operation unit 505, a DRAM 506, a flash memory 507, and a recording unit 508. . The control unit 502 includes a compression unit that compresses moving image data as described later. Therefore, the configuration including at least the control unit 502 in the electronic device 500 is a moving image compression apparatus.

The imaging optical system 501 is composed of a plurality of lenses, and forms an object image on the imaging surface 200 of the imaging element 100. In FIG. 5, the imaging optical system 501 is illustrated as a single lens for the sake of convenience.

The imaging device 100 is, for example, an imaging device such as a complementary metal oxide semiconductor (CMOS) or a charge coupled device (CCD), captures an object image formed by the imaging optical system 501, and outputs an imaging signal. The control unit 502 is an electronic circuit that controls each unit of the electronic device 500, and includes a processor and its peripheral circuits.

A predetermined control program is written in advance in the flash memory 507, which is a non-volatile storage medium. The control unit 502 controls each unit by reading a control program from the flash memory 507 and executing it. This control program uses a DRAM 506 which is a volatile storage medium as a work area.

The liquid crystal monitor 503 is a display device using a liquid crystal panel. The control unit 502 causes the imaging device 100 to repeatedly capture a subject image at predetermined intervals (for example, 1/60 second). Then, the image pickup signal output from the image pickup element 100 is subjected to various image processing to create a so-called through image, which is displayed on the liquid crystal monitor 503. In addition to the above-described through image, a setting screen for setting an imaging condition is displayed on the liquid crystal monitor 503, for example.

The control unit 502 creates an image file to be described later based on the imaging signal output from the imaging element 100, and records the image file on a memory card 504, which is a portable recording medium. The operation unit 505 includes various operation members such as a push button, and outputs an operation signal to the control unit 502 in response to the operation of the operation members.

The recording unit 508 is, for example, a microphone, converts environmental sound into an audio signal, and inputs the audio signal to the control unit 502. The control unit 502 may record the moving image file in a recording medium (not shown) built in the electronic device 500 such as a hard disk instead of recording the moving image file in the memory card 504 which is a portable recording medium.

<Example of movie file configuration>
FIG. 6 is an explanatory view showing a configuration example of a moving image file. The moving image file 600 is generated during compression processing in a compression unit 902 described later in the control unit 502, and is stored in the memory card 504, the DRAM 506, or the flash memory 507. The moving image file 600 is composed of two blocks of a header portion 601 and a data portion 602. The header unit 601 is a block located at the beginning of the moving image file 600. In the header portion 601, a file basic information area 611, a mask area 612, and an imaging information area 613 are stored in the order described above.

In the file basic information area 611, for example, the size and offset of each part (header section 601, data section 602, mask area 612, imaging information area 613, etc.) in the moving image file 600 are recorded. In the mask area 612, imaging condition information, mask information, and the like described later are recorded. In the imaging information area 613, information related to imaging such as a model name of the electronic device 500 or information of the imaging optical system 501 (for example, information regarding optical characteristics such as aberration) is recorded. The data unit 602 is a block located behind the header unit 601, and stores image information, audio information, and the like.

<Relationship between imaging plane and subject image>
FIG. 7 is an explanatory view showing the relationship between the imaging plane and the subject image. (A) schematically shows an imaging surface 200 (imaging range) of the imaging element 100 and a subject image 701. In (a), the control unit 502 captures a subject image 701 once before capturing in (c). The imaging of (a) may also be performed, for example, for creating a live view image (so-called through image).

The control unit 502 executes predetermined image analysis processing on the subject image 701 obtained by the imaging in (a). The image analysis process is a process of detecting the main subject area and the background area by, for example, a known subject detection technology (a technology for calculating a feature amount and detecting a range in which a predetermined subject is present). By the image analysis processing, the imaging surface 200 is divided into a main subject region 702 in which a main subject is present and a background region 703 in which a background is present.

Although the area roughly including the subject image 701 is illustrated as the main subject area 702 in (a), the main subject area 702 may have a shape along the outer shape of the subject image 701. That is, the main subject region 702 may be set so as to include as little as possible other than the subject image 701.

The control unit 502 sets different imaging conditions for each unit group 202 in the main subject region 702 and each unit group 202 in the background region 703. For example, in the former unit group 202, a shutter speed faster than that of the latter unit group 202 is set. In this way, in the imaging of (c) taken after the imaging of (a), image blurring is less likely to occur in the main subject region 702.

Further, when the main subject area 702 is in a backlit state due to the light source such as the sun present in the background area 703, the control unit 502 makes the ISO relatively higher for each unit group 202 of the former. Set the sensitivity or set a slow shutter speed. Further, the control unit 502 sets a relatively low ISO sensitivity or sets a high shutter speed to each of the latter unit groups 202. In this way, in the imaging of (c), it is possible to prevent blackout of the main subject region 702 in a backlit state and overexposure of the background region 703 having a large amount of light.

The image analysis process may be different from the process of detecting the main subject area 702 and the background area 703 described above. For example, processing may be performed to detect a portion where the brightness is equal to or more than a predetermined level (a portion that is too bright) or a portion where the brightness is less than a predetermined level (a too dark portion). When the image analysis processing is such processing, the control unit 502 causes the exposure value (Ev value) to be lower for the unit group 202 included in the former region than for the unit group 202 included in the other region. , Shutter speed and ISO sensitivity.

Further, the control unit 502 sets the shutter speed and the ISO sensitivity so that the exposure value (Ev value) of the unit group 202 included in the latter region is higher than that of the unit group 202 included in the other region. . By doing this, the dynamic range of the image obtained by the imaging of (c) can be expanded beyond the original dynamic range of the imaging device 100.

(B) of FIG. 7 shows an example of the mask information 704 corresponding to the imaging surface 200 shown in (a). “1” is stored at the position of the unit group 202 belonging to the main subject area 702, and “2” is stored at the position of the unit group 202 belonging to the background area 703.

The control unit 502 executes an image analysis process on the image data of the first frame to detect the main subject area 702 and the background area 703. As a result, the frame obtained by the imaging in (a) is divided into the main subject area 702 and the background area 703 as shown in (c). The control unit 502 sets different imaging conditions for each unit group 202 in the main subject area 702 and each unit group 202 in the background area 703, performs imaging in (c), and creates image data. . An example of the mask information 704 at this time is shown in (d).

In the mask information 704 of (b) corresponding to the imaging result of (a) and the mask information 704 of (d) corresponding to the imaging result of (c), imaging is performed at different times (the time difference is Therefore, for example, when the subject is moving or when the user moves the electronic device 500, the two mask information 704 have different contents. In other words, the mask information 704 is dynamic information that changes as time passes. Therefore, in a certain unit group 202, different imaging conditions are set for each frame.

<Specific example of video file>
FIG. 8 is an explanatory view showing a specific configuration example of the moving image file 600. As shown in FIG. In the mask area 612, identification information 801, imaging condition information 802, and mask information 704 are recorded in the order described above.

The identification information 801 indicates that the moving image file 600 is created by the multi-imaging condition moving image pickup function. The multi-imaging condition moving image imaging function is a function of shooting a moving image with the imaging element 100 in which a plurality of imaging conditions are set.

The imaging condition information 802 is information indicating what use (purpose, role) exists in the unit group 202. For example, as described above, when the imaging plane 200 (FIG. 7A) is divided into the main subject area 702 and the background area 703, each unit group 202 belongs to the main subject area 702, or It belongs to the area 703.

That is, when creating the moving image file 600, the imaging condition information 802 uses the unit group 202, for example, “moving image shooting of main subject area at resolution A” and “moving image shooting of background area at resolution B” Is information that represents the unique number assigned to each of these uses. For example, the number 1 is assigned to "use moving image of main subject area at resolution A" and the number 2 is assigned to "use moving image at background B to resolution B".

The mask information 704 is information representing the use (purpose, role) of each unit group 202. The mask information 704 is “information represented by a number assigned to the imaging condition information 802 in the form of a two-dimensional map in accordance with the position of the unit group 202”. That is, when the two-dimensional array of unit groups 202 is specified by two integers x and y at two-dimensional coordinates (x, y), the use of the unit group 202 at the position of (x, y) is It is expressed by the number existing at the position (x, y) of the mask information 704.

For example, if the position of coordinates (3, 5) of the mask information 704 contains a number “1”, the unit group 202 located at the coordinates (3, 5) is “image main subject area” It can be seen that the application has been given. In other words, it can be understood that the unit group 202 located at the coordinates (3, 5) belongs to the main subject region 702.

Since the mask information 704 is dynamic information that changes for each frame, it is recorded during compression processing for each frame, that is, for each data block Bi described later (not shown).

In the data unit 602, data blocks B1 to Bn are stored as moving image data in the order of imaging for each frame F (F1 to Fn). Data block Bi (i is an integer of 1 ≦ i ≦ n) includes mask information 704, image information 811, Tv value map 812, Sv value map 813, Bv value map 814, and Av value information 815, Audio information 816 and additional information 817 are included.

The image information 811 is information obtained by recording an image pickup signal output from the image pickup element 100 by the image pickup of FIG. 7C in a form before performing various image processing, and is so-called RAW image data.

The Tv value map 812 is information in which a Tv value representing a shutter speed set for each unit group 202 is represented in the form of a two-dimensional map in accordance with the position of the unit group 202. For example, the shutter speed set to the unit group 202 located at the coordinates (x, y) can be determined by examining the Tv value stored at the coordinates (x, y) of the Tv value map 812.

The Sv value map 813 is information in which the Sv value representing the ISO sensitivity set for each unit group 202 is expressed in the form of a two-dimensional map, similarly to the Tv value map 812.

The Bv value map 814 is a Tv value map 812 for the subject brightness measured for each unit group 202 at the time of imaging in FIG. 7C, that is, the Bv value representing the brightness of the subject light incident on each unit group 202. And is information expressed in the form of a two-dimensional map.

The Av value information 815 is information representing the aperture value at the time of imaging in (c) of FIG. 7. Unlike the Tv value, the Sv value, and the Bv value, the Av value is not a value that exists for each unit group 202. Therefore, unlike the Tv value, the Sv value, and the Bv value, only a single value of the Av value is stored, and the information is not information obtained by mapping a plurality of values in a two-dimensional manner.

The audio information 816 is divided into information of one frame, easily multiplexed with the data block Bi, and stored in the data unit 602 so as to facilitate moving image reproduction. The audio information 816 may be multiplexed not for one frame but for a predetermined number of frames. Note that the voice information 816 does not necessarily have to be included.

The additional information 817 is information representing, in the form of a two-dimensional map, the resolution set for each unit group 202 at the time of imaging in (c) of FIG. 7. The additional information 817 may be held in the frame F, but may be held in a cache memory of the processor 1001 described later. In particular, when performing compression processing in real time, it is preferable to use a cache memory from the viewpoint of high-speed processing.

As described above, the control unit 502 performs image pickup with such a moving image pickup function, and thereby, the image information 811 generated by the image pickup element 100 in which the image pickup condition can be set for each unit group 202, and A moving image file 600 associated with data relating to the imaging conditions (imaging condition information 802, mask information 704, Tv value map 812, Sv value map 813, Bv value map 814, etc.) is recorded in the memory card 504.

<Imaging Example on Imaging Surface 200 with Different Resolutions Set>
Next, an imaging example on the imaging surface 200 in which different resolutions are set will be described. The moving picture compression apparatus of the present embodiment compresses moving picture data including a plurality of frames generated from the output of the imaging device 100. The imaging element 100 has a plurality of imaging areas in which different resolutions can be set. Specifically, for example, according to the above setting, the imaging element 100 includes a first imaging area for imaging an object at a first resolution and a second imaging area for imaging an object at a second resolution different from the first resolution. Have.

As described above, in the present embodiment, since imaging regions of different resolutions are set in the imaging surface 200, the frames to be output are also expressed with different resolutions. Therefore, the video compression apparatus applies different intra-frame prediction for each resolution to compress a frame. As a result, the low resolution image area in the frame can be significantly compressed compared to the high resolution image area, and the load on the compression processing can be reduced.

FIG. 9 is an explanatory view showing an imaging example on the imaging plane 200 in which different resolutions are set. In FIG. 9, two types of resolutions A and B are set on the imaging surface 200 as an example. Resolution A is higher than resolution B. For example, in the case of resolution A, the imaging device outputs 16 × 16 pixels of the imaging region 901A of resolution A in an image region 910A of 16 × 16 pixels. On the other hand, in the case of the resolution B, the imaging device 100 thins out the 16 × 16 pixels of the imaging region 901B of the resolution B and outputs the thinned image in the image region 910b of 1 × 1 pixel. The resolutions A and B are not limited to the above, and the resolution A may be higher than the resolution B.

The moving picture compression apparatus divides a block of 16 × 16 pixels into 16 blocks of 4 × 4 pixels in the image area 910A of resolution A in the frame F output from the imaging device 100, X4 Perform prediction. Since each of the 16 blocks is 4 × 4 pixels, 4 × 4 pixels are the prediction processing unit in 4 × 4 prediction.

Also, the moving picture compression apparatus copies the image area 910b of one pixel output to the defect area 910c for the image area 910b of resolution B in the frame F output from the imaging element 100 and After generating an image area 910B to be a block, so-called 16 × 16 prediction is performed. Since the block generated by copying in this way is one block of 16 × 16 pixels, in 16 × 16 prediction, 16 × 16 pixels become a prediction processing unit.

In both predictions, scanning is performed rightward (white thick arrow) from the upper left block of frame F, and when reaching the right end block, it is shifted downward by one block and scanned from the left end block to the right end block (Raster scan).

Since the 16 × 16 prediction requires fewer bits for encoding in the prediction direction than the 4 × 4 prediction, the moving image compression apparatus compresses the image area of resolution B as compared to the image area of resolution A. Rate can be improved. That is, rather than 4 × 4 prediction of the entire image area of the frame F, it is possible to improve the compression rate and reduce the processing load of the compression process. The image area 910A and the image area 910B may be hereinafter referred to as a block 910A and a block 910B, respectively.

FIG. 10 is an explanatory view showing a prediction example of 16 × 16 prediction. (A) shows mode 0 (vertical prediction), (b) shows mode 1 (horizontal prediction), (c) shows mode 2 (average value prediction), and (d) shows mode 3 (planar prediction). A block of 16 × 16 pixels to be predicted is referred to as a target block 1000.

(A) Mode 0 is applied when there is a predicted block of the same resolution adjacent to the target block 1000 and no predicted block of the same resolution adjacent to the left.

(B) Mode 1 is applied when there is a predicted block of the same resolution adjacent to the left of the target block 1000 and there is no predicted block of the same resolution adjacent to the top.

(C) Mode 2 is applied when there is a predicted block of the same resolution adjacent above and to the left of the target block 1000.

(D) Mode 3 is also applied when there is a predicted block of the same resolution adjacent above and to the left of the target block 1000. Which one of the mode 2 and the mode 3 is to be applied may be set in advance, or may be set by the user operating the operation unit 505.

FIG. 11 is an explanatory view showing a prediction example of 4 × 4 prediction. (A) mode 0 (vertical prediction), (b) mode 1 (horizontal prediction), (c) mode 2 (average value prediction), (d) mode 3 (diagonal left lower prediction), (e) Mode 4 (diagonal lower right prediction), (f) mode 5 (vertical right prediction), (g) mode 6 (horizontal lower prediction), (h) mode 7 (vertical left prediction), (i) mode 8 (horizontal upper prediction) is shown. A block of 4 × 4 pixels to be predicted is referred to as a target block 1100.

(A) Mode 0 is applied when there is a predicted block of the same resolution adjacent to the target block 1100 and no predicted block of the same resolution adjacent to the left.

(B) Mode 1 and (i) Mode 8 is applied when there is a predicted block of the same resolution adjacent to the left of the target block 1100 and there is no predicted block of the same resolution adjacent above. Ru. Which one of the mode 1 and the mode 8 is applied may be set in advance, or may be set by the user operating the operation unit 505.

(C) mode 2, (e) mode 4, (f) mode 5 and (g) mode 6 are applied when there is a predicted block of the same resolution adjacent to the top and left of the target block 1100 . Which one of mode 2, mode 4, mode 5 and mode 6 is to be applied may be set in advance, or may be set by the user operating the operation unit 505.

(D) Mode 3 and (h) Mode 7 are applied when there is a predicted block of the same resolution adjacent on the upper and upper right of the target block 1100. Which one of the mode 3 and the mode 7 is to be applied may be set in advance, or may be set by the user operating the operation unit 505.

<Configuration Example of Control Unit 502>
FIG. 12 is a block diagram showing a configuration example of the control unit 502 shown in FIG. The control unit 502 includes a preprocessing unit 1210, an image processing unit 1220, an acquisition unit 1230, and a compression unit 1240, and is configured by a processor 1201, a memory 1202, an integrated circuit 1203, and a bus 1204 connecting these. Be done.

The preprocessing unit 1210, the image processing unit 1220, the acquisition unit 1230, and the compression unit 1240 may be realized by causing the processor 1201 to execute a program stored in the memory 1202, and may be realized by an application specific integrated circuit (ASIC) or an FPGA (FPGA). It may be realized by an integrated circuit 1203 such as a field-programmable gate array). Also, the processor 1201 may use the memory 1202 as a work area. The integrated circuit 1203 may use the memory 1202 as a buffer that temporarily holds various data including image data.

The preprocessing unit 1210 executes preprocessing of image processing by the image processing unit 1220 on moving image data including a plurality of frames F from the imaging element 100. Specifically, for example, when moving image data (here, a set of RAW image data) is input from the imaging device 100, the preprocessing unit 1210 uses a known object detection technology to identify a specific subject such as a main subject. To detect.

For example, in the case where resolution B is set over the entire imaging surface 200, when a specific subject such as a main subject is detected and imaged, the preprocessing unit 1210 resolves the imaging area of the imaging element 100 which images the specific subject. The image is output to the image sensor 100 so as to be A. As a result, the imaging area of the specific subject is set to the resolution A, and the other imaging areas are set to the resolution B.

Further, specifically, the preprocessing unit 1210 calculates, for example, the motion vector of the specific subject from the difference between the imaging area where the specific subject in the input frame is detected and the imaging area where the specific subject in the input completed frame is detected. It is possible to detect and specify an imaging region of a specific subject in the next input frame. In this case, the preprocessing unit 900 outputs, to the imaging element 100, an instruction to change the identified imaging area to the resolution A. As a result, the imaging area of the specific subject is set to the resolution A, and the other imaging areas are set to the resolution B.

The image processing unit 1220 performs image processing such as demosaicing processing, white balance adjustment, noise reduction, and debayering on the moving image data input from the imaging element 100. Specifically, for example, the image processing unit 1220 executes known image processing such as demosaicing processing and white balance adjustment. Further, as described with reference to FIG. 9, the image processing unit 1220 copies the image data of the image area 910 b output from the pixel of resolution B to generate the image area 910 B of resolution B.

The acquisition unit 1230 holds the moving image data output from the image processing unit 1220 in the memory 1202, and outputs a plurality of frames F included in the moving image data one frame at a time in chronological order to the compression unit 1240 at a predetermined timing.

The compression unit 1240 compresses the moving image data input from the acquisition unit 1230. Specifically, for example, compression section 1240 compresses frame F by inter-frame prediction and intra-frame prediction, for example. In the inter-frame prediction, the compression unit 1240 compresses the frame F by hybrid coding combining entropy coding with motion compensation inter-frame prediction (Motion Compensation: MC) and discrete cosine transform (DCT). . In intra-frame prediction, as shown in FIGS. 9 to 11, the compression unit 1240 compresses the

image areas

910A and 910B of the resolution for each resolution.

Note that the control unit 502 may execute compression processing of moving image data from the imaging element 100 in real time processing, or may execute it in batch processing. For example, the control unit 502 temporarily stores moving image data from the imaging device 100, the pre-processing unit 1210, or the image processing unit 1220 in the memory card 504, the DRAM 506, or the flash memory 507, and automatically or by user operation. When there is a trigger, moving image data may be read out and the compression unit 1240 may execute compression processing.

<Configuration Example of Compression Unit 1240>
FIG. 13 is a block diagram showing a configuration example of the compression unit 1240. As described above, the compression unit 1240 compresses the frame F by, for example, inter-frame prediction and intra-frame prediction.

The compression unit 1240 includes a subtraction unit 1301, a DCT unit 1302, a quantization unit 1303, an entropy coding unit 1304, a code amount control unit 1305, an inverse quantization unit 1306, an inverse DCT unit 1307, and a generation unit. A frame memory 1309, a motion detection unit 1310, a motion compensation unit 1311, a determination unit 1320, and an intra-frame prediction processing unit 1330 are included. The subtractor unit 1301 to the motion compensation unit 1311 and the determination unit 1320 have the same configuration as the existing compressor. Further, the DCT unit 1302, the quantization unit 1303, the entropy coding unit 1304, and the code amount control unit 1305 are referred to as a coding unit 1340.

The subtracting unit 1301 subtracts the prediction frame from the motion compensating unit 1311 that predicts the input frame from the input frame, and outputs difference data. The DCT unit 1302 performs discrete cosine transform on the difference data from the subtracting unit 1301.

The quantization unit 1303 quantizes the discrete cosine transformed difference data. The entropy coding unit 1304 entropy codes the quantized difference data, and also entropy codes the motion vector from the motion detection unit 1310.

The code amount control unit 1305 controls the quantization by the quantization unit 1303. The inverse quantization unit 1306 inversely quantizes the difference data quantized by the quantization unit 1303 to obtain discrete cosine transformed difference data. The inverse DCT unit 1307 inverse discrete cosine transforms the dequantized difference data.

The generation unit 1308 adds the inverse discrete cosine transformed difference data and the prediction frame from the motion compensation unit 1311 to generate a reference frame to which a frame input temporally after the input frame refers. . The frame memory 1309 holds the reference frame obtained from the generation unit 1308.

The motion detection unit 1310 detects a motion vector by block matching, for example, using the input frame and the reference frame. The motion compensation unit 1311 generates a predicted frame using the reference frame and the motion vector. Specifically, for example, the motion compensation unit 1311 performs motion compensation using a specific reference frame and a motion vector among the plurality of reference frames stored in the frame memory 1309.

By making the reference frame a specific reference frame, it is possible to suppress high-load motion compensation using another reference frame other than the specific reference frame. Also, by setting a specific reference frame as one reference frame obtained from the temporally previous frame of the input frame, heavy processing of motion compensation is avoided, and processing load on motion compensation is reduced. Can be

The inter-frame prediction is realized by the subtraction unit 1301, the inverse quantization unit 1306, the inverse DCT unit 1307, the generation unit 1308, the frame memory 1309, the motion detection unit 1310, and the motion compensation unit 1311 described above.

The determination unit 1320 uses the input frame and the difference data from the subtraction unit 1301 to determine which of intra-frame prediction and inter-frame prediction is more efficient to select, thereby performing intra-frame prediction and intra-frame prediction. Select one of the inter-frame predictions. If intra-frame prediction is selected, the determination unit 1320 outputs the input frame to the intra-frame prediction processing unit 1330. Further, the determination unit 1320 may select intra-frame prediction at the insertion timing of the I picture. On the other hand, when inter-frame prediction is selected, determination section 1320 outputs differential data to DCT section 1302.

The intraframe prediction processing unit 1330 performs intraframe prediction of an input frame. The in-frame prediction processing unit 1330 includes a setting unit 1331 and a prediction unit 1332. The setting unit 1331 sets a prediction processing unit for predicting the prediction target image area based on the resolution of the prediction target image area in the plurality of image areas corresponding to the plurality of imaging areas in the prediction target frame among the plurality of frames. Do.

The prediction target frame is an input frame which is input to the compression unit 1240 and is a target of compression processing. The imaging area is an area of pixels having a predetermined number of pixels in the imaging device 100. For example, in the example of FIG. 9, 16 × 16 pixels are taken as one imaging area. The size of the imaging area is not limited to 16 × 16 pixels, and may be an integral multiple of the unit group 202 (in this example, 4 × 4 pixels as an example). In the example of FIG. 9, the number of imaging areas 901A of resolution A is four, and the number of imaging areas 901B of resolution B is twenty-one.

The image area is an area of pixel data in the frame F corresponding to the imaging area. That is, the subject imaged in the imaging area is expressed as image data (set of pixel data) in the image area. In the example of FIG. 9, the image area 910A of resolution A corresponds to the imaging area 901A, and the image area 910B of resolution B corresponds to the imaging area 901B. In the example of FIG. 9, the number of image areas 910A of resolution A is four, and the number of image areas 910B of resolution B is twenty-one.

The prediction target image area is an image area which has not been predicted yet and is to be currently predicted among the plurality of image areas in the frame F. In this example, any prediction of 4 × 4 prediction and 16 × 16 prediction is scanned from the upper left block of frame F to the right, and when it reaches the right end block, it is shifted downward by one block to the right end block from the left end block A raster scan is applied which is scanned to.

Therefore, an image area having the same resolution as the prediction target image area and having the same resolution located on the left side, or an image area having the same resolution image area in the row above the prediction target image area has already been predicted (the above-described predicted block) It becomes. Since intra-frame prediction is performed, it is preferable that the prediction target image region and the predicted image region be closer. For example, the most preferable image region as a predicted image region is an adjacent image region of a prediction target image region.

The prediction processing unit is a processing unit for predicting a prediction target image area, and is the target blocks 1000 and 1100 shown in FIG. 10 and FIG. In the case of 4 × 4 prediction, a 16 × 16 pixel prediction target area is divided into 16 blocks. Since each of the 16 blocks is 4 × 4 pixels, 4 × 4 pixels are the prediction processing unit in 4 × 4 prediction. In the case of 16 × 16 prediction, a 16 × 16 pixel prediction target area is one block. Since this block is 16 × 16 pixels, in 16 × 16 prediction, 16 × 16 pixels become a prediction processing unit. That is, the higher the resolution, the smaller the prediction processing unit, and the lower the resolution, the larger the prediction processing unit.

The prediction unit 1332 predicts a prediction target image region based on the prediction processing unit set by the setting unit 1331. Specifically, for example, when the prediction processing unit is 16 × 16 pixels, the prediction unit 1332 performs 16 × 16 prediction as shown in FIG. 10, and the prediction processing unit is 4 × 4 pixels. In the case, as shown in FIG. 11, 4 × 4 prediction is performed. The prediction unit 1332 outputs the prediction result to the DCT unit 1302 of the coding unit 1340. The output may be output as it is.

<Example of pre-processing procedure>
FIG. 14 is a flowchart illustrating an example of a preprocessing procedure by the preprocessing unit 1210. In FIG. 14, the resolution B is set in advance in the imaging device 100, and an example in which the image area of the resolution A is tracked by the subject detection technology of the preprocessing unit 1210 and fed back to the imaging device 100 will be described. The image areas of resolutions A and B may be fixed at all times.

The preprocessing unit 1210 waits for the input of the frame F constituting the moving image data (step S1401: No), and when the frame F is input (step S1401: Yes), the detection unit detects a specific subject such as a main subject. It is determined whether or not it is (step S1402). When the specific subject is not detected (step S1402: No), the process proceeds to step S1401.

On the other hand, when the specific subject is detected (step S1402: YES), the preprocessing unit 1210 compares the temporally previous frame (for example, a reference frame) with the input frame to detect a motion vector, An image area of resolution A in the next input frame is predicted and output to the imaging device 100 (step S1403), and the process proceeds to step S1401. Thereby, the imaging device 100 sets the resolution of the unit group 202 constituting the imaging area corresponding to the predicted image area to the resolution A, sets the resolution of the remaining unit groups 202 to the resolution B, and sets the object. Take an image.

Then, the process returns to step S1401. When the frame is not input (step S1401: No) and the input of all the frames constituting the moving image data is completed, the series of processing is ended.

<Example of image processing procedure>
FIG. 15 is a flowchart showing an example of the image processing procedure by the image processing unit 1220. In FIG. 15, the process of copying the image data of the image area 910b of the resolution B described above will be described. If the frame F is input (step S1501: YES), the image processing unit 1220 determines whether there is an unselected block in the frame (step S1502). The block is an image area of 16 × 16 pixels as an example. Unselected blocks are blocks that have not been selected in step S1503.

If there is an unselected block (step S1502: YES), the image processing unit 1220 selects one unselected block (step S1503). The selected block is referred to as a selected block. The image processing unit 1220 determines whether the resolution of the selected block is the resolution B (step S1504). Specifically, for example, the image processing unit 1220 refers to the information of the resolution set in each unit group 202 of the imaging device 100 in the pre-processing unit 1210 to determine the resolution of the selected block by specifying.

If the resolution of the selected block is not the resolution B (step S1504: NO), the image processing unit 1220 returns to step S1502. On the other hand, if the resolution of the selected block is the resolution B (step S1504: YES), the image processing unit 1220 duplicates the inside of the selected block with the image data of the image area 910b to generate a block 910B (step S1505) , And return to step S1502.

If there is no unselected block in step S1502 (step S1502: NO), the process returns to step S1501. When the frame F is not input (step S1501: No) and the input of all the frames constituting the moving image data is completed, the series of processing is ended.

<Example of intraframe prediction processing procedure>
FIG. 16 is a flowchart of an example of the intra-frame prediction processing procedure by the intra-frame prediction processing unit 1330. If the frame F is input (step S1601: YES), the intra-frame prediction processing unit 1330 determines, by the setting unit 1331, whether there is an unselected block in the frame (step S1602).

The block is an image area of 16 × 16 pixels as an example. If there is an unselected block (step S1602: YES), the intra-frame prediction processing unit 1330 selects one unselected block using the setting unit 1331 (step S1603), and determines the resolution of the selected block (step S1604) . Specifically, for example, the image processing unit 1220 refers to the information of the resolution set in each unit group 202 of the imaging device 100 in the pre-processing unit 1210 to determine the resolution of the selected block by specifying.

If the resolution of the selected block is resolution A (step S1604: A), the intra-frame prediction processing unit 1330 sets the prediction processing unit of the selected block to 4 × 4 pixels by the setting unit 1331 (step S1605). The intra-frame prediction processing unit 1330 causes the prediction unit 1332 to divide the selected block in the set prediction processing unit (step S1606). In this case, the selected block of 16 × 16 pixels is divided into 16 blocks of 4 × 4 pixels (hereinafter referred to as divided blocks).

The intra-frame prediction processing unit 1330 determines, with the prediction unit 1332, whether there is an undivided selected block (step S1607). If there is an unselected divided block (step S1608: YES), the intra-frame prediction processing unit 1330 causes the prediction unit 1332 to select one unselected divided block (step S1608). Then, the intra-frame prediction processing unit 1330 causes the prediction unit 1332 to determine the prediction mode of the selected divided block (step S1609). Specifically, for example, as shown in FIG. 11, the intra-frame prediction processing unit 1330 determines applicable prediction modes from a plurality of prediction modes 0 to 9 by the prediction unit 1332.

Then, the intra-frame prediction processing unit 1330 generates a prediction block for predicting the selected divided block in the prediction mode determined by the prediction unit 1332 (step S1610). The generated prediction block is the prediction result of the prediction unit 1332. Thereafter, the process returns to step S1607. If there is no unselected divided block in step S1607 (step S1607: NO), the process returns to step S1602.

When the resolution of the selected block is resolution B in step S1604 (step S1604: B), the intra-frame prediction processing unit 1330 sets the prediction processing unit of the selected block to 16 × 16 pixels by the setting unit 1331. (Step S1611). The intra-frame prediction processing unit 1330 causes the prediction unit 1332 to determine the prediction mode of the selected divided block (step S1612). Specifically, for example, as shown in FIG. 10, the intra-frame prediction processing unit 1330 determines an applicable prediction mode from the plurality of prediction modes 0 to 3 by the prediction unit 1332.

Then, the intra-frame prediction processing unit 1330 generates a prediction block for predicting the selected divided block in the prediction mode determined by the prediction unit 1332 (step S1613). The generated prediction block is the prediction result of the prediction unit 1332. Thereafter, the process returns to step S1602. If there is no unselected divided block in step S1607 (step S1607: NO), the process returns to step S1602.

If there is no unselected block in step S1602 (step S1602: NO), the process returns to step S1601. If the frame F is not input in step S1601 (step S1601: NO) and input of all the frames constituting the moving image data is completed, the series of processing ends. The frame predicted by the intraframe prediction processing unit 1330 is output to the coding unit 1340.

(1) As described above, the above-described moving picture compression apparatus is a moving picture compression apparatus that compresses moving picture data including a plurality of frames generated from the output of the imaging device 100 having a plurality of imaging areas in which different resolutions can be set. . The video compression apparatus includes a setting unit 1331, a prediction unit 1332 and an encoding unit 1340.

The setting unit 1331 selects a prediction target image area based on the resolution of the prediction target image area (for example, blocks 910A and 910B) in the plurality of image areas corresponding to the plurality of imaging areas in the prediction target frame among the plurality of frames. The prediction processing unit (for example, 4x4 pixels, 16x16 pixels) which predicts is set. The prediction unit 1332 predicts a prediction target image region based on the prediction processing unit set by the setting unit 1331. The encoding unit 1340 encodes a prediction target frame using the prediction result of the prediction unit 1332.

As a result, in intra-frame prediction, it is possible to partially perform prediction according to the difference in resolution, and it is possible to optimize compression processing according to the resolution.

(2) Further, in the moving picture compression apparatus according to (1), the setting unit 1331 sets the resolution (for example, resolution A) of the prediction target image area (for example, block 910A) to another image area other than the prediction target image area. If the resolution is higher than the resolution (for example, the block 910B) (for example, resolution B), the prediction processing unit for predicting the prediction target image area is more than the prediction processing unit for predicting another image area (for example, 16 × 16 pixels) Is set to a smaller prediction processing unit (for example, 4 × 4 pixels).

As a result, it is not necessary to apply low-resolution encoding to the high-resolution image area in the frame F, and even if the image is complex compared to the low-resolution image area, efficient prediction results are obtained. You can get it.

(3) Further, in the moving picture compression apparatus of (1), the setting unit 1331 sets the resolution (for example, resolution B) of the prediction target image area (for example, block 910B) to another image area other than the prediction target image area. If the resolution (for example, the resolution A) of the block 910A is lower than the resolution (for example, resolution A), the unit of prediction processing for predicting the prediction target image area is Set to a large prediction processing unit (for example, 16 × 16 pixels).

As a result, it is not necessary to apply high-resolution encoding to the low-resolution image area in the frame F, which enables significant compression as compared to the high-resolution image area, and reduces the load of compression processing. be able to.

(4) Further, in the moving picture compression apparatus according to (1), the setting unit 1331 uses the image area predicted by the prediction unit 1332 in the prediction target frame based on the position of the prediction processing unit in the prediction target frame. Among the plurality of prediction modes, a specific prediction mode to be applied to the prediction processing unit is set, and the prediction unit 1332 predicts a prediction target image region by applying the specific prediction mode to the prediction processing unit.

As a result, even when image areas of different resolutions are mixed in one frame F, intra-frame prediction can be appropriately realized.

(5) Further, in the moving picture compression apparatus according to (4), the setting unit 1331 sets a specific prediction mode to be applied to the prediction processing unit based on the resolution of the predicted image area.

As a result, it is possible to selectively realize an efficient or processing load reduction compression processing using the resolution of the predicted image area.

(6) Further, in the moving picture compression apparatus of (5), the resolution of the predicted image area is the same resolution as the resolution of the unit of prediction processing.

As a result, this enables intra-frame prediction to be performed between image areas of the same resolution, and consistent compression processing can be realized. For example, if both the predicted image area and the prediction target image area have resolution A, 4 × 4 prediction is performed, and if both the predicted image area and the prediction target image area have resolution B, 16 × 16 prediction is performed. Is executed.

In other words, when the resolutions of the predicted image area and the prediction target image area are different, it can not be determined which prediction mode should be adopted. Therefore, the compression processing can be made more efficient.

In addition, when the resolution of the prediction target image area is resolution A and the resolution of the predicted image area is resolution B, when 16 × 16 prediction is applied, the prediction target image area is low despite the high resolution. The application of the resolution prediction mode leads to a reduction in prediction accuracy. Therefore, the prediction accuracy can be improved.

In addition, when the resolution of the prediction target image area is resolution A and the resolution of the predicted image area is resolution B, when 4 × 4 prediction is applied, the predicted image area with coarse resolution is referred to. , Cause a decrease in prediction accuracy. Therefore, the prediction accuracy can be improved.

In addition, when the resolution of the prediction target image area is resolution B and the resolution of the predicted image area is resolution A, when 16 × 16 prediction is applied, the resolution refers to the predicted image area with fine resolution. , Cause a decrease in prediction efficiency. Therefore, the prediction efficiency can be improved.

Also, when the resolution of the prediction target image area is resolution B and the resolution of the predicted image area is resolution A, if 4 × 4 prediction is applied, the prediction target image area is high despite the low resolution. The application of the resolution prediction mode leads to a reduction in prediction accuracy. Therefore, the prediction accuracy can be improved.

(7) Further, in the moving picture compression apparatus according to (4), the setting unit 1331 uses the adjacent area of the prediction processing unit as the predicted image area.

Thereby, the prediction accuracy can be improved in any prediction mode.

(8) Further, in the moving picture compression apparatus according to the above (1), the image processing unit 1220 receives image data (for example, image data of the image area 910b) from the corresponding imaging area in each of the plurality of frames. As for the missing area 910c which has not been output, a plurality of frames are output by duplicating based on the image data. Then, the setting unit 1331 selects a prediction target based on the resolution of the prediction target image area in the plurality of image areas corresponding to the plurality of imaging areas in the prediction target frame among the plurality of frames output from the image processing unit 1220 Set a prediction processing unit for predicting an image area.

Thereby, the low resolution image area is restored, and the prediction mode can be applied.

(9) Further, the electronic device described above includes the imaging device 100 having a plurality of imaging regions in which different resolutions can be set, a setting unit 1331, a prediction unit 1332, and an encoding unit 1340. The imaging element 100 has a plurality of imaging areas in which different resolutions can be set. The setting unit 1331 sets a prediction processing unit for predicting the prediction target image area based on the resolution of the prediction target image area in the plurality of image areas corresponding to the plurality of imaging areas in the prediction target frame among the plurality of frames. Do. The prediction unit 1332 predicts a prediction target image region based on the prediction processing unit set by the setting unit 1331. The encoding unit 1340 encodes a prediction target frame using the prediction result of the prediction unit 1332.

Thereby, in the intra-frame prediction, prediction according to the difference in resolution can be partially performed, and the electronic device 500 capable of optimizing compression processing according to the resolution can be realized. Examples of the electronic device 500 described above include a digital camera, a digital video camera, a smartphone, a tablet, a surveillance camera, a drive recorder, and a drone.

(10) Also, the above-described moving picture compression program causes moving picture compression that causes the processor 1201 to compress moving picture data including a plurality of frames generated from the output of the imaging device 100 having a plurality of imaging areas in which different resolutions can be set. It is a program. The moving picture compression program causes the processor 1201 to predict the prediction target image area based on the resolution of the prediction target image area in the plurality of image areas corresponding to the plurality of imaging areas in the prediction target frame among the plurality of frames. An encoding process for encoding a frame to be predicted using a prediction process for predicting an image area to be predicted based on a setting process for setting a prediction process unit and a unit area set by the setting process Execute the processing.

As a result, in intra-frame prediction, it is possible to partially perform prediction according to the difference in resolution, and it is possible to optimize compression processing according to the resolution. The moving picture compression program may be recorded on a portable recording medium such as a CD-ROM, a DVD-ROM, a flash memory, or a memory card 504. Also, the moving picture compression program may be recorded in a moving picture compression apparatus or a server that can be downloaded to the electronic device 500.

Reference Signs List 100 imaging device, 200 imaging plane, 202 unit group, 500 electronic device, 502 control unit, 600 moving image file, 1210 pre-processing unit, 1220 image processing unit, 1230 acquisition unit, 1240 compression unit, 1310 motion detection unit, 1311 motion compensation Unit, 1320 determination unit, 1330 intraframe prediction processing unit, 1331 setting unit, 1332 prediction unit, 1340 encoding unit

Claims

A moving image compression apparatus that compresses moving image data including a plurality of frames generated from an output of an imaging device having a plurality of imaging regions in which different resolutions can be set.
Setting a prediction processing unit for predicting the prediction target image area based on the resolution of the prediction target image area in the plurality of image areas corresponding to the plurality of imaging areas in the prediction target frame among the plurality of frames Department,
A prediction unit that predicts the prediction target image area based on the prediction processing unit set by the setting unit;
An encoding unit that encodes the frame to be predicted using a prediction result from the prediction unit;
A video compression device.
The video compression apparatus according to claim 1, wherein
The setting unit, when the resolution of the prediction target image area is higher than the resolutions of other image areas other than the prediction target image area, the setting processing unit for predicting the prediction target image area is the other image area A moving picture compression apparatus that sets a prediction processing unit smaller than a prediction processing unit that predicts.
The video compression apparatus according to claim 1, wherein
The setting unit, when the resolution of the prediction target image area is lower than the resolution of another image area other than the prediction target image area, the setting processing unit for predicting the prediction target image area is the other image area A moving picture compression apparatus that sets a prediction processing unit larger than a prediction processing unit that predicts.
The video compression apparatus according to claim 1, wherein
The setting unit is configured to set the prediction processing unit among the plurality of prediction modes that use an image region predicted by the prediction unit in the prediction target frame based on a position of the prediction processing unit in the prediction target frame. Set the specific prediction mode to apply,
The prediction unit predicts the prediction target image area by applying the particular prediction mode to the prediction processing unit.
The video compression apparatus according to claim 4, wherein
The setting unit sets a specific prediction mode to be applied to the prediction processing unit based on the resolution of the predicted image area.
The video compression apparatus according to claim 5, wherein
The moving picture compression apparatus, wherein the resolution of the predicted image area is the same resolution as the resolution of the prediction processing unit.
The video compression apparatus according to claim 4, wherein
The setting unit may use an adjacent area of the unit of prediction processing as the predicted image area.
The video compression apparatus according to claim 1, wherein
An image for outputting the plurality of frames by copying based on the image data with respect to a defective area in which image data is not output from the corresponding imaging area in the image area of each of the plurality of frames Has a processing unit,
The setting unit is configured to perform the prediction based on a resolution of a prediction target image area in a plurality of image areas corresponding to the plurality of imaging areas in the prediction target frame among the plurality of frames output from the image processing unit. A moving picture compression apparatus for setting a prediction processing unit for predicting a target image area.
A moving image compression apparatus that compresses moving image data including a plurality of frames generated from an output of an imaging device having a plurality of imaging regions in which different resolutions can be set.
A setting unit configured to set a prediction processing unit for predicting an image of the target area based on the resolution of the target area in the target frame among the plurality of frames;
A prediction unit that predicts the prediction target image area based on the prediction processing unit set by the setting unit;
An encoding unit that encodes the frame to be predicted using a prediction result from the prediction unit;
A video compression device.
An imaging element having a plurality of imaging areas in which different resolutions can be set;
The prediction target image area is predicted based on the resolution of the prediction target image area in the plurality of image areas corresponding to the plurality of imaging areas in the prediction target frame among the plurality of frames generated from the output of the imaging element A setting unit for setting a prediction processing unit to be
A prediction unit that predicts the prediction target image area based on the prediction processing unit set by the setting unit;
An encoding unit that encodes the frame to be predicted using a prediction result from the prediction unit;
Electronic equipment having.
A moving image compression program that causes a processor to execute compression of moving image data including a plurality of frames generated from an output of an imaging device having a plurality of imaging regions in which different resolutions can be set.
In the processor,
Setting a prediction processing unit for predicting the prediction target image area based on the resolution of the prediction target image area in the plurality of image areas corresponding to the plurality of imaging areas in the prediction target frame among the plurality of frames Processing and
Prediction processing for predicting the prediction target image area based on the unit area set by the setting process;
An encoding process for encoding the frame to be predicted using a prediction result by the prediction process;
A video compression program that runs