JP2016009957A - Image processing apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus which reduces an area of a memory to be used when processing a moving image and a still image.SOLUTION: An image processing apparatus includes: a frame memory which has a plurality of frames that store images; means for inputting image data; storage means which stores the input image data to a first frame of the frame memory; resolution conversion means which reads the image stored in the storage means and converts resolution of the image; and means for writing the image with converted resolution to a second frame in the frame memory. An image to be read and an image to be written are divided into a plurality of areas in a horizontal direction, and when performing sequential resolution conversion processing, in using areas of the first frame and the second frame while partially overlapping them, the first and second frames are arranged on the frame memory so that an area of an image to be written and an area of an image not to be read yet do not overlap with each other on the frame memory, and resolution is converted.

Description

  The present invention relates to an image processing apparatus.

  A DRAM is mainly used as an intermediate buffer for processing images taken with a video camera or a digital camera. Since the number of pixels is increasing in recent years, the capacity used for DRAM tends to increase. The price of DRAM is generally higher as the capacity is larger. Therefore, it is important in terms of cost to design a product using a memory area efficiently and using a small capacity DRAM.

  Conventionally, there is an attempt to arrange the memory so that the write address does not overtake the read address. Also, there is an attempt to reduce the capacity by partially using a buffer for storing input / output images for image processing and using the input / output areas in an overlapping manner. For example, in an image processing apparatus that compresses uncompressed image data stored in a buffer into JPEG or the like and writes it back to the buffer as compressed image data, the image compression rate is set, the buffer area of the uncompressed image data and the compressed image data There is an attempt to reduce the memory usage capacity by overlapping the buffer areas (Patent Document 1).

  In general, when performing resolution conversion processing using dedicated hardware, line memory is used as a buffer to perform resolution conversion. Since the number of pixels has been increasing in recent years, the number of pixels of an image handled in resolution conversion processing tends to increase. When processing an image with the number of pixels in the horizontal direction exceeding the capacity of the line memory, it is necessary to divide the image into a plurality of regions, divide the processing into a plurality of times, and sequentially perform resolution conversion.

JP 2008-176481 A

According to Patent Document 1, the amount of image data compression and the amount of buffer to be overlaid are set in advance. When compression is not sufficient and the output data is likely to overwrite the input data, the compression ratio is increased to prevent overwriting. ing. Therefore, it is not possible to determine the overlay amount according to the size of the output data.
Although it is possible to superimpose the areas where the input / output data is stored according to the enlargement magnification, there are the following problems.

  When performing resolution conversion to the number of pixels in which the number of pixels in the horizontal direction exceeds the capacity of the line memory, the image is divided into a plurality of areas and the resolution conversion is performed in multiple times.

  In such a case, when the input and output addresses are simply determined only by the magnification and the memory is overlaid, the area that has not yet been read out of the divided areas is written. The amount of overlap cannot be optimized.

  An object of the present invention is to reduce the memory area used when processing a moving image or a still image.

  The configuration of the image processing apparatus according to the present invention includes a frame memory having a plurality of frames for storing images, means for inputting image data, and storage means for storing the input image data in the first frame of the frame memory; A resolution conversion unit that reads an image stored in the storage unit and converts the resolution of the image; and a unit that writes the resolution-converted image to the second frame in the frame memory. When the image is divided into multiple regions in the horizontal direction and the resolution conversion process is performed sequentially, when the first frame and second frame regions are partially overlapped, the image to be written and the image to be read are still read. Place the first and second frames on the frame memory and convert the resolution so that the unexposed areas are not the same on the frame memory. And features.

  According to the present invention, even when resolution conversion is performed by dividing into a plurality of areas, it is possible to maximize the amount of overlap of the input frame and the output frame, and the effect of reducing the memory area used is achieved. can get.

It is a figure which shows the structure of an imaging device. It is a figure which shows the structure of a resolution conversion part. It is a figure which shows the structure of a frame memory. It is a flowchart of a resolution conversion process. It is a flowchart of an input / output frame address calculation process. It is a figure which shows the state of the frame memory at the time of video recording. It is a figure which shows the state of the frame memory at the time of still image recording during video recording. It is a figure which shows the relationship of the input / output frame of a frame memory. It is a figure which shows arrangement | positioning of the input-output frame of a frame memory. It is a figure which shows the state of the frame memory when the resolution conversion magnification changes.

  The present invention is applicable to devices that perform image resolution conversion, such as video cameras, digital still cameras, and mobile phones.

(Example)
FIG. 1 shows a configuration example of a digital video camera 100 having means for recording still images and moving images to which the present invention is applied.

  A configuration of the video camera 100 will be described. The digital video camera 100 includes a lens unit 101, an image sensor 102, an analog front end (hereinafter referred to as AFE) 103, an image input unit 104, a system control unit 105, an operation unit 106, a resolution conversion unit 107, and an output unit 108. Furthermore, a ROM 109, a RAM 110, a frame memory 111, a resolution conversion unit 112, a still image encoding unit 113, a recording medium control unit 114, a recording medium 115, and an internal bus 116 are provided.

  The lens unit 101 is a correction lens group that has both a function of correcting the imaging position moved by the movement of the fixed lens group, the variable power lens group, the stop, and the variable power lens group for focusing, and the function of performing focus adjustment. The imaging sensor 102 configured by the above converts the optical image formed by the lens unit 101 into an electrical signal. The AFE 103 reads the image signal by driving the image sensor 102, and samples and holds the image signal, amplifies it, and converts it into a digital signal.

  The image input unit 104 performs color conversion processing on the image signal input from the AFE 103 and converts the image signal into YCC (4: 2: 2) image data. The system control unit 105 is connected to the RAM 110, the ROM 109, and other blocks via the internal bus 116, and controls each block.

  The operation unit 106 transmits input contents of user operations to the zoom lever, buttons, and the like, and the system control unit 105 performs various processes according to user instructions. In response to an instruction from the system control unit 105, the resolution conversion unit 107 performs resolution conversion on the image data read from a desired position in the frame memory 111 and outputs the image data to the output unit 108.

  The output unit 108 outputs the image data input from the resolution conversion unit 107. The ROM 109 stores a control program for the system control unit 105 to control each block. In the RAM 110, the system control unit 105 is used as a temporary data storage location during operation.

  The frame memory 111 is a memory composed of a plurality of frames. Each frame can be arranged at a desired position on the memory by the system control unit 105, and is used to store image data for one frame. Each frame can occupy the same area of the frame memory 111. Each frame can be set to a desired size by the system control unit 105. The address and size of each frame can be dynamically changed, and can be switched between recording and playback of moving images.

  The resolution converter 112 is mainly used for still image shooting, reads the image data stored in the frame memory 111, converts the resolution to the size designated by the system controller 105, and then writes it to the frame memory 111. Returned.

  The still image encoding unit 113 encodes the image data stored in the frame memory 111 according to the target compression rate instructed by the system control unit 105, and stores the encoded image data in the frame memory 111 as encoded still image data. Retained.

  The recording medium control unit 114 writes the data stored in the frame memory 111 to the recording medium 115 together with the management information according to an instruction from the system control unit 105.

  The recording medium 115 is a detachable recording medium. A memory card such as an MMC, SD, compact flash, or PC card, or an optical (or magneto-optical) recording medium such as a hard disk, CD-RW, DVD-R / RW, or MD can be considered.

  Next, the configuration of the resolution conversion units 107 and 112 will be described with reference to FIG. The configurations of the resolution conversion units 107 and 112 are the same. The resolution conversion unit includes a zoom control unit 201, a memory interface 202, a line memory 203, a horizontal zoom unit 204, and a vertical zoom unit 205.

  The zoom control unit 201 controls the start and end timing of resolution conversion and controls the memory interface 202, the line memory 203, the horizontal zoom unit 204, and the vertical zoom unit 205 according to instructions from the system control 105. The zoom control unit 201 determines the address of the frame in the frame memory 111 that stores the input image and output image from the system control unit 105, the reading and writing position in the frame, the horizontal and vertical resolutions of the input and output images, and the zoom magnification. Instructed. The zoom control unit 201 issues a zoom magnification instruction to the horizontal zoom unit 204 and the vertical zoom unit 205. The zoom control unit 201 also instructs the memory interface about the frame address of the input / output image and the reading and writing position within the frame.

  The memory interface 202 is an interface with the frame memory 111 and the output unit 108. In accordance with an instruction from the zoom control unit 201, image data reading and writing control are performed.

  The line memory 203 can store a predetermined amount of image data in the horizontal direction. The maximum horizontal resolution of image data output in one process is determined by the size of the line memory 203.

  The horizontal zoom unit 204 performs resolution conversion in the horizontal direction. Outputs data obtained by performing interpolation processing such as linear interpolation and bicubic interpolation (Bi-Cubic) using a plurality of points of input data.

  The vertical zoom unit 205 performs resolution conversion in the vertical direction. A plurality of line memories are provided inside, and output is obtained by performing interpolation processing such as linear interpolation and bicubic interpolation (Bi-Cubic) using a plurality of points of input data as in the horizontal direction.

  Next, resolution conversion processing by the resolution conversion units 107 and 112 will be described. The system control unit 105 reads data necessary for zoom processing such as the resolution of the input image and the output image from the ROM 109 to the resolution conversion unit 107.112, and issues an instruction to the zoom control unit 201. In response to an instruction from the zoom control unit 201, the memory interface 202 reads several pixels of data in the horizontal direction of the image from the frame memory 111 and inputs them to the horizontal zoom unit 204. The input data is converted into a horizontal resolution output by the horizontal zoom unit 204 and stored in the line memory 203. The data stored in the line memory 203 is input to the vertical zoom unit 205 according to an instruction from the zoom control unit 201. After a plurality of lines are read into the line memory inside the vertical zoom unit 205, they are converted to the vertical resolution to be output. The converted data is written back to the frame memory 111 by the memory interface 202.

  Next, the operation when the horizontal resolution of the output image is larger than the size of the line memory will be described. First, the configuration of the frame memory and the input / output image size will be described with reference to FIG.

  Assume that the size of the input image is stored in the frame 303 of the frame memory 111 as horizontal 1600 pixels and vertical 900 pixels, and the output image size is horizontal 1920 pixels and vertical 1080 pixels, and is stored in the frame 304. . It is assumed that the size of the line memory can store 640 pixel data. The frame 303 can store an image of horizontal 1600 pixels and vertical 900 pixels, and the frame 304 can store an image of horizontal 1920 pixels and vertical 1080 pixels.

  Next, the flow of processing will be described using the flowchart of FIG.

STEP: S401 The horizontal resolution is the amount that can be accommodated in the line memory. When performing resolution conversion processing, the system control unit 105 acquires the size of the output image, and the size of the line memory 203 is smaller than the horizontal size of the output image. Judge whether or not.

STEP: S402 Resolution conversion processing If the size of the line memory is smaller than the size of the output image, the resolution conversion processing is performed without dividing the region in the horizontal direction.

STEP: S403 Dividing in the horizontal direction When the horizontal size of the output image is larger than the size of the line memory 203 in S401, the system control unit 105 divides the image into a plurality of areas in the horizontal direction. In this example, the horizontal size of the output image is 1920 pixels and the size of the line memory is 640, so 1920 is divided by 640 to be 3. Therefore, as shown in FIGS. 3B and 3C, the frame memory is divided into three regions, resizing is performed three times so that the output is 640 pixels, and an image of 1920 pixels is output.

STEP: S404 Specify the first area Next, specify the area to be processed first. In this case, it is assumed that resolution conversion processing is sequentially performed from the left area among the divided areas. An area 3031 of the input frame 303 is an input image area, and an area 3041 of the output frame 304 is an output image storage destination.

STEP: S405 Resolution Change Processing The resolution conversion units 107 and 112 perform the above-described resolution conversion processing on the area determined in S404 according to an instruction from the system control unit 105.

STEP: S406 Have you converted all the areas?
The system control unit 405 determines whether the resolution conversion processing for all areas in the input image has been completed. When the resolution conversion process for all the areas is completed, the resolution conversion process ends.

STEP: S407 Specify the next area When the zoom process has not been completed for all areas in S406, an area that has not been subjected to the resolution conversion process is selected and used as the input and output of the resolution converter.

  By performing the processing as described above, the resolution conversion processing can be performed even if the horizontal resolution of the output image is larger than the size of the line memory.

  Next, an arrangement method of input frames and output frames at the time of resolution conversion processing according to the present invention will be described with reference to the flowchart of FIG.

STEP: S501 Acquisition of Image Resolution First, the system control unit 105 acquires the resolution of the image output by the resolution conversion units 107 and 112. The resolution of the image to be output is instructed from the operation unit 106 by the user, or the resolution determined by the operation mode such as moving image shooting or still image shooting is stored in the ROM 109.

STEP: S502 The horizontal resolution of the output image is larger than the capacity of the line memory. The system control unit 105 determines whether the horizontal resolution of the output image is larger than the capacity of the line memory.

STEP: S503 Size adjustment in the horizontal direction of the frame memory This processing is performed when the horizontal resolution of the image output in S502 is larger than the capacity of the line memory. When the resolution of the output image in the horizontal direction is large, it is necessary to divide the region in the horizontal direction and perform resolution conversion processing in multiple steps.

  Hereinafter, the same resolution conversion process as that used in the description of the flowchart of FIG. 4 will be described. Assume that the size of the input image is stored in the frame 303 of the frame memory 111 as horizontal 1600 pixels and vertical 900 pixels, and the output image size is horizontal 1920 pixels and vertical 1080 pixels, and is stored in the frame 304. It is assumed that the size of the line memory can store 640 pixel data. The frame 303 can store an image of horizontal 1600 pixels and vertical 900 pixels, and the frame 304 can store an image of horizontal 1920 pixels and vertical 1080 pixels.

  FIG. 8 shows the positional relationship of the input / output frames. The upper band represents the frame 303 in FIG. 3 and is an input frame. The lower band represents the frame 304 in FIG. 3 and is an output frame. The address increases as it goes in the horizontal direction, and the area is divided by a predetermined amount.

  Here, it is assumed that the input / output frame has the same start address. If the area 3031 is read and resolution conversion is performed and the area 3041 is written, the area 3032 that has not yet been read is overwritten (FIG. 8A). By shifting the address of the input frame or the output frame, the first area 3032 is not overwritten, but any area of the areas 3031 to 3033 is overwritten thereafter. This occurs because the repetition cycle of the regions 3031 to 3033 and the regions 3041 to 3043 is different.

  Therefore, it is necessary to match the smaller period with the larger period. In this example, since the horizontal size of the input frame 303 is small, the horizontal size of the frame 303 is increased so that data of 1920 pixels can be stored (FIG. 8B). By changing the address of the frame 303, which is the input frame, after adjusting the size in the horizontal direction, it is possible to move to a place where the area of the input frame that has not yet been read is overwritten by the output frame data (FIG. 8 ( c)). Specifically, the start address of the final area of the input frame (here, area 3033) among the divided areas needs to be larger than the start address of the final area of the output frame (here, area 3043). Furthermore, the end address of the last area (here, area 3033) of the input frame needs to be smaller than the start address of the first area (here, area 3041) of the output frame.

  In this example, enlargement is used as an example, but when reducing the size, a frame with a small horizontal size is aligned with a large frame in the same way as when enlarging, and the input data that has not yet been read is overwritten with the resolution conversion output data. You can do it.

STEP: S504 Input / output frame address calculation In S503, an address is calculated so that horizontal data is not overwritten. In this step, an address calculation method when vertical resolution conversion is performed will be described.

  The amount of input / output addresses that can be overlaid will be described with reference to FIG. The same resolution conversion processing conditions as those in S503 are used. Further, it is assumed that the vertical zoom unit 205 of the resolution conversion units 107 and 112 performs interpolation calculation from the data of the two lines before and after the resolution conversion in the vertical direction.

  If the image is enlarged in the vertical direction, more data is output than input data. If the addresses of the input and output frames are the same, the input frame data that has not yet been read is overwritten by the output data of the resolution conversion process.

  If the arrangement is made so that the end address of the input frame is larger than the end address of the output frame, the unread area of the input frame is not destroyed.

  This will be described with reference to FIG. In this example, the start address of the input frame should be set as follows. The number of vertical lines of the image of the input frame is 900, and the number of vertical lines of the image of the output frame is 1080. The horizontal size of the input / output frame is the same in S503. Therefore, the start address of the input frame may be arranged at an offset larger than 180 lines from the start address of the output frame. The amount of overlay can be maximized when the offset is 180 lines.

  If the data is reduced in the vertical direction, the input data is less than the data to be output, so the input data is not overwritten. As shown in FIG. 9B, the start addresses of the input frame and the output frame can be made the same.

STEP: S505 Input / output frame address setting The system control unit 105 sets the input frame and output frame addresses calculated in the previous step.

STEP: S506 Input / output frame address calculation This process is performed when the horizontal resolution of the output image is smaller than the capacity of the line memory. Since it is smaller than the capacity of the line memory, it is not necessary to divide the area in the horizontal direction during the resolution conversion process. If the arrangement is made so that the end address of the input frame is larger than the end address of the output frame, the unread area of the input frame is not destroyed.

  When the frames are overlapped according to the flowchart, the relationship between the frame 303 and the frame 304 is as shown in FIG. Here, it is assumed that the resolution conversion magnification is changed in the resolution conversion units 107 and 112 as shown in FIG. 10B and the size of the image input from the input frame 303 is reduced. Furthermore, if the size of the output image is not changed, it is necessary to perform resolution conversion processing by dividing the output image into a plurality of areas. When the horizontal size of the input image changes, the addresses of the areas 3031 to 3033 change. The areas 3031 to 3033 are sequentially read, resolution conversion processing is performed, and the areas 3041 to 3043 are written. When the resolution conversion from the area 3032 to the area 3042 is performed, the area 3033 that has not yet been read is overwritten.

  Therefore, when the size of the input frame is different from the size of the input image, it is necessary to rearrange the input / output frames as shown in FIG. By aligning the horizontal and vertical end addresses of the input image with the horizontal and vertical end addresses of the output frame, the output image will not overwrite an unread area of the input image. Even when the resolution conversion magnification is changed, by using the above method, it is possible to reduce the amount of memory required by multiplying the horizontal size of the frame, which is the shaded area in FIG. 10D, by the vertical size of the input image. Savings can be made. By arranging the memory by the memory arrangement method according to the present invention, the free space of the memory increases and the area for storing the encoded image data such as JPEG increases. As a result, it is possible to increase the number of continuous shots of simultaneous still image recording when performing simultaneous recording of still images during moving image shooting. Here, the input / output frame arrangement method is determined every time the resolution conversion process is performed. However, in a system in which the resolution conversion magnification is made clear in advance, it is possible to construct a system equipped with a minimum DRAM by determining the input / output frame addresses in advance according to the above flowchart.

  Next, the description at the time of moving image shooting will be made with reference to the block diagram of FIG. 1 and FIG. At the time of moving image shooting, the light transmitted through the lens unit 101 forms an image on the imaging surface of the imaging sensor 102, and the optical image on the imaging surface is converted into an electrical signal every predetermined time. The converted electrical signal is converted into a digital signal by the AFE 103. The converted digital signal is subjected to color conversion processing by the image input unit, and after conversion, is stored in the frame memory 111 as YCC (4: 2: 2) image data.

  The stored YCC image data is subjected to resolution conversion processing by the resolution conversion unit 107 and output to the output unit 108. Output destinations include a monitor that displays image data, an image data recording device that records image data, and the like.

  FIG. 6 shows the flow and timing of shooting data during moving image shooting. It shows the state of the frame memory 111 every predetermined time, and the system controller 105 controls the time interval. Each block performs processing based on this time interval, and processing described below is assumed to be completed at this time interval unless otherwise specified.

  A signal captured from the image sensor 102 at a certain time T1 (FIG. 6) is converted into image data according to the above processing and stored in the frame 301 in the frame memory 111.

  At the next time T2 (FIG. 6), the resolution conversion unit 107 performs resolution conversion processing on the image data stored in the frame 301 and outputs it to the output unit. A signal captured from the image sensor 102 is converted into image data and stored in the frame 302.

  At the next time T3 (FIG. 6), the input of the resolution converting unit 107 is set to the frame 302, the image taken from the sensor is stored in the frame 301, and the same processing is performed. By alternately using the two frames, it is possible to prevent the image captured from the sensor from overwriting the area while the resolution conversion unit 107 reads the image data.

  Next, the operation of each block and the flow of shooting data when shooting still images simultaneously during moving image shooting will be described with reference to FIGS. Similar to FIG. The state of the frame memory 111 for every predetermined time is shown. The frames 301 and 302 are arranged at different addresses, and the areas do not overlap. Since the frames 303 and 304 are arranged using the frame memory arrangement method according to the present invention, a part of each area is shared.

  Simultaneous shooting during moving image shooting is to record a still image at a desired timing during moving image shooting. Assume that the user makes a still image shooting request at the timing T1 (FIG. 7) during the moving image shooting.

  The image data captured from the image sensor 102 at T2 is stored in the frame 303. Frames 301 and 302 are alternately used during normal moving image shooting, but when an image to be recorded as a still image is stored in the frame, resolution conversion takes time when the image size to be handled is large. A frame 303 different from the frames 301 and 302 is used so that the sensor captured image is not overwritten during resolution conversion.

  The image captured from the sensor at T3 is stored in the frame 301. After this, unless there is a new request for still image shooting, images captured from the sensor are alternately stored in 301 and 302 every predetermined time. The image data stored in the frame 303 is subjected to resolution conversion processing by the resolution conversion unit 107 and output to the output unit 108.

  At T 4, the resolution conversion unit 107 performs resolution conversion processing on the image data stored in the frame 301 and outputs it to the output unit 108. The resolution conversion unit 112 performs resolution conversion processing of image data for still image recording. The image data stored in the frame 303 is subjected to resolution conversion processing by the resolution conversion unit 112 and stored in the frame 304.

  The size of the image stored in the frame 303 is 1600 pixels horizontally and 900 pixels vertically, and the image size stored in the frame 304 is 1920 pixels horizontally and 1080 pixels vertically. It is assumed that the size of the line memory can store 640 pixel data. Assume that the size of the frame 303 can store an image of horizontal 1600 pixels and vertical 900 pixels, and the size of the frame 304 can store an image of horizontal 1920 pixels and vertical 1080 pixels. Since the horizontal resolution of the output image is 1920 and the data can be stored only up to 640 pixels in the line memory size, it is necessary to execute the resolution conversion process by dividing the area at least three times.

  Here, it is assumed that resolution conversion is performed in three steps, and the resolution conversion processing is performed by dividing the image into a plurality of areas as described above during the period of T4, T5, and T6. T5 and T6 perform the same processing as T4.

  In T7, the still image encoding unit 113 reads the image subjected to the resolution conversion processing of the frame 304, performs encoding according to the target compression rate instructed by the system control unit 105, and generates encoded still image data. Stored in the RAM 110.

  The reason why the resolution conversion unit 112 for moving picture output and the resolution conversion unit 112 for still image shooting are not simultaneously processed at T4 is that the areas of the frames 304 and 305 partially overlap. Since they overlap, when the resolution conversion unit 112 performs resolution conversion processing from the frame 304 to the frame 305, a part of the frame 304 is updated. If the resolution conversion unit 107 is updated before reading image data from the frame 304, the image written by the resolution conversion unit 112 is read. Therefore, resolution conversion for creating a still image is input to the output unit. Needs to be created after creating.

100 video camera, 101 lens unit, 102 image sensor,
103 analog front end, 104 image input unit, 105 system control unit,
106 operation unit, 107 resolution conversion unit, 108 output unit, 109 ROM1,
110 RAM, 111 frame memory, 112 resolution converter,
113 still image encoding unit, 114 recording medium control unit, 115 recording medium,
116 Internal bus

Claims (4)

A frame memory having a plurality of frames for storing images;
Means for inputting image data;
Storage means for storing input image data in a first frame of the frame memory;
A resolution conversion unit that reads the image stored in the storage unit and converts the resolution of the image; and a unit that writes the image whose resolution is converted to the second frame in the frame memory;
When the image to be read and the image to be written are divided into a plurality of areas in the horizontal direction and the resolution conversion process is sequentially performed, the image to be written is read when the areas of the first frame and the second frame are partially overlapped. An image processing apparatus comprising: a first frame and a second frame arranged on a frame memory so that an unread region of an image is not the same on the frame memory; and resolution conversion.   Making the horizontal size of the first frame and the second frame the same; calculating the difference between the vertical size of the read image and the vertical size of the image to be written; and 2. The image processing apparatus according to claim 1, wherein the n-1th area of the image to be written is arranged so that the horizontal position does not overlap with the nth image of the read image.   Means for acquiring the vertical size of the read image and the image to be written; and means for acquiring addresses on the frame memory of the read image and the image to be written; and the vertical size of the read image and the image to be written; The image processing apparatus according to claim 1, wherein an arrangement of the first frame and the second frame is determined.   When recording the first frame in the moving image stream when simultaneously capturing a still image while shooting a moving image, the resolution conversion processing is performed after the image of the first frame is recorded in the moving image stream. The image processing apparatus according to claim 1.