JP2008011224A - Image processing device, image transmitting/receiving system, and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To enable image processing to be easily carried out when an image is partially enlarged. <P>SOLUTION: Enlarging region information indicating a region that is enlarged in an image is stored in a header when image data are encoded. The image region indicated by the enlarging region information is set at one of encoding regions. The image data are encoded by the encoding region and outputted together with the header. When the encoded image data are decoded, partial image data corresponding to the encoding region indicated by the enlarging region information are decoded when it is detected that the enlarging region information indicating one of the encoding regions is contained in the header. Then, the decoded partial image data are magnified as large as prescribed magnification for performing display on a display device. Therefore, when a part of an image is magnified, image processing can be easily carried out. The enlarged image and original image can be selectively displayed without increasing a load imposed on a controller that processes the image data. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image processing technique for capturing a moving image and enlarging and displaying the captured moving image.

When a subject is photographed using a zoom function in a video camera or the like, only an image enlarged by zooming is recorded in a memory or the like and displayed on a monitor. Images that are not displayed on the monitor are not recorded in the memory (see, for example, Patent Documents 1 and 2). For this reason, when the subject is photographed using the zoom function, the entire image cannot be displayed. In order to solve this problem, there has been proposed a method of recording an original image that has not been zoomed and zoom information indicating a zoomed image area in a memory or the like (see, for example, Patent Document 3). In general, since the amount of data is large, the image data is encoded and recorded in a memory. The image data read from the memory is displayed after being decoded.
JP-A-8-294120 JP 2005-192097 A JP 2004-248171 A

  However, in the above-described method, after decoding the image data read from the memory, it is necessary to cut out the partial image data from the decoded image data according to the zoom information, and to enlarge the cut out partial image data. Since an extra process for cutting out the partial image data is necessary, the load on the controller such as a CPU that processes the image data increases. In particular, when the video is displayed on a monitor or the like while shooting a subject, the controller needs to display an enlarged image in real time in addition to shooting control. For this reason, an increase in load is not preferable. When the processing capacity of the controller is insufficient by the above method, it is necessary to use a higher-performance controller, and the cost of the system increases.

  An object of the present invention is to simplify image processing when a part of an image is enlarged.

  When encoding image data, enlarged area information indicating an area to be enlarged in the image is stored in the header. The image area indicated by the enlarged area information is set as one of the encoding areas. The image data is encoded for each encoding area, and the encoded image data is output together with the header. When decoding encoded image data, if it is detected that the header includes expanded area information indicating any of a plurality of encoded areas, a portion corresponding to the encoded area indicated by the expanded area information Image data is decoded. Then, the decoded partial image data is enlarged to a predetermined magnification for display on the display device.

  By matching the area to be enlarged with the coding area, it is not necessary to store the coordinates of the image to be enlarged in a special area and to pass it to the decoding processing side. That is, a special image cutout process for enlarging the image can be eliminated. Therefore, image processing can be simplified when enlarging an image. The enlarged image and the image before enlargement can be selectively displayed without increasing the load on the controller that processes the image data.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 shows a first embodiment of the present invention. In this embodiment, the invention is applied to a digital video camera VCAM. The digital video camera VCAM has an imaging unit 10 having an imaging device such as a CCD, an operation unit 12, a memory 14, a display unit 16 having an LCD or the like on which a captured moving image is displayed, a CPU, a video CODEC, and the like. Yes. The operation unit 12 has a switch operated by a user. The switch is used to set shooting conditions, adjust the zoom amount (specify the enlargement ratio), and specify the zoom position (enlarged display area). The memory 14 has, for example, a buffer area for storing encoded image data (stream data) and the like, a program area for storing a program executed by the CPU, and the like.

  The CPU includes a zoom control unit 18, a header analysis unit 20, and an enlargement unit 22. The zoom control unit 18, the header analysis unit 20, and the enlargement unit 22 are realized by a program executed by the CPU. The CODEC includes a header encoding unit 24 (storage unit), an image encoding unit 26, a header decoding unit 28, and an image decoding unit 30. For example, the CODEC is formed as a macro mounted on the system LSI together with the CPU. The CODEC may be configured with one chip or may be configured as software. In this embodiment, the image data is encoded and decoded using, for example, the H.264 standard. In the H.264 standard, slices (video packets) can be arranged not only in a regular order (raster scan) but also in an arbitrary order (ASO: Arbitrary Slice Order). In addition, the H.264 standard has a higher compression rate than other video coding standards.

  The zoom control unit 18 outputs enlarged information indicating an enlarged display area (enlarged area information) and an enlargement rate set by the operation of the operation unit 12 to the header encoding unit 24. The header encoding unit 24 sets the enlarged display area to one of a plurality of slices (encoding areas), as will be described with reference to FIG. The header encoding unit 24 stores the slice number and the enlargement ratio corresponding to the enlarged display area in the header, and encodes the header. The image encoding unit 26 encodes image data for each slice according to the set slice. The image encoding unit 26 outputs the encoded header and image data to the memory 14 as stream data. As described above, the header encoding unit 24 and the image encoding unit 26 store the expansion information in the header, set the expansion area to one of the slices, and encode each slice (image processing apparatus). ).

  The header decoding unit 28 decodes the header included in the stream data received from the memory 14 and outputs the decoded header to the image decoding unit 30 together with the encoded image data. At this time, the header analysis unit 20 of the CPU analyzes the contents of the header decoded by the header decoding unit 28, and controls the operation of the image decoding unit 30 according to the analysis result.

  The image decoding unit 30 receives an instruction from the header analysis unit 20 and decodes the image data based on the header information supplied from the header decoding unit 28. For example, when the slice number is stored in the header, the image decoding unit 30 decodes only the slice indicated by the slice number as partial image data. The image decoding unit 30 outputs the decoded image data to the display unit 16. The enlargement unit 22 functions only when the slice number and the enlargement rate are stored in the header, and enlarges the image data decoded by the image decoding unit 30 to a predetermined magnification indicated by the enlargement rate. As described above, the header analysis unit 20, the image decoding unit 30, and the enlargement unit 22 decode the image data based on the enlargement information included in the header and generate an enlarged image (image processing device). Function. The header encoding unit 24 and the image encoding unit 26, the header analysis unit 20, the image decoding unit 30 and the enlargement unit 22 constitute an image transmission / reception system.

  Although not shown, a line memory or the like for temporarily storing image data to be displayed on the display unit 16 is disposed between the image decoding unit 30 and the display unit 16. The enlargement unit 22 enlarges the image data before being transferred from the image decoding unit 30 to the line memory. Each functional block described above operates when displaying a captured image on the display unit 16 and when displaying image data recorded in the memory 14 after shooting on the display unit 16.

  FIG. 2 shows the structure of the access unit constituting the stream data generated by the image encoding unit 26. The access unit includes an AU delimiter (Access Unit Delimiter), SPS (Sequence Parameter Set), PPS (Picture Parameter Set), SEI (Supplemental Enhancement Information), n slices, EOS (End of Sequence), and EOS (End of Stream). ). The header is composed of an AU delimiter, SPS, PPS, and SEI.

  The AU delimiter is a start code indicating the head of the access unit. The SPS is an area for storing information related to the sequence. The PPS is an area for storing information on the coding mode of the entire image (picture). The SEI is an area for storing additional information that is not directly related to the encoding process and the decoding process. The SEI has initial time information, time information, a user data area (User Data Unregistered SEI), an extension area, and the like. The user data area is an area where the usage (designer of the video camera VCAM) can be uniquely defined by the user. In the present invention, the slice number and the enlargement ratio are stored in the user data area.

  FIG. 3 shows an example of slice settings. In this embodiment, “type 2” is adopted as a slice specification among a plurality of slice types that can be specified by the H.264 standard. In “Type 2”, a rectangular area surrounding a main subject such as a person can be set as a foreground slice, and the remaining area can be set as a background (Leftover) slice. Each slice is configured by combining macroblocks composed of 16 pixels × 16 pixels, for example.

  The picture PCT is an image to be encoded. In this example, the picture PCT is an image of one frame of a moving image. An area indicated by a thick broken line is an area to be enlarged and set by the operation unit 12, and is designated as an enlarged display area. Specifically, an area (enlarged display area) indicated by a thick broken line is set to a slice SL2 of “Foreground”. The picture PCT is composed of, for example, three slices SL1-SL3 including the enlarged display area SL2. The slices SL1 and SL3 are “Leftover” slices. The slices SL1 to SL3 may have overlapping portions, and may be set without overlapping each other.

  FIG. 4 shows a procedure for encoding image data by the CPU and CODEC shown in FIG. Steps S102 and S112 are performed by CODEC, and other processing is performed by the CPU. The flow of FIG. 4 shows an image encoding method for encoding image data. According to the flow of FIG. 4, image data of one picture PCT (one frame of image data) is encoded.

  First, in step S100, the zoom control unit 18 monitors the state of the operation unit 12 and detects whether there is an enlargement process. If there is no enlargement process, a normal encoding process is performed in step S102. That is, for example, the picture PCT is divided into a plurality of slices according to the raster scan order, and encoding is performed for each slice to generate stream data. The slice number and the enlargement ratio are not stored in the user data area of the header.

If there is an enlargement process, it is determined in step S104 whether or not to enlarge the image by automatically setting the image size and enlargement ratio. When the automatic enlargement is selected by the mode setting of the digital video camera VCAM, the quick enlargement button, or the like, in step S106, the image size and the enlargement ratio indicating the size of the image to be enlarged are read out from the CPU register or the like. In this case, for example, the zoom control unit 18 aligns the center of the slice (enlarged display area) for enlarging processing with the center of the captured image, and reads the size of the slice (enlarged display area) from the register. Set to size. Then, the zoom control unit 18 requests the header encoding unit 24 to perform slice encoding processing in an arbitrary slice order (ASO).

  If the automatic enlargement function is not designated, in step S108, the zoom control unit 18 causes the header encoding unit 24 to set the enlargement display area and the enlargement ratio set by the operation of the operation unit 12 as enlargement information. The header encoding unit 24 is requested to perform slice encoding processing in an arbitrary slice order (ASO). The enlarged display area is defined by, for example, the number of pixels in the horizontal direction, the number of pixels in the vertical direction, and the display start position of the picture PCT. Next, in step S110, the header encoding unit 24 stores the slice number and the enlargement ratio indicating the enlarged display area instructed from the zoom control unit 18 in the header, and encodes the header. The header encoding unit 24 also performs setting of other slices. In step S112, the image encoding unit 26 performs an encoding process for each slice, and generates stream data. At this time, the image area (enlarged display area) to be enlarged is set to one of the slices. That is, the image to be enlarged is cut out as a slice before the image is encoded. Therefore, in the decoding process shown in FIG. 5, the CPU simply selects a slice corresponding to the slice number stored in the SEI area from the image data composed of a plurality of slices. The cut-out process to be performed on the conversion processing side can be performed in advance.

  FIG. 5 shows a procedure of decoding image data by the CPU and CODEC shown in FIG. Steps S200, S208, and S204 are performed by CODEC, and other processing is performed by the CPU. The flow of FIG. 5 shows an image decoding method for encoding image data. According to the flow of FIG. 5, image data of one picture PCT (one frame of image data) is generated.

  First, in step S200, the header decoding unit 28 decodes the header in the stream data. Next, in step S202, the header analysis unit 20 determines whether or not enlargement information (slice number and enlargement ratio) is stored in the user data area of the SEI area. That is, it is determined whether or not an enlarged display of the image has been instructed when the subject is photographed. When enlargement display is not instructed, the header analysis unit 20 notifies the image decoding unit 30 that a plurality of slices are encoded according to the raster scan order. In this case, in step S204, the image decoding unit 30 performs a decoding process for each slice, and generates image data for one frame.

  If enlargement display is instructed at the time of shooting, in step S206, the CPU determines whether the image is to be enlarged and reproduced or whether the image is reproduced without being enlarged. In this embodiment, for example, when displaying an image taken after photographing of a subject on the display unit 16, enlarged display or normal display (not enlarged) can be selected according to the setting of the playback mode of the digital video camera VCAM. When the enlarged display is selected, the process proceeds to step S204 described above. If normal display is selected, the process proceeds to step S208. Through the processing in step S206, the enlarged image and the entire image before enlargement can be selectively displayed on the display unit 16 regardless of whether enlargement display is instructed when the subject is photographed.

In step S208, the image decoding unit 30 decodes only the slice indicated by the slice number stored in the SEI area. The image enlarged by this decoding is
Automatically cut out. In other words, an image clipping process is included in the standard decoding process of CODEC. For this reason, the CPU does not need to cut out the image data to be enlarged using the coordinates of the image. Next, in step S210, the enlarging unit 22 performs image data enlarging processing on the decoded image data in accordance with the enlarging rate stored in the SEI area. Next, in step S212, the CPU outputs the decoded image data to the display unit 16 via, for example, a line memory. Then, the enlarged image or the image before enlargement is displayed on the display unit 16.

  As described above, in the first embodiment, the image area to be enlarged is matched with the slice (encoding area) that is a unit of image encoding / decoding, so that the coordinates of the image to be enlarged are specially set in the encoding process. It is no longer necessary to store the data in the area and pass it to the decryption processing side. Further, in the decoding process, a special clipping process for obtaining image data to be enlarged can be eliminated. In other words, the CPU does not have to perform the process of cutting out the image to be enlarged. As a result, image processing can be simplified when a part of the image is enlarged. Further, the enlarged image and the image before being enlarged can be selectively displayed without increasing the load on the CPU.

  FIG. 6 shows a second embodiment of the present invention. The same elements as those described in the first embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted. In this embodiment, the invention is applied to an image transmission / reception system CSYS such as a video conference system. The image transmission / reception system CSYS includes, for example, a plurality of image transmission / reception apparatuses that transmit / receive images to / from each other via a network NET.

  Each image transmission / reception apparatus includes the imaging unit 10, the operation unit 12, the display unit 16, the CPU, and the video CODEC of the first embodiment. Although not particularly illustrated, each image transmission / reception apparatus has a memory for temporarily storing image data and a memory for storing a program executed by the CPU. In this embodiment, one image transmission / reception apparatus performs the processing shown in FIG. 4, encodes image data including enlarged information, and transmits the stream data generated by the encoding to the other image transmission / reception via the network NET. Output to the device. The other image transmission / reception apparatus performs the processing shown in FIG. 5, decodes the stream data, enlarges the decoded image data according to the enlargement information, and displays it on the display unit 16.

  As mentioned above, also in 2nd Embodiment, the effect similar to 1st Embodiment mentioned above can be acquired. Furthermore, in this embodiment, the image transmission / reception system CSYS such as a video conference system can also eliminate the need to cut out image data for enlarging the image. As a result, the enlarged image and the image before being enlarged can be selectively displayed without increasing the CPU load.

  In the first embodiment described above, an example in which the present invention is applied to a digital video camera has been described. The present invention is not limited to such an embodiment. For example, the present invention may be applied to a digital camera having a moving image shooting function or a mobile phone having a moving image shooting function.

  In the above-described embodiment, the example in which the enlargement information is stored in the header defined by the H.264 standard and the image is enlarged is described. The present invention is not limited to such an embodiment. The present invention can be applied to a moving image encoding method in which an image is divided into a plurality of encoding regions (slices), image data is encoded for each encoding region, and user information can be stored in a header. In this case, the same effect as that of the first embodiment can be obtained.

In the above-described embodiment, the example in which the enlargement factor set by the operation unit 12 or the preset enlargement factor is stored in the header and the image is enlarged using the enlargement factor stored in the header has been described. The present invention is not limited to such an embodiment. For example, only the slice number may be stored in the header, and the image may be enlarged using a preset enlargement ratio. At this time, the enlargement ratio may be automatically adjusted so that the image is always displayed at the maximum size on the display unit 16.

  In the above-described embodiment, the example in which the presence or absence of the automatic enlargement function can be selected has been described. The present invention is not limited to such an embodiment. For example, a digital video camera VCAM that does not have the processing functions of steps S104 and S106 illustrated in FIG. 4 and does not have an automatic enlargement function in which step S108 is always performed may be configured. Alternatively, a digital video camera VCAM that does not have the processing functions of steps S104 and S108 but has only an automatic enlargement function in which step S106 is always performed may be configured.

  In the above-described embodiment, the example in which the image is encoded / decoded using “type 2” in the slice specification of the H.264 standard has been described. The present invention is not limited to such an embodiment. For example, an image may be encoded / decoded using “type 6” of the slice specifications of the H.264 standard. In “Type 6”, a table (correspondence table) indicating slices to which each macroblock belongs is stored in the header and transmitted from the image coding apparatus to the image decoding apparatus. By adopting “Type 6”, it is possible to specify a slice of any shape.

The invention described in the above embodiments is organized and disclosed as an appendix.
(Appendix 1)
An image processing apparatus that divides an image into a plurality of encoding regions and encodes image data for each of the encoding regions,
A storage unit that stores, in a header, enlarged area information indicating an area to be enlarged in the image;
An image encoding unit configured to encode image data by setting an image region indicated by the enlarged region information as one of the encoding regions, and to output the encoded image data together with a header; An image processing apparatus.
(Appendix 2)
In the image processing apparatus according to attachment 1,
The enlargement area information includes an enlargement ratio when the encoded image data is decoded and displayed as an enlarged image.
(Appendix 3)
An image processing device for decoding encoded image data for display on a display device,
A header analysis unit that detects that the extended region information indicating any of the plurality of encoded regions is included in the header;
An image decoding unit that decodes partial image data corresponding to the encoding region indicated by the expansion region information when the expansion region information is included in the header;
An image processing apparatus, comprising: an enlargement unit that enlarges the decoded partial image data to a predetermined magnification for displaying on the display device.
(Appendix 4)
In the image processing device according to attachment 3,
The enlargement area information includes an enlargement ratio when the encoded image data is decoded and displayed as an enlarged image,
The image processing apparatus, wherein the predetermined magnification is an enlargement ratio included in the enlargement area information.
(Appendix 5)
An image encoding device that divides an image into a plurality of encoding regions and encodes image data for each of the encoding regions, and an image decoding device that decodes the encoded image data for display on a display device An image transmission / reception system comprising:
The image encoding device includes:
A storage unit that stores, in a header, enlarged area information indicating an area to be enlarged in the image;
An image encoding unit that sets an image region indicated by the enlarged region information as one of the encoding regions, encodes image data, and outputs the encoded image data together with a header;
The image decoding device includes:
A header analysis unit that detects that the extended region information indicating any of the plurality of encoded regions is included in the header;
An image decoding unit that decodes partial image data corresponding to the encoding region indicated by the expansion region information when the expansion region information is included in the header;
An image transmission / reception system comprising: an enlargement unit for enlarging to a predetermined magnification in order to display the decoded partial image data on a display device.
(Appendix 6)
In the image transmission / reception system according to attachment 5,
The enlargement area information includes an enlargement ratio when the encoded image data is decoded and displayed as an enlarged image,
The image transmission / reception system, wherein the predetermined magnification is an enlargement ratio included in the enlargement area information.
(Appendix 7)
An image processing method for dividing an image into a plurality of encoding regions and encoding image data for each of the encoding regions,
The enlarged area information indicating the area to be enlarged in the image is stored in the header,
An image processing method comprising: encoding an image data by setting an image area indicated by the enlarged area information as one of the encoding areas, and outputting the encoded image data together with a header.
(Appendix 8)
In the image processing method according to attachment 7,
The enlargement area information includes an enlargement ratio when the encoded image data is decoded and displayed as an enlarged image.
(Appendix 9)
An image processing method for decoding encoded image data for display on a display device,
When the header includes enlarged region information indicating one of a plurality of encoded regions, decodes the partial image data corresponding to the encoded region indicated by the expanded region information;
An image processing method, wherein the decoded partial image data is enlarged to a predetermined magnification for display on a display device.
(Appendix 10)
In the image processing method according to attachment 9,
The enlargement area information includes an enlargement ratio when the encoded image data is decoded and displayed as an enlarged image,
The image processing method, wherein the predetermined magnification is an enlargement ratio included in the enlargement area information.

  The present invention is applicable to an apparatus that captures a moving image and enlarges and displays the captured moving image.

It is a block diagram which shows the 1st Embodiment of this invention. It is explanatory drawing which shows the access unit structure which comprises the stream data produced | generated by an image coding part. It is explanatory drawing which shows an example of the setting of a slice. It is a flowchart which shows the encoding of the image data by CPU shown in FIG. 1, and CODEC. It is a flowchart which shows decoding of the image data by CPU shown in FIG. 1, and CODEC. It is a block diagram which shows the 2nd Embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ... Imaging part; 12 ... Operation part; 14 ... Memory; 16 ... Display part; 18 ... Zoom control part; 20 ... Header analysis part; 22 ... Enlargement part; 24 ... Header encoding part; 28. Header decoding unit; 30. Image decoding unit

Claims (5)

An image processing apparatus that divides an image into a plurality of encoding regions and encodes image data for each of the encoding regions,
A storage unit that stores, in a header, enlarged area information indicating an area to be enlarged in the image;
An image encoding unit configured to encode image data by setting an image region indicated by the enlarged region information as one of the encoding regions, and to output the encoded image data together with a header; An image processing apparatus. An image processing device for decoding encoded image data for display on a display device,
A header analysis unit that detects that the extended region information indicating any of the plurality of encoded regions is included in the header;
An image decoding unit that decodes partial image data corresponding to the encoding region indicated by the expansion region information when the expansion region information is included in the header;
An image processing apparatus, comprising: an enlargement unit that enlarges the decoded partial image data to a predetermined magnification for displaying on the display device. An image encoding device that divides an image into a plurality of encoding regions and encodes image data for each of the encoding regions, and an image decoding device that decodes the encoded image data for display on a display device An image transmission / reception system comprising:
The image encoding device includes:
A storage unit that stores, in a header, enlarged area information indicating an area to be enlarged in the image;
An image encoding unit that sets an image region indicated by the enlarged region information as one of the encoding regions, encodes image data, and outputs the encoded image data together with a header;
The image decoding device includes:
A header analysis unit that detects that the extended region information indicating any of the plurality of encoded regions is included in the header;
An image decoding unit that decodes partial image data corresponding to the encoding region indicated by the expansion region information when the expansion region information is included in the header;
An image transmission / reception system comprising: an enlargement unit for enlarging to a predetermined magnification in order to display the decoded partial image data on a display device. An image processing method for dividing an image into a plurality of encoding regions and encoding image data for each of the encoding regions,
The enlarged area information indicating the area to be enlarged in the image is stored in the header,
An image processing method comprising: encoding an image data by setting an image region indicated by the enlarged region information as one of the encoding regions, and outputting the encoded image data together with a header. An image processing method for decoding encoded image data for display on a display device,
When the header includes expanded area information indicating one of a plurality of encoded areas, decodes the partial image data corresponding to the encoded area indicated by the expanded area information;
An image processing method, wherein the decoded partial image data is enlarged to a predetermined magnification for display on a display device.

Images