JP2008053884A - Image processing method and apparatus and electronic device utilizing them - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To efficiently generate images for simple browsing. <P>SOLUTION: In order to adjust the size of a reduced image I2 to be the object of encoding, a dummy region addition part 34 adds a dummy region Rdmy to the reduced image I2 as needed. An encoder 36 encodes a provisional image I3 in the state that the dummy region Rdmy is added, and also records the position information Ddmy of the dummy region Rdmy. A decoder 40 decodes encoded data Denc. A dummy region information acquisition part 42 eliminates the dummy region Rdmy from the decoded provisional image I3 on the basis of the recorded position information Ddmy of the dummy region Rdmy and then outputs a thumbnail image It2 to a display process. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a technique for generating an image intended for display, particularly an image for simple browsing, from images of various sizes and formats.

  2. Description of the Related Art Electronic devices such as personal computers, hard disk video recorders, DVD (Digital Versatile Disc) recorders, and game machines in recent years include recording devices such as hard disk devices and DVD devices. Various still images and moving images (also referred to simply as images including still images and moving images in the present specification) created or recorded by the above are recorded.

  When the number of images recorded in the recording device increases, the user cannot recall the contents of the file from the file name, and it becomes difficult to access a desired image. Therefore, a technique is known in which for each recorded image, data obtained by reducing the original image, called a thumbnail, is known. The user interface of each electronic device displays a thumbnail instead of displaying an icon for each image, so that the user can visually grasp the contents of the image, and can easily create a desired file. It becomes possible to access.

  However, with the recent trend toward multimedia, the size and type of images recorded on a recording device are diversifying. For example, even in the case of data recorded on a television broadcast, the image size differs between HD (High Definition) image quality and conventional SD (Standard Definition) image quality. Is set freely.

  The present invention has been made in view of these problems, and one of its purposes is to easily display images, particularly from images of different sizes, in order to easily and quickly grasp the contents of the images, and capable of being displayed at high speed. It is to provide a technique for generating an image for display.

  According to an image processing method of an aspect of the present invention, a technique for encoding an image to be encoded and sending it to a display process is provided. In this image processing method, as a first step, in order to adjust the size of the image to be encoded, a dummy area is added to the image as necessary, and the image with the dummy area added is encoded. Turn into. At this time, the position information of the added dummy area is recorded. As a second step, the encoded image is decoded, and an area excluding the dummy area from the decoded image is displayed based on the recorded position information of the dummy area.

  “Dummy area position information” means information that can distinguish the original image from the added dummy area. That is, the position information of the dummy area may be, for example, the coordinates of the vertices of the original image, the vertical and horizontal widths, or a combination thereof, the coordinates of the vertices of the dummy area, the vertical and horizontal widths, or these It may be a combination. “Image” includes both still images and moving images. In addition, “display area” means that signal processing is performed so that the area is finally displayed on the display, and it is necessary to actually display the image on the display. is not.

  According to the present invention, an image for display can be efficiently generated for images of different sizes.

  First, the outline of the image processing method according to the embodiment of the present invention will be described. The image processing method according to the present embodiment is mounted on an electronic device that outputs video data, such as a personal computer having an animation playback function, an HDD recorder, a DVD recorder, or a game device. As a premise, in these electronic devices, a plurality of images are recorded in various sizes and formats.

The electronic device displays a moving image file recorded therein to the user via a GUI (Graphical User Interface). At this time, a simple browsing moving image (hereinafter also referred to as a thumbnail moving image) for display is generated so that the user can grasp the contents of the moving image file at a glance. The size of the thumbnail moving image is preferably a reduced size of the original moving image in order to reduce the capacity, and is preferably an extracted part of the content of the moving image. The electronic device arranges and plays a thumbnail video on the GUI.
The techniques of the following various aspects are suitably used when generating such a simple browsing moving image, or when reproducing and displaying it. The same applies to still images, not moving images.

  According to an image processing method of an aspect of the present invention, in order to adjust the size of an image to be encoded, a dummy area is added to an image as necessary and positional information of the dummy area is recorded. A first step of encoding the image with the region added, and decoding the encoded image, and displaying the region excluding the dummy region from the decoded image based on the recorded dummy region position information And a second step.

  According to this aspect, the size of the image to be encoded is standardized, and it is not necessary to check the number of pixels, the angle of view, and the aspect ratio at the time of encoding. Processing can be simplified and efficient. Furthermore, by recording information about the dummy area, the dummy area can be deleted when the image is displayed on the GUI, and the occurrence of a band-like area corresponding to the dummy area at the corner of the image is suppressed. And improve the appearance.

In one aspect, when processing a plurality of images having different image sizes, the first step is performed for each image so that the size of the images matches the reference image size in a state where dummy regions are added. A dummy area may be added.
According to this aspect, since the size of the image to be encoded becomes the reference image size, it is possible to simplify the processing, program code, and apparatus necessary for encoding.

In one aspect, the first step includes a step of resizing an image to be encoded so as to fit within a reference image size. In the first step, a dummy area may be added to the difference area between the resized image and the reference image size, and the image with the dummy area added may be encoded.
In this case, it is possible to unify the sizes of images to be encoded even when images having completely different sizes are input.

In a certain aspect, when displaying the area | region remove | excluding the dummy area | region from the decoded image, a 2nd step may make the reference position correspond with the reference position of a predetermined | prescribed display frame. The “reference position” may be, for example, the center (center of gravity) of the image, or an apex of an image including a dummy area or an image excluding the dummy area.
In this case, since the position of the image displayed on the display screen can be unified, the appearance can be improved.
In this case, in the second step, a plurality of display frames may be arranged on a display screen in a straight line or in a matrix.

  The image may be moving image data, and the second step may reproduce the moving image from which the dummy area has been deleted within a predetermined display frame.

In one aspect, the second step includes a step of generating a polygon having a size proportional to an area obtained by removing the dummy area from the decoded image, and a texture obtained by adding the texture obtained by removing the dummy area from the decoded image to the generated polygon. Mapping as follows.
In this case, the image from which the dummy area is removed can be displayed in the display frame corresponding to the polygon. Furthermore, by moving and deforming the polygon, it is possible to move and deform the image from which the dummy area has been deleted.

In one aspect, the second step includes a step of generating a polygon of a predetermined size, a step of setting the α value of the pixel in the dummy area in the decoded image to a transparent value (= 0), and α Mapping the entire decoded image with the value set to a polygon as a texture.
In this case, since the portion of the dummy area is displayed through the background, substantially only the area excluding the dummy area can be displayed on the display.

In the second step, the outer periphery of the data from which the dummy area has been deleted may be trimmed and displayed, and the trimming rate may correspond to the trimming rate when displaying the encoded original image. .
When displaying an image on a cathode ray tube television or the like, a few percent of the outer edge of the image may not be displayed. Therefore, an area to be surely displayed is prepared as a safety frame, and the image is placed inside the safety frame. In some cases, the image may be trimmed and displayed. In this case, also in the second step, by displaying the image at the same trimming rate, an image having the same angle of view as the image that is actually reproduced can be displayed as a thumbnail.

In yet another aspect of the present invention, the second step may draw the predetermined ground image over the entire predetermined area of the display screen and overwrite the predetermined ground image with the image from which the dummy area has been deleted. .
According to this aspect, since the predetermined ground image is displayed as the background without an unnecessary band corresponding to the deleted dummy area appearing around the image, the appearance can be improved.

Yet another embodiment of the present invention is also an image processing method. This method includes a step of decoding an encoded image, a step of acquiring position information of the dummy region when a dummy region is added to the decoded image, and a position of the acquired dummy region. And displaying a region excluding the dummy region from the decoded image.
According to this aspect, it is possible to prevent the dummy area from being visually displayed on the display.

Yet another embodiment of the present invention is also an image processing method. This method corresponds to the difference between the step of acquiring the size of the image to be encoded, the step of calculating the difference between the acquired image size and the reference image size, and the image to be encoded. Adding a dummy area, and encoding the image with the dummy area and recording the position information of the dummy area separately.
According to this aspect, at the time of encoding, by retaining information about the dummy area added to adjust the size of the image, when decoding and displaying the encoded image, only the area from which the dummy area has been removed is displayed. Can be displayed. In addition, since the size of an image to be encoded and further subjected to subsequent decoding processing is determined, processing such as checking the number of pixels and the angle of view can be omitted, and the processing can be simplified and made efficient. .

  Another embodiment of the present invention also relates to an image processing method. In this method, a dummy area is added to a reduced image obtained by reducing the original image to generate a provisional image for encoding, and the position information of the dummy area is recorded, and the provisional image is encoded. And when the encoded provisional image is decoded, based on the recorded dummy region position information, deleting the dummy region from the decoded provisional image and outputting to the display processing block; including. In this case, since the size of the provisional image to be encoded is unified, the encoding process can be made more efficient, and the dummy area can be prevented from being displayed.

  According to still another aspect of the image processing method, there is provided a technique for encoding an image to be encoded and sending it to a display process. In this image processing method, as a first step, in order to adjust the size of the image to be encoded, a dummy area is added to the image as necessary, and the image with the dummy area added is encoded. Turn into. At this time, the position information of the added dummy area is recorded. As a second step, the encoded image is decoded, the dummy area is deleted from the decoded image based on the recorded position information of the dummy area, and then the image is output to the display process.

  It should be noted that any combination of the above processes, procedures, and constituent elements, or processes, procedures, and constituent elements that are mutually replaced among methods, programs, apparatuses, systems, etc. are also effective as an aspect of the present invention. It is.

  Hereinafter, an image processing method and an image processing apparatus according to embodiments of the present invention will be described with reference to the drawings. The same or equivalent components, members, and processes shown in the drawings are denoted by the same reference numerals, and repeated descriptions are omitted as appropriate. The embodiments do not limit the invention but are exemplifications, and all features and combinations thereof described in the embodiments are not necessarily essential to the invention.

  FIG. 1 is a block diagram illustrating a configuration of an electronic device 1 having a moving image playback function according to an embodiment. The electronic device 1 is, for example, a personal computer, a DVD player, a game device, or the like, and a device having a video data reproduction function. In the present embodiment, electronic device 1 is a game device. This game device has a function of reproducing a moving image in addition to the function as an original game device.

  The electronic device 1 includes a CPU (Central Processing Unit) 10, a DVD device 12, a hard disk 14, an I / O interface 16, a main memory 18, a graphic processor 20, a graphic memory 22, and a bus 24. Each block in FIG. 1 is connected via a bus 24 and configured to be able to transmit and receive data to and from each other. The electronic device 1 outputs video data to a display device (not shown) that is a display device connected to the outside. The display device may be built in the electronic device 1 as a liquid crystal panel.

  The CPU 10 is an arithmetic processing device that controls the entire electronic device 1 in an integrated manner. The CPU 10 loads an OS (Operating System) program stored in the hard disk 14 or a ROM (Load Only Memory) (not shown) into the main memory 18 and manages each block of the electronic device 1 based on the loaded OS. Perform the necessary arithmetic processing. Further, the CPU 10 executes arithmetic processing requested by other blocks.

  The DVD device 12 is used to read a game program to be executed in the electronic device 1 from the inserted disc, or to reproduce video data from a disc on which movies and music are recorded. When a game program is recorded on the DVD medium inserted in the DVD device 12, the CPU 10 loads and executes the game program. Further, when video content such as a movie or a concert (hereinafter also simply referred to as an image) is recorded, the CPU 10 activates a necessary application and reproduces it.

  The hard disk 14 records application programs necessary for the operation of the electronic device 1, data used by the applications, and the like. Further, various images I are recorded on the hard disk 14. In this specification, the term “image” means still image data or moving image data. The image I is recorded by a user on a television broadcast, captured by a user with a digital video camera or digital still camera, or captured from a personal computer. These images are recorded in various sizes and encoding formats. An external device such as a digital video camera or a personal computer is connected to the electronic apparatus 1 via the I / O interface 16, and an image is transferred to the hard disk 14.

  The main memory 18 includes a work area used by each block of the electronic device 1. The main memory 18 is loaded with an application program and stores the result of arithmetic processing by the CPU 10 or the like, or is used for transmission / reception of data between blocks.

  The graphic memory 22 includes a frame buffer (not shown) in which final video data to be output to the display device is stored. The graphic processor 20 is a block that performs arithmetic processing related to an image. The graphic processor 20 performs predetermined processing on the video data generated as a result of executing the game program and the video data read from the DVD device 12 or the hard disk 14, and the final video data is stored in the graphic memory. Write to 22 frame buffers. Data on the frame buffer is output to a display device by a display driver (not shown).

  As described above, in the electronic device 1 according to the present embodiment, many videos are recorded on the hard disk 14. The electronic device 1 has a function of reproducing an image (video) recorded on the hard disk 14. The video playback function of the electronic device 1 is executed by a dedicated application program. Therefore, in the present specification, the “electronic device 1” means a function realized by cooperation of the CPU 10 and other blocks and a necessary program.

  When the user of the electronic device 1 reproduces a plurality of images recorded on the hard disk 14, first, it is necessary to select an image to be reproduced. The electronic device 1 generates a simple browsing image (hereinafter also referred to as a thumbnail image) for each image recorded on the hard disk 14 so that the user can intuitively understand the content of the video data. Display on a display device.

  Since the method for generating the thumbnail image It is known, it will be described briefly. The electronic device 1 associates the thumbnail image It with the original image (hereinafter also simply referred to as an original image) I1, and generates the thumbnail image It at a predetermined timing. The predetermined timing is a timing at which a new image I1 is stored in the hard disk 14, a timing at which a user gives an instruction to create a thumbnail image, a time predetermined in the electronic device 1, and the like. The thumbnail image It once generated is deleted together with the original image I1 when the original image I1 is deleted from the hard disk 14.

  The size of the image I1 recorded on the hard disk 14 varies. Electronic device 1 according to the present embodiment generates thumbnail image It having a uniform size regardless of the size of original image I1. Hereinafter, a method for generating and reproducing a thumbnail image It according to the present embodiment will be described.

  FIG. 2 is a block diagram illustrating a configuration of the image processing apparatus 100 mounted on the electronic apparatus 1. FIG. 3 is a flowchart of image processing according to the embodiment. In FIG. 2, each processing block can be configured by a CPU, a memory, and other LSIs in hardware, and is realized by a program loaded in the memory in software. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof, and is not limited to any one. That is, each processing block shown in FIG. 2 corresponds to a code module of a program executed in the electronic device 1 in terms of software, and is a processing unit configured as a CPU or DSP (Digital Signal Processor) in terms of hardware. It is realized by these cooperation.

  The image processing apparatus 100 includes a hard disk 14, a thumbnail image generation unit 102, and a thumbnail image playback unit 104. The thumbnail image generation unit 102 reads the image I1 recorded on the hard disk 14 and generates a thumbnail image It. The thumbnail image playback unit 104 plays back the generated thumbnail image It and outputs it to the display device.

  First, the configuration of the thumbnail image generation unit 102 will be described. The thumbnail image generation unit 102 includes an image acquisition unit 30, a resize processing unit 32, a dummy area adding unit 34, an encoder 36, and an output processing unit 38.

  The image acquisition unit 30 selects and reads out an image for which the thumbnail image It is to be created from images recorded on the hard disk 14 (S90). The image I1 read by the image acquisition unit 30 is output to the resize processing unit 32. Specifically, this processing is realized by the image acquisition unit 30 loading the image I1 into the main memory 18 and the resize processing unit 32 accessing the main memory 18. Alternatively, the image acquisition unit 30 may notify the address of the image I1 on the hard disk 14 to the resizing processing unit 32, and the resizing processing unit 32 may access this address.

  The resizing processing unit 32 performs a resizing process S100. The resizing process S100 is a process for adjusting the size of the image I1 so as to be within a predetermined unified reference size. For example, the number of horizontal and vertical pixels is 432 × 240, and the reference size will be described below using the number of 432 × 240 pixels as the reference size. The number of pixels of 432 × 240 is preferable because the aspect ratio is approximately 16: 8.89, and is highly compatible with a 16: 9 wide HD image. Since it is a multiple of 16, it has the advantage of high consistency with encoding and decoding processes such as MPEG (Moving Picture Expert Group). It should be noted that the number of pixels of 432 × 240 multiplied by a predetermined coefficient (for example, 216 × 120) may be used as the reference size. The selection of the reference size may be set according to the screen size of the display device connected to the electronic device 1. A plurality of reference sizes may be prepared.

  The resizing process itself may use a general image reduction / enlargement technique. The reduction and enlargement processes here are simple enlargement and simple reduction. In the resizing process S100, it is desirable that the original image I1 is reduced and enlarged while the aspect ratio is fixed, so that the maximum size x1 × y1 can be accommodated in the reference size x0 × y0 in both vertical and horizontal directions. That is, the original image I1 may be reduced within a range where x1 ≦ x0 and y1 ≦ y0. A virtual image having a reference size is referred to as “reference image Istd”.

  For example, when the number of pixels of the original image I1 is x1 × y1 = 640 × 480 pixels, the vertical and horizontal directions are halved to generate a reduced image I2 having 320 × 240 pixels. The resized reduced image I2 is output to the dummy area adding unit 34. Since the size of the thumbnail image is almost smaller than the size of the original image, it is called a reduced image here. However, the original image with a low resolution is resized by enlargement.

Next, the dummy area adding unit 34 performs a dummy area adding process S110 to add the dummy area Rdmy to the reduced image I2 and generate the provisional image I3. At this time, the shape and size of the dummy area are determined so that the size of the provisional image I3 is constant. The reduced image I2 becomes a provisional image I3 for encoding in a state where a dummy area is given. “For encoding” is to make the size of the provisional image I3 constant for efficient encoding. As will be described in detail later, when the image is finally displayed to the user, since the dummy area is deleted from the provisional image I3, the provisional image exists only from encoding to storage and decoding. It does not touch the user's eyes. “A state in which a dummy area is given” refers to a state in which the image size of the reduced image I2 is changed by the dummy area adding unit 34. To change the image size,
1) Rewrite the size specifying information in the header area of the image, and
2) The image data area is expanded, and the pixel values of the pixels included in the dummy area are written into the expanded image data area.
Is made by The pixel value of the pixel in the dummy region Rdmy (hereinafter referred to as “dummy value”) may be arbitrary, but a value that makes the encoding faster is desirable in connection with the encoding method. As an example, it is conceivable to set the dummy value to a fixed value. This corresponds to a state in which the dummy area is filled with one color in the provisional image I3.

  That is, the dummy area adding process S110 is a process for adding the dummy area Rdmy to the difference area ΔR (= Istd−I2) between the resized reduced image I2 and the reference image Istd so as to match the reference size Istd.

  Since the resized reduced image I2 is composed of 320 × 240 pixels, a dummy region Rdmy for 102 pixels is added in the horizontal direction to match the reference size 432 × 240. The dummy areas may be arranged equally at both ends of the reduced image I2, or may be arranged together on either the left or right side. For example, in the case where the left and right sides are evenly arranged, 51 (= 102/2) pixel dummy areas are added to the left and right sides of the reduced image I2.

  Further, depending on the size of the resized reduced image I2, the dummy area Rdmy may be added in the vertical direction or both in the vertical direction and the horizontal direction.

  The provisional image I3 having the reference size output from the dummy area adding unit 34 is input to the encoder 36. The encoder 36 performs an encoding process S120 on the provisional image I3. In the encoding process S120, the provisional image I3 to which the dummy area Rdmy is added and the size is unified is encoded by a predetermined format, that is, an encoding method. The format may be the same format as the original image I1 recorded on the hard disk 14, or may be a format predetermined in the electronic device 1. In the present embodiment, in order to simplify data processing (decoding) when displaying the generated thumbnail image, compression encoding is performed by a predetermined method, for example, MPEG2 or MPEG4. If the encoding method of the thumbnail image It to be displayed is unified, the decoding method can be fixed in the later-described decoding process S140, so that the program can be simplified or the configuration is simplified when a dedicated DSP is provided. Can be As an example, the encoder 36 may compress and encode the provisional image I3 with MPEG4 and store it in the MP4 format. However, the method used for compression encoding is not limited to this, and various methods that can be used now or in the future can be used. For still images, JPEG (Joint Photographic Experts Group) or the like can be used.

  When the thumbnail image It is generated, if encoding is performed for the same full playback time as the original image I1 which is a moving image, the data size of the thumbnail image It becomes large. Therefore, the encoder 36 encodes a frame over a predetermined period, for example, several seconds to several tens of seconds in order to prevent the data size of the thumbnail image It from becoming large. Encoding may be performed over a predetermined period from the first frame, or the range to be encoded by the user may be selectable. Alternatively, the original image I1 may be analyzed to extract a highlight scene or a rising part, and the extracted period may be encoded to generate a thumbnail image It as a digest. For example, it is possible to extract a highlight scene or a climax part by analyzing audio data of a moving image and extracting a period during which the volume is high, or by analyzing image data and extracting a period during which movement is intense.

  Further, the encoder 36 encodes an image of a predetermined frame, for example, the top frame as a still image thumbnail image, and records it separately from the moving image thumbnail image. Later, when displaying a thumbnail image on the display, if the CPU 10 of the electronic device 1 performs another heavy process, it may be assumed that the arithmetic process required for the decoding process of the moving image cannot be performed. In such a case, it is meaningful to generate a thumbnail image of a still image that requires less calculation amount than the moving image. The encoder 36 outputs the encoded image (hereinafter also referred to as encoded data) Denc to the output processing unit 38. The encoded data Denc includes data obtained by encoding the moving image and the still image.

  The output processing unit 38, along with the encoded data Denc obtained as a result of the encoding process S120 in the encoder 36, indicates the position information of the dummy region Rdmy added in the dummy region adding unit 34 (hereinafter referred to as dummy region data). Ddmy (simply referred to as dummy area data Ddmy) is output (S130). The dummy area data Ddmy includes information that can distinguish the original image I1 from the added dummy area Rdmy.

For example, the dummy area data Ddmy is the coordinates of four vertices (p1, p2, p3, p4) of the reduced image I2 obtained by removing the dummy area Rdmy from the provisional image I3. Alternatively, it may be two vertices (p1, p4) present on the diagonal, or the coordinates of one of the vertices (p1) and the vertical and horizontal widths Δx, Δy of the reduced image I2. Alternatively, the dummy area data Ddmy may be the coordinates of each vertex of the dummy area Rdmy, the vertical and horizontal widths, or a combination thereof. The dummy area data Ddmy is written in the hard disk 14 of FIG. 1 together with the encoded data Denc. The recording of the dummy area data Ddmy may be an entry to a table (not shown) held on the hard disk 14. As an example, this table
(1) Information relating to the original image I1 itself, that is, its name, size, creation time, and address indicating the storage location of the image (2) Address indicating the storage location of the encoded data Denc of the thumbnail image It corresponding to the original image I1 3) Dummy area data Ddmy
including. The encoded data Denc and the dummy area data Ddmy are data constituting a thumbnail image.

  The above is the configuration of the thumbnail image generation unit 102. Next, the configuration of the thumbnail image reproduction unit 104 used when the user selects the image I1 to be reproduced based on the thumbnail image It2 will be described. The thumbnail image reproduction unit 104 outputs the thumbnail image It2 to the display based on the encoded data Denc and the dummy area data Ddmy recorded on the hard disk 14.

  The decoder 40 reads the encoded data Denc from the hard disk 14 and performs a decoding process S140. The decoding process here corresponds to the encoding process by the encoder 36. The encoded data Denc can be acquired by referring to the table. Since the encoding format is unified in the encoding process S120, it is not necessary to confirm the format each time in the decoding process S140, and the process here is only a substantial decoding process. As a result of decoding by the decoding process S140, a thumbnail image It1 including the dummy area Rdmy is generated. The contents of the thumbnail image It1 correspond to the contents of the provisional image I3 output from the dummy area adding unit 34.

  The dummy area information acquisition unit 42 acquires dummy area data Ddmy corresponding to the input thumbnail image It1 with reference to the above table (S150). The dummy area information acquisition unit 42 outputs the acquired dummy area information Ddmy and the thumbnail image It1 to the display processing unit 44.

  Based on the dummy area information Ddmy, the display processing unit 44 displays an area obtained by removing the dummy area Rdmy from the decoded thumbnail image It1 on a display device (not shown) (S160). The area obtained by removing the dummy area Rdmy from the decoded thumbnail image It1 corresponds to the thumbnail image It2 to be reproduced. Hereinafter, display processing of the thumbnail image It2 in the display processing unit 44 will be described.

(First display method)
Based on the dummy area information Ddmy, the display processing unit 44 acquires the size of the area excluding the dummy area Rdmy from the decoded thumbnail image It1 (hereinafter referred to as the thumbnail size), and obtains a polygon having a size proportional to the thumbnail size. create. Polygons are arranged in a three-dimensional space, and the size expressed by the number of pixels when actually displayed on the display, depending on the display state such as the position of the polygon itself and the position of the viewpoint, is Change. Therefore, the “polygon size” here means the size in a certain reference display state, and “generate a thumbnail size polygon” means a pixel proportional to the thumbnail size in the reference display state. This means that a polygon whose size is displayed as a number is generated. In the following description, it is assumed that the polygon size matches the thumbnail size in the reference display state.

  Subsequently, an area It2 obtained by removing the dummy area Rdmy from the decoded thumbnail image It1 is mapped to this polygon as a texture. By rendering the polygon in which the thumbnail image It2 is mapped as a texture, only the thumbnail image It2 can be visually displayed on the display device.

For example, in the specific example described above, the reference size is 432 × 240 pixels, the size of the region excluding the dummy region, that is, the thumbnail size is 320 × 240 pixels, and the dummy region Rdmy is 51 at both the left and right ends. Consider a case where pixels are arranged. In this case, a rectangular polygon is generated so as to be displayed in 320 × 240 pixels in a reference display state, that is, in a state where the polygon is arranged at a predetermined position and the viewpoint is fixed at the predetermined position.
On the other hand, the region It2 obtained by removing the dummy region Rdmy from the thumbnail image It1 is mapped to this polygon. The texture mapping process is executed by designating the UV coordinates to the graphic processor 20 of FIG. 2, and among the 432 × 240 thumbnail images It1, x = 52, y = 0 are the vertices, the width Δx = 320, the high The area of Δy = 240 is texture-mapped to the polygon. The graphic processor 20 of FIG. 2 renders the polygon and writes it in the frame buffer of the graphic memory 22, so that only the region It2 excluding the dummy region Rdmy from the thumbnail image It1 is visibly displayed on the display device.

(Second display method)
The display processing unit 44 generates a polygon having a predetermined size. The size of the polygon only needs to be larger than the thumbnail size, and is set to a reference size, for example. Subsequently, the display processing unit 44 sets the α value indicating the transparency of the pixels in the dummy area Rdmy in the decoded thumbnail image It1 to a transparent value (= 0). In the thumbnail image It1, the α value is set to a value corresponding to non-transparency (for example, the maximum value) for the region excluding the dummy region Rdmy. The display processing unit 44 maps the entire thumbnail image It2 as a texture on the generated polygon.

  As a result, by rendering the polygon to which the thumbnail image It2 is mapped, the portion of the dummy area Rdmy can be seen through the background, so that substantially only the thumbnail image It2 excluding the dummy area is displayed on the display device. be able to.

(Third display method)
Based on the dummy area information Ddmy, the display processing unit 44 generates a thumbnail image It2 in which the dummy area Rdmy is deleted from the decoded thumbnail image It1. Here, “deletion” is not image processing such as simply clearing pixel values to zero, but image reconstruction processing that changes the image size itself. In order to change the image size, the display processing unit 44 rewrites the image size specifying information in the header area of the image. Subsequently, the display processing unit 44 executes a drawing process for the thumbnail image It2 from which the dummy area Rdmy has been deleted. The third display method is an effective technique in the image processing apparatus 100 in which a function relating to three-dimensional graphics is not implemented.

  The operation of the image processing apparatus 100 configured as described above will be described. 4A to 4C are diagrams showing each image generated in the process of the image processing apparatus 100 in FIG. 4A shows the original image I1 recorded on the hard disk 14, and FIG. 4B shows the reduced image I2 resized by the resize processing unit 32 or the thumbnail that is finally displayed on the display device. The image It2 shows the temporary image I3 to which the dummy area Rdmy is added and the size is unified, or the thumbnail image It1 decoded by the decoder 40. FIG.

When the user instructs generation of a thumbnail image, the thumbnail image generation unit 102 of the image processing apparatus 100 starts a thumbnail image generation process. First, the image acquisition unit 30 selects an original image I1 for creating a thumbnail image shown in FIG.
Subsequently, by the resizing process S100 in the resizing processing unit 32 in FIG. 2, the original image I1 in FIG. 4A is reduced and enlarged so as to fit in the reference size. The resulting image is a reduced image I2 shown in FIG. A reduced image I2 in FIG. 4B is a simple enlargement and reduction of the image in FIG. 4A, and has the same aspect ratio.

  Subsequently, the dummy area adding unit 34 adds the dummy area Rdmy to the reduced image I2 to generate a provisional image I3 that is standardized to the reference size (FIG. 4C). The encoder 36 encodes the provisional image I3 and generates encoded data Denc. The encoded data Denc is recorded on the hard disk 14 by the output processing unit 38 together with the dummy area data Ddmy.

  Thereafter, when the user selects an image to be viewed from a plurality of images recorded on the hard disk 14, the thumbnail image reproduction unit 104 of the image processing apparatus 100 operates.

  The decoder 40 of the thumbnail image reproduction unit 104 reads out the encoded data Denc and decodes it. The image It1 obtained as a result of the decoding corresponds to the provisional image I3 (FIG. 4C). The dummy area information acquisition unit 42 acquires dummy area data Ddmy corresponding to the provisional image I3 from the hard disk 14. The display processing unit 44 displays the area obtained by deleting the dummy area Rdmy from the image of FIG. 4C, that is, the thumbnail image It2 on the display device (FIG. 4B).

  According to the image processing technique described above, the size of an image to be encoded can be unified by adding a dummy area as necessary during encoding when generating a thumbnail image. In the conversion process, the image sizes are unified. As a result, it is not necessary to check the number of pixels, the angle of view, the aspect ratio, and the like, and it is sufficient to perform the same process every time. Therefore, the encoding process can be made efficient and faster. Furthermore, the algorithm and hardware can be optimized so that the encoding process can be performed at the highest speed for a unified image size.

  Furthermore, since information related to the dummy area Rdmy is recorded together with the encoded data Denc as metadata, the dummy area Rdmy can be removed when displaying the thumbnail image It. Thereby, it is possible to suppress the occurrence of a band-like area corresponding to the dummy area Rdmy (a white band if the dummy area is white) at the corner of the thumbnail image It, and to improve the appearance.

Next, data processing when displaying the decoded thumbnail image on the display will be described.
FIG. 5 is a diagram showing a display area of the display 50 on which thumbnail images are displayed. A predetermined background image 52 is displayed on the display 50. The background image 52 may be filled with a single color or a predetermined ground image, or an image such as wallpaper may be displayed. When using the first and second display methods, a polygon may be arranged at a position sufficiently distant from the viewpoint and colored, or a wallpaper may be mapped to the polygon.

  On the display 50, a plurality of predetermined display frames FRM1 to FRM4 are arranged on a straight line so that the centers are aligned. The size of the display frame FRM is preferably the same as the reference size described above. Note that the display frames FRM1 to FRM4 are not drawn as line drawings on the display 50, but are virtually arranged. Further, they may be arranged in the horizontal direction instead of the vertical direction, or may be arranged in a matrix. Further, the size of the display frame FRM may be a size obtained by multiplying the reference size by a predetermined amount.

  The electronic device 1 displays thumbnail images obtained by the above-described image processing technique in each of the plurality of display frames FRM1 to FRM4. Further, the file name of the image corresponding to the thumbnail image displayed in the display frame, the reproduction time, and other information may be displayed in the vicinity of each display frame FRM, for example, on the right side.

  The plurality of display frames correspond to the polygons described in the first and second display methods, and the polygons are arranged at positions closer to the viewpoint than the polygons for displaying the background. The display processing unit 44 maps thumbnail images to polygons corresponding to these display frames.

  Since the time required for decoding a still image is shorter than that for a moving image, the thumbnail image reproduction unit 104 in FIG. 2 may generate a thumbnail image of a moving image after first generating a thumbnail image of the still image. By performing this process, the thumbnail image can be displayed on the display with good response, and the user's experience processing speed can be improved.

  Further, a cursor 54 may be displayed on the display 50. The position of the cursor 54 may move up and down in accordance with a user operation, and an image selected by the cursor 54 may be highlighted. In this case, the user can select an image to be reproduced with the cursor 54.

In order to provide a more intuitive user interface, the size of the image selected with the cursor 54 may be made relatively different from the sizes of the images displayed in other display frames. For example, the image selected by the cursor 54 may be enlarged and displayed. When the display processing unit 44 adopts the first and second display methods, the size of the thumbnail image It2 mapped as the texture can be increased by increasing the size of the polygon. Alternatively, the polygon on which the image to be enlarged is mapped is moved to a position close to the viewpoint, so that it can be displayed in a large size.
When the third display method is employed, the thumbnail image It2 is enlarged or reduced as appropriate and drawn directly on the frame buffer.

Further, instead of the cursor 54, a selection frame 56 fixed at a predetermined position may be provided. In this case, the display position of each thumbnail image may be moved according to the user's selection operation. For example, when the user selects the upward direction from the display state of FIG. 5, the thumbnail images ItA to ItD are shifted to the upper display frame. As a result, the thumbnail image It4 is displayed in the selection frame 56. The user may move a desired thumbnail into the selection frame 56 and select an image to be reproduced.
Of course, the method of selecting one image from a plurality of thumbnail images may be performed using another GUI.

Thumbnail images ItA to ItD generated based on the first to fourth original images are displayed in the first display frame FRM1 to the fourth display frame FRM4, respectively. The displayed thumbnail images ItA to ItD are thumbnail images from which the dummy area Rdmy has been removed by the display processing unit 44 of FIG.
The user can intuitively grasp the contents of the original image I corresponding to each of the thumbnail images ItA to ItD displayed in the display frames FRM1 to FRM4.

  By performing the above display processing, the dummy area Rdmy added when generating the thumbnail image is not displayed on the display 50, and the background image 52 is displayed in the area where the dummy area Rdmy exists. . As a result, it is possible to prevent a black or white band corresponding to the dummy area Rdmy from being displayed on the display, and to suppress the appearance from deteriorating.

  The above display processing can be realized as follows at the hardware or software level. First, the program draws the background image 52 in the entire frame buffer on the graphic memory 22 of FIG. 1 where the display content of the display 50 is drawn. Subsequently, in order to display the thumbnail image It on the display 50, the thumbnail image It obtained by the dummy area information acquisition unit 42 in FIG. 2 is drawn by overwriting in the area on the frame buffer corresponding to the display frame FRM. To do. These drawing processes are executed by the CPU 10, the graphic processor 20, and the like.

  As shown in FIG. 5, when displaying a plurality of thumbnail images ItA to ItD, the decoder 40 and the dummy area information acquisition unit 42 described in FIG. 2 perform parallel processing on the plurality of thumbnail images. The parallel processing may be realized by time-sharing processing by a single processor, may be realized by complete parallel processing by a plurality of processors, or a combination thereof.

  When displaying the thumbnail image It in the display frame FRM, it is preferable to perform data processing so that the reference position of the image It from which the dummy area Rdmy is deleted matches the reference position of the display frame FRM. In the example of FIG. 5, the center coordinates (that is, the center of gravity) of the image are selected as the reference position. By matching the reference positions, the thumbnail images can be aligned and the appearance can be improved. The user may freely set the reference position.

  In the background of the processing for displaying thumbnail images, when processing with a large amount of computation is being executed, the CPU 10 is used for all the thumbnail image decoders 40, dummy area information acquisition unit 42, and display processing unit 44. It is also assumed that hardware resources of the graphic processor 20 cannot be allocated. For this purpose, the thumbnail image playback unit 104 may include a calculation monitor unit (not shown) for determining whether the hardware resources necessary for the processing of the thumbnail image playback unit 104 are sufficient. For example, the calculation monitor unit acquires the calculation amount of the decoder 40 and compares the acquired calculation amount with a predetermined threshold value. When the decoder 40 is configured by a combination of the CPU 10 and a program, the calculation amount of the CPU 10 is monitored. As a result of monitoring, when the calculation amount exceeds the threshold, it is determined that the resources of the decoder 40 are insufficient. The decoder 40 determined to have insufficient resources performs a decoding process of a still image thumbnail image instead of a moving image thumbnail image decoding process for some images. Note that the calculation monitoring unit may determine the resource shortage by comparing another parameter such as a memory free area with a threshold value instead of or in addition to the calculation amount.

  When a resource shortage occurs, only a certain thumbnail image, for example, a thumbnail image ItC selected by the user, is subjected to a moving image decoding process to display a moving image thumbnail image in the display frame FRM3. A thumbnail image of a still image prepared separately may be displayed in the other display frames FRM1, 2, and 4. By performing this process, it is possible to reduce the amount of calculation processing required for displaying the thumbnail image, so that the process can be prevented from failing.

When generating and displaying thumbnail images, it is preferable to perform processing in consideration of so-called safety frames. In general, a cathode ray tube television or the like cannot display information on the outer periphery thereof. Therefore, no matter what kind of display is connected to the electronic device 1, a safety frame in which the outer periphery of the actual frame size is trimmed at a trimming rate of about several to 20% in order to reliably display necessary data. (Safety Frame) may be prepared, and processing for outputting data in the safety frame to the display 50 may be performed.
In the following, processing that considers a safety frame will be described.

In the present embodiment, a safety frame is considered when generating a thumbnail image. In this method, the resizing processing unit 32 trims the outer periphery of the actual frame size of the image I1 by several% in order to delete the region outside the safety frame. The trimming rate may be set as appropriate according to the size of the safety frame. Subsequently, the thumbnail image having the same size as the safety frame remaining as a result of the trimming is enlarged and reduced so as to be within the above-described reference size, and the resizing process is performed to obtain a reduced image I2. Thereafter, the dummy area adding unit 34 adds a dummy area Rdmy to the reduced image I2 to generate a provisional image I3.
In this way, the thumbnail image finally obtained by considering the safety frame when generating the thumbnail image does not include the data outside the safety frame, so the original image was reproduced by reproducing it as it is. A video with the same angle of view can be obtained.

  As a modification, a safety frame may be considered when displaying thumbnail images. In this modification, when the thumbnail image is displayed in the display frame FRM, the outer periphery of the thumbnail image It from which the dummy area Rdmy has been deleted may be trimmed and displayed. When displaying the thumbnail image It, it is desirable to trim the thumbnail image at the same trimming rate and display it in the display frame FRM. In this case, the area outside the safety frame is trimmed on the display 50, and an image inside the safety frame is displayed. By this processing, the angle of view of the thumbnail image can be aligned with the angle of view of the original image when it is reproduced at full size on the display 50. Note that the trimming process can be realized by mapping only the pixels in the safety frame of the thumbnail image as a texture on the polygon or by directly writing to the frame buffer.

  Note that this process can be skipped when a safety frame is not set in the original image, that is, when all pixels are displayed.

  The present invention has been described based on the embodiments. This embodiment is an exemplification, and it will be understood by those skilled in the art that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are also within the scope of the present invention. is there.

  In the embodiment, in order to generate the provisional image I3, the dummy image is added to the lacking region after the original image I1 is resized and brought close to the reference size. However, the present invention is not limited to this. For example, the reference image is enlarged to include the original image I1, and a virtual image close to the size of the original image I1 is generated. A difference area between the original image I1 and the virtual image may be calculated and added to the original image I1 as a dummy area. The provisional image I3 can be obtained by reducing the size after adding the dummy area.

  Further, in the embodiment, the case where the process of displaying the area obtained by removing the dummy area Rdmy from the thumbnail image It1 has been mainly described at the stage of performing the display process on the final display. It is not limited. That is, after decoding the thumbnail image It1, an image from which the dummy area Rdmy is deleted may be generated and input to the final display process.

It is a block diagram which shows the structure of an electronic device provided with the moving image reproduction function which concerns on embodiment. It is a block diagram which shows the structure of the image processing apparatus mounted in an electronic device. 5 is a flowchart of image processing according to the embodiment. 4A to 4C are diagrams showing each image generated in the process of the image processing apparatus of FIG. It is a figure which shows the display with which the thumbnail image was displayed.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Electronic device, 10 ... CPU, 12 ... DVD apparatus, 14 ... Hard disk, 16 ... I / O interface, 18 ... Main memory, 20 ... Graphic processor, 22 ... Graphic memory, 24 ... Bus, 30 ... Image acquisition unit, 32 ... Resize processing unit, 34 ... Dummy area adding unit, 36 ... Encoder, 38 ... Output Processing unit, 40 ... Decoder, 42 ... Dummy area information acquisition unit, 44 ... Display processing unit, 50 ... Display, 100 ... Image processing device, 102 ... Thumbnail image generation unit 104 thumbnail image reproduction unit.

Claims (17)

In order to adjust the size of the image to be encoded, a dummy area is added to the image, the positional information of the dummy area is recorded, and the image with the dummy area added is encoded. Steps,
A second step of decoding the encoded image and displaying an area excluding the dummy area from the decoded image based on the recorded position information of the dummy area;
An image processing method comprising:   The method according to claim 1, wherein when the plurality of images having different image sizes are processed, the first step is configured so that the sizes of the images match a reference image size in a state where the dummy regions are added. An image processing method characterized by adding a dummy area to each image. 3. The method of claim 2, wherein the first step is
Resizing an image to be encoded to fit within the reference image size;
An image processing method, wherein the dummy area is added to a difference area between the resized image and the reference image size, and the image with the dummy area added is encoded. 4. The method of claim 3, wherein the first step is
An image processing method, wherein when a safety frame is set in an image to be encoded, an image from which an area outside the safety frame has been removed is resized to fit within the reference image size. The method of claim 1, wherein the second step comprises:
An image processing method characterized by matching a reference position with a reference position of a predetermined display frame when displaying an area excluding a dummy area from the decoded image.   6. The image processing method according to claim 5, wherein in the second step, the plurality of display frames are arranged on a display screen in a line or in a matrix.   6. The image processing method according to claim 5, wherein the image is moving image data, and the second step reproduces an area excluding the dummy area within the predetermined display frame. . The method of claim 1, wherein the second step comprises:
Generating a polygon having a size proportional to an area excluding a dummy area from the decoded image;
Mapping an area obtained by removing a dummy area from the decoded image as a texture onto the polygon;
An image processing method comprising: The method of claim 1, wherein the second step comprises:
Generating a polygon of a predetermined size;
Setting the α value of the pixels in the dummy area of the decoded image to a transparent value;
mapping the entire decoded image with an α value set as a texture on the polygon;
An image processing method comprising: Decoding the encoded image; and
When a dummy area is added to the decoded image, obtaining the position information of the dummy area;
A step of displaying an area obtained by removing the dummy area from the decoded image based on the obtained position information of the dummy area;
An image processing method comprising: Obtaining a size of an image to be encoded;
Calculating a difference between the size of the acquired image and a reference image size;
Adding a dummy area corresponding to the difference to the image to be encoded;
Encoding the image with the dummy area added, and recording the position information of the dummy area separately;
An image processing method comprising: Adding a dummy area to the reduced image obtained by reducing the original image to generate a provisional image for encoding, and recording position information of the dummy area;
Encoding a provisional image;
When the encoded provisional image is decoded, displaying a region excluding the dummy region from the decoded provisional image based on the recorded dummy region position information;
An image processing method including: In order to adjust the size of the image to be encoded, a dummy area adding unit that adds a dummy area to the image and records position information of the dummy area;
An encoder for encoding an image with the dummy region added thereto;
A decoder for decoding the encoded image;
A display processing unit that outputs, to a display device, an area obtained by removing the dummy area from the image decoded by the decoder based on the position information of the dummy area recorded by the dummy area adding unit;
An image processing apparatus comprising: A decoder for decoding the encoded image;
When a dummy area is added to the decoded image, a position information acquisition unit that acquires position information of the dummy area;
A display processing unit that outputs, to the display device, an area obtained by removing the dummy area from the decoded image, based on the acquired position information of the dummy area;
An image processing apparatus comprising: An image processing program that generates and reproduces an image for display with respect to an image recorded in a recording device.
Adding a dummy area to the image and recording the position information of the dummy area in order to adjust the size of the image to be encoded;
Encoding the image with the dummy region added thereto;
Decoding the encoded image; and
Based on the recorded location information of the dummy area, displaying the area excluding the dummy area from the decoded image;
A program that executes An image processing program for displaying an image on a computer
Decoding the encoded image; and
When a dummy area is added to the decoded image, obtaining the position information of the dummy area;
A step of displaying an area obtained by removing the dummy area from the decoded image based on the obtained position information of the dummy area;
A program that executes A recording device for recording a plurality of images;
Generating an image for simple viewing of the plurality of images and recording the image on the recording device;
A playback unit for playing back an image for simple browsing recorded in the recording device;
With
The image generation unit
In order to adjust the size of the image, a dummy area adding unit for adding a dummy area to the image and recording position information of the dummy area;
An encoder that encodes the image with the dummy region added thereto,
The playback unit
A decoder for decoding the encoded image;
A display processing unit that outputs, to the display device, an area obtained by removing the dummy area from the decoded image based on the position information of the dummy area recorded in the recording device;
An electronic device comprising:

Images