JP5685916B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program.

  Conventionally, there is a technique for synthesizing an arbitrary background with a foreground object such as a person and printing a resultant synthesized image.

  For example, Patent Document 1 discloses a technique for directly projecting another background image onto a screen behind a person as a technique for combining and printing a person photograph or the like with another background. Patent Document 1 also discloses a technique for projecting a blue back background for use in chroma key composition and a technique for combination with a so-called chroma key composition process often used in movies.

JP 2004-77958 A

  However, in recent years, there is a demand for synthesizing a foreground object cut out from an arbitrary image on an arbitrary background only by image processing without using a screen, a projection device, or the like. Can not respond to.

  The present invention has been made in view of such a situation, and realizes image processing necessary for synthesizing a foreground object cut out from an arbitrary image on an arbitrary background without using a screen or a projection device. In addition, it is an object of the present invention to suppress a decrease in output quality while realizing an increase in the speed of the image processing.

According to one aspect of the invention,
First conversion means for converting the data size of the processing target image from the first size to the second size;
Output means for outputting a mixture ratio and foreground pixel values in units of pixels for the data of the processing target image converted to the second size by the first conversion means;
Of the pixels which are output by said output means, said the area of the larger pixel than mixing ratio threshold, on the basis of the image pixel value of the first size of the processing target image, the region of the pixel below the mixing ratio threshold A generating means for generating output image data based on the foreground pixel values;
An image processing apparatus is provided.

  According to one aspect of the present invention, an image processing method and a program corresponding to the above-described image processing apparatus according to one aspect of the present invention are provided.

  According to the present invention, it is possible to realize image processing necessary for synthesizing a foreground object cut out from an arbitrary image on an arbitrary background without using a screen, a projection device, or the like, and thus increase the speed of the image processing. However, a decrease in output quality can be suppressed.

1 is a block diagram illustrating a configuration of an image processing system according to an embodiment of the present invention. It is a block diagram which shows the hardware constitutions of the client apparatus of the image processing system of FIG. It is a block diagram which shows the hardware constitutions of the server of the image processing system of FIG. FIG. 4 is a functional block diagram illustrating a functional configuration of an image processing system when a client device having the hardware configuration of FIG. 2 and a server having the hardware configuration of FIG. 3 cooperate to execute a clipping process. FIG. 5 is a state transition diagram illustrating an example of each state that can be taken when the client device of FIG. 4 executes a clipping designation process. It is a figure which shows an example of the basic screen used by the clipping process of the image processing system of FIG. It is a figure which shows an example of the cutting area designation | designated screen used by the clipping designation | designated process of FIG. It is a figure which shows an example of the cut correction designation | designated screen used by the cut designation | designated process of FIG. FIG. 5 is a state transition diagram illustrating an example of each state that can be taken when the client device of FIG. 4 executes a finish adjustment process. FIG. 10 is a flowchart for explaining a flow of area calculation processing executed in the finish adjustment processing of FIG. 9. FIG. It is a figure which shows an example of the finishing confirmation screen used by the finishing adjustment process of FIG. It is a figure which shows an example of the finishing adjustment screen used by the finishing adjustment process of FIG. FIG. 2 is a flowchart for explaining a relationship between processes when a client device and a server in FIG. 1 cooperate to execute a clipping process. It is a flowchart explaining the flow of the clipping process of the client side process of FIG. FIG. 5 is a diagram illustrating an example of a display screen when the image processing system in FIG. 4 executes a clipping process. FIG. 5 is a diagram illustrating an example of a display screen when the image processing system in FIG. 4 executes a clipping process. FIG. 5 is a diagram illustrating an example of a display screen when the image processing system in FIG. 4 executes a clipping process. FIG. 5 is a diagram illustrating an example of a display screen when the image processing system in FIG. 4 executes a clipping process. FIG. 5 is a diagram for explaining a method applied to the clipping method of the image processing system of FIG. 4, in which mat processing is performed at medium resolution and is combined with a high-resolution original image at the time of output such as display or printing.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram showing a configuration of an image processing system according to an embodiment of the present invention.

The image processing system illustrated in FIG. 1 includes client devices 11-1 to 11 -N (N is an arbitrary integer value of 1 or more) and a server 12.
Each of the client devices 11-1 to 11-N and the server 12 is connected to each other via a predetermined network 21 such as the Internet.
Hereinafter, when it is not necessary to distinguish each of the client devices 11-1 to 11-N, these are collectively referred to as “client device 11”.

The client device 11 cooperates with the server 12 connected via the network 21 to perform image processing for cutting out the foreground from the original image using an image including a target object such as a person (hereinafter referred to as “foreground”) as an original image. Can be executed. Such image processing is hereinafter referred to as “cutout processing”.
The foreground cut out from the original image by such a clipping process is combined with a new background different from the original image, and the resultant combined image is printed on a paper medium or the like and provided to the user.

  FIG. 2 is a block diagram showing a hardware configuration of the client device 11 for executing such a clipping process. The client device 11 can be constituted by a personal computer with a camera, for example.

  The client device 11 includes a CPU (Central Processing Unit) 31, a ROM (Read Only Memory) 32, a RAM (Random Access Memory) 33, a bus 34, an input / output interface 35, an imaging unit 36, and an operation unit 37. A display unit 38, a storage unit 39, a communication unit 40, and a drive 41.

The CPU 31 executes various processes according to a program recorded in the ROM 32 or according to a program loaded from the storage unit 39 to the RAM 33.
The RAM 33 also appropriately stores data necessary for the CPU 31 to execute various processes.

  For example, in the present embodiment, a program for realizing the communication control unit 101 shown in FIG. 4 to be described later is stored in the ROM 32 or the storage unit 39. Also, a program that realizes a cutout processing unit 102 in FIG. 4 to be described later (a program that executes cutout processing, hereinafter referred to as a “cutout processing program”) is provided from the server 12 and expanded in the RAM 33. Therefore, each function of the communication control unit 101 and the cutout processing unit 102 in FIG. 4 can be realized by the CPU 31 executing processes according to these programs.

  The CPU 31, ROM 32, and RAM 33 are connected to each other via a bus 34. An input / output interface 35 is also connected to the bus 34. An imaging unit 36, an operation unit 37, a display unit 38, a storage unit 39, a communication unit 40, and a drive 41 are connected to the input / output interface 35.

  Although not shown, the imaging unit 36 includes an optical lens unit and an image sensor.

The optical lens unit is configured with a lens that collects light, such as a focus lens and a zoom lens, in order to capture an image of the subject.
The focus lens is a lens that forms a subject image on the light receiving surface of the image sensor. The zoom lens is a lens that freely changes the focal length within a certain range.
The optical lens unit is also provided with a peripheral circuit for adjusting setting parameters such as focus, exposure, and white balance as necessary.

The image sensor includes a photoelectric conversion element, an AFE (Analog Front End), and the like.
The photoelectric conversion element is composed of, for example, a CMOS (Complementary Metal Oxide Semiconductor) type photoelectric conversion element. A subject image is incident on the photoelectric conversion element from the optical lens unit. Therefore, the photoelectric conversion element photoelectrically converts (captures) a subject image at regular time intervals to accumulate image signals, and sequentially supplies the accumulated image signals to the AFE as analog signals.
The AFE performs various signal processing such as A / D (Analog / Digital) conversion processing on the analog image signal. Through various signal processing, a digital signal is generated and output as an output signal of the imaging unit 36.
Hereinafter, the output signal of the imaging unit 36 is referred to as “captured image data”. Accordingly, captured image data is output from the imaging unit 36 and is appropriately supplied to the CPU 31 and the like.

The operation unit 37 includes a keyboard, a mouse, and the like, and accepts user instruction operations.
The display unit 38 is configured by a liquid crystal display or the like and displays various images.
The storage unit 39 is configured with a hard disk or the like, and temporarily stores image data to be subjected to image processing, such as captured image data output from the imaging unit 36. The storage unit 39 also stores various data necessary for various image processing, for example, image data, various flag values, threshold values, and the like.
The communication unit 40 controls communication with other devices such as the server 12 (FIG. 1) via the network 21.
For example, in this embodiment, in addition to various data, the cutout processing program may be transmitted from the server 12 as described above. In such a case, the communication unit 40 receives the clipping processing program and temporarily stores it in the storage unit 39. The cutout processing program stored in the storage unit 39 is expanded in the RAM 33 and executed by the CPU 31.

A removable medium 42 made of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately attached to the drive 41.
The program read from the removable medium 42 by the drive 41 is installed in the storage unit 39 as necessary. The removable medium 42 can also store various data such as image data stored in the storage unit 39 in the same manner as the storage unit 39.

  FIG. 3 is a block diagram showing a hardware configuration of the server 12 for executing the clipping process in cooperation with the client device 11.

  The server 12 includes a CPU 51, a ROM 52, a RAM 53, a bus 54, an input / output interface 55, an input unit 56, an output unit 57, a storage unit 58, a communication unit 59, and a drive 60. Yes.

The CPU 51 executes various processes according to a program recorded in the ROM 52 or according to a program loaded from the storage unit 58 to the RAM 53.
The RAM 53 also appropriately stores data necessary for the CPU 51 to execute various processes.

  For example, in the present embodiment, programs for realizing the functions of the communication control unit 151 and the main control unit 152 shown in FIG. 4 to be described later are stored in the ROM 52 and the storage unit 58. Therefore, each function of the communication control unit 151 and the main control unit 152 in FIG. 4 can be realized by the CPU 51 executing processing according to these programs.

  The CPU 51, ROM 52, and RAM 53 are connected to each other via a bus 54. An input / output interface 55 is also connected to the bus 54. An input unit 56, an output unit 57, a storage unit 58, a communication unit 59, and a drive 60 are connected to the input / output interface 55.

The input unit 56 includes a keyboard, a mouse, and the like, and inputs various information.
The output unit 57 includes a liquid crystal display, a speaker, a printer, and the like, and outputs various types of information. For example, the foreground clipped from the original image by the clipping process of the client device 11 and a new background different from the original image are combined, and the resultant combined image is printed on a paper medium or the like by the printer and output Is done.
The storage unit 58 includes a hard disk and stores various information.
The communication unit 59 controls communication with other devices such as the client device 11 (FIG. 1) via the network 21.

A removable medium 61 composed of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately attached to the drive 60.
The program read from the removable medium 61 by the drive 60 is installed in the storage unit 58 as necessary. The removable medium 61 can store various data such as image data stored in the storage unit 58 in the same manner as the storage unit 58.

  4 shows a functional configuration of the image processing system when the client device 11 having the hardware configuration shown in FIG. 2 and the server 12 having the hardware configuration shown in FIG. It is a block diagram.

[Functional configuration of client device 11]
First, the functional configuration of the client device 11 among the components of the image processing system will be described.

When the client device 11 communicates with the server 12 via the network 21, the communication control unit 101 functions in the CPU 31 of the client device 11.
That is, the communication control unit 101 is connected to the server 12 via the communication unit 40 and the network 21 and executes control for transferring various information.
For example, when a cutout processing program is transmitted from the server 12, the communication control unit 101 causes the communication unit 40 to receive the cutout processing program via the network 21 and temporarily stores the program in the program storage unit 131 of the storage unit 39.
When the cutout processing program stored in the program storage unit 131 is developed in the RAM 33 (FIG. 2) and executed in this way, the cutout processing unit 102 functions in the CPU 31.

  The cutout processing unit 102 includes a cutout specifying unit 111 and a finish adjusting unit 112 in order to execute cutout processing. The finish adjustment unit 112 includes an area calculation processing unit 121.

  That is, the cutout processing programs for realizing the cutout processing are roughly classified into a cutout designation program and a finish adjustment program.

When a cut designation program is executed in the cut processing program, the cut designation section 111 of the cut processing section 102 functions.
The clipping designation program designates the foreground area (hereinafter referred to as “selected area”) selected by the user as the clipping target from the original image to be processed, and finishes the boundary portion of the selected area neatly (boundary area) Is a program that performs a series of processing (hereinafter referred to as “cutout designation processing”) until the calculation result is stored and the calculation result is stored.
Note that the calculation for finely finishing the boundary portion of the selected region (smoothing the boundary region) is hereinafter referred to as “region calculation”.
In the clipping specification program, data can be saved even in the middle of processing, and when data saved in the middle of processing is read again, processing can be continued from the state at the time of saving. . That is, it becomes possible for the user to continue working from the stored state.

On the other hand, when a finishing adjustment program is executed in the cutout processing program, the finishing adjustment unit 112 in the cutout processing unit 102 functions.
That is, even if only the above-described cutout designation program is executed, data that can specify the selected area after finishing the boundary area can be obtained. However, if the user desires to obtain a higher-definition composite image, A program executed when it is desired to adjust the boundary region is a finishing adjustment program.
In the finish adjustment program, the result of the cut designation process by the cut designation program is read, and various processes involving area calculation (collectively referred to as “finish adjustment process”) are executed. Since the area calculation in the finishing adjustment process is accompanied by a mat process to be described later, the area calculation including this mat process is hereinafter referred to as “area calculation process”.
When this area calculation process is executed, the area calculation processing unit 121 of the finish adjustment unit 112 functions.
In the finishing adjustment program, data can be saved even in the middle of processing, and when data saved in the middle of processing is read again, processing can be continued from the saved state. . That is, it becomes possible for the user to continue working from the stored state.

  Further details of the functions of each of the cutout designation unit 111, the finish adjustment unit 112, and the area calculation processing unit 121 will be described later with reference to the drawings from FIG.

As described above, in the present embodiment, the communication control unit 101 and the clipping processing unit 102 are configured as a combination of the hardware called the CPU 31 and the program (software) in the configuration illustrated in FIG.
However, this is merely an example, and it is naturally possible to transfer at least some of the functions of the communication control unit 101 and the cutout processing unit 102 to other components other than the CPU 31.

In the present embodiment, a program storage unit 131 and an image storage unit 132 are provided as one area of the storage unit 39.
The program storage unit 131 stores a program for realizing the communication control unit 101 in advance. Further, as described above, when a cutout processing program is received under the control of the communication control unit 101, the cutout processing program is stored in the program storage unit 131.
The image storage unit 132 stores various image data to be processed when the cutout processing unit 102 executes cutout processing in a predetermined file format. Specific examples of image data that can be stored in the image storage unit 132 will be described later.
Note that the program storage unit 131 and the image storage unit 132 are provided as one area of the storage unit 39, but may be provided as one area of the removable medium 42, for example. The image storage unit 132 is not particularly required to be provided in the client device 11, and may be provided, for example, in another device connected via the communication unit 40 and the network 21.

  The functional configuration of the client device 11 among the components of the image processing system has been described above. Next, the functional configuration of the server 12 among the components of the image processing system will be described.

[Functional configuration of server 12]
When the client device 11 executes the clipping process in cooperation with the server 12 connected via the network 21, the communication control unit 151 and the main control unit 152 function in the CPU 51 of the server 12.

The communication control unit 151 is connected to the client device 11 via the communication unit 59 and the network 21 and executes control for transferring various information.
For example, when the communication control unit 151 is accessed from the client device 11 (for example, when a log-in process described later is performed), the communication control unit 151 reads out the cutout processing program stored in the program storage unit 161 of the storage unit 58 and performs communication. The data is transmitted from the unit 59 to the client device 11 via the network 21 (see steps S201, S221, and S223 in FIG. 13 described later).
Further, for example, when various image data such as an image after the clipping process is transmitted from the client device 11, the communication control unit 151 causes the communication unit 59 to receive the various image data via the network 21, and the storage unit 58 image storage units 162.
Thereafter, for example, based on the image data stored in the image storage unit 162, the foreground clipped from the original image by the clipping process and a new background different from the original image are combined, and the resultant composite image obtained as a result Is printed on a paper medium or the like by a printer and output.

When the client device 11 is connected to the network 21 and cooperates with the server 12 to execute the clipping process, the main control unit 152 controls the entire server 12.
Details of the control of the main control unit 152 will be described later with reference to FIG.

As described above, in this embodiment, the communication control unit 151 and the main control unit 152 are configured as a combination of the hardware called the CPU 51 and the program (software) in the configuration shown in FIG.
However, this is merely an example, and it is naturally possible to transfer at least some of the functions of the communication control unit 151 and the main control unit 152 to other components than the CPU 51.

In the present embodiment, a program storage unit 161 and an image storage unit 162 are provided as one area of the storage unit 58.
The program storage unit 161 stores a program for realizing each of the communication control unit 151 and the main control unit 152 in advance. The program storage unit 161 also stores in advance a cutout processing program that can be downloaded to the client device 11 as described above.
As described above, the image storage unit 162 stores various image data transmitted from the client device 11 and received by the communication unit 59 under the control of the communication control unit 151 in a predetermined file format. Specific examples of image data that can be stored in the image storage unit 162 will be described later.
Note that the program storage unit 161 and the image storage unit 162 are provided as one area of the storage unit 58, and may be provided as one area of the removable medium 61, for example. Further, the image storage unit 162 is not particularly required to be provided in the server 12, and may be provided, for example, in another device connected via the communication unit 59 and the network 21.

Next, details of the clipping process executed by the image processing system having the functional configuration of FIG. 4 will be described.
First, with reference to FIG. 5 thru | or FIG. 8, the detail of a clipping designation | designated process is demonstrated among clipping processes.

  FIG. 5 is a state transition diagram showing an example of each state that can be taken when the client device 11 executes the clipping designation process.

In FIG. 5, each state is indicated by one ellipse, and is determined by a code including “S” drawn on the ellipse.
The state transition from one state to one state (including the case where the state remains the same) is executed when a predetermined condition (hereinafter referred to as “state transition condition”) is satisfied.
In FIG. 5, such a state transition condition is represented by adding a symbol including “C” to an arrow representing a transition from one state to one state.
Note that the contents of this paragraph also apply to the state transition diagram of FIG. 9 described later.

As a premise, in this embodiment, it is assumed that the data of the processing target image is read from the image storage unit 132 in advance before the start of the clipping designation process.
That is, details will be described later with reference to the flowchart of FIG. 13, but the foreground to be cut out from among a plurality of image data stored in the image storage unit 132 before the cut-out process (cut-out designation process). Image data including (person or the like) is selected by the user (see step S204 in FIG. 13 described later).
The image data selected in this way is read from the image storage unit 132 as data of the processing target image and transmitted to the server 12 (see step S205 in FIG. 13 described later). As a result, the cut designation process of the cut process is started (see step S206 in FIG. 13 and step S241 in FIG. 14 described later).
Then, the clipping designation unit 111 (FIG. 4) determines that the state transition condition C1 is satisfied, and transitions the state of the client device 11 to the processing target image acquisition processing state S11.

When a transition is made to the processing target image acquisition processing state S11, the clipping designation unit 111 acquires the data of the processing target image.
Here, the data of the processing target image is not particularly limited, and may be, for example, captured image data obtained as a result of imaging a person by the imaging unit 36 (FIG. 2), or may be the storage unit 39 or the removable medium 42. The image data stored in the memory may be used.
However, if the resolution (size) of the read image remains as it is, the clipping designation process may take a long time. Therefore, in the present embodiment, the clipping designation unit 111 reduces the resolution (size) of the read image data.
The resolution (size) after being reduced by the cutout specifying unit 111 in this way is hereinafter referred to as “reduced size”. For example, if the notation of (the number of pixels in the horizontal direction) × (the number of pixels in the vertical direction) is used, the size of 640 × 480 or the size of 480 × 640 is adopted as the reduction size in this embodiment. .

Here, the following two types exist as types acquired as data of the processing target image.
That is, the first type of data is original image data such as a captured image.
Mainly, when the clipping designation process for the processing target image is performed for the first time, the first type of data, that is, the original image data itself is acquired. In this case, when the original image data is converted into reduced-size data, the clipping designation unit 111 determines that the state transition condition C2 is satisfied, and changes the state of the client device 11 to the clipping region designation screen operation processing state S12. Transition to.
On the other hand, the second type of data is data of an image (such as a cut image) saved during the cut designation process for the previous processing target image.
When the data of the second type of image is acquired and the data is converted into reduced size data as necessary, the clipping designation unit 111 determines that the state transition condition C3 is satisfied, and the client device 11 The state is changed to the cutout correction screen operation processing state S14.

In this way, the processing target image acquisition processing state S11 transits to the clipping region designation screen operation processing state S12 or the clipping correction screen operation processing state S14.
Here, the “cutout area designation screen operation processing state” means that a GUI (Graphical User Interface) screen named “Cutout area designation screen” is displayed on the display unit 38 of the client device 11 (FIGS. 2 and 4). When a GUI operation by the user is performed on the screen, a process corresponding to the content of the GUI operation is being executed.
Accordingly, the user performs various GUI instructions using the mouse or the like of the operation unit 37 (FIGS. 2 and 4) using the cutout area designation screen to give various instructions necessary for executing the cutout process to the client device. 11 can be performed interactively.
Note that the items described in this paragraph also apply to other screen operation process states such as a cut correction screen operation process state in the cut process.
Since there are a plurality of types of screens used in the clipping process, different types of screens are displayed on the display unit 38 in each state.
However, a plurality of types of screens used in the clipping process are all generated based on a basic screen as shown in FIG. In other words, among the elements (software buttons, sliders, etc.) that make up the basic screen, when a predetermined screen is displayed, it is not usable or there is no need to select the icon, and the icon is selected. The predetermined type of screen is generated so that the display (such as a movement icon) is prohibited.

FIG. 6 shows an example of a basic screen used in the clipping process.
The basic screen 201 is divided into six display areas 211 to 216.

  The display area 211 is located at a substantially central portion of the basic screen 201, and displays a portion that is a user's operation target in the processing target image.

In the display area 212, for example, a software button 231 (hereinafter referred to as a “color change button 231”) displayed as “color change” is arranged. The color change button 231 is assigned a color change instruction function for the boundary area and the background area outside the boundary area (non-selected area described later).
Further, for example, although not shown in FIG. 6 (see FIG. 11), a software button 351 displaying “change background” (hereinafter referred to as “background change button 351”) is arranged at the same position in the display area 212. May be. The background change button 351 is assigned a function of instruction such as display switching of backgrounds (of different colors).
That is, the user can instruct the contents assigned to them by pressing the color change button 231 or the background change button 351 using the mouse of the operation unit 37 or the like. Details of these various instructions will be described later.

In the display area 212, a slider 232 having “transparency” displayed on the left side (hereinafter referred to as “transparency change slider 232”) is arranged. The transparency changing slider 232 is a slider for changing the transparency of the boundary area and the background area outside the boundary area (non-selected area described later).
Here, the slider refers to a software control that simulates a slide volume seen in an audio device or the like. In general, a slider is composed of a small knob and a display imitating a groove for moving the slider. This slider has a function of selecting one value from a certain range. Of the both ends of the groove, one end has a minimum value and the other end has a maximum value. That is, the user can select an arbitrary value within the range of the groove (1 transparency in the transparency changing slider 232) by dragging and moving the small knob using the mouse of the operation unit 37 or the like. it can. If you focus only on the function of selecting one of the options, you may adopt radio buttons or list boxes, but by using a slider as in this embodiment, the user has a continuous relationship. A predetermined value can be selected from a large number of values.

In the display area 213, a slider 241 (hereinafter referred to as “enlargement / reduction slider 241”) displayed as “enlargement / reduction” is arranged.
The enlargement / reduction slider 241 is a slider that enlarges or reduces the image displayed in the display area 211. In the enlargement / reduction slider 241 of this embodiment, the minimum magnification of 1 is associated with the left end of the groove, and the maximum magnification of 8 is associated with the right end of the groove. ing. That is, in this embodiment, selection less than 1 time cannot be performed. When a small knob is put on the left end of the groove, the display area 211 is displayed as a 1 × display. Here, the 1 × display means a state in which the entire edited image (processing target image) is displayed in the full display area 211.

Drawing tools are arranged in the display area 214.
The drawing tool refers to a tool that is used by a user to change an operation of a mouse operation on the display area 211 by clicking an icon or a pen using the mouse of the operation unit 37 or the like.
The type and number of elements constituting the drawing tool are not particularly limited, but in this embodiment, as shown in FIG. 6, the movement icon 251, the background pen 252, the boundary pen 253, the eraser 254, and the pen size 255 are displayed in the display area 214. Is arranged.
The movement icon 251 is an icon having a function of changing (moving) the display area of the screen that is being enlarged when the user performs mouse dragging using the mouse or the like of the operation unit 37.
The background pen 252 is a software pen used for designating a background.
The boundary pen 253 is a software pen used to specify the boundary between the background and the foreground.
The eraser 254 is used for erasing the area drawn with the boundary pen 242 on the cutout area designation screen (see FIG. 7 described later), and on the screen other than the cutout area designation screen, the selected area (the foreground area to be cut out) is used. It is a tool for use to specify.
The pen size 255 is an icon for selecting the size of the background pen 252, the boundary pen 253, and the eraser 254. The user can change the paint size by clicking the icon of the desired size among the pen sizes 255 using the mouse of the operation unit 37 or the like.
In this embodiment, the setting of the pen size 255 (paint size) is reset when the screen is shifted, and the pen size 255 is changed when the area calculation (including the mat area calculation) is executed. The setting (paint size) is maintained, and the setting (paint size) of the pen size 255 at the time of display enlargement is enlarged together with the enlargement of the screen.

Various software buttons such as software buttons 261 to 264 are arranged in the display area 215.
Since the software buttons arranged in the display area 215 vary depending on various screens, they will be described together with the description of various screens to be described later. Note that the software buttons 261 to 264 shown in FIG. 6 are used on a finishing area designation screen described later.

  The display area 216 is provided as a spare area, and various images such as message images are appropriately displayed.

  Next, among the screens generated based on the basic screen 201 of FIG. 6, the screen displayed in the cut region designation screen operation processing state S12 of FIG. 5, that is, the cut region designation screen will be described.

FIG. 7 shows an example of the cutout area designation screen.
The cutout area designation screen 202 is divided into six display areas 211 to 216 as in the basic screen 201 (FIG. 6).
However, nothing is displayed in the display areas 212, 213, and 216 of the cutout area designation screen 202.

In the display area 211, the entire processing target image such as a captured image (however, a reduced size) is displayed as an initial state.
The display area 214 displays a boundary pen 253, an eraser 254, and a pen size 255 among the drawing tools.

  Therefore, the user uses the mouse or the like of the operation unit 37 to specify that the boundary area between the foreground and the background in the processing target image displayed in the display area 211 is surrounded by the boundary pen 253. At this time, the boundary area designated by the user is displayed in the display area 211 with the thickness of the fill size designated by the pen size 255. For example, in the example of FIG. 7, since the boundary area between the foreground (sea turtle) and the background is being specified, a partial area 301 corresponding to the specified part is displayed as the boundary area.

When the user designates an incorrect part with the boundary pen 253 during the operation of specifying the boundary area, the wrong part of the boundary area displayed in the display area 211 using the eraser 254 is used. The area corresponding to can be erased.
In addition, when the user wants to change the drawing size (painting size) of the boundary pen 253 and the eraser 254, the pen size 255 may be set.

  The display area 215 of the cutout area designation screen 202 includes a software button 311 (hereinafter referred to as “redo button 311”) displayed as “redo” and a software button 312 (hereinafter referred to as “area calculation”). Are called area calculation buttons 312 ”.

The redo button 311 is assigned a function for canceling the previous operation.
That is, the user can cancel the previous operation by pressing the redo button 311 using the mouse or the like of the operation unit 37 when the previous operation content is incorrect.

The area calculation button 312 is assigned a function for instructing start of area calculation.
That is, the user can issue a region calculation start instruction for the boundary region by pressing the region calculation button 312 using the mouse or the like of the operation unit 37 after the boundary region designation operation.

  When the area calculation button 312 is pressed, the clipping designation unit 111 (FIG. 4) determines that the state transition condition C4 is satisfied as shown in FIG. 5, and the state of the client device 11 is changed to the area calculation processing state. Transition to S13.

When transitioning to the area calculation processing state S13, the clipping designation unit 111 performs area calculation for the boundary area designated by the user based on the data (reduction size) of the processing target image.
Specifically, in this embodiment, the clipping designation unit 111 performs a region calculation by executing a series of processes as follows.
That is, the clipping designation unit 111 determines the area inside the boundary area designated by the user as an area that can be determined as the foreground (hereinafter referred to as “absolute foreground area”) based on the data of the processing target image. An area outside the boundary area is determined as an area (hereinafter referred to as “absolute background area”) that can be determined as the background.
Next, the clipping designation unit 111 calculates a cost function for evaluating the goodness of the binary label indicating the foreground or the background for each pixel constituting the processing target image (reduced size) as a first-order MRF (Markov Random Field). ) Model the model and find the optimal solution.
Specifically, the clipping designation unit 111 creates a data term using information on the absolute foreground region and the absolute background region, and creates an adjacent term using the edge property of the processing target image.
As a result, an optimized reduced size binary label is obtained. Therefore, the cutout designation unit 111 stores the binary label in the image storage unit 132 (FIG. 4) or the like of the storage unit 39.
In this way, in the present embodiment, the area calculation is performed by the cutout designation unit 111.

When such region calculation is completed, the clipping designation unit 111 (FIG. 4) determines that the state transition condition C5 is satisfied as shown in FIG. 5, and the state of the client device 11 is changed to the clipping correction screen operation processing state. Transition to S14.
Although not shown in FIG. 5, if the area calculation is not performed due to a defect in the boundary area specified by the boundary pen 253 (for example, the boundary area is not closed), the clipping designation unit 111 displays the area calculation. Is displayed on the display unit 38, and the clipping region designation screen operation processing state is maintained.

  When the transition is made to the cutout correction screen operation processing state S14, the cutout designation unit 111 causes the display unit 38 to display a cutout correction screen as shown in FIG.

FIG. 8 shows an example of the cutout correction designation screen.
The cutout correction screen 203 is divided into six display areas 211 to 216, similarly to the basic screen 201 (FIG. 6).

In the display area 211, as an initial state, an image obtained as a result of performing the above-described area calculation by pressing the area calculation button 312 on the cut-out area designation screen 202 in FIG. 7 is displayed.
That is, in the display area 211, a composite image in which a binary label (reduction size) is combined with a processing target image (reduction size) is displayed, and a boundary area between the foreground and the background (in the example of FIG. A surrounding area 302) is displayed.
In this case, the area in the boundary area is a foreground area selected by the user as a clipping target, that is, a selection area. The selected area has a fixed transmittance of 100%, and the processing target image itself is always displayed.
A screen for the user to correct the selection area using the mouse or the like of the operation unit 37 is a cutout correction screen 203.

In the display area 212, a color change button 231 and a transparency change slider 232 are arranged.
Here, a region outside the boundary region (region 302 surrounding the periphery of the sea turtle in the example of FIG. 8), that is, a background region that is not selected by the user as a clipping target is hereinafter referred to as a “non-selected region”. In this case, the images displayed in the display area 211 of the cutout correction screen 203 are classified into a selection area as a foreground, a non-selection area as a background, and a boundary area serving as a boundary between these areas.
In the present embodiment, each of the boundary area and the non-selected area is displayed in the first color and the second color so that the user can visually recognize such classification. Hereinafter, the first color for displaying the boundary area is referred to as “boundary color”, and the second color for displaying the non-selected area as the background is referred to as “background color”.

An enlargement / reduction slider 241 is disposed in the display area 213.
Accordingly, by operating the enlargement / reduction slider 241, the user can enlarge and display a portion of interest such as a desired region or a combined portion of two or more regions among the selected region, the boundary region, and the non-selected region. . Thereafter, the user can operate the enlargement / reduction slider 241 to reduce and display the image after enlargement display within the range up to the same magnification.

  In the display area 214, among the drawing tools, a movement icon 251, a background pen 252, an eraser 254, and a pen size 255 are displayed.

Therefore, the user uses the mouse of the operation unit 37 or the like to specify that the area that is supposed to be the foreground but the background should be erased with the eraser 254 in the image displayed in the display area 211. The area can be newly designated as the selection area.
On the other hand, the user uses the mouse or the like of the operation unit 37 to designate the area displayed in the display area 211 to be drawn with the background pen 252 in the foreground that should be the background. Thus, the area can be designated as a non-selected area.
In short, since the non-selected area is displayed in the second color in the display area 211, the user has an image of an area that is cut out when the selected area (foreground) is cut out. Then, it is possible to work to correct the selected area.
When the display area 211 is being enlarged, the user can change (move) the display area (work target area) using the movement icon 251. In addition, when the user wants to change the drawing size (painting size) of the background pen 252 or the eraser 254, the pen size 255 may be set.

  In the display area 215 of the cutout correction screen 203, a redo button 311 and an area calculation button 312 are arranged as in the cutout area designation screen 202. Further, the display area 215 includes a software button 313 (hereinafter referred to as “area designation button 313”) displayed as “area designation” and a software button 314 (hereinafter referred to as “save button”) displayed as “save”. 314 ") is arranged.

  The user can cancel the previous operation by pressing the redo button 311 using the mouse or the like of the operation unit 37 when the previous operation content is wrong.

  In addition, the user can issue an instruction to start recalculation of the area calculation by pressing the area calculation button 312 again using the mouse of the operation unit 37 after correcting the selected area.

  That is, when the area calculation button 312 on the cutout correction screen 203 is pressed, the cutout designation unit 111 (FIG. 4) determines that the state transition condition C6 is satisfied as shown in FIG. The state is changed again to the region calculation processing state S13.

  When the transition is made again to the area calculation processing state S13, the cutout designation unit 111 performs recalculation of the area calculation for the selected area after being corrected by the user based on the data (reduction size) of the processing target image.

  When such recalculation of the area calculation is completed, the clipping designation unit 111 determines that the state transition condition C5 is satisfied again as shown in FIG. 5, and changes the state of the client device 11 to the clipping correction screen operation processing state. Transition to S14 again.

  When the transition is made again to the cutout correction screen operation processing state S14, the cutout specifying unit 111 causes the display unit 38 to display a cutout correction screen. In this case, an image showing the result of the recalculation of the area calculation is displayed in the display area 211.

  Here, there is no particular limitation on the number of times the area calculation button 312 is pressed. Accordingly, when the user performs an operation for correcting the selected area, presses the area calculation button 312, and the result of the area calculation recalculation is displayed in the display area 211, the correction result is displayed based on the display content of the display area 211. It is possible to repeat a series of operations such as confirming a number of times.

Note that if a desired selection area cannot be obtained even after repeating such a series of operations, the user may perform the boundary area designation operation from the beginning.
In such a case, the user may press down the area designation button 313.
Then, as shown in FIG. 5, it is determined that the state transition condition C7 is satisfied, and the state of the client device 11 transits again to the cutout area designation screen operation processing state S12, and the cutout area designation screen 202 shown in FIG. It will be displayed again.
Thereby, the user can perform GUI operation with respect to the clipping area designation screen 202 again.

On the other hand, when such a series of operations are repeated and a desired selection area is obtained, the user presses the save button 314 on the cutout correction screen 203 in FIG. 8 to save the result of the cutout designation process. Good.
Then, as illustrated in FIG. 5, it is determined that the state transition condition C8 is satisfied, and the state of the client device 11 transitions to the boundary area finishing process state S15.

When the transition is made to the boundary area finishing process state S15, the clipping designation unit 111 executes the boundary area finishing process.
Thereby, it is determined that the state transition condition C9 is satisfied, and the state of the client device 11 transitions to the result storage processing state S16. When transitioning to the result storage processing state S16, the clipping designation unit 111 generates a file including at least the data of the processing target image (original size) and the binarized label (reduced size) obtained by the area calculation, The image is stored in the image storage unit 132 of the storage unit 39.
Such a file is hereinafter referred to as a “cutout designation processing result file”.
As a GUI operation for the cutout correction screen 203 in FIG. 8, after the correction with the background pen 252 or the eraser 254, the area calculation button 312 is not pressed (the area calculation is not performed), and the save button 314 is pressed. In this case, the clipping designation unit 111 creates data (binary label) that reflects the last modified state in the selected area and the non-selected area without performing the area calculation, and performs the clipping designation process. Include in the result file.

  When the clipping designation process result file is stored in the storage unit 39 in this way, it is determined that the state transition condition C10 is satisfied as shown in FIG. 5, and the clipping designation process ends.

As described above, the details of the clipping designation process among the clipping processes executed by the image processing system having the functional configuration of FIG. 4 have been described with reference to FIGS. 5 to 8.
Next, with reference to FIGS. 9 to 12, the details of the finish adjustment process in the clipping process will be described.

  FIG. 9 is a state transition diagram illustrating an example of each state that can be taken when the client device 11 executes the finish adjustment process.

The finish adjustment process is executed after the cutout designation process. That is, even if the cutout specification processing result file is used, it is possible to cut out the foreground from the processing target image and combine it with another new background. However, some users desire a high-precision finish of the boundary region and a higher-definition image finish. Therefore, a finishing process is executed for such a user.
The instruction to start the finish adjustment process is performed by a predetermined operation on the operation unit 37.
When such an instruction is given, the finish adjustment unit 112 (FIG. 4) determines that the state transition condition C41 is satisfied, and transitions the state of the client device 11 to the file reading processing state S41.

When the transition is made to the file reading processing state S41, the finish adjustment unit 112 reads from the storage unit 39 a cut-out designation processing result file including data of the processing target image.
Thereby, it is determined that the state transition condition C42 is satisfied, and the state of the client device 11 transits to the size setting processing state S42.

When the transition is made to the size setting processing state S42, the finish adjusting unit 112 executes the following series of processing.
That is, although not shown, the finish adjustment unit 112 displays a screen for selecting an output size on the display unit 38 and waits for a user's selection operation.
The user operates the operation unit 37 to select a desired size as an image size when performing printout or the like.
The finish adjustment unit 112 sets the size selected by the user as the output size. Here, the output size is generally equal to or smaller than the original size of the processing target image such as a captured image, but is not particularly limited to this, and may be any size.
When such a series of processing is executed, it is determined that the state transition condition C43 is satisfied, and the state of the client device 11 transitions to the region calculation processing state S43.

  When the transition is made to the mat processing state S43, the area calculation processing unit 121 of the finish adjustment unit 112 executes the area calculation process on the read cut designation processing result file.

In the area calculation, mat processing is executed.
The essence of the matte process is to separate the mixed color of the foreground and background boundary pixels.
Specifically, mat processing is processing that estimates the unknowns α and F (also B in some cases) from the observed value P by simultaneously satisfying the condition of the following expression (1).
P [x, y, c] = α [x, y] F [x, y, c] + (1-α [x, y]) B [x, y, c] (1)
In the left side of Equation (1), P [x, y, c] represents the pixel value of coordinates x, y (hereinafter also referred to as “pixel position x, y”) and color component c.
In the right side of the equation (1), α [x, y] is a ratio at the pixel position x, y and is a ratio that varies within a range of 0 to 1, such as “mixing ratio α” and “alpha value”. And so on. That is, the mixture ratio α [x, y] has a different value for each pixel. F [x, y, c] is a foreground pixel value, and is hereinafter referred to as a “foreground pixel value”. B [x, y, c] is a background pixel value, and is hereinafter referred to as a “background pixel value”.
In the case where it is not necessary to individually distinguish the pixel position x, y and the color component c, the notation of x, y, c will be omitted as appropriate.

Equation (1) shows that the foreground pixel value F [x, y, c] and the background pixel value B [x, y, c] are mixed at the ratio α [x, y] at the pixel position x, y in the boundary region As a result, the pixel value P [x, y, c] is observed.
Here, since equation (1) is an insufficient condition equation when simply coupled, it cannot be solved as it is. However, by specifying the possible range of the background pixel value B by another means (chroma key), or by making assumptions such as the foreground pixel value F and the background pixel value B changing smoothly (locally equal) Equation (1) can be solved.

For this reason, in the mat processing, the mixture ratio α (alpha value) and the foreground pixel value F are estimated. If the mixture ratio α (alpha value) and foreground pixel value F can be correctly estimated by mat processing in this way, the following equation (2) is calculated for the new background pixel value B ′ [x, y, c]. Thus, the pixel value P ′ [x, y, c] at the pixel position x, y of the composite image P ′ can be easily obtained.
P ′ [x, y, c] = α [x, y] F [x, y, c] + (1-α [x, y]) B ′ [x, y, c] (2)
Since the background pixel value B [x, y, c] of the original image (the processing target image in the present embodiment) does not ideally remain in the composite image P ′ obtained in this way, it is beautiful and natural. Result.

  For further details of mat processing, reference may be made to “Bayesian approach to digital matting”, 2001 (IEEE CVPR) and the like.

Here, in the image processing system (FIG. 1) of the present embodiment, it is also assumed that the composite image P ′ is printed out.
When considering a sufficient number of pixels as a print output, the biggest problem with performing mat processing is that a huge amount of calculation is required according to the number of output pixels.
However, the real-time mat processing of the number of pixels of recent digital still cameras and digital printers that greatly exceed 10 million pixels is performed by a personal computer (for example, the client device 11 in FIG. 1) or a web server (for example, the server 12 in FIG. 1). It is difficult for them.
Therefore, in the present embodiment, the area calculation process as shown in FIG. 10 is executed to greatly reduce the calculation amount of the mat process.

  FIG. 10 is a flowchart for explaining the flow of the area calculation process.

In step S101, the area calculation processing unit 121 sets a medium resolution size.
Here, the medium resolution size refers to the smaller one of the output size and a predetermined size (which is equal to or larger than the reduced size used in the above-described clipping designation process).
Accordingly, the area calculation processing unit 121 sets the medium resolution size by comparing the output size with a predetermined size set in advance and selecting the smaller size.

In step S102, the area calculation processing unit 121 reads out the data of the processing target image from the clipping designation processing result file, and converts the size of the processing target image data from the original size to the medium resolution size.
Since the original size is usually larger than the medium resolution size, the process in step S102 is a so-called size reduction process.

In step S103, the area calculation processing unit 121 reads the binary label from the clipping designation processing result file, and converts the size of the binarized label from the reduced size to the medium resolution size, thereby generating a medium resolution binary label. To do.
In general, since the reduced size is smaller than the medium resolution size, the process in step S103 is a so-called size enlargement process.
Thus, since the medium resolution size binary label needs to be a binary label as the name suggests, the method of converting the size of the binary label employs, for example, the nearest neighbor method or, for example, bicubic. It is advisable to adopt a method of re-binarizing the result such as the method.

In step S <b> 104, the area calculation processing unit 121 creates a medium resolution size trimap from the medium resolution binary label.
The trimap has foreground, background, and intermediate ternary labels for each pixel position x, y.
Here, the intermediate label region of the trimap is a set sum region of the following first region and second region.
That is, the foreground label area of the medium resolution binary label is expanded by a predetermined width (dilation in the morphological process), and the resulting expanded area (not including the original binary foreground label area) is used as the first area. Adopted.
On the other hand, the background label area of the medium resolution binary label is expanded (same) by a predetermined width, and the resulting expanded area (not including the original binary background label area) is adopted as the second area.
An intermediate label area is obtained as a set sum area of the first area and the second area. This intermediate label area is an area in which the boundary area designated by the user with the boundary pen 253 is slightly exaggerated. Therefore, the user specifies the boundary area with the boundary pen 253, and also specifies the area serving as a reference for the intermediate label area in the mat processing.
Of the areas not included in the intermediate label area obtained in this way, the binary foreground label area becomes the foreground label area, and the binary background label area becomes the background label area.

In step S105, the region calculation processing unit 121 sets a set of blocks having a predetermined number and a predetermined shape covering the medium resolution size for the trimap.
Each block is set to have an overlap area with a predetermined width at the boundary with the adjacent block.

  In step S106, the area calculation processing unit 121 sets a block to be processed from the block aggregate set in the process in step S105.

In step S107, the region calculation processing unit 121 inputs a trimap block and a corresponding medium resolution processing target image block as processing target blocks, and executes mat processing based on these inputs.
By the mat processing in step S107, the mixture ratio α (alpha value) and foreground pixel value F are obtained for each of the pixels constituting the block to be processed.

In step S108, the area calculation processing unit 121 determines whether all blocks have been set as processing targets.
If there is a block that has not yet been set as a processing target, it is determined as NO in step S108, the process returns to step S106, and the subsequent processes are repeated. That is, by repeatedly executing the loop processing of steps S106 to S108, each of the blocks set in the processing of step S105 is sequentially set as a processing target, and mat processing is executed each time.
When the mat process for the last block is executed, it is determined as YES in the next step S108, and the process proceeds to step S109.

In step S109, the region calculation processing unit 121 mosaics the mixture ratio α (alpha value) and foreground pixel value F of each block, and collects the mixture of medium resolution size mixture ratio α (hereinafter referred to as “medium resolution α image”). And an aggregate of foreground pixel values F of medium resolution size (hereinafter referred to as “medium resolution foreground F image”).
Note that the area calculation processing unit 121 performs blending with a gentle blend ratio for the overlap area between blocks, and alleviates boundary mismatch.

  As a result, the area calculation process is completed. Then, the finish adjustment unit 112 (FIG. 4) determines that the state transition condition C44 is satisfied as shown in FIG. 9, and transitions the state of the client device 11 to the finish confirmation screen operation processing state S44.

  When a transition is made to the finish confirmation screen operation processing state S44, the finish adjustment unit 112 causes the display unit 38 to display a finish confirmation screen as shown in FIG.

FIG. 11 shows an example of the finishing confirmation screen.
The finish confirmation screen 204 is a screen for confirming an image in which the boundary region is finished by the region calculation process, and is divided into six display regions 211 to 216 as in the basic screen 201 (FIG. 6).

In the display area 211, the result of the mat processing is displayed.
That is, the intermediate resolution processing target image data and the new background image data prepared in advance (converted to the intermediate resolution as necessary) are combined according to the above-described equation (2), and the result of the combination is obtained. The composite image P ′ is displayed in the display area 211.
In this case, each pixel value constituting the data of the new background image is substituted into the expression (2) as the background pixel value B ′ [x, y, c]. In addition, each pixel value of the medium resolution α image obtained by the mat processing is substituted into the equation (2) as a mixture ratio α [x, y]. Each pixel value of the medium resolution foreground F image obtained by the mat processing is substituted into the equation (2) as the foreground pixel value F [x, y, c].
Note that a gray (single color) image is displayed in the display area 211 as the new background image in the initial state.

In the display area 212, a background change button 351 is displayed.
Therefore, the user can change the background image displayed in the display area 211 to a desired one of the plurality of new background images by pressing the background change button 351.
The new background image is not particularly limited, and an arbitrary image desired by the user can be set in addition to an image set as a default in advance. However, the setting of the new background image is performed on another screen (not shown).
In this way, the user can instantly change to various new background images simply by pressing the background change button 351, that is, by simply clicking the mouse on the operation unit 37, etc. It is easy to visually recognize the processing result as an actual composite image.

An enlargement / reduction slider 241 is disposed in the display area 213.
Accordingly, by operating the enlargement / reduction slider 241, the user can enlarge and display a portion of interest in the composite image P ′ that is the result of the mat processing. Thereafter, the user can operate the enlargement / reduction slider 241 to reduce and display the combined image P ′ after the enlargement display within a range up to the same magnification.

  The display area 215 includes a software button 361 (hereinafter referred to as “finish adjustment button 361”) displayed as “finish adjustment” and a software button 362 (hereinafter referred to as “save button 362”) displayed as “save”. Are arranged).

  When the user confirms the composite image P ′ displayed in the display area 211 and determines that the partial correction of the finishing of the boundary area is to be further performed, the finish adjustment button 361 is pressed to start finishing adjustment. Can be instructed.

  That is, when the cut finish adjustment button 361 is pressed, the finish adjustment unit 112 (FIG. 4) determines that the state transition condition C45 is satisfied, as shown in FIG. Transition to the adjustment screen operation processing state S45.

  When a transition is made to the finish adjustment screen operation processing state S45, the finish adjustment unit 112 causes the display unit 38 to display a finish adjustment screen as shown in FIG.

FIG. 12 shows an example of the finish adjustment screen.
The finishing adjustment screen 205 is a screen for performing partial finishing on the processing target image that has been subjected to the finishing of the boundary area by the mat processing. For example, the finish adjustment screen 205 is used to correct the smoothness of the boundary of a part in order to clean out a thin part such as a hair part. The finishing adjustment screen 205 is divided into six display areas 211 to 216, similarly to the basic screen 201 (FIG. 6).

The display area 211 includes a selection area as a foreground, a non-selection area as a background, and a boundary area that is a boundary between these areas (area 381 in the example of FIG. 11). ) And categorized and displayed respectively.
That is, in the display area 211, a composite image in which a trimap (medium resolution size) is combined with a processing target image (medium resolution size) is displayed.
In this case, the transparency is set on the trimap side. That is, the selected area has a fixed transmittance of 100%, and the processing target image itself (a sea turtle in the example of FIG. 12) is always displayed. On the other hand, the non-selected region of the trimap is a background color (single color) region, and as the transparency increases, the background portion of the processing target image becomes visible to the user. Similarly, the boundary region (intermediate label region) of the trimap is a boundary color (single color) region, and as the transparency increases, the boundary portion of the processing target image is visually recognized by the user.
Therefore, similarly to the cutout correction screen 203 in FIG. 8, a color change button 231 and a transparency change slider 232 are arranged in the display area 212, and an enlargement / reduction slider 241 is arranged.
However, since the parts of the color change button 231, the transparency change slider 232, and the enlargement / reduction slider 241 have been described as the parts of the cutout correction screen 203 in FIG. 8, the description thereof is omitted here.

The display area 215 includes a software button 371 (hereinafter referred to as “redo button 371”) displayed as “redo” and a software button 372 (hereinafter referred to as “confirmation screen button 372”) displayed as “confirmation screen”. Are arranged).
The user can cancel the previous operation by pressing the redo button 371 when the previous operation content is wrong.
The confirmation screen button 372 will be described later.

By the way, in the state where the finish adjustment screen 205 of FIG. 12 is displayed on the display unit 38, the user uses the mouse of the operation unit 37 to select a portion to be corrected in the image displayed in the display area 211. By clicking, it is possible to instruct the display of the correction area designation screen.
That is, when such a click operation is performed to instruct the display of the correction area designation screen, the finish adjustment unit 112 (FIG. 4) satisfies the state transition condition C46 as shown in FIG. And the state of the client device 11 is changed to the finishing area designation screen operation processing state S46.

  When a transition is made to the finish area designation screen operation processing state S46, the finish adjustment unit 112 causes the display unit 38 to display a finish area designation screen having the same configuration as the basic screen 201 shown in FIG.

A display area 211 of the finish area designation screen (same configuration as the basic screen 201 in FIG. 6) is an area of a predetermined size centered on the place where the click operation is performed on the finish adjustment screen 205 in FIG. An image is displayed.
Specifically, the finish adjustment unit 112 blocks an area in the vicinity of the place where the click operation is performed on the finish adjustment screen 205 of FIG. Here, this block does not necessarily match the size, position, and shape of the block set in the process of step S105 in FIG.
The finish adjustment unit 112 transparently displays the original image (processing target image) and the trimap of the block as a region image on the display region 211.
The user can correct the selected area or the like within the range of the area image using the mouse or the like of the operation unit 37.

Therefore, similarly to the cutout correction screen 203 in FIG. 8, a color change button 231 and a transparency change slider 232 are arranged in the display area 212, and an enlargement / reduction slider 241 is arranged in the display area 213. .
However, since the parts of the color change button 231, the transparency change slider 232, and the enlargement / reduction slider 241 have been described as the parts of the cutout correction screen 203 in FIG. 8, their descriptions are omitted here.

Of the area images displayed in the display area 211, the user uses the background pen 252 for the non-selected area as the background, the boundary pen 253 for the boundary area, and the eraser for the selected area as the foreground. Using 254, a correction operation can be performed.
By such a correction operation, the trimap displayed as the area image in the display area 211 is corrected.
Note that the user can change (move) the display area (work target area) using the move icon 251, but may prohibit the change (move) of the display area as necessary. In addition, when the user wants to change the drawing size (painting size) of the background pen 252, the boundary pen 253, or the eraser 254, the pen size 255 may be set.

  In the display area 215 of the finishing area designation screen (same configuration as the basic screen 201 in FIG. 6), a software button 261 displayed as “Redo” (hereinafter referred to as “Redo button 261”), “Area calculation”, and A displayed software button 262 (hereinafter referred to as “area calculation button 262”), a software button 263 (hereinafter referred to as “finish adjustment button 263”) displayed as “finish adjustment”, and a “confirmation screen”. A displayed software button 264 (hereinafter referred to as “confirmation screen button 264”) is arranged.

  The user can cancel the previous operation by pressing the redo button 261 when the content of the previous operation is wrong.

  Further, the user can issue an instruction to start recalculation of the mat area calculation by pressing the area calculation button 262 after correcting the selected area or the like, that is, after correcting the trimap.

  That is, when the area calculation button 262 on the finish area designation screen (same configuration as the basic screen 201 in FIG. 6) is pressed, the finish adjustment unit 112 (FIG. 4), as shown in FIG. Is satisfied and the state of the client device 11 is changed to the partial mat area calculation processing state S47.

  When the state transitions to the partial region calculation processing state S47, the region calculation processing unit 121 of the finishing adjustment unit 112 executes partial region calculation processing for the selected region after being corrected by the user.

However, as the partial region calculation processing in this case, the following processing is executed in this embodiment instead of the processing according to the flowchart of FIG. 10 described above.
That is, the area calculation processing unit 121 of the finishing adjustment unit 112 performs mat processing on the blocks displayed in the display area 211 of the finishing area designation screen (same configuration as the basic screen 201 in FIG. 6) using the modified trimap. I do.
The area calculation processing unit 121 mosaics the mixture ratio α (alpha value) and foreground pixel value F obtained as a result of the mat processing into the overall result. That is, the area calculation processing unit 121 gently blends the vicinity of the predetermined width boundary of the block and overwrites the area inside.
When the medium resolution α image and the medium resolution foreground F image are reconstructed in this way, the finish adjustment unit 112 (FIG. 4) determines that the state transition condition C48 is satisfied as shown in FIG. Then, the state of the client device 11 is changed again to the finishing adjustment screen operation processing state S45.

  When the transition is made again to the finishing adjustment screen operation processing state S45, the finishing adjustment screen 205 (FIG. 12) is displayed again on the display unit 38. In this case, a partial mat area calculation process result is displayed in the display area 211 of the finish adjustment screen 205.

Even when the finishing adjustment button 263 on the finishing area designation screen (same configuration as the basic screen 201 in FIG. 6) is pressed, it is determined that the state transition condition C49 is satisfied as shown in FIG. Changes again to the finishing adjustment screen operation processing state S45.
Also in this case, the finish adjustment screen 205 (FIG. 12) is redisplayed on the display unit 38, but the image in the previous state is displayed in the display area 211 of the finish adjustment screen 205.

Here, when the user who has viewed the display area 211 of the finish adjustment screen 205 (FIG. 12) desires to make further corrections, the user clicks again on the part to be corrected using the mouse of the operation unit 37. do it.
In this case, as shown in FIG. 9, it is determined that the state transition condition C46 is satisfied, and the state of the client device 11 transits again to the finishing area designation screen operation processing state S46, and the finishing area designation screen (FIG. 6). The same configuration as that of the basic screen 201) is redisplayed.
The user presses the area calculation button 262 after repeating the above-described work on the finishing area designation screen and re-correcting it, that is, after correcting the trimap again. Then, as shown in FIG. 9, it is determined that the state transition condition C47 is satisfied, and the transition is made again to the partial region calculation processing state S47.
When the transition is made again to the partial region calculation processing state S47, the partial region calculation processing for the selected region after re-correction by the user is executed again. That is, the medium resolution α image and the medium resolution foreground F image are reconstructed. Thereby, it is determined again that the state transition condition C48 is satisfied, and the state of the client device 11 transits again to the finish adjustment screen operation processing state S45.
When the transition is made again to the finishing adjustment screen operation processing state S45, the finishing adjustment screen 205 (FIG. 12) is displayed again on the display unit 38. In this case, the display area 211 of the finish adjustment screen 205 displays the result of partial area reprocessing.
By repeatedly executing such a series of operations, the user can correct the selected area and the like as many times as desired until a desired result is obtained.

When the user determines that the desired result has been obtained by repeating such correction operations, the user presses the confirmation screen button 372 while the finish adjustment screen 205 (FIG. 12) is displayed on the display unit 38. By doing so, the finish confirmation screen display instruction is given.
When the confirmation screen button 372 is pressed in this manner, as shown in FIG. 9, it is determined that the state transition condition C49 is satisfied, and the state of the client device 11 is returned to the finishing confirmation screen operation processing state S44. Transition. As a result, the finish confirmation screen 204 (FIG. 11) is displayed again on the display unit 38.

  Even if the confirmation screen button 264 on the finishing area designation screen (the same configuration as the basic screen 201 in FIG. 6) is pressed, it is determined that the state transition condition C50 is satisfied as shown in FIG. This state again transitions to the finishing confirmation screen operation processing state S44, and the finishing confirmation screen 204 (FIG. 11) is displayed again on the display unit 38.

When a desired correction result is obtained, the user may press the save button 362 on the finish confirmation screen 204 (FIG. 11) to save the result of the finish adjustment process.
Then, as shown in FIG. 9, it is determined that the state transition condition C51 is satisfied, and the state of the client device 11 transitions to the result storage processing state S48.

In the result storage processing state S48, when the output size is equal to the medium resolution size, the finish adjustment unit 112 generates a file including at least the data of the medium resolution α image and the medium resolution foreground F image, and the storage unit 39. Are stored in the image storage unit 132.
Such a file is hereinafter referred to as a “finishing adjustment result file”.

However, since the output size is generally larger than the medium resolution size, the finish adjustment unit 112 generates a finish adjustment result file by executing the following series of processes, and The image is stored in the image storage unit 132.
That is, the finish adjustment unit 112 converts the size of each data of the medium resolution α image and the medium resolution foreground F image into an output size. The medium resolution α image converted to the output size is hereinafter referred to as “output size α image”. The medium resolution foreground F image converted to the output size is hereinafter referred to as “output size foreground F image”.
The finish adjustment unit 112 converts the data size of the processing target image P (input image P) into an output size.
For each pixel of the output size, the finish adjustment unit 112 outputs the foreground pixel value F [x at the pixel position x, y where the mixing ratio α [x, y]> t (t is a predetermined threshold, 1 or less and close to 1). , y, c] is updated to the value of the pixel value P [x, y, c] at the corresponding pixel position x, y of the processing target image P.
The finishing adjustment unit 112 generates a finishing adjustment result file including at least each data of the output size α image and the updated output size foreground F image, and stores the result in the image storage unit 132 of the storage unit 39.

  When the finish adjustment result file is stored in the storage unit 39 in this way, it is determined that the state transition condition C52 is satisfied, and the finish adjustment process in FIG. 9 ends.

In the present embodiment, the client device 11 or the server 12 in FIG. 1 (hereinafter referred to as the server 12 for convenience of description) can subsequently start a program (not shown) and execute the following processing. . That is, the server 12 accepts user selection of a new background to be printed and a special effect filter to be applied to the new background or foreground, and presents the screen image of the final print output to the user for user confirmation. If the user approves, the server 12 makes a payment and prints a composite image in which a new background is combined with the cut out foreground on a paper medium or the like.
Specifically, each pixel value constituting the new background data is substituted into the above equation (2) as the background pixel value B ′ [x, y, c]. In addition, each pixel value of the output size α image is substituted into Equation (2) as a mixture ratio α [x, y]. Each pixel value of the output size foreground F image is substituted into equation (2) as the foreground pixel value F [x, y, c]. Then, Expression (2) is calculated, and the resultant composite image P ′ is printed on a paper medium.

As described above, the essence of the mat processing is to solve the equation (1), but the solution speed of the equation is naturally governed by the number of unknowns.
In the present embodiment, since processing at a medium resolution size (hereinafter referred to as “intermediate resolution processing”) can be performed, the number of intermediate label area mixture ratios α and foreground pixel values F that are unknown can be greatly reduced. As a result, the processing speed is greatly improved.
Here, in the simple intermediate resolution process, the resolution of the foreground pixel value F is inferior to the resolution of the processing target image (an original image such as a captured image, that is, an input image), so that output quality such as printing is deteriorated.
Therefore, in the clipping process according to the present embodiment, the pixel value of the medium-resolution foreground F image having no color mixture is used in the boundary region by the α value determination, and the high-resolution processing target image is used in the inner region (the foreground region). Pixel values of (original images such as captured images, that is, input images) are used. As a result, it is possible to minimize a decrease in output quality while realizing high-speed processing.
Furthermore, since the processing target image (original image such as a captured image, that is, an input image) is usually imaged with the main part of the main subject in focus, the edge part of the main subject may be out of focus. . In such a case, the result of the clipping process of the present embodiment gives a result that is completely inferior to the high resolution process.
Further, even if the matte processing result of the boundary region is processed at a high resolution due to the noise of the processing target image (original image such as a captured image, that is, the input image) or the essential similarity of the foreground / background, etc. The resolution may not be accurate. In such a case, the result of the clipping process of the present embodiment is a result that is completely inferior to the result of the high resolution process.

The details of the clipping process executed by the image processing system having the functional configuration of FIG. 4 have been described above with reference to FIGS.
Next, with reference to FIG. 13, a description will be given of the relationship between the processing of the client device 11 and the server 12 when the clipping process is executed in the image processing system having the functional configuration of FIG.

FIG. 13 is a flowchart for explaining the relationship between processes when the client device 11 and the server 12 cooperate to execute the clipping process.
A flowchart showing the process flow on the client device 11 side is shown on the left side of FIG. 13, and a flowchart showing the process flow on the server 12 side is shown on the right side of FIG.
Of the client device 11 side and the server 12 side, an arrow drawn from one to the other means that information is transmitted in that direction.

  In step S <b> 201, the communication control unit 101 of the client device 11 in FIG. 4 executes login processing for the server 12 based on the operation of the operation unit 37 by the user.

In step S <b> 221, the communication control unit 151 of the server 12 accepts login processing by the client device 11.
In step S <b> 222, the communication control unit 151 of the server 12 reads the clipping processing program from the program storage unit 161 and causes the communication unit 59 to transmit it to the client device 11.

In step S <b> 202, the communication control unit 101 of the client device 11 causes the communication unit 40 to receive a clipping processing program transmitted from the server 12 via the network 21.
In step S203, the communication control unit 101 of the client device 11 temporarily stores the cutout processing program in the program storage unit 131, and then expands and executes it in the RAM 33 (FIG. 2) or the like. As a result, the cutout processing unit 102 functions in the CPU 31 of the client device 11.

In step S204, the clipping processing unit 102 of the client device 11 determines whether or not a processing target image has been selected.
That is, the user can select a desired image such as a captured image as a processing target image by operating the operation unit 37.
When the selection operation by the user is not performed, it is determined as NO in Step S204, and the process returns to Step S204 again. That is, until the selection operation by the user is performed, the determination process in step S204 is repeatedly executed, and the process on the client device 11 side is in a standby state.
Thereafter, when a selection operation is performed by the user, YES is determined in step S204, and the process proceeds to step S205.
In step S <b> 205, the communication control unit 101 of the client device 11 reads the processed image data from the image storage unit 132 and causes the communication unit 40 to transmit the data to the server 12.

  In step S <b> 223, the communication control unit 151 of the server 12 causes the communication unit 59 to receive the processing target image data transmitted from the client device 11 via the network 21, and stores it in the image storage unit 162.

  On the other hand, in step S <b> 206, the clipping processing unit 102 of the client device 11 performs the above-described clipping process on the processed image data.

  FIG. 14 is a flowchart for explaining the flow of the clipping process in step S206 of the client side process.

  In step S241, the cutout designation unit 111 of the cutout processing unit 102 executes cutout designation processing. The details of the clipping designation process have already been described with reference to FIGS. 5 to 8, and the description thereof is omitted here.

In step S242, the cutout processing unit 102 determines whether or not the finishing adjustment process is to be executed.
In other words, as described above, it is sufficiently possible to extract the foreground from the processing target image only by the result of the clipping designation process, and therefore the finish adjustment process is not an essential process, and the user can freely select whether or not to execute it. It is possible processing.

Therefore, when the user operates the operation unit 37 to instruct execution of the finish adjustment process, it is determined as YES in Step S242, and the process proceeds to Step S243. In step S243, the finish adjustment unit 112 of the cutout processing unit 102 performs a finish adjustment process. Details of the finish adjustment processing have already been described with reference to FIGS.
When the finish adjustment process ends, the process proceeds to step S244 described later.

  On the other hand, when the user operates the operation unit 37 to instruct that the finish adjustment process is not executed, it is determined NO in step S242, and the finish adjustment process in step S243 is not executed. Advances to step S244.

In step S244, the communication control unit 101 of the client device 11 causes the communication unit 40 to transmit the processing result data to the server 12 as cut-out image data.
For example, when the finish adjustment process is not executed (when it is determined NO in the process of step S242), the above-described clipping designation processing result file is transmitted to the server 12 as the data of the clipping image.
On the other hand, for example, when the finish adjustment process is executed (when it is determined as YES in the process of step S242 and the process of step S243 is executed), the finish adjustment result file is used as cut-out image data. It is transmitted to the server 12.
As described above, in the clipping designation process or the finish adjustment process, data can be saved even in the middle of the process, and the data saved in the middle of the process is saved when it is read again. Processing can be continued from the current state. Therefore, the data in the middle of such processing is also transmitted to the server 12 as cut-out image data as necessary.

  Returning to FIG. 13, in step S <b> 224, the communication control unit 151 of the server 12 causes the communication unit 59 to receive cutout image data transmitted from the client device 11 via the network 21, and temporarily stores the data in the image storage unit 162. save.

During this time, in step S207, the cutout processing unit 102 of the client device 11 determines whether or not there has been an instruction to end the process.
The method for instructing the end of the process is not particularly limited, but in the present embodiment, it is assumed that a method in which the user operates the operation unit 37 to instruct the end of the process is employed.
If such an end instruction operation has not been performed, it is determined as NO in step S207, the process returns to step S204, and the subsequent processes are executed. That is, until the end instruction operation is performed, the loop processing of steps S204 to S207 is repeatedly executed, and the clipping process is executed each time, and the data of the clipped image is transmitted to the server 12.
Thereafter, when an operation for instructing termination is performed, it is determined as YES in Step S207, and the process proceeds to Step S208.
In step S <b> 208, the communication control unit 101 of the client device 11 causes the communication unit 40 to transmit a notification indicating the end of processing (hereinafter referred to as “end notification”) to the server 12.
Thereby, the processing on the client device 11 side ends.

Meanwhile, in step S225, the communication control unit 151 of the server 12 determines whether an end notification has been received.
If the end notification has not been received, it is determined as NO in step S225, the process returns to step S223, and the subsequent processes are repeated. That is, as described above, since the end notification is not transmitted while the loop processing of steps S204 to S207 is repeatedly executed on the client device 11 side, the loop processing of steps S223 to S225 is repeatedly executed and processed from the client device 11. Each time data of an image or a cutout image is transmitted, it is received and saved.
Thereafter, when an end notification is received, it is determined as YES in the next step S225, and the process proceeds to step S226.
In step S226, the main control unit 152 of the server 12 stores the clipped image data temporarily stored so far.
Thereby, the processing on the server 12 side ends.

Thereafter, although not shown, the server 12 accepts user selection of a new background to be printed and a special effect filter to be applied to the new background or foreground, and presents the screen image of the final print output to the user to confirm the user. Ask for.
If the user approves, the server 12 makes a payment and prints a composite image in which a new background is combined with the cut out foreground on a paper medium or the like.

In the above, one embodiment of the image processing system capable of executing the clipping process has been described.
Here, the features (effects) that the image processing system has for performing the clipping process will be described with reference to FIGS. 15 to 19 as appropriate.

  The image processing system of the present embodiment has a first feature that a user who is not familiar with the operation only needs to perform a simple interactive operation because the clipping process can be executed.

That is, as described above, the clipping process is divided into a clipping designation process and a finishing adjustment process. The finishing adjustment process is a process for performing fine finishing adjustment and is not an essential process. That is, the foreground can be cut out from the processing target image and combined with the new background only by the result of the cut designation process (cutout designation process result file).
As described above, the user's GUI operation required when executing this clipping designation process is an operation that is very easy and hassle-free for the user, and can be performed without being confused even by a user who is not used to the operation. .
Hereinafter, a specific example of the user's GUI operation during the clipping designation process will be described with reference to FIGS. 15 to 17.

FIG. 15 shows a state of the GUI operation of the user on the cutout area designation screen displayed in the cutout area designation screen operation processing state S12 (FIG. 5).
As shown in FIG. 15, the user can designate a boundary area 501 between the foreground and the background from the processing target image displayed on the cutout area designation screen 202.
Here, the boundary region 501 does not need to be strictly specified by the true boundary line (thin line) between the foreground and the background, and may be specified by a rough thick line as shown in FIG. In short, the user can designate the boundary region 501 with a line having a certain thickness so as to include the true boundary line between the foreground and the background and the background.
Of course, the user may use a thin line for a portion that is likely to fail or a portion that has a special intention and is desired to be specified in detail.
In other words, the boundary area 501 between the foreground and the background is designated by the boundary pen 253 in FIG. 7, and the thickness of the coating of the boundary pen 253 can be freely variably set by the pen size 255 in FIG. it can.

The user depresses the area calculation button 312 after the operation of specifying the boundary area 501.
Thereby, it changes to area | region calculation process state S13 (FIG. 5), and area | region calculation is performed.
When the area calculation is completed, the process proceeds to a cutout correction screen operation processing state S14 (FIG. 5), and a cutout correction screen 203 as shown in FIG. 16 is displayed.

FIG. 16 shows the initial state of the cutout correction screen displayed in the cutout correction screen operation processing state S14 (FIG. 5).
As shown in FIG. 16, a boundary area 502 is displayed based on the result of area calculation (binary label). However, the region 503 is a part of the turtle fin, and has become the background (outside the boundary region 502) where it should originally be the foreground (inside the boundary region 502). This is because the foreground color and the background color are similar in the processing target image.
In such a case, the user newly designates the area 503 as the selected area by designating the area 503 that should be the foreground to be the background with the eraser 254 to be erased. Thereafter, the user presses the area calculation button 312.
As a result, the region calculation processing state S13 (FIG. 5) is entered again, and the region calculation is recalculated.
When the recalculation of the area calculation is completed, the screen shifts again to the clipping correction screen operation processing state S14 (FIG. 5), and a clipping correction screen 203 as shown in FIG. 17 is displayed.

FIG. 17 shows an execution result of the GUI operation on the cutout correction screen displayed in the cutout correction screen operation processing state S14 (FIG. 5).
As shown in FIG. 17, a boundary region 502 is displayed based on the result of region operation recalculation (binary label after reconstruction). It can be seen that the region 503, which is a part of the sea turtle fin, is now included in the boundary region 502, that is, in the foreground.
In the state of FIG. 17, it seems that the desired result for the user is obtained. At this point, the user presses the save button 314.
Then, the process proceeds to the boundary area finishing process state S15 (FIG. 5), and subsequently transitions to the result storage process state S16 (FIG. 5).
Thereafter, the server 12 (FIG. 1) or the like can cut out at least the sea turtle image in the boundary region 502 of FIG. It is possible to print the synthesized image.
Note that the user may cause the client device 11 to perform finish adjustment processing so that the desired selection area is obtained.

  Since the image processing system of the present embodiment can execute the clipping process, in addition to the first feature described above, the width of the boundary region (corresponding to the boundary of the binary label) on the clipping correction screen 203 is appropriate. It has the 2nd feature that it is.

FIG. 18 shows the initial state of the cutout correction screen 203 displayed in the cutout correction screen operation processing state S14 (FIG. 5).
As shown in FIG. 18, the boundary area 510 is displayed based on the result of the area calculation. However, the result of the area calculation here is a binary label obtained in the area calculation processing state S13 (FIG. 5), and the boundary area 510 originally has no width, but the boundary area 510 has a certain width. ing.
That is, the boundary area 510 is obtained by giving a width to the binary label boundary of the binary optimization result obtained by the area calculation, and is created by, for example, expansion processing (dilation in morphology processing) having a predetermined width.
Here, the boundary area 510 can be created as a set sum area of the first area and the second area. That is, the binary foreground label area is expanded by a predetermined width, and the expansion area (not including the original binary foreground label area) obtained as a result is the first area. On the other hand, the binary background label area is expanded by a predetermined width, and the resulting expanded area (not including the original binary background label area) is the second area. A boundary area 510 is created as a set sum area of the first area and the second area. In the present embodiment, an intermediate label region for mat processing is created with a width obtained by expanding the boundary region 510 as it is or once (from one pixel to several pixels).

Adopting such a boundary area 510 also has an effect of displaying the shape of the binary boundary in an easy-to-see manner, but also has an effect that the width of the boundary area 510 indicates the required accuracy of the user interaction operation.
In other words, it is sufficient for the boundary region 510 to have a foreground inside and a background to the outside, and a binary label accurate to the pixel unit on the boundary line is necessary (because mat processing is performed later). It is possible to produce an effect that the user is intuitively clearly indicated that there is not. In other words, if there is no such clarification, there is a possibility that a user unfamiliar with the operation may try to perform a fine interactive operation, so that such a fear can be eliminated.
In addition, binary optimization using the 1st floor MRF model may cause fine irregularities due to noise at the boundary due to the limitations of the model, but it is also possible to conceal it so that it does not matter. become.
The meaning that it is sufficient if this accuracy is satisfied is because the intermediate label area for mat processing is created with a width obtained by expanding the boundary area 510 as it is or once (from one pixel to several pixels). It means that.
Thereby, for example, the width for determining the intermediate label area of the mat processing can be set as a variable setting value as a software configuration. Specifically, for example, it can be set with a numerical value or a slider on a software setting panel. In this case, the width of the intermediate label area of the mat processing and the thickness of the boundary area 510 can be changed in conjunction with the single set value.

Since the image processing system according to the present embodiment can perform the clipping process, in addition to the first and second features described above, the image processing system further performs a mat process at a medium resolution (for example, about 30,000 pixels), It has a third feature that it is combined with a high-resolution original image (120,000 pixels or more) at the time of output such as printing.
FIG. 19 shows a specific example for explaining a method of performing mat processing at medium resolution and combining it with a high-resolution original image at the time of output such as display or printing.
In FIG. 19, an image 521 shows a medium resolution foreground F image obtained as a result of the finish adjustment process. An image 522 shows a high-resolution original image (an output size processing target image).
In this case, for each high resolution pixel (output size pixel), when the mixture ratio α is an intermediate value, the foreground pixel value of the medium resolution foreground F image that is the medium resolution result is upsampled and adopted. When the mixing ratio α is approximately 1, the pixel value of the high-resolution original image is embedded. As a result, a high-resolution composite image 531 is obtained.
Here, as the mixture ratio α, a pixel value of an output size α image obtained by converting a medium resolution α image obtained as a result of the finish adjustment process into a high resolution (output size) is used.
More specifically, for each high-resolution (output size) pixel, a pixel position x, y with a mixing ratio α [x, y]> t (t is a predetermined threshold, a value less than 1 and close to 1) Foreground pixel value F [x, y, c] is updated to the value of pixel value P [x, y, c] at the corresponding pixel position x, y of the high-resolution original image (processing target image P) .

Here, the essence of the mat processing is to solve the equation (1) described above, but the solution speed of the equation is naturally governed by the number of unknowns.
In the present embodiment, since the intermediate resolution processing can be performed by the third feature, the mixing ratio α of the intermediate label area and the number of foreground pixel values F that are unknown can be greatly reduced, and as a result, the processing speed can be greatly increased. Will improve.
Here, in simple intermediate resolution processing, the resolution of the foreground pixel value F is inferior to the resolution of the high-resolution original image (the processing target image of the output size), so that output quality such as printing is degraded.
Therefore, in the clipping processing according to the present embodiment, the pixel value of the medium-resolution foreground F image having no color mixture is used in the boundary region based on the α value determination, and the high-resolution original image (region in the foreground) is used in the inner region (the foreground region). The pixel value of the output size processing target image) is used. As a result, it is possible to minimize a decrease in output quality while realizing high-speed processing.
Furthermore, since the high-resolution original image (the processing target image of the output size) is usually imaged with the main part of the main subject in focus, the edge part of the main subject may be out of focus. In such a case, the result of the clipping process of the present embodiment gives a result that is completely inferior to the high resolution process.
In addition, due to the noise of the high-resolution original image (the processing target image of the output size) and the essential similarity of the foreground and background, even if the matte processing result of the boundary region is processed at high resolution, May not be accurate. In such a case, the result of the clipping process of the present embodiment having the sixth feature is a result that is completely inferior to the result of the high resolution process.

  It should be noted that the present invention is not limited to the above-described embodiment, but includes modifications and improvements as long as the object of the present invention can be achieved.

For example, the finish adjustment unit 112 (FIG. 4) determines the sharpness for each block set in step S104 of the area calculation process of FIG. It is desirable to ignore it by the above.
For example, when the finish adjustment unit 112 performs determination using a bandpass filter or dispersion and determines that there are many high resolution components, the mat processing in step S106 is performed for the block as follows. Also good. In other words, the finish adjustment unit 112 may perform a matting process in which a block of a processing target image having a corresponding output size and a block of a trimap are input, and α and F blocks having an output size are output.
In this case, a medium resolution size block and an output size block coexist. Therefore, the finish adjustment unit 112 first resizes the medium resolution size block to the output size as the processing of step 108, and then performs mosaic with the output size. At this time, α only needs to be resized, but regarding F, the pixel position x, y in the block where α [x, y]> t within the block is described above with reference to FIG. In accordance with the method according to the sixth feature, it is necessary to replace the output size with the pixel value of the processing target image.
In this case, the confirmation in the finishing confirmation screen operation processing state S44 (FIG. 9) and the correction in the finishing adjustment screen operation processing state S45 and the finishing area designation screen operation processing S46 can also be performed with the output size. Therefore, after that, the processing according to the method according to the sixth feature described above with reference to FIG. 19 is not necessary, and if the user's consent is obtained, printing or the like can be output as it is.
The effect of adopting such a series of processing is as follows. In other words, even in the case of an input in which the subject boundary is in focus and the boundary has a resolution (in the case of the difference in the above embodiment), only a necessary area can be processed with a high resolution (output size). As a result, it is possible to calculate a result equivalent to the result of processing at high resolution at high speed.

Further, for example, the finish adjustment unit 112 (FIG. 4) performs the medium resolution processing as it is for the region calculation processing of FIG. 10, and immediately follows the method according to the sixth feature described above with reference to FIG. Processing (upsampling processing) is performed. Thereafter, the finishing adjustment unit 112 outputs the medium resolution size for the confirmation in the finishing confirmation screen operation processing state S44 (FIG. 9) and the correction in the finishing adjustment screen operation processing state S45 and the finishing area designation screen operation processing S46. Replace with size. Thereafter, when the user confirms, output such as printing is performed.
The effect of adopting such a series of processing is as follows. That is, high resolution processing (output size processing) is performed by the user's explicit selection, and high resolution processing of only a necessary and sufficient area is possible.

Further, for example, in the above-described embodiment, the clipping process is performed by the cooperation of the client device 11 and the server 12 (FIG. 4), but may be performed by either the client device 11 or the server 12. Good.
Here, if the system is defined to represent the entire apparatus composed of a plurality of apparatuses and processing units, and it is assumed that one apparatus is within a range that can be accommodated in one enclosure, the present invention The applied image processing system may be composed of a plurality of devices or a single device. In this case, if it is configured by a plurality of devices, it is sufficient that the entire functions of the cutout processing unit 102 in FIG. 4 can be exhibited by the plurality of devices as a whole. One function of the cutout processing unit 102 in FIG. Any device of any number of devices may be included.

  For example, in the above-described embodiments, the information processing apparatus to which the present invention is applied has been described as an example configured as a personal computer or the like. However, the present invention is not particularly limited to this, and can be applied to general electronic devices having an image processing function. For example, the present invention can be widely applied to video cameras, portable navigation devices, portable game machines, and the like. is there.

  The series of processes described above can be executed by hardware or can be executed by software.

  When a series of processing is executed by software, a program constituting the software is installed on a computer or the like from a network or a recording medium. The computer may be a computer incorporated in dedicated hardware. The computer may be a computer capable of executing various functions by installing various programs, for example, a general-purpose personal computer.

  The recording medium including such a program is not only constituted by the removable media 42 (FIG. 2) and 61 (FIG. 3) distributed separately from the apparatus main body in order to provide the program to the user, but also in the apparatus main body. It is composed of a recording medium or the like provided to the user in a preinstalled state. The removable medium is composed of, for example, a magnetic disk (including a floppy disk), an optical disk, a magneto-optical disk, or the like. The optical disk is configured by, for example, a CD-ROM (Compact Disk-Read Only Memory), a DVD (Digital Versatile Disk), or the like. The magneto-optical disk is configured by an MD (Mini-Disk) or the like. The recording medium provided to the user in a state of being incorporated in advance in the apparatus main body includes, for example, the ROM 32 (FIG. 2), 52 (FIG. 3) and the storage units 39 (FIG. 2), 58 (in which programs are recorded). The hard disk included in FIG. 3).

  In the present specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in time series along the order, but is not necessarily performed in time series, either in parallel or individually. The process to be executed is also included.

DESCRIPTION OF SYMBOLS 11 ... Client device, 12 ... Server, 21 ... Network, 101 ... Communication control part, 102 ... Switching processing part, 111 ... Clipping designation | designated part, 112 ... Finish adjustment part 121... Region calculation processing unit

Claims (6)

First conversion means for converting the data size of the processing target image from the first size to the second size;
Output means for outputting a mixture ratio and foreground pixel values in units of pixels for the data of the processing target image converted to the second size by the first conversion means;
Of the pixels which are output by said output means, said the area of the larger pixel than mixing ratio threshold, on the basis of the image pixel value of the first size of the processing target image, the region of the pixel below the mixing ratio threshold A generating means for generating output image data based on the foreground pixel values;
An image processing apparatus comprising: A second conversion means for converting the output result of the output means and the size of the data of the processing target image into a third size;
It said generating means, said converted into the third size by the second conversion means, based on the data of the output results and the processing target image of the output unit, the area of the pixel the mixing ratio is greater than the threshold value, The output image data is generated by adopting the pixel value of the third size processing target image and adopting the third size foreground pixel value for the pixel region whose mixing ratio is equal to or less than the threshold value. ,
The image processing apparatus according to claim 1. Tri-map means for generating a tri-map having a foreground, background, or intermediate ternary label in pixel units for the data of the processing target image of the second size,
The output means outputs the data mixture ratio and foreground pixel value of the processing target image in units of pixels using the trimap of the second size.
The image processing apparatus according to claim 1. Further comprising a correcting means for correcting the trimap,
The output means further outputs a mixing ratio and foreground pixel value of the data of the processing target image in units of pixels using the trimap corrected by the correction means.
The image processing apparatus according to claim 3 . In an image processing method in which an image processing apparatus performs image processing on data of a processing target image,
A conversion step of converting the data size of the processing target image from the first size to the second size;
An output step of outputting a mixture ratio and a foreground pixel value in units of pixels with respect to the data of the processing target image converted into the second size by the processing of the conversion step;
Among the pixels output by the processing of the outputting step, the the area of the larger pixel than mixing ratio threshold, the first based on the image pixel value of the size of the processing target image, pixel below the mixing ratio threshold For the region , a generation step of generating output image data based on the foreground pixel values;
An image processing method including: A computer for controlling the image processing apparatus ;
Conversion means for converting the data size of the processing target image from the first size to the second size;
Output means for outputting a mixture ratio and foreground pixel value in units of pixels for the data of the processing target image converted to the second size by the conversion means;
Among the pixels output by the output means, the the area of the larger pixel than mixing ratio threshold, on the basis of the image pixel value of the first size of the processing target image, the region of the pixel below the mixing ratio threshold Generating means for generating output image data based on the foreground pixel values;
Program to function as.