JP2011139307A - Moving picture generation system, moving picture display method, and web server - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a moving picture generation system which creates moving pictures based on continuous still images for enjoyment, to provide a moving picture display method, and to provide a Web server to support it. <P>SOLUTION: There are created a plurality of time-series image data continuously photographed with a digital camera (50) by a user, or a cut out image data sequence (n pieces of continuous images) where background images are removed from the image data using an image processing technology and then objects are cut out or a composite image data sequence where the cut out image sequences are pasted onto the other background images. They are transmitted (uploaded) from a user PC (10) to a Web server (20) as still image data sequences. Then, the Web server (20) converts the received cut out image data sequences to moving pictures so that the user PC (10) or a cellular phone (40) can download the converted moving pictures through the Web server (20). <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a moving image generation system, a moving image display method, and a web server.

  Conventionally, there are various techniques for processing or processing an image captured using an electronic imaging device (digital camera). For example, a digital camera is used to capture an image with a subject in the background, and then a background image with no subject in the background is captured to generate difference information from the background image and the image with the subject. One of them is an invention related to an image processing technique in which only the image is cut out (for example, Patent Document 1).

  In addition, when a plurality of still image data is stored as a moving image in the storage unit of the imaging device in accordance with the moving image capturing operation, the plurality of stored still image data is transmitted to another electronic device. There is a technique for creating moving image data based on a plurality of still image data transmitted from an imaging device (Patent Document 2).

Japanese Patent Laid-Open No. 10-21408 JP 2003-250119 A

  With the technology disclosed in Patent Document 1 or Patent Document 2, a general moving image can be synthesized, but a moving image is cut out from an image photographed with a digital camera, and a moving image created by combining it with a background image is created. However, there is a problem in that it cannot be transferred to an external device as a continuous still image, and a moving image file cannot be generated by combining the continuous still image and a background image on the external device side. In other words, it is difficult to display a clipped image on the transmitting side as a moving image on the receiving side even though a plurality of still images can be played back continuously and played back like a flip book. It was.

  The main object of the present invention is to cut out a moving image from an image taken with a digital camera, and transfer a moving image created by combining it with a background image to an external device as a continuous still image. An object of the present invention is to provide a moving image generation system capable of generating a moving image file by combining a still image and a background image, a moving image display method, and a web server that supports the moving image display method.

In order to achieve the above object, a moving image generating system according to claim 1,
Imaging means capable of continuous shooting;
A cut-out image generating means for generating a cut-out image data string obtained by cutting out a subject from the continuous images picked up by the image pickup means;
Image combining means for generating a composite image data string obtained by combining the cut image generated by the cut image generating means with another background image;
An imaging device comprising:
Receiving means for receiving the cut-out image data sequence generated by the cut-out image generating means or the composite image data sequence generated by the image combining means;
Moving image conversion means capable of creating moving image data of a predetermined format based on the cut-out image data sequence or the composite image data sequence received by the receiving means;
Data storage means for storing moving picture data created by the moving picture conversion means;
A video transmission means for transmitting the video data created by the video conversion means;
An electronic device with
The cut-out image generation unit assigns the same group identification information to each image in one image data sequence, and assigns image order identification information indicating which image in the image data sequence is,
The moving image conversion means converts the image data having the same group identification information into data that can be reproduced as a moving image in accordance with the order of the image order identification information.

The invention according to claim 2 is the moving image generation system according to claim 1,
A computer capable of transmitting the cut-out image data sequence or the composite image data sequence received from the transmission unit of the imaging apparatus to the electronic device via the Internet;
The electronic device converts the moving image data into the moving image data by the moving image conversion unit based on the generated cutout image data sequence or the synthesized image data sequence transmitted from the computer, and converts the converted moving image data to the Internet. And transmitting to the computer.

The invention according to claim 3 is the moving image generation system according to claim 1 or 2,
The electronic device includes moving image posting means for transmitting the moving image data created by the moving image conversion means to a website published on the Internet.

The moving image display method according to claim 4 comprises:
A cut-out image generation step for generating a cut-out image data sequence in which a subject is cut out from a plurality of continuous images captured by an imaging device capable of continuous shooting;
An image compositing step for generating a composite image data sequence in which a plurality of cut-out images generated by the cut-out image generating step are combined with other background images;
Group identification information indicating that each image data in the cut-out image data sequence or the composite image data sequence belongs to the same image data sequence, and an image indicating the number of images in the image data sequence Providing order identification information;
The cut-out image data sequence generated by the cut-out image generation step to which the group identification information and the image order identification information are added or the composite image data sequence generated by the image combination step is sent to a web server via the Internet. An image data transmission step for transmission;
A moving image conversion step of creating moving image data in a predetermined format based on the cut-out image data sequence or the composite image data sequence transmitted in the image data transmission step, the group identification information, and the image order identification information;
A video transmission step of transmitting the video data created by the video conversion step via the Internet;
A video playback step for playing back a video on a monitor based on the video data transmitted by the video transmission step;
It is characterized by including.

The web server according to claim 5 is:
It consists of a plurality of continuous still images to which group identification information indicating that they belong to the same image data sequence and image order identification information indicating the number of images in the image data sequence are given. Receiving means for receiving continuous image data via the Internet;
Moving image conversion means for creating moving image data of a predetermined format based on the continuous image data received by the receiving means and the group identification information and image order identification information;
Data storage means for storing moving picture data created by the moving picture conversion means;
Address generating means for assigning a URL to moving image data in a predetermined format generated by the moving image converting means and creating a two-dimensional barcode from the URL;
Two-dimensional barcode transmission means for transmitting the two-dimensional barcode created by the address generation means to a device that has sent the continuous image data via the Internet;
Video transmission means for transmitting the video data designated by the URL corresponding to the two-dimensional barcode created by the address generation means via the Internet;
It is characterized by providing.

The invention according to claim 6 is the web server according to claim 5,
The data storage means includes
Storage means for assigning an identification code to each user and storing the moving image data created by the moving image conversion means based on the identification code;
Charge requesting means for performing an output requesting a usage fee to a user corresponding to the identification code when the storage capacity of the moving image data stored in the storage area for each identification code of the storage means exceeds a predetermined amount;
A payment confirmation means for confirming that the user has paid the usage fee;
A storage capacity increasing means for increasing the storage capacity of the storage area of the corresponding identification code when the payment confirmation means confirms the payment;
It is characterized by providing.

The invention according to claim 7 is the web server according to claim 5 or 6,
It is characterized by comprising moving image posting means for transmitting the moving image data created by the moving image conversion means to a website published on the Internet.

  According to the present invention, there is an effect that it is possible to realize a system, a moving image display method, and a web server supporting the moving image display method that can be created and enjoyed by another electronic device such as a mobile phone. is there.

1 is a system configuration diagram showing an overall configuration of a network system using a web server according to a first embodiment of the present invention. It is a block diagram which shows the structural example of the computer for management which manages a website. It is a system explanatory view showing one embodiment of a network service (network business) according to the present invention provided in a network system. It is a flowchart which shows the procedure of the main processes performed in a web server in order to provide a network service. It is explanatory drawing which shows the structural example of the main screen displayed on the monitor screen of user PC when starting the support software for receiving the network service installed in user PC. FIG. 6 is an explanatory diagram illustrating an example of a dialog displayed in a window when an image addition button displayed on the main screen of FIG. 5 is clicked. FIG. 6 is an explanatory diagram illustrating an example of a dialog displayed in a window when a moving image creation button displayed on the main screen of FIG. 5 is clicked. It is a block diagram which shows schematic structure of the imaging device which has a function which produces | generates a cutting image data sequence. FIG. 9 is an image explanatory diagram illustrating an example of images continuously captured by the imaging device of FIG. 8. FIG. 9 is an image explanatory diagram illustrating an example of a subject cut-out image data sequence generated from images continuously captured by an image processing function included in the imaging device of FIG. 8. It is explanatory drawing which shows an example of the image image | photographed with the imaging device of this embodiment, and the information added to this image data. It is explanatory drawing which shows the detail of the header added to the moving image pre-processing image data processed by the imaging device of this embodiment. It is a system block diagram which shows the whole network system structure using the web server which concerns on the 2nd Embodiment of this invention.

DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments of the invention will be described with reference to the drawings.
FIG. 1 shows the overall configuration of a network system using a web server according to the first embodiment of the present invention.

As shown in FIG. 1, the network system includes personal computers (hereinafter abbreviated as user PCs) 10a to 10d owned by a plurality of users, and server computers owned by service providers that provide services related to image processing ( (Hereinafter referred to as a web server) 20 is configured to be communicably connected via the Internet 30. In the network system of this embodiment, the mobile phone 40 owned by the user can also be connected to the web server 20 via the Internet 30.

  Digital cameras 50a to 50c owned by the user are connected to the user PCs 10a to 10c via communication means such as a USB cable, and images captured by the camera or images processed in the camera are stored in the user PCs 10a to 10c. The data can be taken in, processed as it is or in a PC, and transmitted to the web server 20 via the Internet 30.

  Further, a printer 60 capable of color printing is connected to the user PC 10a so that an image can be output and printed out. Although not shown, the same applies to the user PCs 10b and 10c. Further, the user PCs 10a to 10c may include an adapter or the like that can be loaded with a memory card built in the digital cameras 50a to 50c, obtain image data captured by the camera from the memory card, and store the image data in a hard disk or the like. It may be configured to be able to.

  The web server 20 is connected to a non-volatile storage device (storage) 21 such as a hard disk device that stores image data sent from the user PC via the Internet 30 as a file. The web server 20 is connected to a management computer (hereinafter referred to as a site management PC) 23 for managing the website by controlling the server via a LAN (local area network) 22. The web server 20 according to the present embodiment has a function of converting an image in a predetermined form sent from the user PCs 10a to 10c into a moving image of various formats, a function of publishing the converted image on a website, and a mobile phone. Management software (program) for realizing a function of transmitting an image converted into a format for a mobile phone to the mobile phone is incorporated.

  As shown in FIG. 2, the site management PC 23 is a CPU 201 that performs image data processing in accordance with a program, and an occasional write that temporarily stores image data to be processed or processed by the CPU or provides a work area for the CPU. A readable RAM 202, a read-only ROM 203 that stores programs executed by the CPU and fixed data, a hard disk 204 that stores image data and the like in a nonvolatile manner, and a CPU connected to the web server 20 and the Internet 30 via the LAN 23 For example, a LAN adapter 205 as a communication interface and a keyboard 206 as an input operation device for giving a command to the CPU 201 from the outside are connected to each other via a system bus 207.

  The CPU 201 reads out and executes a program stored and stored in the ROM 203 or the hard disk 204, and performs a moving image conversion process described later. In addition to the hard disk 204, the site management PC 23 may be provided with an external storage device such as a CD-ROM or DVD.

  Next, an embodiment of a network service (network business) according to the present invention provided in the network system will be described with reference to FIG. In the network service of the present embodiment, the image data is generated or transmitted according to the order of the numbers with the symbol # in FIG.

In the network service of the present embodiment, first, a user obtains a plurality of time-series image data continuously photographed with the digital camera 50, or a background image is removed from the image data by image processing technology, and the subject is cut out. A cut-out image data sequence (n consecutive images) or a composite image data sequence in which the cut-out image sequence is pasted to another background image is created. Then, it is transmitted (uploaded) as a still image data sequence to the web server 20 by the user PC 10.
Then, the web server 20 converts the received cut-out image data sequence into a moving image, and the user PC 10 or the mobile phone 40 can download the converted moving image from the web server 20. As a result, the user can enjoy moving images not only on the digital camera but also on the monitor of the personal computer or the screen of the mobile phone. In the present specification, an object that mainly plays a role such as a person for clipping is called a subject. The web server 20 publishes a service website on the network system in order to provide the above services to the user.

  In addition, the web server 20 has a function to support the user so that the user can easily post the video converted as described above to a website that can be viewed by an unspecified number of third parties in the network system. . This makes it easy for beginners unfamiliar with personal computers to easily post videos and enjoy their work. Further, the web server 20 stores image data (still images and moving images) owned by the service provider in the storage device 21, and provides the data that can be used (downloadable) by the user, that is, online material. Is selected as a background image, and a cut-out image data sequence received from the user PC 10 is combined and converted into a moving image.

  Here, in the web server 20, the process of converting the image data sequence composed of a plurality of continuous images into a moving image is performed by connecting n still images to a time sequence and generating a frame for generating a moving image. In addition to processing to convert to a JPEG video format that can be played back by a digital camera, processing to convert video data to a format that can be played back by a personal computer or a mobile phone is included. In addition to the image data sequence received from the user PC 10, the storage device 21 of the web server 20 can also store video converted data, image data obtained by the user on the website, and data synthesized and converted using the data. It can be constituted as follows.

In addition, the web server 20 assigns a URL (Uniform Resource Locator) to the moving image after the moving image is created so that the moving image data stored in the storage device 21 can be easily downloaded by the mobile phone. A two-dimensional bar code represented by a (Quick Response) code (trademark; the same applies hereinafter) is created and transmitted to the user PC 10 to have a function of displaying a QR code on the monitor of the user PC. As a result, the user can read the QR code with a mobile phone having a camera function, and download a moving image from a URL corresponding to the code.
In addition, if the URL is displayed together with the QR code, and the URL is sent to family members and acquaintances by e-mail, etc., the recipient can open an Internet browser on a personal computer and copy or enter the URL in the address field. Thus, the moving image created by the web server 20 can be viewed on the monitor of the personal computer. In other words, if the user transmits the QR code, the user can display the moving image on various terminals without transmitting the moving image data itself.

  Furthermore, by pasting the URL on the blog, a moving image can be automatically displayed on the blog. Here, the web server 20 is configured so that the image data can be uploaded to the website by the user PC using a browser installed in the user PC or dedicated software provided by the service provider.

  The web server 20 has a function of converting the image data sequence received from the user PC into a moving image. That is, it is configured such that the cut image data or the composite image data sequence created by the digital camera can be received via the Internet 30 and converted into a moving image. As a theme for a moving image, a moving image may be created using not only the cut-out image and the background image taken by the user but also the online material as a theme.

Further, the web server 20 includes a storage device 21 that stores the moving image created by the moving image conversion function. When the web server 20 stores the moving image in the storage device 21 of the web server 20, an ID (personal identification code) is assigned to each user. Then, a storage area (folder) is created for each user to store moving image data. At this time, the storage capacity may be rented for a fee, or the storage capacity may be increased for a fee for a user who wants to rent to a certain storage capacity free of charge and store data of a certain capacity or more. This makes it possible to build a business model that can increase profits.
In this case, the web server 20 includes a charge billing means (function) for performing an output requesting a usage fee from the user corresponding to the ID as the identification code, and a payment confirmation means for confirming that the usage fee has been paid by the user. (Function) and storage capacity increasing means (function) for increasing the storage capacity of the storage area of the corresponding identification code when the deposit confirmation means confirms the deposit. In addition, it is preferable that the output requesting the usage fee and the confirmation of payment can be performed without sending a manpower by, for example, sending an e-mail via the Internet.

  Furthermore, when a video is generated from the cut-out image data sequence received from the user PC 10 and the online material in a service using online material, a plurality of videos are arranged as a movie album, not just as a video. The web server 20 is provided with a function for storing or generating videos in various greeting card formats, videos used for mobile phone standby screens, pictograms used for mobile phone mail, etc. You may build the system which charges and provides the service by a function. In addition, a service is provided in which a printer of a printing provider (including a development shop or a convenience store) can be connected to the Internet 30 in FIG. 3, and an image data sequence is transmitted from the user PC 10 to a printer on the network and printed out as a seal. May be provided.

Here, a procedure of main processing performed in the web server 20 for providing the network service will be described with reference to a flowchart of FIG.
This process is started when there is a video conversion request from the user PC 10 on the website. The web server 20 first determines whether or not there is a moving image conversion request (step S1). If there is a request, the web server 20 determines whether or not image data has been transmitted (step S2). Here, if it is determined that there is transmission, image data reception processing is performed in step S3, and it is determined whether or not all n image data have been received in the next step (step S4). If “No” in steps S2 and S4, the process returns to step S2, and if it is determined in step S4 that all data has been received, it is determined whether or not an online material has been selected in the next step (step S5).

  If an online material is selected, the image data of the selected online material is read and combined with the image data received in step S3 (step S6). If it is determined in step S5 that no online material is selected, step S6 is skipped and the process proceeds to step S7. In step S7, the image data received in step S3 or the image data synthesized in step S6 is converted into a moving image. Thereafter, it is determined whether or not there is a moving image download request from the user PC (step S8). If there is a request, moving image data is transmitted (step S9). When there is no download request in step S8 or after the moving image data is transmitted in step S9 in response to the request, the moving image data is stored in the storage and a series of processing is terminated (step S10).

  Next, in the network service of this embodiment, a function of software suitable for supporting receiving a video conversion service by transmitting image data from the user PC 10 to the web server 20 will be described. The present inventors considered two methods as the support software. The first is a method that uses the browser function of the user PC 10, and is a simple method that is suitable for providing only a video conversion service. The second is a method that uses service support software that does not use the browser function, and not only the video conversion service but also the conversion of the video synthesized with the online material provided by the site as mentioned above and the created video are open to the public This is a method suitable for realizing advanced functions for providing a plurality of services such as a function for providing a place to perform. In either method, the program is installed in the user PC 10 as support software.

Hereinafter, the second method will be specifically described.
In the second method, when the program installed in the user PC 10 is started up, a main screen as shown in FIG. 5 is displayed on the monitor screen of the user PC 10. (* Please delete the description of DP Manager at the top of the screen in Figure 5)

  The main screen is provided with an image display area (preview area) A1 for displaying an image selected from the center to the upper right, and a folder display area A2 for displaying a list of folders is selected on the left side of the screen in order from the top. An information display area A3 for displaying detailed information on files and images, and a function button display area A4 for displaying a plurality of function buttons for selecting a service provided by the site are provided. Further, immediately below the image display area A1, a navigation button display area A5 for displaying a plurality of navigation buttons for inputting commands such as playback, stop, and fast-forward of the converted video is also provided at the lower side of the screen. Is provided with a thumbnail list display area A6 for reducing and displaying images in the selected folder. Further, a task bar A7 is displayed in the upper right corner of the screen.

  Two tabs (composite image and online material) are displayed at the upper end of the folder display area A2, and when the left tab is selected by clicking with the mouse, the photographed data uploaded by the user or the video data converted on the site is displayed. A folder in which the file is classified and stored by year and month is displayed, and when the tab on the right side is selected by clicking with the mouse, a folder storing the online material provided by the site is displayed. When a button provided in the menu selection field (filter) displayed in the thumbnail list display area A6 is clicked, “all display”, “continuous image”, “moving image file”, “image” storing still images are stored. A pull-down menu for displaying a menu such as “File” appears, and images in the folder are displayed in the thumbnail list display area A6 according to the menu selected from the menu.

  In the function button display area A4, an image addition button B1 for instructing to upload an image taken by the user from the user PC 10 to the site, and a movie creation button B2 for instructing creation of a movie from the selected “continuous image” An animation creation button B3 for instructing creation of an animation GIF from the selected “continuous image”, a mail transmission button B4 for instructing to send image data of the selected file by email, and image data of the selected file Camera transmission button B5 for instructing transmission to the camera, portable browsing button B6 for instructing conversion of image data of the selected file so that it can be viewed on a mobile phone, and continuous images of the selected file to the public site A posting button B7 for instructing posting is provided.

  Here, when the image addition button B1 is clicked, a dialog as shown in FIG. 6 is displayed on the monitor in a window that displays folders stored in the hard disk in the user PC. In this dialog, when the folder storing the image data (continuous image / moving image) to be added is designated with the mouse and the “OK” button is clicked, the selected image data is automatically uploaded. When the moving image creation button B2 is clicked, a moving image conversion setting dialog as shown in FIG. 7 is displayed in a window on the monitor. When an “OK” button is clicked in this dialog, a moving image is automatically created based on the selected image data (continuous image). In addition, this dialog has a "Detailed Settings" button. Clicking this button displays another dialog that allows you to change settings such as format, image size, playback speed, and repeat count. It is like that.

  When the animation creation button B3 is clicked, an animation is created from the selected “continuous image”. This animation creation is to convert it into a kind of animation format called animation GIF, but animation GIF is a format of a category different from a general animation format that is infinite in length in time series, and is a series corresponding to a flip book. It is a mechanism that packages still images and displays them sequentially. By converting to this format, it can be used for such purposes as attaching a moving character in a sentence attached to an HTML-based mail.

  Also, if you select an image you want to see on your mobile phone on the main screen and click the mobile browsing button B6, a screen for selecting the playback speed is displayed for continuous images. Select the speed and click the “OK” button. When is clicked, the selected image data (continuous image) is uploaded to the conversion site and automatically converted into a portable image. The converted data is stored in a folder for portable images, and the URL is The QR code is created based on the URL and is displayed on the monitor. By reading this QR code with the camera function of the mobile phone and downloading the image data, a moving image is displayed on the screen of the mobile phone.

  Next, an image data sequence transmitted from the user PC 10 to the web server 20 and a generation method thereof in the network service of the present embodiment will be described. In the present embodiment, the digital camera is provided with a function for generating a cut-out image data sequence in which a background image is removed by image processing technology from a plurality of time-series image data continuously photographed by the camera, and a subject is cut out. .

FIG. 8 shows a schematic configuration of the imaging device 50 having a function of generating a cut-out image data sequence.
The imaging apparatus 50 according to the present exemplary embodiment includes a lens unit 501, an electronic imaging unit 502, an imaging control unit 503, an image data generation unit 504, an image memory 505, a feature amount calculation unit 506, and a block matching unit 507. , An image processing unit 508, a recording control unit 509, a recording medium 510, a display control unit 520, a display unit 521 including a liquid crystal display device, an operation input including a shutter button 531 and a mode selection determination button 532 530 and a CPU (microprocessor) 540 as control means.

  The imaging apparatus 50 according to the present embodiment determines whether or not the subject S exists in the background based on a plurality of image frames f1 to fn (FIG. 9A) continuously captured by the electronic imaging unit 502. Based on this determination result, a background image P1 (see FIG. 9B) for subject area extraction is acquired. And based on the difference information of each pixel corresponding between this background image P1 and subject presence image P2, it has a function which extracts subject image S from subject presence image P2, and produces | generates a cutout image data sequence.

The lens unit 501 includes a plurality of lenses and includes a zoom lens, a focus lens, and the like. Although not shown, the lens unit 501 includes a zoom drive unit that moves the zoom lens in the optical axis direction and a focus drive unit that moves the focus lens in the optical axis direction when imaging a subject. Yes.
The electronic imaging unit 502 includes, for example, an image sensor such as a charge coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS), and converts an optical image that has passed through various lenses of the lens unit 1 into a two-dimensional image signal. To do.

  Although not shown, the imaging control unit 503 includes a timing generator, a driver, and the like. Then, the imaging control unit 503 scans and drives the electronic imaging unit 502 with a timing generator and a driver, converts the optical image into a two-dimensional image signal with the electronic imaging unit 502 every predetermined period, and the electronic imaging unit 502. The image frames for one screen are read out from the imaging area and output to the image data generation unit 504. The imaging control unit 503 performs adjustment control of imaging conditions such as AE (automatic exposure processing), AF (automatic focusing processing), and AWB (automatic white balance).

  The image data generation unit 504 appropriately adjusts the gain for each RGB color component with respect to the analog value signal of the image frame transferred from the electronic imaging unit 502, and then performs sample holding by a sample hold circuit (not shown). The digital signal is converted into digital data by a / D converter (not shown), color processing including pixel interpolation processing and γ correction processing is performed by a color process circuit (not shown), and then a digital luminance signal Y and color difference signal Cb , Cr (YUV data).

  The luminance signal Y and the color difference signals Cb and Cr output from the color process circuit are DMA-transferred to an image memory 505 used as a buffer memory via a DMA controller (not shown). The image memory 505 is constituted by, for example, a DRAM or the like, and temporarily stores data processed by the feature amount calculation unit 506, the block matching unit 507, the image processing unit 508, the CPU 540, and the like. The image memory 505 includes a ring buffer that can temporarily store 20 image frames, for example, and sequentially stores a plurality of image frames generated by the image data generation unit 504.

  The feature amount calculation unit 506 performs feature extraction processing for extracting feature points from the background image P1 with reference to the background image P1. Specifically, for example, the feature amount calculation unit 506 selects a predetermined number (or a predetermined number or more) of highly featured block regions (feature points) based on the YUV data of the background image P1, and then selects the block. Is extracted as a template (for example, a square of 16 × 16 pixels). Here, the feature extraction process is a process of selecting a feature having a high characteristic convenient for tracking from a large number of candidate blocks.

  The block matching unit 507 performs block matching processing for aligning the background image P1 and the subject presence image P2 when generating the subject cutout image. Specifically, the block matching unit 507 determines where the template extracted in the feature extraction process corresponds in the subject existing image P2, that is, the template pixel value optimally matches in the subject existing image P2. Search for a position (corresponding region). Then, an optimum offset between the background image P1 and the subject presence image P2 having the best evaluation value (for example, sum of squared differences (SSD), sum of absolute differences (SAD), etc.) of the pixel value is determined as the template. Calculate as a motion vector.

The image processing unit 508 includes a determination unit 581 that determines whether or not the subject S exists in the background based on a plurality of image frames circulated and stored in the image memory 505. Specifically, the determination unit 581 performs moving body detection by using a predetermined moving body analysis technique for each of the image frames whose time series moves back and forth based on a plurality of image frames circulated and stored in the image memory 505. When the moving object is not detected, the determination unit 581 determines that the subject S does not exist in the background. On the other hand, when the moving object is detected, the determination unit 581 determines that the subject S exists in the background. Further, the determination unit 581 determines whether or not the state has changed from a state in which the subject S is not present in the background to a state in which the subject S is present in accordance with a determination result of whether or not the subject S is present in the background.
In addition, the image processing unit 508 acquires, based on the determination result by the determination unit 581, the background image P <b> 1 in which no subject exists and the continuous image P <b> 2 in which the subject exists from the image frames stored in the image memory 505. Part 582.

  When the determination unit 581 determines that the subject S has changed from the non-existing state to the existing state, the acquisition control unit 582 captures the subject presence image captured by the electronic imaging unit 502 and stored in the image memory 505 after the change. Acquisition of the P2 image frame is started. For example, the acquisition control unit 582 ends the acquisition of the image frame of the subject existing image P2 at a predetermined acquisition end timing defined according to the memory capacity and the acquisition specified period. That is, after the determination unit 581 determines that the subject S has changed from the non-existing state to the existing state, the acquisition control unit 582 acquires the image frame of the subject existing image P2 for a predetermined period. A period during which the shutter button 531 is operated may be set as a shooting period, and an image frame during that period may be stored as a shot image.

  If the determination unit 581 determines that the subject S has changed from a non-existing state to an existing state, the acquisition control unit 582 is generated by the image data generation unit 504 and cyclically stored in the image memory 505 before the determination. One acquired image frame, for example, the image frame immediately before the determination is acquired as a background image P1 in which the subject S does not exist. Note that the image processing unit 508 may extract a portion where no subject exists from the continuous image P2 where the subject exists, and add them to generate a background image P1 where no subject exists. The image processing unit 508 includes a subject extraction unit 583 that extracts a subject area including the subject S from the subject presence image P2 acquired by the acquisition control unit 582.

  The subject extraction unit 583 includes the subject S from the subject presence image P2 based on the difference information of the corresponding pixels between the background image P1 and the subject presence image P2 in which no subject is acquired, acquired by the acquisition control unit 582. To extract a subject area to be detected. Specifically, the subject extraction unit 583 applies a low-pass filter to each of the YUV data of each of the image frames f1 to fn and the YUV data of the background image P1 in the subject presence image P2 acquired by the acquisition control unit 582. The high frequency component of each image is removed.

  Thereafter, the CPU 540 calculates a degree of difference for each pixel corresponding to each of the image frames f1 to fn and the background image P1 in the subject existing image P2 from which noise has been removed by applying a low-pass filter to the subject extraction unit 583. Thus, a difference degree map for each image frame f1 to fn is generated. In addition, the subject extraction unit 583 binarizes the dissimilarity map relating to each pixel with a predetermined threshold value, and then performs a contraction process in order to remove an area in which the difference is caused by fine noise or camera shake from the dissimilarity map. Further, the subject extraction unit 583 performs a labeling process, removes an area below a predetermined value or an area other than the maximum area, identifies the largest island pattern as the subject area, and corrects the shrinkage. Perform expansion treatment.

  Further, the image processing unit 508 specifies the position of the subject area extracted in the subject existing image P2, and generates position information (for example, an alpha map) indicating the position of the subject area in the subject existing image P2. A generation unit 584 is provided. The alpha map represents the weight when alpha blending the image of the subject area with respect to a predetermined background for each pixel of the subject existing image P2 as an alpha value (0 ≦ α ≦ 1).

Specifically, the position information generation unit 584 applies a low-pass filter to the binarized difference map in which the largest island portion is 1 and the other portions are 0, and the boundary portion is intermediate. Create an alpha value by producing a value. In this case, the alpha value of the subject area is 1, and the transparency of the subject existing image P2 with respect to a predetermined background is 0%. On the other hand, the alpha value of the background portion of the subject S is 0, and the transparency of the subject existing image P2 is 100%. Since the alpha value near the boundary is 0 <α <1, the subject presence image P2 and the background image P1 are mixed.
Based on the generated alpha map, the image processing unit 508 does not transmit a pixel having an alpha value of 1 among the pixels of the subject existing image P2 to a predetermined single color image, and the alpha value is An image synthesizing unit 585 for synthesizing the image of the subject S with a predetermined single color image to generate image data of the subject synthesized image so as to transmit 0 pixels is provided.

  Specifically, the image composition unit 585 creates an image obtained by cutting out a subject area from a single color image using the one's complement (1-α) in the alpha map, and uses the alpha map for the subject. A subject cut out image is generated by combining the subject S cut out from the existing image P2. Further, the image composition unit 585 generates cut-out images C1 to Cn (see FIG. 10) for each of the plurality of image frames f1 to fn of the subject existing image P2 acquired by the acquisition control unit 582 of the image processing unit 508. The generated n cut images C1 to Cn are recorded in a predetermined area of the recording medium 510 by the recording control unit 509 as a cut image data string. The recording medium 510 is configured by, for example, a nonvolatile memory (flash memory).

  Note that the image processing unit 508 may be provided with a JPEG compression unit, and may be stored in the recording medium 510 as an image data string of a subject cut-out image code-compressed by the compression unit. In this manner, by continuously reading out the cut image data sequence stored in the recording medium 510 and displaying it on the display unit 521, an apparent moving image can be reproduced. Further, each image data of the subject cutout image is stored as one file in which each image frame constituting the subject cutout image is associated with each alpha map generated by the position information generation unit 584 of the image processing unit 508. May be. Further, for example, moving image data of a subject cut-out image code-compressed in the Motion-JPEG format may be recorded, and a moving image can be reproduced by a camera.

  The display control unit 520 performs control to read display image data temporarily stored in the image memory 505 and display it on the display unit 521. Specifically, the display control unit 520 includes a VRAM, a VRAM controller, a digital video encoder, and the like. The digital video encoder periodically reads the luminance signal Y and the color difference signals Cb and Cr read from the image memory 505 and stored in the VRAM (not shown) under the control of the CPU 540 from the VRAM via the VRAM controller. Thus, a video signal is generated based on these data and output to the display unit 521. Based on the video signal supplied from the display control unit 520, the display unit 521 displays a processed image such as a live view image, a captured image, or a cutout image on a display screen. The CPU 540 performs various controls and data processing in accordance with instructions from the operation input unit 530 while executing an operation program (not shown).

  According to the imaging device 50 of the present embodiment having the above functions, when the user acquires the background image P1 and the subject presence image P2 for extracting the subject region, the imaging of the background image P1 and the subject presence image P2 are acquired. The background image P1 and the subject presence image P2 can be easily acquired by simply operating the shutter button once without taking the two steps of imaging, and the background image P1 and the subject presence image P2 are continuously used. The generated subject cut-out images C1 to Cn can be generated. A still image can also be generated from the subject presence image P2 acquired by the recording control unit 509.

  In the above embodiment, in addition to the function of generating n continuous subject cutout image data sequences in the camera, the function of generating moving image data of subject cutout images code-compressed in the Motion-JPEG (M-JPEG) format. However, even in such a case, when configuring the system for providing video conversion by the web service via the Internet according to the embodiment, it is necessary to extract n consecutive subjects instead of video data. It is desirable to convert the moving image by sending the image data string to a website. Web server has more complex compositing processing functions such as video data for PCs, video data for mobile phones, video data for photo frames, etc. This is because a continuous image data sequence is easier to convert than JPEG moving image data.

  Specifically, the M-JPEG video conversion function provided in the camera can only convert under certain conditions, and it cannot change the format itself or make fine adjustments such as the display speed or the number of repetitions. . On the other hand, it is desirable that the moving image conversion performed on the website can support a variety of moving image formats from a format viewed on a mobile phone to a format for a PC. Furthermore, it is desirable to be able to set options with the intention of the user who wants to convert the video format for PC.

  Therefore, in the present embodiment, an image data string composed of n still images before moving image conversion is sent from the user PC to the web server. The web server connects n still images to a time sequence to generate a frame for generating a moving image, and converts the completed frame image into various moving image formats. In addition, since the format for moving images for mobile phones differs depending on the model, the moving image conversion is performed in a moving image format suitable for each model such as 3GP2. By adding video conversion functions to the web server as described above, you can set conversion options that cannot be done with a camera, and generate and transfer videos that can be viewed on mobile phones with greatly different formats. There is an advantage of becoming.

  Here, the case where the digital camera is provided with an image data processing function for generating an image data sequence by extracting the subject as described above from the background image has been described. You may make it provide in 20 side. Accordingly, not only a digital camera user having a special function but also a digital camera user not having such a function can receive the moving image conversion service as described above.

  Next, predetermined image data (a plurality of continuous cut still image sequences or a combination of the cut still image sequence and a background still image; hereinafter referred to as a moving image preprocessed image) transferred by the digital camera is transferred to the web server 20, An example of a moving image creation procedure on the server side when creating a moving image on the web server 20 will be described with reference to FIGS. 11 and 12. In FIG. 11, images denoted by reference numerals # 1, # 5, # 6, and # 10 are single-shot images, and # 2 to # 4 and # 7 to # 9 are continuous images. In FIG. 11, three continuous images are shown for the sake of space, but it is desirable that the image is composed of about 8 to 30 continuous images.

  In the present embodiment, among the images photographed and processed by the digital camera, the moving image preprocessed image is added with a header as shown in FIG. 12 and stored in a storage medium in the digital camera. In FIG. 11, the header is shown in a simplified manner. Further, an image data of a normal still image taken by a camera has an extension “jpg”, whereas an extracted image or a composite image at the time of shooting has an extension “jpe”. Note that both images with the extension jpg and images with jpe are jpeg format data and are recorded in the same folder. However, as shown in FIG. 11, there is no “Group ID” and “ID” information in the header of the image with the extension “jpg”, and “Group ID” is in the header of the image with the extension “jpe”. "And" ID "information is added. Here, the “group ID” is identification information unique to a group indicating to which “cut-out image data sequence or composite image data sequence” the still image to which the ID is attached belongs.

  Therefore, a different ID is generated and given each time a “cutout image data sequence or composite image data sequence” is generated. On the other hand, “ID” is identification information indicating the number of images in the “cut image data sequence or composite image data sequence” specified by the group ID. By adding such information, it is possible to reliably distinguish and recognize a normal still image, a cut-out image and a synthesized image related to a moving image preprocessed image, and facilitate moving image conversion processing. In the present embodiment, a moving image in which a user converts a moving image into a desired format (mov or gif) by designating a desired image from stored image data with a group ID and transmitting it to a web server. It is intended to obtain data. At that time, it is preferable that the user does not directly specify the ID number, but can select it by a user interface such as a dialog screen displayed on the monitor.

Here, a supplementary explanation will be given regarding the composite display of the image data sequence. FIG. 11 is a simplified drawing for synthesizing three cutout moving images and one background still image. For example, in a certain group ID “3012C3”, cutout images are A1, A2, A3, A4, A5, A6. , A7, A8, A9, and A10. Further, it is assumed that the still image as the background is B1. In this case, the synthesized images are A1B1, A2B1, A3B1, A4B1, A5B1, A6B1, A7B1, A8B1, A9B1, and A10B1. If this is continuously reproduced, it is possible to obtain a moving image in which the background is a still image and only the cut image is moving.
Also, if you want to switch the background halfway, prepare another still image B2 and switch halfway. For example, A1B1, A2B1, A3B1, A4B1, A5B1, A6B2, A7B2, A8B2, A9B2, and A10B2.
In addition, if the background is a moving image consisting of B1, B2, B3, B4, B5, B6, B7, B8, B89, B10, A1B1, A2B2, A3B3, A4B4, A5B5, A6B6, A7B7, A8B8, A9B9, A10B10 If combined in this way, the cut-out image will appear to move in the background of the moving image.

It is not necessary to match the number of frames of the cutout image with the number of frames of the background video. When the number of frames of the background video is high, the frames may be dropped like A1B1, A2B1, A3B3, A4B3, A5B5, A6B5, A7B7, A8B7, A9B9, A10B9, or A1B1, A1B2, A1B3, A2B4, A2B5 , A2B6, A3B7, A3B8, A3B9, and A3B10, it is possible to display a clipped moving image in a smooth background moving image.
Cut-out images and background images may be online materials.
The combined display may be combined while being reproduced, or may be displayed after being created in advance as a moving image file. If combined display is performed during playback, processing takes time, but only one still image is stored, and at the time of playback, as in A1B1, A2B1, A3B1, A4B1, A5B1, A6B1, A7B1, A8B1, A9B1, and A10B1 Just synthesize.
When creating a moving image file in advance, it is necessary to store and store the 10 composite images, for example, but it can be transferred to another device as an M-JPEG or GIF file.

Next, a second embodiment of the network service (network business) according to the present invention will be described with reference to FIG.
The network service of this embodiment connects an image display terminal 70 called a photo frame to the Internet 30, and the web server 20 as described in the above embodiment via a website opened to the network by the web server 20. The moving image converted using the function of the image is transmitted to the image display terminal 70 so that the moving image can be displayed.

  Accordingly, in this embodiment, the image display terminal 70 has a communication function. As the communication function, in addition to the function of the image display terminal 70 itself connecting to the Internet 30 for communication, an Internet-applicable home appliance (for example, a TV or a set-top box) or a game machine (hereinafter referred to as an Internet device) 80. A function capable of communicating with a communication standard called Bluetooth or a communication standard called Wi-Fi of the IEEE 802.11 standard is provided in the image display terminal 70, and moving image data is transferred from the web server 20 via the Internet device 80. May be transmitted to the image display terminal 70 so that a moving image can be displayed.

Although the invention made by the present inventor has been specifically described based on the embodiment, the present invention is not limited to the embodiment. For example, in the above-described embodiment, the case where a continuous image (still image sequence) photographed by a digital camera is converted to a moving image by a web server has been described. However, images such as a plurality of illustrations created by a user on a personal computer are displayed. It can also be applied when converting to a video.
In the above embodiment, a case has been described in which a continuous image (still image sequence) captured by a digital camera is transmitted to a web server and converted to a moving image by the web server. However, a continuous image (still image sequence) is transmitted to the user PC. ) May be converted into a moving image so that a continuous image (still image sequence) can be converted into a moving image on the user PC.

10 User PC
20 Web server 30 Internet 40 Mobile phone 50 Digital camera (imaging device)
60 Printer 70 Image display terminal 80 Internet device 501 Lens unit 502 Electronic imaging unit (imaging means)
503 Imaging control unit 504 Image data generation unit 505 Image memory 506 Feature amount calculation unit 507 Block matching unit 508 Image processing unit 509 Recording control unit 510 Recording medium 520 Display control unit 521 Display unit 530 Operation input unit 540 CPU (microprocessor; control) means)

Claims (7)

Imaging means capable of continuous shooting;
A cut-out image generating means for generating a cut-out image data string obtained by cutting out a subject from the continuous images picked up by the image pickup means;
Image combining means for generating a composite image data string obtained by combining the cut image data string generated by the cut image generating means with another background image;
An imaging device comprising:
Receiving means for receiving the cut-out image data sequence generated by the cut-out image generating means or the composite image data sequence generated by the image combining means;
Moving image conversion means capable of creating moving image data of a predetermined format based on the cut-out image data sequence or the composite image data sequence received by the receiving means;
Data storage means for storing moving picture data created by the moving picture conversion means;
A video transmission means for transmitting the video data created by the video conversion means;
An electronic device with
The cut-out image generation unit assigns the same group identification information to each image in one image data sequence, and assigns image order identification information indicating which image in the image data sequence is,
The moving image generation system, wherein the moving image conversion means converts the image data having the same group identification information into data that can be reproduced as a moving image in accordance with the order of the image order identification information. A computer capable of transmitting the cut-out image data sequence or the composite image data sequence received from the transmission unit of the imaging apparatus to the electronic device via the Internet;
The electronic device converts the moving image data into the moving image data by the moving image conversion unit based on the generated cutout image data sequence or the synthesized image data sequence transmitted from the computer, and converts the converted moving image data to the Internet. The moving image generating system according to claim 1, wherein the moving image generating system transmits the image to the computer.   The moving image generating system according to claim 1, wherein the electronic device includes moving image posting means for transmitting the moving image data created by the moving image conversion means to a website published on the Internet. A cut-out image generation step for generating a cut-out image data sequence in which a subject is cut out from a plurality of continuous images captured by an imaging device capable of continuous shooting;
An image compositing step for generating a composite image data sequence in which a plurality of cut-out images generated by the cut-out image generating step are combined with other background images;
Group identification information indicating that each image data in the cut-out image data sequence or the composite image data sequence belongs to the same image data sequence, and an image indicating the number of images in the image data sequence Providing order identification information;
The cut-out image data sequence generated by the cut-out image generation step to which the group identification information and the image order identification information are added or the composite image data sequence generated by the image combination step is sent to a web server via the Internet. An image data transmission step for transmission;
A moving image conversion step of creating moving image data in a predetermined format based on the cut-out image data sequence or the composite image data sequence transmitted in the image data transmission step, the group identification information, and the image order identification information;
A video transmission step of transmitting the video data created by the video conversion step via the Internet;
A video playback step for playing back a video on a monitor based on the video data transmitted by the video transmission step;
A moving image display method comprising: It consists of a plurality of continuous still images to which group identification information indicating that they belong to the same image data sequence and image order identification information indicating the number of images in the image data sequence are given. Receiving means for receiving continuous image data via the Internet;
Moving image conversion means for creating moving image data of a predetermined format based on the continuous image data received by the receiving means and the group identification information and image order identification information;
Data storage means for storing moving picture data created by the moving picture conversion means;
Address generating means for assigning a URL to moving image data in a predetermined format generated by the moving image converting means and creating a two-dimensional barcode from the URL;
Two-dimensional barcode transmission means for transmitting the two-dimensional barcode created by the address generation means to a device that has sent the continuous image data via the Internet;
Video transmission means for transmitting the video data designated by the URL corresponding to the two-dimensional barcode created by the address generation means via the Internet;
A web server comprising: The data storage means includes
Storage means for assigning an identification code to each user and storing the moving image data created by the moving image conversion means based on the identification code;
Charge requesting means for performing an output requesting a usage fee to a user corresponding to the identification code when the storage capacity of the moving image data stored in the storage area for each identification code of the storage means exceeds a predetermined amount;
A payment confirmation means for confirming that the user has paid the usage fee;
A storage capacity increasing means for increasing the storage capacity of the storage area of the corresponding identification code when the payment confirmation means confirms the payment;
The web server according to claim 5, comprising:   The web server according to claim 5, further comprising a moving image posting unit that transmits the moving image data created by the moving image conversion unit to a website open to the Internet.