JP2016054524A - Program and display device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a presentation capable of drawing an interest of viewer's with respect to an image that is reproduced and displayed, and making the viewer to feel interesting and not to feel boring.SOLUTION: A presentation is provided in which a subject is moved or deformed on an image to match a music by making a computer to perform: an image analysis procedure for analyzing an image photographed by an imaging apparatus; a region extraction procedure for extracting one or a plurality of subject regions out of the image on the basis of the analysis result of the image analysis procedure; an image generation procedure for generating a plurality of new images by moving or deforming the subject regions on the image; and an image reproduction procedure for sequentially reproducing and displaying the plurality of new images generated in the image generation procedure while reproducing a preselected music.SELECTED DRAWING: Figure 5

Description

  The present invention relates to a program and a display device.

  As a reproduction effect device that produces effects on captured images and presents them to viewers, when switching images, effects are produced by special effects such as fade effect, wipe effect, and overlap effect on images before and after switching. Is known.

Japanese Patent Laying-Open No. 2005-167689

  Compared to simply switching and presenting a plurality of images, the above-described conventional reproduction effect device can achieve a certain level of visual effect, but it is not enough to attract viewers' interest. For example, in a photo frame or the like installed on a wall where people gather, there is a demand for an effect that attracts the viewer's interest to the reproduced and displayed image, makes it feel interesting, and does not get bored.

  The program according to the first aspect of the present invention includes an extraction procedure for extracting a specific area or subject from an image, and a blank area generated by movement or deformation of the specific area or subject within an interpolable range. The image generation procedure for generating an image obtained by moving or deforming the specific area or subject and interpolating the blank area using an image of the area around the blank area, and the image generated by the image generation procedure Let the computer execute the display procedure to display.

  According to the program of the present invention, it is possible to produce an effect that attracts the viewer's interest, makes it feel interesting, and does not get bored with respect to the reproduced and displayed image.

The figure which shows the structure of one embodiment The figure explaining the reproduction production method at the time of switching and displaying an image The figure explaining the reproduction production method at the time of switching and displaying an image Flow chart showing operation of reproduction effect method 1 The figure explaining the reproduction production | presentation method which moves and deform | transforms the to-be-photographed object in an image The figure explaining the reproduction reproduction method of the hiding accompanying the movement or deformation of the subject in the image The figure which shows the example which moves the subject in a picture according to music, and transforms The figure which shows the example which moves the subject in a picture according to music, and transforms Flow chart showing operation of reproduction effect method 4

  FIG. 1 is a block diagram illustrating a configuration of a reproduction effect device 1 according to an embodiment. The storage device 2 includes a slot into which various recording media 2a can be inserted. The storage device 2 stores image data and music data read from the recording medium 2a, while recording image data and music data that have been rendered. Write to 2a. It should be noted that the image to be produced by the reproduction effect device 1 includes a frame image in an image or a moving image taken by a digital still camera, a frame image in a video taken by a video camera, or the like. A microphone (hereinafter referred to as a microphone) 3 collects sounds around the reproduction effect device 1, that is, human voice, car sound, music sound, and the like. The camera 4 picks up images of people who view the image and music reproduced and displayed around the reproduction effect device 1, particularly in front of the reproduction effect device 1.

  The operation unit 5 includes operation members for the user or viewer to perform various operations on the reproduction effect device 1. The computer 6 includes hardware such as a CPU and a memory (not shown), a control unit 6a configured in software form, an analysis unit 6b, a processing unit 6c, and the like, performs various effects on the image, and displays the image after the effect processing on the LCD. Are reproduced and displayed on the display 7, and the selected or identified music (details will be described later) is reproduced from the speaker 8. The control unit 6a controls a series of effect processes. The analysis unit 6b analyzes the image and music to be produced, and the processing unit 6c performs various types of production processing on the image based on the analysis result.

  A reproduction effect method according to an embodiment by the above-described reproduction effect device 1 will be described.

<< Playing production method 1 >>
First, referring to FIG. 2 and FIG. 3, a description will be given of a reproduction effect method when a display image is switched from an image being displayed to another image, that is, when a scene change is performed. 2A and 3A show images before switching, and FIGS. 2B and 3B show images after switching. First, each feature is detected from the images before and after switching, and a region where the features are similar is extracted. Image features include brightness, color components, edges, textures, etc. If the difference in image feature values is less than or equal to each feature threshold in the image regions before and after switching, these regions are similar. It is determined that

  In the example shown in FIG. 2, the blue sky region A of the image (a) and the blue sky region B of the image (b) are similar to each other with a difference in image feature amount being equal to or less than a threshold value. Next, in the pre-switching image (a) displayed on the display 7, trimming is performed so that the cutout area becomes smaller as indicated by the broken line frame in (a), and the image is reproduced and displayed on the display 7. A pseudo zoom-in is performed on a blue sky region A in the original image (a). In FIG. 2A, the cutout area indicated by the broken line frame is reduced in stages, but in actuality, the cutout area indicated by the broken line frame is reduced smoothly.

  Here, the pseudo zoom-in means that a part of the captured original image (a) is cut out and displayed without performing camera work for zooming in by changing the focal length of the zoom lens during imaging by the imaging device. 7 is a method of zooming up the area A of a part of the subject area (here, the blue sky) by sequentially reducing the cutout area while zooming in on the display. When a part of the original image (a) is cut out and displayed on the display screen of the display 7 by trimming, the number of pixels decreases as the cut-out area decreases, and this is displayed on the display 7 as it is. The displayed image of the cutout area becomes small. Therefore, resolution conversion (number of pixels conversion) is performed on the image data of the cutout area in accordance with the pixel resolution of the display screen of the display 7, and the image of the cutout area is always reproduced and displayed on the display 7 with a constant pixel resolution. . Thereby, pseudo zoom-in is realized.

  After pseudo zooming in to the blue sky area A in the image (a) before cutting out, the image in the blue sky area A is switched to the image cut out by trimming the blue sky area B in the image (b) after switching. To display on the display 7. After that, in the image (b) after the switching, trimming is performed so that the cutout area is sequentially enlarged as shown by the broken line frame in (b), and the result is displayed on the display 7 and finally the entire range of the original image (b). Zoom out pseudo. In FIG. 2 (b), the cutout area indicated by the broken line frame is shown in a stepwise manner, but the cutout area indicated by the broken line frame is actually enlarged smoothly.

  Here, the pseudo zoom-out means that a part of the captured original image (b) is cut out without performing the camera work of zooming out by changing the focal length of the zoom lens during imaging by the imaging device. This is a technique of zooming down to a wide range of subject areas by sequentially increasing the cutout area while enlarging and displaying on the display 7. Note that when a part of the original image (b) is cut out by trimming and displayed on the display screen of the display 7, the number of pixels increases as the cutout area increases, and the image of the displayed cutout area becomes finer. Therefore, resolution conversion (pixel number conversion) is performed on the image of the cutout region in accordance with the pixel resolution of the display screen of the display 7, and the image of the cutout region is always displayed on the display 7 with a constant pixel resolution. Thereby, pseudo zoom-out (zoom-down) is realized.

  On the other hand, in the example shown in FIG. 3, the difference in image feature amount between the black eye area A of the image (a) before switching and the night sky area B of the image (b) after switching is less than the threshold value. It was determined that they were similar to each other. In addition, although the fireworks are rising in the area B of the night sky of the image (b), the determination of the result of comparing the image feature amount of the dark night sky when there is no fireworks and the image feature amount of the black eye of the image (a) It is. Next, in the image (a) before switching displayed on the display 7, the trimming area is trimmed so as to be sequentially reduced and displayed on the display 7, and finally the black eye area in the original image (a) is displayed. Zoom in on A pseudo. In FIG. 3A, only the solid line frame indicating the final cutout area before the image switching is shown, and the stepped cutout area frame as shown in FIG. 2A is not shown. Yes.

  After performing pseudo zoom-in to the black-eye area A in the image (a) before cropping, the image of the black-eye area A is converted into an image cut out by trimming the area B in the night sky of the image (b) after switching. Switch to display on the display 7. Thereafter, in the image (b) after the switching, trimming is performed so that the cut-out area becomes sequentially larger and the result is displayed on the display 7, and finally the image is zoomed out to the entire range of the original image (b). In FIG. 3B, only a solid line frame indicating a cutout region immediately after image switching is shown, and a frame indicating a stepped cutout region as shown in FIG. 2B is omitted. Further, as described above, the image data in each size cut-out region is converted in the number of pixels when displayed on the display 7, and is also shown in the examples of FIGS. 3 (a) and 3 (b). As described above, the final size (number of pixels) of the pre-switching image and the initial size (number of pixels) of the post-switching image need not necessarily be the same.

  FIG. 4 is a flowchart showing a control operation performed by the control unit 6a in the reproduction effect method 1 described above. In step S001, the control unit 6a causes the analysis unit 6b to analyze the first image previously displayed on the display 7, and extracts the feature amount for each region that can be regarded as having the same feature amount. In subsequent step S002, the control unit 6a stores, for the image other than the first image stored in the storage device 2, the feature amount for each region (for example, main color, edge amount, edge direction, etc.) similar to that in step S001. Perform extraction. In step S002 and step S003, which will be described later, in order to select the second image to be reproduced and displayed on the display 7 after the first image, the same subject as the first image, such as a continuous shot image, is similar to the first image. It is preferable that an image shot with such a composition is not a target of processing in steps S002 and S003. Judgment whether or not the same subject as the first image was shot with a composition similar to the first image is, for example, as SIFT (Scale Invariant Feature Transform) feature amount proposed by David Lowe in 1999 as feature information Etc. can be used. Since this SIFT feature quantity is local information that is invariant to changes in rotation and scale, robust object recognition is possible even for complex subjects. By comparing the two images based on this feature amount, it is possible to determine whether the same subject is photographed with the two images and whether the composition is similar.

  In step S003, an image (second image) having an area (second area) most similar to the feature quantity extracted in step S001 among the feature quantities extracted in step S002, and information about the position in the screen To get. In step S004 following step S003, the control unit 6a causes the display 7 to display the full screen of the first image. In step S005, the cutout area of the image displayed on the display 7 is gradually reduced toward the first area in the first image that has a similar feature amount to the second area of the second image extracted in step S003. The generated images are generated and sequentially displayed on the display 7 to perform pseudo zoom-in. In a succeeding step S006, the display screen of the display 7 is switched to the second area of the second image. The screen aspect ratio of the first area and the second area is preferably set to be the same as the aspect ratio of the display 7. In the subsequent step S007, the control unit 6a sets a cutout region that includes the second region and gradually increases in size on the second image, and then generates display image data by the above-described pixel number conversion. This is sequentially displayed on the display 7 to perform pseudo zoom-out.

  In the scene change reproduction effect example 1 shown in FIG. 2 and FIG. 3, the blue sky area, the black eye area, and the night sky area have been described as examples of areas having similar image feature amounts in the images before and after switching. However, instead of using an image of a region having a similar feature amount as a switching image, an image of a common subject region in the images before and after switching may be used as the switching image. For example, building windows and doors may be cut out from images before and after switching, and these cut-out images may be used as images before and after switching. To cut out areas such as windows and doors of buildings from the image, extract the edge components in the image to detect the outline of the building, and extract an edge closed curve of a predetermined size inside the building. That's fine.

  In a scene change that artificially zooms in to the window area in the image before switching and zooms out from the window area in the image after switching, zooms in to the window area in the image before switching After gradually increasing the brightness of the image in the process, the image of the window area that was finally zoomed in is “exposed” (saturates the brightness of the image data to the maximum level), and then in the image after switching As an effect of displaying the image of the window area in a “out-of-white” state and gradually changing the brightness of the image to an appropriate brightness in the process of pseudo zooming out from the window area in the switched image Good. Or, instead of “blowing out”, in the process of zooming in on the tunnel or cave area of the image before switching, finally “blackout” (reducing the brightness of the image data to the minimum level) And display the tunnel or cave area in the image after switching in “blackout”, then gradually increase the brightness of the image in the process of zooming out from the tunnel or cave area in the image after switching It may be configured to increase.

  In this way, in a scene change in which one image of a plurality of images is displayed and then switched to the next image, regions with similar feature amounts or regions of the same type of subject are extracted from the images before and after switching. After performing pseudo zoom-in to the image extraction area before switching, and then performing pseudo zoom-out from the image extraction area after switching, viewing is performed when switching and playing back multiple images. It attracts the people's interests, makes them feel interesting, and gives them a stunning performance.

  In the playback effect method 1 described above, the effect process when switching between two different images has been described. However, two regions having similar feature amounts are extracted from one image, and the one region is simulated. It is also possible to produce an effect such that a pseudo zoom-out is performed from the other area. As a result, even when one image is reproduced and displayed, it is possible to produce an effect that attracts the viewer's interest, makes it feel interesting, and does not get bored.

<< Playing production method 2 >>
Next, a reproduction effect method for moving or transforming a part of the subject in the image will be described. FIG. 5 is a diagram illustrating an effect example in which a road sign shown in the image is moved and deformed. An image to be reproduced and displayed is analyzed to identify one or more subjects based on features such as color components, edges, and texture. Further, it is determined whether or not these subjects can be moved and deformed. That is, when the subject is moved and deformed on the image, a blank area is generated after the movement and deformation. However, it is necessary to interpolate the blank area using image data around the subject area. If interpolation of a blank area accompanying movement or deformation is possible using image data adjacent to the object to be moved or deformed, the object can be moved or deformed.

  In FIG. 5 (a), attention is paid to the traffic sign 200 shown in the image as an object to be moved and deformed. The feature point of the image is detected by extracting the intersection of the edges from the image, and the subject and its area are identified. Feature points of black circles (●) and crosses (x) are detected from the image shown in FIG. 5A, and the traffic sign 200 is identified as a subject. In the image area adjacent to the traffic sign 200, there is an image area of a tree leaf on the upper background, and an image area of a child and a tree trunk on the lower background. When the traffic sign 200 is moved or deformed on the image, the blank area generated by the movement or deformation must be interpolated using the image data of the leaves of the background tree or the image data of the child and the tree trunk and repaired. .

  Here, the area adjacent to the subject area to be moved or deformed is (1) an area in which the low frequency component of the image in the area is small, the high frequency component is large, and the directionality is random (such as a leaf) (Region of the image), or (2) the region of the image of the region having a small number of high frequency components, a large amount of low frequency components, and a random direction (for example, a region of an image such as a cloudy sky), or ( 3) If the high frequency component and low frequency component of the image in the region are either low regions (for example, an image region such as a blue sky), the subject is moved or deformed using the image in such a region. Later blank areas can be repaired by interpolation. In addition, even if there is an image area in which a blank area can be interpolated around the subject area to be moved or deformed, “blurring” is performed to make the boundary between the blank area and the surrounding interpolable area inconspicuous. In order to include "," the area of the surrounding interpolable area must be at least sufficiently larger than the blank area. Moreover, it is preferable that the blank area is included in the surrounding interpolable area.

  In the example shown in FIG. 5A, there is an image area of a leaf in the vicinity (background) of the upper part of the traffic sign 200, and this image area of the leaf moves or deforms (in this case, reduced) in the upper part of the traffic sign 200. ), Which is sufficiently larger than the blank area at that time and includes most of the blank area. Further, as described in the above (1), the leaf image is an image in which the blank area can be easily interpolated. Therefore, it is possible to move or deform the upper part of the traffic sign 200 (the area surrounded by the black circle feature points). On the other hand, there is an image area of a child and a tree trunk in the lower part (background) of the traffic sign 200, and this child and tree trunk image area is moved or deformed (reduced) in the lower part of the traffic sign 200. The blank area is not sufficiently wide than the blank area, and is complicatedly intertwined around the blank area, and the image of the child and the trunk of the tree is not a uniform image as described above. It is not an image that can be easily interpolated. Therefore, it is difficult to move or deform the lower part of the traffic sign 200 (the area surrounded by the cross feature points). Only the upper part of the traffic sign 200 shown in FIG. 5 (a) can move as shown in FIG. 5 (b), and can be deformed or reduced as shown in FIG. 5 (c). Note that the lower part of the traffic sign 200 can be enlarged and deformed. Here, ideally, it is desirable that all of the border lines of the blank area are in contact with the image area of the leaf of the tree, but for example, as shown in FIG. It may not be in contact with the image area. In general, it is considered that it is sufficient if the length is 60% or more of the closed curve representing the boundary and is in contact with an interpolable region (in the above example, an image region of a leaf).

  The movement or deformation process of the subject on the image can be realized by a known method such as a polygon method. The image captured by the camera is analyzed to extract one or more subject areas from the image, and the subject area is moved or deformed little by little on the image to newly generate a plurality of images. Then, the generated plurality of images are sequentially reproduced and displayed on the display 7, and a moving image is presented as if the subject is moving or deforming dynamically over time. At this time, as the amount of movement or deformation of the subject is reduced and more images are newly generated until the predetermined amount of movement or deformation is reached, the image of the subject when the images are sequentially reproduced and displayed is increased. The movement becomes smooth.

  In this way, in the effect of moving or deforming the subject in the image, the image adjacent to the subject is an image that can interpolate a blank area due to the movement and deformation of the subject, and the area sufficiently includes the blank area. If it is a wide area, the subject can be moved or deformed. In other words, when an area of an image that can interpolate a blank area due to movement or deformation of the subject is adjacent to the blank area and the blank area is within the range of the adjacent image area, the subject can be easily moved or Deformation is possible. By extracting the subject that meets the above-mentioned conditions that can be moved or deformed from the image to be reproduced and displayed, and applying the effect of moving or deforming the subject, the viewer's interest is attracted and displayed when the image is reproduced and displayed. You can make them feel and get bored. Therefore, a subject whose amount of movement or deformation of the subject within the above-described range is less than a predetermined amount is not targeted for movement or deformation. In addition, an image in which there is no subject whose amount of movement or deformation of the subject is greater than or equal to a predetermined amount can be set as a reproduction effect target.

  Next, an effect method for hiding a subject moved in an image behind another subject will be described. In general, a subject farther from the camera is reflected closer to the vanishing point of the image. For example, in an image in which the ground is shown, the farther from the camera in the field, the closer the boundary between the subject and the ground is closer to the vanishing point on the imaging screen. A vanishing point is, for example, a point that intersects when an edge is detected from a captured image, a plurality of straight lines are extracted by performing a Hough transform on the detected edge, and the plurality of straight lines are extended based on the plurality of extracted straight lines. The position on the captured image can be calculated by extracting. Therefore, by analyzing the captured image and extracting a plurality of subject areas, the relative position of the plurality of subjects in the direction of the optical axis of the imaging lens of the imaging device that captured the captured image is determined from the position on the lower end image of each subject area. Can be obtained.

  In FIG. 6A, the boundary X between the traffic sign 300 and the ground is on the lower side of the image than the boundary Y between the tree 201 and the ground. In this embodiment, the position in the depth direction in the object scene of each subject is obtained based on the position in the vertical direction on the image at the lower end of each subject area, and based on the position in the depth direction when moving the subject. Hide behind other subjects. When the traffic sign 300 is moved, the boundary of the traffic sign 300 with the ground is moved from the position X shown in FIG. 6A to the vanishing point side (X ′) from the position of the boundary Y between the tree 201 and the ground. It is moved and a part of the traffic sign 300 is hidden behind the tree 201 as shown in FIG. In this way, based on the position of the subject in the depth direction, an effect (occlusion) that hides behind the other subject when the subject is moved or deformed is provided. Along with the deformation, things that should be hidden behind other subjects can be seen, preventing the viewer from feeling uncomfortable.

  In the example of the reproduction, rendering and hiding of the subject in the image shown in FIG. 6, only the traffic sign 300 is moved, deformed and hidden. However, the subject that meets the above-described moving, deforming and hiding conditions is shown. Any object may be used as a target to be moved, deformed, and hidden, and there may be a plurality of target objects.

<< Playing production method 3 >>
A reproduction effect method for providing an effect to an image while reproducing a music piece when the image is reproduced and displayed will be described. The music data is read and stored in the storage device 2 from the recording medium 2a, and is reproduced by the speaker 8 when an image is reproduced and displayed on the display 7. Note that music data may be acquired from an external web, for example, via a communication device (not shown). The music data stores the genre of music such as standard jazz, ballad, and rock as metadata, and the rhythm and tempo of the music such as 3 and 4 beats. The music data (metadata) is analyzed by the computer 6. Such information can be acquired by analyzing by the part 6b. When these pieces of information are not stored as metadata, the rhythm and tempo can be grasped by, for example, bass drum sounds. The sound of the bass drum can be extracted by performing frequency analysis of music data (music data).

  The music to be reproduced together with the image may be manually selected by the user or the viewer, or may be automatically selected by the control unit 6a of the computer 6. As an automatic music selection method, there is a method of selecting music according to imaging date information (year / month / date / time information) recorded in association with image data. For example, if the image is reproduced together with an image picked up from mid-December, a music related to Christmas is selected. Alternatively, there is a method of selecting music according to imaging location information recorded in association with image data. For example, if you want to play music along with images taken in Okinawa, select Okinawan folk songs. By selecting music based on the date and time of image capture and the location of the image, the viewer's interest in the image can be further enhanced when viewing travel photos. Furthermore, the music may be selected according to the place where the image is reproduced. The reproduction place can be specified by the destination of the product (the reproduction effect device 1 or a reproduction effect program that causes the device to perform a reproduction effect operation). The user or the viewer may manually set the playback location using the operation unit 5. Furthermore, based on the analysis result of one or a plurality of images to be reproduced, for example, an event shown in the image such as “athletic meet” may be identified, and a music suitable for the event may be automatically selected. Alternatively, the music flowing around the image reproduction apparatus 1 is collected by the microphone 3, and the analysis unit 6b of the computer 6 analyzes the genre, rhythm, tempo, etc. to identify the music, and according to the analysis result The image may be rendered. If the above metadata is not stored in association with the music data and it is difficult to identify the music, the image is rendered according to the rhythm and tempo information obtained as a result of analyzing the music data. You may do it.

  The movement and deformation pattern of the subject in the image is generated according to the selected or identified music, and the rendered image is displayed on the display 7, and the selected music, the identified music, or the analysis result The music that matches the rhythm and tempo is played from the speaker 8.

  For example, in the image shown in FIG. 6A, the traffic signs 300 are copied to make a total of three, and the three traffic signs are moved and deformed in accordance with the music as shown in FIGS. 7 and 8 show only movements in accordance with the music of the three traffic signs, and illustration of other subjects shown in FIG. 6A is omitted. FIG. 7 (a) shows an example of the rhythm of a certain musical piece, and FIGS. 7 (b) to (f) show that three traffic signs move and deform according to the rhythm of FIG. 7 (a). The traffic signs on the book show how they “dancing” to the music. In line with the rhythm progression shown in Fig. 7 (a), in Fig. 7 (b), (c), (d), the three traffic signs change the direction of the traffic sign arrows one by one. In FIG. 7 (e), the direction of the arrow of the traffic sign is fixed and the direction of the traffic sign is bent and deformed, and in FIG. The position of the traffic sign (arrow display part) is changed. In FIG. 7, the rhythm of (a) is repeated at the time of movement and deformation of (c) and (d) as in the case of movement and deformation of (b). Moreover, it is good also as a structure which changes the number of traffic signs according to the rhythm of a music. FIG. 8 shows an example of this. In accordance with the rhythm progression of the music shown in (a), the number of traffic signs is gradually increased as shown in (b), and all traffic signs are displayed with the last sound. Erasing from the screen.

  In addition, the subject in the image may be moved and deformed according to the melody of the music, or the subject may be moved and deformed according to the melody, rhythm and tempo of the music. Alternatively, the production may be changed according to the tone of the music. For example, for a slow ballad-like song, the red component of the sky color is strengthened from the daytime image and gradually changed to an image of the sunset sky, and then the entire image is darkened to make a night view. It is good to produce lighting in which the windows and street lights of houses in the recognized image are turned on sequentially. In addition, the presentation of the image may be changed according to the title or lyrics of the selected or identified music. In some cases, an image that matches the music is selected after the music is selected or identified, and the image matches the music. You may make it perform the production process. Such music-tone recognition may be configured to analyze and recognize music data itself, regardless of music metadata.

  The storage device 2 stores in advance various effect templates that prescribe the movement, deformation or hiding of the subject, and the processing unit 6c of the computer 6 selects from the various effect templates stored in the storage device 2. An effect template that matches the selected or identified music piece is selected, and effect processing is performed on the image. At this time, since the frame rate of the image display on the display 7 is constant, it is desirable to determine the playback speed of the effect image, that is, the number of effect images generated according to the rhythm of the music.

<< Playing production method 4 >>
A reproduction effect method will be described in which an animation that moves (changes) in time according to the type of subject recognized by image analysis is superimposed and superimposed on a region related to the subject region of the image. For example, when it is recognized by image analysis that a dish such as yudofu is reflected in the image, an effect process is performed to change the steam rising from the dish over time. In this case, the steam image is stored in advance in the storage device 2 as an effect template image. In addition, when a car is reflected in an image captured in the daytime, a lighting template is turned on or blinked when performing an effect process for changing this image into a night view, or an effect template is created in advance in an image showing a blue sky. The cloud image stored as an image is superimposed, and the cloud is moved over time, or the snow image stored in advance as a production template image is superimposed on the image snap-photographed at the street corner to make snow fall May be.

  FIG. 9 is a flowchart showing a control operation performed by the control unit 6a in the reproduction effect methods 2 to 4 described above. In step S100, the control unit 6a selects one frame image to be reproduced and displayed on the display 7 from the images stored in the storage device 2. In step S101, the control unit 6a analyzes the image reproduced and displayed on the display 7, identifies one or a plurality of subjects based on characteristics such as color components, edges, and textures, and can the subject be moved and deformed? Judge whether or not. In this determination, as described above, a determination may be made based on whether or not movement or deformation of a predetermined amount or more is possible. If it is determined in step S101 that the subject cannot be moved and deformed, an image that has not yet been selected is selected as an image to be reproduced and displayed on the display 7 in step S102. On the other hand, if it is determined in step S101 that the subject can be moved and deformed, in step S103, a subject region that can be moved and deformed, a possible amount of movement, and information on the amount of deformation that can be obtained are acquired. Furthermore, the separation information (distance information) on the screen between the boundary between the movable subject and the ground and the vanishing point is also acquired.

  In step S104, it is determined whether or not the metadata is stored in the music data that is stored in the storage device 2 and is to be reproduced together with the image. If the determination in step S104 is affirmative, the determination is made in step S106. Information such as rhythm (time signature) and tempo is acquired from the metadata. On the other hand, if a negative determination is made in step S104, information such as rhythm (time signature) and tempo is acquired by analyzing the music data in step S105. In subsequent step S107, the control unit 6a generates image data for moving and deforming the object that can be moved and deformed according to the selected music. At this time, when the subject is moved from the original position to a certain position, for example, between 4 beats, the frame rate of the display 7 is fixed. The number of image frames generated in the process is determined by the tempo of the music. The control unit 6a obtains information such as the tempo by analyzing the metadata of the music data or the music data, and based on this information, the movement of the subject, the deformation pattern, and the image of the transition process in each movement and deformation process are obtained. The number of generated images is determined and image generation is performed. In addition, when moving or deforming the subject, in order to determine whether or not the subject is hidden by another subject, in order to obtain depth information of subjects other than the subject to be moved or deformed in the screen, these subjects If the distance information between the boundary with the ground and the vanishing point is acquired, and the distance information acquired in step S103 is smaller than the distance information acquired in step S107, the movement or movement of the deformation target subject is performed. Then, it is determined that the subject to be moved or deformed is hidden by the deformation, and such an image is generated.

On the other hand, if the separation information acquired in step S103 is larger than the separation information acquired in step S107, it is determined that the movement or deformation target subject is not hidden by the movement or deformation of the subject. Then, such an image is generated. Similarly, in the case where it is recognized in step S107 that a specific subject such as yudofu or oden is captured, an animation image such as steam is superimposed on a region related to the subject region (for example, the upper part). Generate an image.
In step S108, the control unit 6a sequentially reproduces the selected music and the image generated in step S107. Note that the music selected in step S104 described above may be selected based on the shooting date / time, shooting location, shooting event, playback date / time, playback location, and the like of the image to be played back.

  In the above-described embodiment, the example of incorporating the reproduction effect program and the reproduction effect device of the present invention into the image reproduction device has been described. However, the reproduction effect program and the reproduction effect device of the present invention may be incorporated into the imaging device. Further, the reproduction effect program of the present invention may be executed on a general personal computer.

  In addition, in the reproduction | regeneration production | generation method of embodiment mentioned above and those modifications, every combination with the reproduction | regeneration production | generation methods or a reproduction | regeneration production | generation method and a modification is possible.

  According to the reproduction effect method of the embodiment described above and its modifications, the following operational effects can be obtained. First, an image captured by a camera is analyzed to extract first and second regions having similar image characteristics from the image, and a first cutout region including the first region is set from the image. Then, after gradually reducing the first cutout area to the first area, the image of the first cutout area is displayed on the display, and then the second cutout area including the second area is selected from the image. Since the second cutout area is gradually enlarged from the second area and the image of the second cutout area is displayed on the display 7, the viewer's image is reproduced and displayed. It is possible to create a production that attracts interest, makes you feel interesting, and does not get bored.

  In addition, the image captured by the camera is analyzed, one or more subject areas are extracted from the image based on the analysis result, and the subject areas are moved or deformed on the image to obtain a plurality of new images. An image is generated, and a plurality of new images are reproduced and displayed sequentially on the display while broadcasting a pre-selected song through a speaker, so that the subject moves or deforms on the image according to the song. This makes it possible to attract the viewer's interest in the image that is reproduced and displayed, to make it feel interesting, and not to get bored.

DESCRIPTION OF SYMBOLS 1; Reproduction production | presentation apparatus, 2; Memory | storage device, 6a; Control part, 6b; Analysis part, 6c; Processing part, 7; Display, 8;

Claims (13)

An extraction procedure to extract a specific area or subject from the image;
The specific area or the subject is moved or deformed in the image within a range in which the blank area caused by the movement or deformation of the specific area or the subject can be interpolated, and the blank area is changed to an area around the blank area. An image generation procedure for generating an interpolated image using the image;
A display procedure for displaying the image generated in the image generation procedure;
A program that runs a computer. The program according to claim 1,
In the extraction procedure, an intersection of edges is detected as a feature point from the image and a region surrounded by the feature point is extracted as the specific region, or a subject is detected based on a color component, edge, or texture. A program to extract as a subject. In the program according to claim 1 or 2,
In the image generation procedure, a plurality of images in which the movement amount or deformation amount of the specific area is changed, or a plurality of images in which the movement amount or deformation amount of the specific subject is changed, are generated.
In the display procedure, a program for sequentially displaying a plurality of images generated in the image generation procedure. In the program according to any one of claims 1 to 3,
In the image generation procedure, a program for moving or deforming the specific region or subject of an image in which the interpolable range is a predetermined range. In the program according to any one of claims 1 to 4,
In the extraction procedure, the position in the depth direction in the object scene of the subject is estimated based on the position of the lower end of the region of the subject on the image,
In the image generation procedure, when moving or deforming the specific area or the specific subject, an image in which the subject is hidden behind other subjects based on the position of the subject in the depth direction in the field of view. The program to generate. In the program according to any one of claims 1 to 5,
Let the computer execute the playback procedure to play music,
In the display procedure, a program for displaying the image generated in the image generation procedure in accordance with the music reproduced in the reproduction procedure. The program according to claim 6,
In the reproduction procedure, a program for selecting music corresponding to the date when the image was captured or music corresponding to the location where the image was captured. The program according to claim 6,
In the reproduction procedure, a program for selecting music corresponding to a date on which an image generated in the image generation procedure is displayed or music corresponding to a place where the image generated in the image generation procedure is displayed. The program according to claim 6,
In the reproduction procedure, a program for identifying an event shown in the image and selecting a music piece corresponding to the identified event. In the program according to any one of claims 6 to 9,
In the display procedure, a program for displaying an image generated in the image generation procedure while moving or deforming the specific area or subject in accordance with the music reproduced in the reproduction procedure. In the program according to any one of claims 7 to 9,
In the image generation procedure, a program for determining a generation number of images generated in the image generation procedure according to the selected music piece. In the program according to any one of claims 1 to 11,
In the display procedure, a subject identified in the image is identified, an animation corresponding to the identified subject is superimposed on the image, and the animation is displayed while being temporally changed. A storage unit for storing images;
A computer that executes the program according to any one of claims 1 to 12,
A display device comprising a display unit for displaying an image.

Images