JP3523784B2 - Interactive image operation display apparatus and method, and program storage medium - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention displays a background image on a computer screen or the like and associates it with a position interactively designated by a pointing device such as a mouse or a touch screen. The present invention relates to an apparatus and a method for expressing the behavior of a series of partial images in a background image by combining and displaying the partial images, and a medium for storing a program for realizing the method.

[0002]

2. Description of the Related Art Conventionally, looking at an example in which some operation is performed on a computer for displaying an image, there has been one in which a corresponding frame image is displayed by operating a time axis with respect to a moving image. For example, in QuickTime player of Apple Inc., a frame image corresponding to each time is displayed in the display window by operating a slider (time axis cursor) corresponding to the time axis as shown in FIG. In this example, there is no operation related to the image space, and the moving image is expressed on the time axis basis. See also Apple
QuickTimeVR player of the same company has an interface for viewing a panoramic image with a large aspect ratio in a window, and as shown in FIG. The part that is displayed is displayed. In this case, the window is virtually moved with respect to the background image according to the relative movement direction of the mouse and the amount of displacement. In this example, there is no time axis information and there is no operation on the display image itself, and the field of view of the window is operated.

[0003]

From the viewpoint of interactive display of moving images, most of the above-mentioned prior arts simply operate the time axis based on the time axis as shown in the example of FIG. Also, in the example of FIG. 9 which is not a time axis standard operation, as a still image display operation, the displacement amount relative to the operation direction of the mouse cursor is used for the field of view operation of the window, and the position on the actual background image is directly displayed. It is not manipulating the associated information. As described above, as a conventional technique, there is no interface for interactively operating the display at the position of the background image.

In view of the above, the present invention takes the above-mentioned restrictions of the prior art into consideration, and when an object behaves in a series in a background image, the object image is interactively displayed at the place where the object occupies every moment on the background image. An object is to provide an interactive image operation display device and method and a program storage medium that enable display.

[0005]

The present invention solves the above problems by the following means.

(1) A device for synthesizing and displaying on a background image a series of partial images that change in time and are associated with the spatial position of the background image in the displayed background image space,
Background image storage means for storing the background image, display means for displaying the background image stored in the background image storage means, and partial image storage means for storing partial images associated with the spatial position of the background image. A setting means for setting an operation target space position on the screen of the display means, a specifying means for uniquely specifying a partial image to be composite-displayed from the operation target space position, and a partial image to be composite-displayed. An interactive image operation display device comprising: a composite display unit that compositely displays the read partial images at the operation target space position on the background image that is read from the partial image storage unit and displayed.

(2) In the means (1), an interactive image operation display device characterized in that a display integrated touch screen is used as the setting means.

(3) In the means (1), the interactive image operation display device is provided with means for creating a background image stored in the background image storage means from a moving image composed of continuous frame images. .

(4) The interactive image operation display device according to (1), further comprising means for extracting the partial image stored in the partial image storage means from the original image from which the background image was created.

(5) In the means (1), the specifying means defines a continuous mapping from the operation object space position to the time position for specifying the partial image, and continuously from the first operation object space position to the neighborhood. An interactive image operation display device characterized in that, when a position is designated, a partial image is specified by continuously tracing the mapping line.

(6) In the means (5), when the mapping from the operation target space position to the time position is multi-valued, the specifying means defines a one-to-one mapping for the mapping of temporally close intervals. An interactive image operation display device, characterized in that conversion is performed so that

(7) In the means (5), when the mapping from the operation target spatial position to the time position is multi-valued, the specifying means has a predetermined rule for mapping the sections that are not temporally close to each other. An interactive image operation display device, wherein one value is selected from multiple values according to the following.

(8) In the means (5), the specifying means sets a continuous operation target space position from the immediately previous operation target space position in the case where the mapping from the operation target space position to the time position is multivalued. If the operation target space position is mapped to the corresponding continuous time position and the operation target space position that is not continuous from the immediately previous operation target space position is set, one of the multivalues should be selected according to a predetermined rule. An interactive image operation display device characterized by:

(9) In the means (1), the composite display means displays the partial image together with the background image by enlarging or reducing the composite image by designating the enlarging or reducing mode, and displaying the interactive image operation display device. .

(10) In the means (1), when the spatial position to be operated is set, the composite display means sets a predetermined time or a predetermined time ahead of the set position by setting the arbitrary position reproduction mode and the reproduction direction. Partial images in succession,
Alternatively, an interactive image operation display device characterized by performing composite display at fixed time intervals.

(11) The interactive image operation display device according to the above-mentioned (1), wherein the composite display means erases the composite-displayed partial image after a preset predetermined time.

(12) In the means (1), the time positions at which the partial images to be combined are present are sequentially obtained from the operation target spatial position at predetermined time intervals, and the present time position and the time position immediately before are obtained. An interactive image operation display device comprising sound reproduction means for reading sound data corresponding to the intervals and reproducing the sound data within the predetermined time interval.

(13) In the means (12), the sound reproducing means compresses the sound data when the difference between the current time position and the time position obtained immediately before is longer than the predetermined time interval. An interactive image operation display device, characterized in that when the audio data is short, the audio data is expanded and reproduced so that the reproduction of the sound data is completed within the predetermined time interval.

(14) In the means (1), the setting means sets an operation target space position using a pointing device which moves a cursor displayed by the display means, and the display means sets the operation target space position. An interactive image operation display device, wherein when set, only the outline is displayed with the cursor being transparent.

(15) A method of synthesizing and displaying on the displayed background image space a series of partial images which are associated with the spatial position of the background image and change with time, the background image storing means. A step of reading out the background image accumulated in the image and displaying it on the image of the display means, a step of setting the operation target space position on the display screen of the background image, and a part to be synthesized and displayed from the operation target space position. An interactive image operation display method comprising: a step of uniquely specifying an image; and a step of reading out the partial image to be combined and displayed from the partial image storage means and combining and displaying the partial image in the operation target spatial position.

(16) An interactive image operation display method characterized in that the means (15) further comprises a step of creating a background image to be accumulated from a moving image consisting of continuous frame images.

(17) An interactive image operation display method characterized by further comprising the step of obtaining the partial image to be accumulated from the original image from which the background image was created in the means (15).

(18) In the means (15), in the specifying step, a continuous mapping from the operation target spatial position to the temporal position for specifying the partial image is defined, and the first operation target spatial position is continuously adjacent. An interactive image operation display method characterized in that, when the position is specified, the partial image is specified by continuously tracing the map line.

(19) In the means (18), when the mapping from the operation target spatial position to the time position is multi-valued, the mapping of the temporally adjacent sections is one-to-one mapping in the specifying step. An interactive image operation display method characterized by performing conversion such that

(20) In the means (18), when the mapping from the operation target spatial position to the time position is multivalued, the mapping of the sections which are not temporally close to each other is determined in the specifying step. An interactive image operation display method characterized by selecting one value from multiple values according to a rule.

(21) In the means (18), in the specifying step, when the mapping from the operation target space position to the time position is multivalued, the operation target position consecutive from the immediately previous operation target space position is set. If the operation target space position is set to a corresponding continuous time position, and if the operation target space position that is not continuous from the immediately previous operation target space position is set, it is necessary to select one value from the multiple values according to a predetermined rule. Characteristic interactive image operation display method.

(22) In the means (15), in the process of the composite display, the enlargement or reduction mode is designated,
An interactive image operation display method characterized by enlarging or reducing a partial image together with a background image for composite display.

(23) In the means (15), in the step of performing the composite display, when the spatial position of the operation target is set, the arbitrary position reproduction mode setting and the reproduction direction setting are performed to advance the set position by a predetermined time. Alternatively, an interactive image operation display method characterized in that partial images are displayed continuously or at fixed time intervals.

(24) An interactive image operation display method characterized in that the means (15) further comprises a step of erasing the combined and displayed partial image after a preset fixed time.

(25) In the means (15), time positions at which the partial images to be combined are present are sequentially obtained from the operation target space position at predetermined time intervals, and the present time position and the time position immediately before are obtained. An interactive image operation display method further comprising a step of reading sound data corresponding to the interval and reproducing the sound data within the predetermined time interval.

(26) In the means (25), in the reproducing step, the sound data is compressed if the difference between the current time position and the time position obtained immediately before is longer than the predetermined time interval. An interactive image operation display method, which is characterized in that when the audio data is short, the audio data is expanded and reproduced so that the reproduction of the sound data ends within the predetermined time interval.

(27) In the means (15), in the setting step, the operation target space position is set by using a pointing device for moving the cursor displayed on the screen, and the cursor is set at the operation target space position. An interactive image operation display method characterized in that only the outline is displayed with a watermark.

(28) A program storage medium for storing a program for synthesizing and displaying on the background image a series of partial images that change in time and are associated with the spatial position of the background image in the displayed background image space. The procedure of reading the background image stored in the background image storage means and displaying it on the screen of the display means, the procedure of setting the operation target space position on the display screen of the background image, and the operation target space position , A procedure for uniquely identifying a partial image to be composite-displayed,
A procedure of reading the partial image to be combined and displayed from the partial image storage means and combining and displaying the partial image at the operation target space position;
A program storage medium in which a program for causing a computer to execute is stored in a computer-readable medium.

(29) A program storage medium, characterized in that, in the means (28), a program for causing the computer to execute a procedure for creating a background image to be accumulated from a moving image composed of continuous frame images is further stored. .

(30) A program characterized by further storing a program for causing the computer to execute a procedure obtained by extracting a partial image to be accumulated from an original image for which a background image is created, in the means (28). Storage medium.

(31) In the means (28), in the specifying procedure, a continuous mapping from the operation target spatial position to the temporal position for specifying the partial image is defined, and the first operation target spatial position is continuously adjacent. A program storage medium characterized in that when a position is designated, a partial image is specified by continuously tracing on a mapping line.

(32) In the means (31), when the mapping from the operation target space position to the time position is multi-valued, in the above-mentioned specifying procedure, one-to-one mapping is performed for the mapping of the temporally close intervals. A program storage medium characterized by performing conversion so that

(33) In the means (31), when the mapping from the operation target spatial position to the time position is multi-valued, in the above-mentioned specifying procedure, the mapping of the sections which are not temporally close to each other is predetermined. A program storage medium, wherein one value is selected from multiple values according to a rule.

(34) In the means (31), in the specifying procedure, when the mapping from the operation target space position to the time position is multivalued, the operation target position consecutive from the immediately previous operation target space position is set. If the operation target space position is set to a corresponding continuous time position, and if the operation target space position that is not continuous from the immediately previous operation target space position is set, it is necessary to select one value from the multiple values according to a predetermined rule. Characteristic program storage medium.

(35) In the means (28), in the procedure of the composite display, the enlargement or reduction mode is designated,
A program storage medium characterized by enlarging or reducing a partial image together with a background image for composite display.

(36) In the means (28), in the procedure of the composite display, when the spatial position of the operation target is set, by the arbitrary position reproduction mode setting and the reproduction direction setting, the predetermined position is preceded by a predetermined time. Alternatively, the program storage medium is characterized in that the partial images are displayed continuously or at fixed time intervals.

(37) A program storage medium, characterized in that, in the means (28), a program for causing the computer to execute a procedure for erasing the combined and displayed partial image after a preset fixed time is further stored.

(38) In the means (28), the time positions at which the partial images to be combined are present are sequentially obtained from the operation space position at predetermined time intervals, and the present time position and the time position immediately before are obtained. A program storage medium further storing a program for causing the computer to execute a procedure of reading sound data corresponding to the interval and reproducing the sound data within the predetermined time interval.

(39) In the means (38), in the reproducing step, the sound data is compressed when the difference between the current time position and the time position obtained immediately before is longer than the predetermined time interval. The program storage medium is characterized in that when it is short, the sound data is expanded and reproduced so that the reproduction of the sound data is completed within the predetermined time interval.

(40) In the means (28), in the setting procedure, the operation target space position is set by using a pointing device for moving the cursor displayed on the screen, and the cursor sets the operation target space position. A program storage medium characterized in that only a contour is displayed as a watermark when the program is executed.

In the present invention, the background image is displayed on the display means, the operator sets a spatial position as an operation target on the display screen with a pointing device such as a mouse, and a portion to be synthesized and displayed from the operation target spatial position. The image is uniquely specified, and the partial images specified in the set spatial position are combined and displayed, so that when the user changes the operation position momentarily, a series of parts associated with each position in the background image. Display an image. From the operator's point of view, when displaying a series of partial images associated with respective positions in the background image, the background image is greatly different from the conventional art in that the background image can be operated by a hand.

[0047]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is a diagram showing a configuration example of an interactive image operation display device according to an embodiment of the present invention. In FIG. 1, 1 is a background image storage unit, 2 is an image display unit, 3 is a partial image storage unit for composition, 4 is an operation target spatial position input unit, 5 is a composite image mapping unit, and 6 is an image composition processing unit. .

2 and 3 are flow charts showing the procedure of the preparation process and the operation event process in the interactive image operation display device. The processing will be described below in order.

First, the preparation process for performing the interactive process is performed in the procedure shown in FIG. In the process initial preparation process 11, various basic data are initially prepared and set. For example, the background image is stored in the background image storage unit 1. An image created by a separate means can be used for this, for example, “Japanese Patent Laid-Open No. 6-98206“ Video shooting state detection method ””.
By the processing by, the shift of each frame image is analyzed by analyzing the striped pattern corresponding to the camera operation created by the background object in the spatiotemporal image from the video image on which the camera operation such as panning (swing the camera left and right) A background image can be automatically created by checking and shifting the frame images by the amount of the shift.

FIG. 4 is a diagram for explaining an example of processing for creating a background image, showing how a background image is created by the above method from a video image taken while the camera is panned to the right. In this example, when the frame images 30 are successively laminated on top of each other, a background image 31 is formed by leaving portions of the frame images 30 that are displaced and do not overlap. In this background image creation processing, by simultaneously or manually recognizing, extracting, and tracking the moving object in the video, the object image (partial image) 32 is recorded with the position information for each frame moment by moment. You can keep it. The partial image 32 corresponding to the object obtained in this way is an image that changes with time including the position, and is recombined at the original extraction position on the background image 31 that was originally the extraction source. As a result, the image 33 useful for grasping the movement of the object in the background image can be created. As described above, the synthesis partial image according to the present invention is intended for an image that is originally inseparable from the background image and is associated with a proper position.

The partial image for combining prepared in advance as described above is stored in the partial image for combining storing unit 3 together with the extracted position information. Extraction position information is the center of gravity of the circumscribing rectangle that contains the object when the image is extracted,
This is information for uniquely specifying the position of the object, such as indicating the point on the lower left gate as the coordinate position on the background image.

FIG. 5A shows a locus 35 of an object that makes a complicated motion in the background image. Process initial preparation process 1
In No. 1, the movement trajectory data 35 on the background image of the partial image (hereinafter, the target object) obtained through the above-described processing method or the like is read as P (ti). ti indicates a time position 40 corresponding to a frame of each partial image, and i is set to 0 to N, for example. Furthermore, FIG.
As shown in, in order to trace the movement trajectory of the target object as a rough trajectory in which small fluctuations are suppressed, a deviation ε (εx38 in vector, εx38,
Considering the enlargement frame 37 in which only εy39) is set, ε is set in advance at the system side or at the user's selection for that purpose.

Next, processing corresponding to each time is performed. The initial position of the target object at the processing target time ti is P (t
i), the movement trajectory trace point position is Q (ti), the enlargement frame enlarged by the deviation ε from the target object position is Wi, and the time position corresponding to the confirmed trace point is ts, and the initial setting process in FIG. 2 is performed. 12, the time is t0
Set the initial value as. Next, as processing in each ti, in processing 13, as shown in FIG.
It is determined whether or not the target object position P (ti + 1) at +1 exceeds the expansion frame centered on the confirmed trace point Q (ts). If it does not overflow, i is incremented in process 15 and the process of 13 is repeated. If it protrudes like the frame of 41,
In the process 14, P (ti + 1) is set again as the trace point Q (ti + 1), and Q (ts) and Q (t
i + 1) is interpolated to determine the trace position at each time position of the section. As the interpolation means, a method such as simple linear interpolation or Bezier curve approximation may be selected according to the required smoothness. The expansion frame is newly set as new ts, and i is incremented in the process 15, and the process of 13 is repeated again. In the process 13, when the final time position of the target object has already been reached, the interpolation process of 14 is performed, and then the entire process ends (end process 16).

As a result of the above processing, a new trace locus (dotted line indicated by 42) as shown in FIG. Compared to FIG. 5A, the part where the loop is drawn is a simple curve. When the above result is drawn as a mapping from the spatial position 44 to the temporal position 43, a diagram as shown in FIG. 5C is obtained. However, in this figure, the spatial position is represented by only the one-dimensional X axis for simplicity.

The above is pre-preparation for interactive processing, and FIG.
Shows the interactive processing (operation event processing) during use.

First, when an event that is defined in advance for this process occurs in the events that occur when the operator performs some operation, 17 this operation event process is activated. This event is set so as to occur when a point in the background image is just pointed in the event processing of mouse input, for example.

Next, a partial image to be combined is determined. Here, the processing in a more complicated mapping example in which a plurality of temporal positions exist near the same spatial position as shown in FIG. 6 will be described. . For example, in the process 18, the operation target space position information is acquired in relation to the event. Further, process 1
9, the operation target spatial position P pointed by the mapping 55 between the temporal position 50 and the spatial position 51 shown in FIG.
The time position 53 in which the target object exists corresponding to i52 is listed (t1, t2, t3 in the example).
In the example of FIG. 6, the space is one-dimensional on the horizontal axis for the sake of explanation, but the case of two-dimensional or more can be similarly considered.
At this time, various rules can be set regarding which time position is selected first in the process 20 from the listed. For example, it is possible to simply select the earliest time t1 first, and change the time sequentially selected by double-clicking from the set of times listed by the display means 57 in FIG. .

Further, when the operation target spatial position is successively designated by the immediately preceding operation, the continuous operation target position is mapped to the corresponding continuous time position, and the spatial position designated by the immediately preceding operation is designated. It is also possible to combine the partial images in succession to the immediately preceding operation by selecting the value that is closest in time to the time position corresponding to.

Through the above processing 20, the partial image 58 which is the target object corresponding to each operation target spatial position.
Is uniquely specified by the time position determined by the mapping 55. Here, the background image portion corresponding to the combined portion is stored by the process 21.

Next, in process 22 of FIG. 3, the partial image is read from the combining partial image storage unit 3 of FIG.
In process 23, the background image is combined and displayed at the position where the partial image should originally be. Processing may be changed depending on the expression purpose, such as composition in which a partial image is overwritten on a background image, or composition in which a background image is transparently mixed in some proportion. In this way, by setting the designated operation target space position as the start position and further shifting the position (for example, while pressing the mouse button), the trace line on the map of FIG. 6 is maintained in a continuous state. Can be traced. When the operation target space position is shifted and the trace line is deviated, the process ends at 25 in FIG. When the position of the operation target space is displaced by the operation, the image already combined at the current location can be immediately erased, left for a certain period of time, or left for a certain period of time, or the like, depending on the expression effect. In the case of erasing, the background image stored in the process 21 following the partial image specifying process 20 is used to erase the already-synthesized partial image in the process 24.

Further, when the arbitrary position reproduction mode setting and the reproduction direction setting are performed by designating one point of the operation target space position and designating the forward direction or the reverse direction, the point is not traced but the point is moved first. It is also possible to display the partial images continuously (forward direction) or forwardly (reverse direction) for a predetermined time, or at a fixed time interval.

Further, the background image to be displayed may not be able to be displayed entirely if the aspect ratio is large, or it may be desired to see the details of a specific portion. Therefore, depending on the user's designation of the enlargement or reduction mode, It is also possible to enlarge or reduce the background image and the partial image in the image synthesizing processing unit and synthesize and display them.

Next, the reproduction of sound during tracing will be described. While tracing, at a suitable short time interval, the time position where the target object exists is obtained from the current operation target space position. The method of obtaining the time position is the same as in the case of the image described above. For example, when the time interval is 0.5 seconds, the time position is obtained every 0.5 seconds. Then 1
The corresponding sound data is read from the previous time position to the current time position, and if the difference between the current time position and the previous time position is longer than the time interval, the read sound data is compressed. If it is short, it is extended and the reproduction is performed so that the reproduction is finished within the time interval.

The sound data can be compressed / decompressed by thinning out or increasing the sound data at appropriate intervals, but when thinned out, the whole sound is high, and when it is increased, it becomes low and difficult to hear. Therefore, the following method of processing only the portion where the frequency characteristic of the sound data is temporally redundant is also possible. First, the read sound data is divided into segments in units of tens of milliseconds, and the autocorrelation between the segments is obtained. When the correlation between adjacent segments is high, it is determined to be redundant, and the data of the segment is thinned out or increased.

In particular, when a person is speaking, there is a lot of redundancy in the sound data corresponding to the vowels of the voice. Therefore, it is necessary to detect the vowels before obtaining the autocorrelation and target that part. It is possible to process efficiently. Since the spectrum of a vowel has a harmonic structure in which a peak appears at an integral multiple of the fundamental frequency, the position of the vowel can be detected by detecting the harmonic structure using a comb filter or the like. However, since the harmonic structure is also observed in musical tones and the like, musical tone components must be removed in advance. Musical sounds are characterized by less temporal frequency fluctuations than voices, so it is possible to remove musical sound components by finding the spectrogram of sound data and deleting the peak of the spectrum that is stable in the frequency direction for a certain period of time. It is possible. still,
The peak of the spectrum can be obtained by utilizing the characteristic that the difference between the values of the spectra adjacent in the frequency direction is large.

FIG. 7 is a diagram showing a state in which an operation corresponding to the above processing is interactively performed using a display-integrated touch screen as a pointing device for inputting an operation target spatial position, and 60 is a background image. , 62 is a state of the finger designating the operation target space position, and 61 is a partial image synthesized at the designated position. By using the display-integrated touch screen as shown in this figure, it is possible to interactively control the moving image in the background image as if the target object is touching and moving.

When performing such an operation, the operation target space position can be separated from the target object by an arbitrary distance in order to prevent the target object from being hidden by the finger and becoming invisible. When a mouse or the like is used as the pointing device, the target object is hidden by the cursor. Therefore, when designating the operation target space position, the cursor can be transparent and only the contour can be specified.

The procedure of the processing flow of the present invention shown in FIGS. 2 and 3 shown in the present embodiment, the procedure of generating a background image to be accumulated from a moving image, and the partial image to be accumulated are also described. The procedure for extracting the background image from the generated original image can be programmed and executed by a computer, and the program is stored in a computer-readable medium (for example, a floppy disk or a CD-ROM). Can be distributed.

[0070]

As described above, according to the present invention,
When an object behaves in a series in the background image,
The effect that the image can be expressed at the place where the object occupies in time by directly instructing the spatial position on the background image such as a panoramic image instead of indirectly playing back the video by time axis control. Is obtained.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration example of an interactive image operation display device according to an embodiment of the present invention.

FIG. 2 is a flowchart showing a process of the interactive image operation display device of FIG.
It is a flow chart which shows the procedure of the preparation processing for dialog processing.

FIG. 3 is a flowchart showing a process of the interactive image operation display device of FIG.
It is a flow chart which shows the procedure of interactive processing (operation event processing).

FIG. 4 is a diagram illustrating the principle of creating a panoramic background image from a moving image that is panned right after the subject in the preparation process of FIG.

5 is a trajectory of an object that makes a complicated motion in the background image in the preparatory processing in FIG. 2, a processing of roughly approximating a temporally continuous neighborhood of the object trajectory, and a background image space position as a result of the processing. It is a figure which shows the mapping from to a partial image time position.

FIG. 6 is a diagram showing an example in which multi-value mapping is performed for the same spatial position in the background image in the interactive processing of FIG.

FIG. 7 is a diagram showing a state in which an interactive operation is performed using a display-integrated touch screen as a pointing device for inputting an operation target spatial position in the interactive process of FIG.

FIG. 8 is a diagram showing an example of operation based on a time axis, among examples of conventional moving image display and its operation.

FIG. 9 is a diagram showing an example of performing a visual field operation, among examples of conventional moving image display and its operation.

[Explanation of symbols]

1 background image storage unit 2 image display unit 3 partial image storage unit for compositing 4 operation target spatial position input unit 5 composite image mapping unit 6 image composition processing unit 30 series of frame images 31 from which background images are created Background image 32 Partial image 33 corresponding to the target object extracted from the original moving image Virtual background image 35 indicating the position of each partial image 35 Trajectory of the target object in a certain background image 36 Background image 37 Operation Enlargement frames 38, 39 surrounding the target space position x, y values of deviation ε centering on the operation target space position 40 for configuring the expansion frame 40 operation target space position 41 Next target in the expansion frame surrounding the operation target space position Object position determined to be out of alignment 42 Object trajectory 43 that was finally traced roughly Axis of time position for specifying partial image 44 Background image Axis 45 indicating spatial position Spatial position with background image 46 Background image Mapping curve from spatial position to partial image specifying time position 50 Axis of time position for specifying partial image 51 Axis 52 showing spatial position of background image Background Image spatial position 53 Time position of partial image corresponding to certain spatial position Pi on background image 54 Point on mapping curve corresponding to certain spatial position Pi on background image 55 Partial image specific time position from background image spatial position Mapping curve 56 to the earliest time position of the partial image corresponding to the point on the mapping curve corresponding to the same spatial position Pi Display example 57 Corresponding to the point on the mapping curve corresponding to the same spatial position Pi Display Example of Time Position of Partial Image 60 Background Image 61 Displayed on Display-Integrated Touch Screen Target Object Combined at Location Specified as Operation Target Space Position The operator's finger to perform the operation for specifying the partial image 62 manipulation target spatial position of the corresponding

Front page continued (72) Inventor Yukinobu Taniguchi 3-19-2 Nishishinjuku, Shinjuku-ku, Tokyo Inside Nippon Telegraph and Telephone Corporation (72) Inventor Takashi Sato 3-19-3 Nishishinjuku, Shinjuku-ku, Tokyo Nihon Telegraph and Telephone Corporation (56) References JP-A-7-84746 (JP, A) JP-A-6-139042 (JP, A) JP-A-8-202897 (JP, A) (58) Field (Int.Cl. ⁷ , DB name) G06T 1/00 G06T 11/60-17/50 H04N 5/262-5/28 G06F 3/00

Claims (21)

(57) [Claims] 1. A device for synthesizing and displaying, on the displayed background image space, a series of time-varying partial images that are associated with the spatial position of the background image, and that accumulates the background image. Background image storage means, display means for displaying the background image stored in the background image storage means, partial image storage means for storing partial images associated with the spatial position of the background image, and the partial image A means for extracting a partial image accumulated in the accumulating means from an original image for which a background image is created, a setting means for setting an operation target space position on the screen of the display means, and a composite display from the operation target space position. Specifying means for uniquely specifying a partial image to be displayed, and reading out the partial image to be combined and displayed from the partial image storing means, and reading the partial image on the displayed background image at the operation target space position. Interactive image manipulation display apparatus comprising: the composite display means for the partial image synthesizing display that issued, the. 2. A device for synthesizing and displaying, on the background image, a series of partial images temporally corresponding to the spatial position of the background image in the displayed background image space, the background image being accumulated. Background image storage means, display means for displaying the background image stored in the background image storage means, partial image storage means for storing a partial image associated with the spatial position of the background image, and the display means Setting means for setting the operation target space position on the screen, and defining a continuous mapping from the operation target space position to the time position for specifying the partial image, and continuously position from the first operation target space position to the vicinity. When there is a designation, a specifying unit that continuously traces on the mapping line to uniquely specify a partial image to be combined and displayed, and the partial image accumulation to the partial image to be combined and displayed. And a composite display unit for compositely displaying the partial images read at the operation target space position on the background image displayed and displayed by the unit, the interactive image operation display device. 3. When the mapping from the operation target space position to the time position is multi-valued, the specifying means performs conversion so that mapping of sections temporally close to each other becomes a one-to-one mapping. The interactive image operation display device according to claim 2. 4. When the mapping from the operation target spatial position to the time position is multi-valued, the specifying means sets one of the multi-valued mappings in accordance with a predetermined rule for the mapping of the sections that are not temporally close to each other. 3. The interactive image operation display device according to claim 2, wherein 5. The specifying means corresponds to the case where the mapping from the operation target space position to the time position is multivalued and when the operation target space position continuous from the immediately previous operation target space position is set. 3. When mapping to consecutive time positions and setting an operation target space position that is not continuous from the immediately preceding operation target space position, one value is selected from multiple values according to a predetermined rule. The interactive image operation display device described. 6. A device for synthesizing and displaying, on the displayed background image space, a series of partial images that change in time and are associated with the spatial position of the background image, and accumulate the background image. Background image storage means, display means for displaying the background image stored in the background image storage means, partial image storage means for storing a partial image associated with the spatial position of the background image, and the display means Setting means for setting the operation target spatial position on the screen, specifying means for uniquely specifying the partial image to be composite-displayed from the operation target spatial position, and the partial image accumulating means for the partial image to be composite-displayed. Composite display means for compositely displaying the partial images read out from the operation target space position on the displayed background image, and the part to be composited from the operation target space position. A time position where an image exists is sequentially obtained at a predetermined time interval, sound data corresponding to a time position between the current time position and the time position immediately before is read, and the sound data is reproduced within the predetermined time interval. An interactive image operation display device, comprising: a reproduction unit. 7. The sound reproducing means compresses the sound data when the difference between the current time position and the time position obtained immediately before is longer than the predetermined time interval, and expands it when the difference is short. 7. The interactive image operation display device according to claim 6, wherein the reproduction of the sound data is performed so as to be completed within the predetermined time interval. 8. A method for synthesizing and displaying a series of partial images, which temporally changes in correspondence with the spatial position of the background image on the displayed background image space, on the background image storage means. A step of reading out the accumulated background image and displaying it on the image of the display means; a step of obtaining the accumulated partial image from the original image that created the background image; and a step of displaying the background image on the display screen. A step of setting an operation target space position; a step of uniquely specifying a partial image to be composite-displayed from the operation target space position; and a step of reading the partial image to be composite-displayed from a partial image storage means to the operation target space. An interactive image operation display method, which comprises: a step of composite display at a position; 9. A method of synthesizing and displaying on a displayed background image space a series of partial images, which are associated with the spatial position of the background image and change with time, on the background image, and comprising: A step of reading out the accumulated background image and displaying it on the image of the display means; a step of setting an operation target spatial position on the display screen of the background image; and a time position for specifying a partial image from the operation target spatial position If a continuous mapping is defined, and if there is continuous position designation from the first operation target spatial position to the vicinity, the mapping line is continuously traced to uniquely identify the partial image to be composite-displayed. And a step of reading the partial image to be combined and displayed from the partial image storage means and displaying the combined image at the operation target spatial position. 10. In the specifying step, when the mapping from the operation target space position to the time position is multi-valued, the mapping of the temporally close intervals is converted to a one-to-one mapping. The interactive image operation display method according to claim 9, wherein 11. In the specifying process, when the mapping from the operation target spatial position to the time position is multivalued, the mapping of the sections that are not temporally close to each other is determined from among the multivalued values according to a predetermined rule. The interactive image operation display method according to claim 9, wherein a value is selected. 12. In the specifying process, when the mapping from the operation target space position to the time position is multi-valued and the operation target space position continuous from the immediately previous operation target space position is set, the correspondence is performed. When the operation target spatial position that is not continuous from the immediately previous operation target spatial position is set, one value is selected from multiple values according to a predetermined rule. 9. The interactive image operation display method described in 9. 13. A method of synthesizing and displaying a series of partial images, which temporally changes in correspondence with the spatial position of the background image on the displayed background image space, on the background image storage means. A process of reading out the stored background image and displaying it on the image of the display means, a process of setting an operation target spatial position on the display screen of the background image, and a partial image to be synthesized and displayed from the operation target spatial position. Uniquely specifying, the step of reading the partial image to be combined and displayed from the partial image storage means, and combining and displaying the partial image at the operation target space position, and the partial image to be combined from the operation target space position. Time positions are sequentially obtained at predetermined time intervals, sound data corresponding to the current time position and the time position obtained immediately before are read, and the sound data are reproduced within the predetermined time intervals. Interactive image manipulation display method characterized by comprising the steps, a. 14. In the step of reproducing, the sound data is compressed when the difference between the current time position and the time position obtained immediately before is longer than the predetermined time interval, and expanded when the difference is short. 14. The interactive image operation display method according to claim 13, wherein the reproduction of the sound data is performed so as to be completed within the predetermined time interval. 15. A program storage medium for storing a program for displaying, in the displayed background image space, a series of partial images corresponding to the spatial position of the background image and changing in time, on the background image. A procedure for reading the background image stored in the background image storage means and displaying the background image on the image of the display means; a procedure for extracting the stored partial image from the original image from which the background image was created; A procedure for setting an operation target spatial position on the image display screen, a procedure for uniquely specifying a partial image to be composite-displayed from the operation target spatial position, and a partial image storage means for the partial image to be composite-displayed. A procedure for reading and synthesizing and displaying in the operation target space position, and a program for causing a computer to execute: stored in a computer-readable medium. A program storage medium characterized by. 16. A program storage medium for storing a program for displaying, in the displayed background image space, a series of partial images corresponding to the spatial position of the background image and changing in time, on the background image. , A procedure of reading the background image stored in the background image storage means and displaying it on the image of the display means, a step of setting an operation target space position on the display screen of the background image, and a part from the operation target space position. When a continuous mapping to the time position that specifies the image is defined, and if there is continuous position designation from the first operation target spatial position to the neighborhood, the part to be traced continuously on the mapping line and displayed as a composite A computer that uniquely identifies the image; and a procedure that reads out the partial image to be combined and displayed from the partial image storage means and combines and displays the partial image in the operation target space Program storage medium characterized by a program for executing the computer stored in the readable medium. 17. In the specifying procedure, when the mapping from the operation target spatial position to the time position is multi-valued, the mapping of the temporally close intervals is converted so as to be a one-to-one mapping. The program storage medium according to claim 16, characterized in that. 18. In the specifying procedure, when the mapping from the operation target spatial position to the temporal position is multivalued, the mapping of the sections that are not temporally close to each other is determined from among the multivalued values according to a predetermined rule. The program storage medium according to claim 16, wherein a value is selected. 19. In the specifying procedure, when the mapping from the operation target space position to the time position is multi-valued and the operation target space position continuous from the immediately previous operation target space position is set, it is possible to correspond to When the operation target spatial position that is not continuous from the immediately previous operation target spatial position is set, one value is selected from multiple values according to a predetermined rule. 16. The program storage medium described in 16. 20. A program storage medium for storing a program for synthesizing and displaying, on the background image, a series of partial images that change in time and are associated with the spatial position of the background image in the displayed background image space. , A procedure of reading the background image stored in the background image storage means and displaying it on the image of the display means, a step of setting an operation target space position on the display screen of the background image, and a step of setting the operation target space position, A procedure for uniquely identifying a partial image to be composite-displayed; a procedure for reading the partial image to be composite-displayed from the partial image storage means and composite-displaying the partial image at the operation-target space position; The time position at which the partial image to be present exists is sequentially obtained at predetermined time intervals, and the sound data corresponding to the time position between the current time position and the time position obtained immediately before is read and the sound data Program storage medium characterized and procedures for reproducing the data within the predetermined time interval, the program causing a computer to execute the fact stored in the computer-readable media. 21. In the step of reproducing, the sound data is compressed when the difference between the current time position and the time position obtained immediately before is longer than the predetermined time interval, and expanded when the difference is short. 21. The program storage medium according to claim 20, wherein the reproduction of the sound data is performed so that the reproduction of the sound data ends within the predetermined time interval.