JP2000270336A - Dynamic picture coding method and photographing form estimating method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To encode a dynamic picture and to reduce an information quantity such as a transmission quantity. SOLUTION: An observation process 11 observes motion vectors of four apexes of a picture obtained through photographing of a camera. The observation process 11 figures out reasons of a change in motion vectors into panning of the camera or a camera-shake by photographing of a person when the motion vectors of the four points in a preceding frame are directed in a same direction. After the observation by the observation process 11 and detection of the motion vectors A-D by a detection process 12, a discrimination process 13 discriminates whether the motion vectors of the current frame are directed in the same direction as or in different directions from those of the preceding frame. The discrimination process 13 discriminates panning of the camera when the motion vectors of the current frame are directed in the same direction as those of the preceding frame, and discriminates a camera shake when the motion vectors of the current frame are directed in the different direction from those of the preceding frame or the magnitude of the motion vectors is zero. A coding omit process 14 omits coding for a motion vector error or the like when the discrimination process 13 discriminates that the motion vectors indicate a camera-shake.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a moving picture coding method and a method for estimating a shooting mode.

[0002]

2. Description of the Related Art In recent years, it has become possible to encode and transmit a photographed image in real time in order to implement an LSI on the encoder side and to improve the processing capability of a CPU. Thus, the present invention can be applied to the use of monitoring and displaying information of devices and the like in the field in real time. A moving image is encoded in various forms such as MPEG and H.261. In addition, there are two types of cameras: a type that is roughly fixed and a type that shoots while moving. With the current technology, there is no change in encoding and transmitting the moving image captured by either of the two types using a conventional method.

[0003]

In MPEG and H.261, which are currently the mainstream of moving image coding, the same algorithm is used to encode a photographed image regardless of the method of photographing. It is configured to be. For this reason, there is a major problem when encoding a moving image captured by a person. The problem is that when a person takes a picture with a camera, camera shake occurs, so that meaningless information (here, additional information due to camera shake) is transmitted, and information such as coding time and transmission amount is transmitted. Waste of quantity,
The decoded image is likely to be difficult to see due to camera shake.

[0004] In the case of shooting while moving, panning (hereinafter referred to as pan) frequently occurs, or when shooting (moving) approaching (enlarging) or moving away (reducing) an object, it takes time. There is also a problem that it is difficult to use the directional correlation.

[0005] For this reason, since an image captured by a person holding a camera and an image captured by a fixed camera have their own characteristics, the method of estimating the image capturing method from the motion vector and capturing the image is described. It is desired to encode a moving image by using the characteristics.

SUMMARY OF THE INVENTION The present invention has been made in view of the above circumstances, and makes it possible to estimate a photographing method from a motion vector, to encode a moving image by utilizing characteristics of the photographing method, and to transmit a moving image. It is an object of the present invention to provide a moving picture coding method and a method of estimating a shooting mode in which the information amount such as the amount is reduced.

[0007]

According to the present invention, in order to achieve the above object, a first aspect of the present invention is to provide an observing step of observing a motion vector of a frame of an image obtained by photographing with a camera; In the detection step of detecting the motion vector observed in the, and after detecting the motion vector in this detection step, determine whether the motion vector is oriented in the same direction as the motion vector of the previous frame, or in a different direction, If the motion vector of the current frame is in the same direction as the motion vector of the previous frame, panning will cause the motion vector of the current frame to be in a different direction from the motion vector of the previous frame, or the motion vector will be " If it is "0", the camera shake is determined based on the camera shake based on the motion vector based on the result of the determination. And when it is determined is characterized in that it comprises motion vectors and coding omit step omitted coding error and the like.

According to a second aspect of the present invention, there is provided a new information input step for incorporating new information obtained by panning a camera, a storage step for storing information once panned input in the new information input step, and a storage step for storing the information. The updated information is supplied, and the image is updated with the information. The image database updating process, the predicted image creating process of creating a predicted image using the updated image database updated in the image database updating process, A first encoding step of encoding position information and difference information of the predicted image on the image database created in the predicted image creation step, and enlargement using data on the image database created in the predicted image creation step; Performs reduction and rotation conversions, and encodes predicted image position information, conversion coefficients, and error information for the converted image. It is characterized in that a second encoding step.

A third invention is a camera photographing start step, a search step of obtaining a motion vector of an image frame shot in the start step, and searching for a direction of a motion vector of a previous frame, and a motion searched in the search step. A first determining step of determining whether the direction of the motion vector is the same or different, and if the direction is the same, determining that the camera is panning in the horizontal direction; In the process, if it is determined that the direction is different, it is determined whether the direction of the vertical motion vector is the same or different, and if the direction is the same, it is determined that the camera is panning in the rotation direction When it is determined that the direction is different in the second determination step and the second determination step,
Depending on the direction of the diagonal motion vector, it is determined that the camera is in the process of enlarging or reducing the object, that the camera is being photographed in a fixed state, or that camera shake is occurring. And three judgment steps.

[0010]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a process explanatory view of a moving picture coding method according to a first embodiment of the present invention. This first
The mode is an encoding method for camera shake.
First, the observation step 11 observes four motion vectors at four vertices of an image (image frame) obtained by photographing with a camera. In this observation step 11, FIG.
If the motion vectors of the four points A, B, C, and D in the previous frame shown in (a) are in the same direction, the camera may pan or be tied to two camera shakes caused by human photography. Observe what you can do. After the observation in the observation step 11, after detecting the motion vectors A to D in the detection step 12, it is determined whether the motion vector is in the same direction as that of the previous frame or in a different direction. 13
To judge.

In this determination step 13, if the motion vector of the current frame shown in FIG. 2B is in the same direction as the motion vector of the previous frame, it is determined that panning is performed in the illustrated arrow direction. If the motion vector of the current frame shown in FIG. 2 (c) is oriented in a different direction from the motion vector of the previous frame, or if the motion vector is "0", then the camera shake caused by human photographing Judge.

It should be noted that, depending on the frame rate, it is unlikely that the motion vector changes so quickly even if it is actually a camera shake. Therefore, the motion vectors pointing in the same direction are counted, and if a certain threshold value is exceeded, it is determined that the motion vector is pan, and if it is smaller than the threshold value, it is determined that the motion vector is caused by camera shake. Here, for the sake of explanation, the previous frame in FIG.
(C) Judgment was made based on the difference in the vector direction of the current frame.

As a result of the judgment in the judgment step 13, when it is judged from the motion vector that the motion vector is caused by camera shake, the coding of the motion vector, the error, etc. is omitted in the coding omission step 14 and these codings are performed in the subsequent stage. To avoid transmission. In order to avoid transmitting unnecessary information due to camera shake caused by human photographing,
This makes it possible to reduce the difficulty of viewing images due to camera shake.

FIG. 3 is an explanatory view of a process of a method of estimating a photographing mode according to a second embodiment of the present invention. FIG. This makes it possible to use various encoding methods.

In FIG. 3, reference numeral 31 denotes a camera photographing start step. This photographing start step 31 includes a step of starting photographing with the camera fixed or moving the camera to start photographing. 4 of the image taken in this step 31
The motion vector of the vertex of the point is obtained as described in the first embodiment, and the direction of the motion vector of the vertex of the previous frame is searched in a search step 32. In the search step 32, FIG.
As shown in (b), motion vectors of four vertices are searched. The result of the search is supplied to the first judgment step 33.

In this first determination step 33, it is determined whether the directions of the motion vectors are the same or different at all four points. If the directions are the same, the camera is panned in the horizontal direction. It is determined in a second determination step 34 whether the directions of the motion vectors of the two vertices are the same or different.

When it is determined in the second determination step 34 that the directions of the motion vectors are the same, it is assumed that the camera is panning in the rotational direction, as shown in FIG. It is determined in a third determination step 35 whether the directions of the motion vectors of the two vertices are opposite or the same.

When it is determined in the third determination step 35 that the directions of the motion vectors are the same, it is determined that the camera is in the process of enlarging or reducing the object. When it is determined in the third determination step 35 that the direction of the motion vector is opposite,
It is determined that the camera is photographed in a fixed state or that camera shake has occurred.

The operation in the second embodiment is shown in the flowchart of FIG. In the second mode, a photographing method is determined based on a motion vector. Generally, there are two methods of photographing with a camera: a method of photographing with the camera fixed and a method of moving the camera. . When a person takes a picture with a camera, it is expected that the number of times of panning will be larger than that of a fixed camera (this is because the number of cases where a person takes a picture while moving) increases.

When a person moves with the camera, not only panning in the horizontal direction, but also expanding (approaching the object), reducing (moving away from the object), and rotating the object. Must also be considered. This is done by examining four vertices as shown in Fig. 4. If the directions of the motion vectors at the diagonal vertices are opposite or the same size, the camera identifies that the object is being scaled up or down. Then, by considering the directions of the motion vectors of the four vertices, it is possible to narrow down whether to enlarge (outward vector) or reduce (inward vector) (see FIG. 4).

The rotation means to pan the camera 360 degrees around the photographing viewpoint, as shown in FIG. For this rotation, two vertical vertices (A,
If only B or C, D) were facing the same direction, I decided to think of it as bread by rotation.

Here, each step of the flowchart shown in FIG. 6 will be described in consideration of the above. In FIG. 6, first, in step S1, the motion vectors of the vertices of the four images are obtained, and the directions of the motion vectors of the vertices of the previous frame are compared. By comparing the directions of the motion vectors, it is determined in step S2 whether the directions of the four motion vectors are the same. If the determination in step S2 is "Y", the camera ends panning in the horizontal direction, and the process ends.
If so, the determination in step S3 is made.

In step S3, it is determined whether the directions of the motion vectors of the two vertical vertices are the same. If "Y", it is determined that the camera is panning in the rotation direction.
The judgment of 4 is performed. The determination in step S4 determines whether the directions of the motion vectors of the two diagonal vertices are opposite or the same,
If "Y", it is determined that the camera is in the process of enlarging or reducing, and if "N", the camera is determined to have been photographed in a fixed state or has been shaken, and the processing is terminated.

If the shooting mode based on the motion vector is known as described above, there is a possibility that a more efficient encoding method corresponding to the shooting mode can be used. This method is also effective when a person pans using a fixed camera other than holding a camera and taking a picture.

FIG. 7 is an explanatory view of the steps of a moving picture coding method according to a third embodiment of the present invention. In FIG. 7, reference numeral 71 denotes a new information input step by panning a camera.
This input step 71 is a step for incorporating this new information because the increase in new information coming into the field of view of the camera by panning the camera is immeasurable. The reason why the input step 71 is provided is that the current encoding method does not use the information of the place where the image is panned once, so that if it disappears from the field of view (active area), it is treated as new information every time the image is panned. This is prevented by using the following steps.

The once panned information input in the new information input step 71 is stored in a memory consisting of an image database in a storage step 72. Therefore, once panned, the next time the same pan is panned, the contents of that memory will be referenced. The information stored in the storage step 72 is supplied to an image database update step 73 to update the image database. This means that the active area is moving (panning) on the image database shown in FIG. 8, and the image data on the image database can be used (the data that actually enters the field of view is But there is already data for it in the image database).

That is, in this process, a common image database is created and updated on the encoder side and the decoder side at the time of photographing. A predicted image is created in the predicted image creation step 74 using this image database. Therefore, if a large capacity image database can be created as long as the capacity of the memory permits, the creation of a predicted image becomes easier.

Here, the created image database is processed by applying the block matching method to the panning in the horizontal direction as it is, and the encoder side obtains information on the position of the predicted image (motion vector) in the image database. ) And the difference information are encoded in a first encoding step 75. In addition, FIG.
For the enlargement / reduction and rotation shown in (b), the data on the image database is subjected to conversion such as enlargement (linear interpolation method), reduction (average value method), rotation (rotation of affine transformation), and the converted image. Is processed by applying the block matching method to Thereafter, in the same way, in the encoder side, similarly, the position information (motion vector) of the predicted image on the image database, the conversion coefficient, and the error information are encoded in the second encoding step 76. The information encoded in the first and second encoding steps 75 and 76 is transmitted to the decoder side.

In the configuration of the third embodiment, it is ideal if a large-capacity memory is mounted and a large-capacity image database capable of covering the panning range of the object to be photographed can be created. However, since the amount of installed memory is actually limited, the image database in a limited range is updated so that the active area becomes the center in consideration of the amount of installed memory.

It is to be noted that the image database may be prepared by preparing a common image as a pre-process at the encoder and the decoder side before photographing starts, or may be prepared at any time during photographing. good. However, in the case of creating and updating as needed, an image on the image database cannot be used as a predicted image for a portion to be panned for the first time.

FIG. 10 is a flowchart of a method of estimating a photographing method from a motion vector of another frame, as described in the second embodiment, of a method of producing a predicted image using an image database applied to the third embodiment. is there.

In FIG. 10, S11 is a step of estimating a photographing method from a motion vector.
In step S12, it is estimated whether the photographing method estimated in step 1 is enlargement from the active area on the image database. In this estimation, if “Y”, step S1
Step 3 is performed. Step S13 is a process for creating an enlarged predicted image using a linear interpolation method.
The created prediction image is sent to the block matching method in step S19.

If "N" in step S12, step S1
In step S14, it is estimated whether or not the area is reduced from the active area.
Is performed. Step S15 is a process for creating a predicted image using the averaging method. Here, the created predicted image is also sent to the block matching method in step S19.

If "N" in step S14, step S1
6, it is estimated that the rotation is from the active area. If “Y” in step S16, the process proceeds to step S17.
Is performed. Step S17 is a process for creating a predicted image rotated by using the affine transformation.
The created predicted image is also sent to the block matching method in step S19.

If "N" in step S16, step S1
In step 8, a process of estimating horizontal panning is performed. Thereafter, the block matching method of step S19 is performed, and it is determined in step S20 whether the error is equal to or less than the threshold. Step S2
As a result of the determination of 0, if “N”, the process is repeated until the value becomes equal to or less than the threshold. If “Y”, the motion vector, the conversion coefficient, and the prediction error are coded in step S21, and
At step 22, the image database is updated, and the process of creating the predicted image ends.

[0036]

As described above, according to the present invention,
The following effects can be obtained. (1) Additional information due to camera shake can be eliminated, and a reduction in the amount of information to be transmitted and an improvement in reproduced images can be expected. (2) A shooting method can be estimated from a motion vector, and coding according to a future shooting method can be performed. (3) Creation of an effective predicted image by panning, scaling, and rotation can be expected, and a reduction in the amount of information can be expected.

[Brief description of the drawings]

FIG. 1 is a process explanatory view showing a first embodiment of the present invention.

FIG. 2 is an explanatory diagram of a motion vector due to panning and camera shake of a camera.

FIG. 3 is a process explanatory view showing a second embodiment of the present invention.

FIG. 4 is an explanatory diagram of a motion vector at the time of enlargement / reduction.

FIG. 5 is an explanatory diagram of a motion vector due to rotation.

FIG. 6 is a flowchart of a method for estimating a shooting mode from a motion vector.

FIG. 7 is a process explanatory view showing a third embodiment of the present invention.

FIG. 8 is an explanatory diagram of an image database.

FIG. 9 is an explanatory diagram of enlargement / reduction and rotation from an active area on an image database.

FIG. 10 is a flowchart of creating a predicted image using an image database.

[Explanation of symbols]

 11: motion vector observation step 12: motion vector detection step 13: determination step 14: encoding omission step

[Procedure amendment]

[Submission date] April 4, 2000 (200.4.4)

[Procedure amendment 1]

[Document name to be amended] Statement

[Correction target item name] Claim 2

[Correction method] Change

[Correction contents]

Claims (3)

[Claims] 1. An observation step of observing a motion vector of a frame of an image obtained by photographing with a camera, a detection step of detecting a motion vector observed in the observation step, and a step of detecting a motion vector in the detection step. , The motion vector points in the same direction as the motion vector of the previous frame,
Judge whether the motion vector of the current frame is in the different direction.If the motion vector of the current frame is in the same direction as the motion vector of the previous frame, if panning, the motion vector of the current frame is different from the motion vector of the previous frame. If the camera is facing the direction, or if the motion vector is “0”, when it is determined from the motion vector that the camera motion is determined to be due to camera motion, , A coding omitting step of omitting coding of a motion vector, an error, and the like. 2. A new information input step for taking in new information obtained by panning a camera, a storage step for storing information once panned in the new information input step, and information stored in the storage step. An image database updating step in which an image is supplied and the image is updated with this information; a predicted image creating step of creating a predicted image using the updated image database updated in the image database updating step; A first encoding step of encoding position information and difference information of the predicted image on the image database created in the step, and enlarging, reducing, and rotating using the data in the image database created in the predicted image creating step And a second encoding scheme for encoding the position information of the predicted image, the conversion coefficient, and the error information with respect to the converted image. And a moving image encoding method. 3. A camera shooting start step, a search step for obtaining a motion vector of an image frame shot in the start step, and searching for a direction of a motion vector of a previous frame, and supplying the motion vector searched for in the search step. It is determined whether the directions of the motion vectors are the same or different, and when the directions are the same, a first determination step of determining that the camera is panning in the horizontal direction, and a first determination step include: When it is determined that the direction is different, it is determined whether the direction of the vertical motion vector is the same or different. When the direction is the same, it is determined that the camera is panning in the rotation direction. When it is determined in the determination step and the second determination step that the directions are different, the camera performs an enlargement or reduction process on the object in accordance with the directions of the diagonal motion vectors. A third determining step of determining that the camera is in a fixed state, that the camera is being shot, or that camera shake is occurring.