JP2020088611A - Image processing apparatus and control method thereof - Google Patents

Abstract

To provide an image processing apparatus that efficiently encodes a moving image.SOLUTION: An image processing apparatus (monitoring camera) comprises: an encoding section that encodes frame images constituting a moving image; an image rotation determining section that determines whether or not image rotation occurs for a preceding frame image in the moving image; storage means that stores a preceding frame image in the frame image, in which it is determined that the image rotation has occurred; and an encoding processing control section that controls encoding means to encode the frame image, in which it is determined that the image rotation has again occurred, using inter-frame prediction in which the frame image stored in the storage means is defined as a reference frame, when it is determined that the image rotation has again occurred after it is determined that the image rotation has occurred.SELECTED DRAWING: Figure 1

Description

The present invention relates to a moving image coding technique.

In recent years, video systems using cameras have been widely introduced along with the widespread use of video conference systems. In addition, surveillance cameras are often installed in facilities not only for crime prevention but also for investigation and management. Among them, there are many applications in which a surveillance camera monitors a surveillance target to transmit video data via the Internet or wireless communication or to store the video data in a storage medium. In recent years, the image quality and resolution of video data shot by surveillance cameras have been increasing, and the amount of video data has been increasing. Therefore, it is required to reduce the data amount.

Video coding techniques include interframe prediction that predicts the pixel value of the target frame from frames before and after the target frame, intraframe prediction that predicts the pixel value of the target frame from within the target frame, and entropy coding and quantization. Is used. By using these video coding techniques, video data can be effectively compressed.

By the way, when a scene change occurs in a video (scene change), it is desirable to encode as an I frame (Intra-coded Frame) in order to avoid deterioration of image quality. The I frame is a frame in which a predicted signal is generated using decoded pixels around the target block in the frame, and has a larger code amount than a P frame (Predicted Frame) or the like. Patent Document 1 proposes a technique of changing a range to be used (changing a scene) in a video conference system based on a trigger such as appearance of a voice or a person and frame information notified in advance from the encoder unit side. ing. More specifically, efficient coding processing is performed by performing a scene change at the timing of an I frame in response to the reception of a trigger.

JP, 2017-28375, A

However, unlike a video conference, in the use of a surveillance camera, an unspecified number of people make or emit audio at irregular timings. Therefore, it is difficult to use a trigger such as the appearance of a voice or a person as in Patent Document 1. Moreover, in a ceiling-suspended surveillance camera, a scene change is likely to occur due to a pan-tilt operation. This is because, in order to output an image that does not cause a sense of discomfort, the image is vertically inverted (the image is rotated by 180°) at the timing when the shooting direction is vertically downward. As a result, the amount of information of the motion vector used in the inter-frame prediction increases between the frame images before and after the inversion. As a result, the encoding efficiency of the entire image is reduced and the encoded data size is increased.

The present invention has been made in view of such problems, and an object thereof is to provide a technique that enables efficient encoding of moving images.

In order to solve the above problems, the image processing device according to the present invention has the following configuration. That is, the image processing device
Encoding means for encoding the frame images forming the moving image,
In the moving image, a determination unit that determines whether or not image rotation has occurred with respect to the preceding frame image,
Storage means for storing a frame image preceding the frame image determined to have image rotation by the determination means;
After the determination means determines that the image rotation has occurred, and when the determination means determines that the image rotation has occurred again in the moving image, the frame image stored in the storage means is used as the reference frame. Control means for controlling the encoding means so as to encode the frame image determined to have re-rotated by using inter-frame prediction;
Have.

According to the present invention, it is possible to provide a technique that enables efficient encoding of moving images.

It is a figure showing the functional composition of the surveillance camera containing the image processing device concerning a 1st embodiment. It is a figure which shows the hardware constitutions of a video processing part. It is a figure explaining the pan tilt operation in a three-dimensional coordinate. It is a figure explaining the image acquired by the pan tilt operation. It is a figure which shows the search range at the time of rotating an image 180 degrees as an example. It is a figure explaining the change of the encoding processing in 1st Embodiment. 3 is an operation flowchart of the image processing apparatus in the first embodiment. It is a figure explaining the reference frame in each frame in 1st Embodiment. It is a figure explaining the relationship between a pan angle and a frame buffer. It is a figure explaining the change of the encoding processing in a modification. It is a figure explaining the change of the encoding processing in a 2nd embodiment. It is a figure explaining the reference frame in each frame in 2nd Embodiment. It is a figure explaining the image processing based on DPTZ setting information. It is a figure explaining the position coordinate of a rectangular area.

Hereinafter, an example of an embodiment of the present invention will be described in detail with reference to the drawings. Note that the following embodiments are merely examples, and are not intended to limit the scope of the present invention.

(First embodiment)
As a first embodiment of an image processing apparatus according to the present invention, an image processing apparatus (video processing unit 200) that encodes a video obtained by a ceiling-suspended surveillance camera will be described below as an example.

<Device configuration>
FIG. 1 is a diagram showing a functional configuration of a surveillance camera including the image processing apparatus according to the first embodiment. The surveillance camera includes a video processing unit 100 which is an image processing device, a camera 300, and an overall control unit 200, and operates based on the settings received from the PTZ setting unit 201. In the following description, the PTZ setting unit 201 is assumed to be configured as a housing separate from the surveillance camera, but it may be configured to be included in the same housing.

The PTZ setting unit 201 receives PTZ setting information (direction information) regarding pan/tilt/zoom (PTZ) in the surveillance camera from the user, and transmits the setting to the overall control unit 200 via, for example, a network. In the following description, it is assumed that the pan/tilt angle (φpan, φtilt) based on the initial position of the surveillance camera is accepted, but it is configured to be accepted as the angle (Δφ) with respect to the current tilt angle of the surveillance camera. You may.

In order to simplify the description, the following description focuses on the setting related to tilt. The PTZ setting unit 201 itself may be configured to calculate instead of receiving the setting from the user. For example, it may be configured to calculate the settings necessary for tracking the object to be captured by the surveillance camera based on the captured image.

Based on the settings received from the PTZ setting unit 201, the overall control unit 200 controls a motor (not shown) or the like to control the shooting direction by pan/tilt. Here, pan means an operation of moving the camera lens direction (shooting direction) in the horizontal direction (horizontal direction), and tilt means an operation of moving the camera lens direction in the vertical direction (vertical direction). is doing. Also, the zoom information is transmitted to the camera 100 to control the zoom. Further, the information is transmitted to the video processing unit 100.

The camera 300 captures a video and transmits the captured video to the video processing unit 100. The camera 300 has a camera control unit 301, a lens 302, and an imaging unit 303. The camera control unit 301 controls the lens 302 and the imaging unit 303, and performs zoom operation and focus control based on the information received from the overall control unit 200.

The image processing unit 100 performs image processing and compression encoding on the image obtained by the camera 300, and outputs an encoded stream. In particular, the video processing unit 100 is characterized in that it performs image processing and compression encoding based on PTZ setting information input from the overall control unit 200.

The image rotation determination unit 101 generates the trigger information of the image rotation timing in the image processing unit 103 based on the instruction from the overall control unit 200. The trigger information is information indicating the timing at which the image is rotated by the tilt operation (the vertical relationship in the captured image is inverted) in the image obtained by the ceiling-suspended surveillance camera (the image output from the imaging unit 303). That is. In the following description, it is assumed that the image rotation is determined inside the video processing unit 100, but the determination may be made by an external device and the determination result may be accepted. For example, the overall control unit 200 may determine the timing at which the image rotates by determining whether or not the tilt setting received from the PTZ setting unit 201 exceeds a preset threshold value.

The encoding processing control unit 102 determines, based on the image rotation timing trigger information output from the image rotation determination unit 101, which of the intraframe prediction and the interframe prediction code is to be executed in the encoding unit 104. .. In particular, a reference frame (LongTerm reference frame described later) used for intra-frame prediction is set. The frame image set as the reference frame may be configured so that the user can arbitrarily set it.

The image processing unit 103 performs image processing on the video obtained by the camera 300. The video is input from the camera 300 as a series of frame images (for example, 30 frames per second). In the following description, the image processing unit 103 is described as performing rotation processing of a predetermined angle (for example, 180°) with respect to the frame image based on the trigger information received from the image rotation determination unit 101, but other image processing will be described. You may comprise so that it may be performed together. The frame image subjected to the rotation processing (or not subjected to the rotation processing) by the image processing unit 103 is output to the encoding unit 104.

The encoding unit 104 performs an encoding process on the frame image received from the image processing unit 103 based on the information received from the encoding process control unit 102 (whether intra-frame prediction or inter-frame prediction is used). To do.

FIG. 2 is a diagram showing a hardware configuration of the video processing unit 100. The image processing unit 100 includes a CPU 151, a ROM 152, a RAM 153, a camera I/F 154, a control I/F 155, and a network I/F 156. The ROM 152 stores programs executed by the CPU 151 and various setting data. The CPU 151 reads a program stored in the ROM 152 into the RAM 153 and executes the program to realize image processing and coding processing for a frame image input via the camera I/F 154, and communication processing via the network I/F 156. .. The camera I/F 154 receives a video (frame image) from the camera 300. The control I/F 155 receives the control signal from the overall control unit 200. The network I/F 155 transmits the coded stream via the network.

<Video captured by a ceiling-mounted surveillance camera>
FIG. 3 is a diagram illustrating a pan/tilt operation in three-dimensional coordinates. Here, in the XYZ orthogonal coordinate system, the pan/tilt operation is shown with the negative direction of the X axis as the initial position of the camera. The shooting direction of the ceiling-suspended surveillance camera is downward from the horizontal direction (XZ plane). Therefore, here, the tilt angle (φtilt) and the pan angle (φpan) are defined with reference to the direction of the initial position of the camera.

FIG. 4 is a diagram illustrating an image acquired by the pan/tilt operation. FIG. 4 corresponds to a state in which the XY plane is observed from the positive direction of the Z axis in FIG. The camera positions shown in (1) to (5) correspond to tilt angles (φtilt) of 30°, 60°, 90°, 120°, and 150°, respectively. For simplicity of explanation, it is assumed here that all pan angles (φpan) are 0°.

When the camera position is continuously changed from (1) to (5), the vertical direction in the frame image is inverted at the position of (3) in which the image is taken vertically downward. FIG. 4 shows a situation in which a person moving in a straight passage extending from west (West Gate) to east (East Gate) is tracked by a tilt operation of a surveillance camera installed on the ceiling of the passage. In this case, frame images F1, F2, FA, FB, and FC are sequentially obtained as the video output from the imaging unit 303. That is, the image is turned upside down in the middle of the image, resulting in an unnatural image. Therefore, a method of rotating the frame images FA, FB, and FC by 180° may be used in order to make the image look natural (preventing vertical inversion). As a result of the method, frame images F1, F2, F3, F4, and F5 are sequentially obtained.

Therefore, in the first embodiment, the image rotation determination unit 101 inputs the frame based on whether or not the information of the tilt angle (φtilt) input from the overall control unit 200 exceeds a predetermined angle (φtrig). A determination process of whether to rotate the image by 180° is performed. For example, when φtrig=90° is set, the image rotation determination unit 101 determines to rotate the frame image by 180° when φtilt=90° (position (3) shown in FIG. 4). Then, the trigger information is generated at the timing when the rotation of 180° is changed from “no” to “Yes” (or vice versa), and is output to the image processing unit 103. The reference initial position of the camera and the tilt angle at which the trigger information is output are not limited to those described above, and can be arbitrarily set by the user.

In the first embodiment, the image rotation determination unit 101 determines whether the instruction is a pan operation, a tilt operation, or a combination thereof based on the pan/tilt angle information set from the PTZ setting unit 201 and received via the overall control unit 200. Determine if there is. The image rotation determination unit 101 does not generate the image rotation trigger information when only the pan operation is performed because the above-described image rotation does not occur when only the pan operation is performed.

FIG. 5 is a diagram exemplarily showing a search range in inter-frame prediction when an image is rotated by 180°. The encoding processing unit 104 uses the immediately preceding preceding frame image as a reference frame when performing the encoding processing using inter-frame prediction. However, as described with reference to FIG. 4, when the image processing unit 103 rotates the image by 180° in the middle of the video, the images input to the encoding processing unit 104 before and after the 180° rotation are It will rotate 180°. Therefore, when encoding the image frame 500b immediately after 180° rotation, the image frame 500a immediately after 180° rotation is used as a reference frame.

Therefore, when calculating the motion vector for encoding the rectangle B of the image frame 500b, the periphery of the rectangle A of the image frame 500a is set as the search range for motion prediction. That is, the rectangle C corresponding to the rectangle B is out of the search range. As a result, when the image frame 500b is encoded using inter-frame prediction, the difference in motion prediction increases.

Therefore, the encoding process control unit 102 stores the frame image preceding the frame image corresponding to the trigger information in the frame buffer, which is the storage unit for the LongTerm reference frame, according to the trigger information for performing the 180° rotation process of the image. .. After that, when the trigger information for performing the 180° rotation processing of the image is input again, the frame image stored in the frame buffer for the LongTerm reference frame is set as the LongTerm reference frame, and the inter-frame prediction is performed. That is, when the image direction is the same as that of the frame image stored in the frame buffer, the control is performed.

<Device operation>
FIG. 6 is a diagram illustrating the change control of the encoding process. A frame image forming a moving image obtained when the camera position is continuously changed from (1) to (5) and then continuously changed from (5) to (2) is shown as an example. There is.

Conventionally, for example, coding is performed using intraframe prediction at a predetermined cycle, and all other coding is performed using interframe prediction. In FIG. 6, I0 is an I frame (I picture) coded using intraframe prediction. Further, P1 to P14 are P frames (P pictures) coded using interframe prediction. In FIG. 6, reference frames used for encoding each frame image are indicated by white arrows.

Conventionally, when encoding a frame image at a timing corresponding to P13, P12, which is the immediately preceding frame image, is set as a reference frame. Then, the frame image is encoded as a P frame (P13) using inter-frame prediction. On the other hand, in the first embodiment, when encoding the frame image at the timing corresponding to P13, the previously stored frame image P3 is set as the LongTerm reference frame. Then, the frame image is encoded as a P frame (P13) using inter-frame prediction. Other frame images are encoded as P frames (P1,..., P12, P14) using the preceding frame image as a reference frame.

FIG. 8 is a diagram illustrating a reference frame in each frame. It is a table showing a relationship between a reference frame corresponding to each processing target frame at each camera position and a 180° rotation trigger. As shown in FIG. 8, the encoding processing unit 104 receives the trigger information for rotating 180° at the timing of the camera position (3). Therefore, the encoding processing unit 104 stores the frame immediately preceding the frame image being encoded at that timing in the frame buffer for the LongTerm reference frame. Specifically, P3 is stored at the timing of the first camera position (3), and P12 is stored at the timing of the second camera position (3). Then, the frame image coded at the timing of the second camera position (3) is coded as P13 by inter-frame prediction using P3 as the LongTerm reference frame. The P12 stored at the timing of P13 will be used as the LongTerm reference frame at the timing when the camera position (3) is reached again.

In the following description, it is assumed that the frame buffer for the LongTerm reference frame is configured to store a plurality of frame images (frame image group) according to the pan angle each time rotation occurs. .. Further, it is assumed that each stored frame image is sequentially updated (overwritten) each time rotation occurs. However, the frame image may be continuously used as the previous LongTerm reference frame without being updated. Further, the frame buffer may be configured to store a frame image group corresponding to a plurality of rotations.

FIG. 9 is a diagram for explaining the relationship between the pan angle and the frame buffer. Here, it is assumed that the reference frame buffer A is used when the pan angle is close to (1). Similarly, when the pan angle is close to (2), (3), and (4), the reference frame buffers B, C, and D are used, respectively. However, the pan angle to be set and the reference frame buffer to be used are not limited to this.

FIG. 7 is an operation flowchart of the image processing apparatus according to the first embodiment. The process is performed, for example, at the start of monitoring by the monitoring camera.

In step S701, the PTZ setting unit 201 receives the initial setting of the surveillance camera from the user. The information set here includes the shooting time of the surveillance camera, the pan/tilt speed, the threshold information about the horizontal swinging angle of the camera, and the like. The PTZ setting unit 201 notifies the overall control unit 200 of the settings received from the user via, for example, a network.

In step S702, the overall control unit 200 starts shooting with the camera 300. Here, the position (1) is assumed as the initial position at the start of shooting, but the position at the start of shooting is not limited to this.

In step S703, the image rotation determination unit 101 receives the moving image (frame image) obtained by shooting, and determines whether the current processing target frame image has been rotated by 180° by the image processing unit 103 with respect to the preceding frame image. To determine. When the frame image to be processed is not rotated by 180° (when the position of the surveillance camera is other than (3) in FIG. 6) (No in S703), the encoding unit 104 encodes the frame to be processed as in the conventional case. Processing is performed, and the process returns to S703 to move to the processing of the next frame image. On the other hand, when the image is not rotated by 180° (when the position of the surveillance camera is (3) in FIG. 6) (Yes in S703), the process proceeds to S704.

In step S704, the image rotation determination unit 101 determines the pan angle when the image is rotated by 180°. Then, according to the pan angle (that is, the angle in the horizontal direction), it is determined which reference frame buffer stores the frame image to be set as the LongTerm reference frame.

In step S705, the image rotation determination unit 101 determines whether the image rotation is the first 180° rotation. If it is the first image rotation (Yes in S705) (camera position (3) in FIG. 6, P4), the process proceeds to S706. If it is the image rotation again (it is not the first image rotation) (No in S705) (camera position (3) in FIG. 6, P13), the process proceeds to S707.

In step S<b>706, the encoding processing control unit 102 stores the frame image one frame before the frame image in the reference frame buffer A and sets it as the LongTerm reference frame. Here, it is assumed that the pan angle is determined to be closest to (1) in S704. This corresponds to the timing of P4 and the camera position (3) in FIGS. 6 and 8, and the reference frame at this time is P3. When the pan angle is close to (2), (3), and (4) in FIG. 9, the frame images are stored in the reference frame buffers B, C, and D, respectively, and set as the LongTerm reference frame. To do.

In step S707, the encoding unit 104 performs the encoding process using the LongTerm reference frame set at the time of the previous image rotation (camera position (3) in FIG. 6, P4). This corresponds to the timing of P13 and camera position (3) in FIGS. 6 and 8, and the reference frame at this time is P3.

In step S708, the encoding processing control unit 102 stores the frame image one frame before the frame image in the reference frame buffer A and sets it as the LongTerm reference frame. As a result, the previously set LongTerm reference frame is overwritten.

In step S709, the overall control unit 200 determines whether or not there is a shooting end setting. If there is no shooting end setting (No in S709), the process proceeds to S703. If there is a shooting end setting (Yes in S709), shooting with the camera 300 ends and the process ends.

As described above, according to the first embodiment, the frame image immediately before the 180° image rotation occurs is stored in the frame buffer. Then, when the 180° image rotation occurs again (that is, the image direction becomes the same as the frame image stored in the frame buffer), the frame image stored in the frame buffer is used as the LongTerm reference frame and interframe prediction is performed. Encode. This makes it possible to reduce the bit rate in the output encoded stream.

(Modification)
In the first embodiment, the frame image one frame before the previous (first) image rotation is set as the LongTerm reference frame, but another frame image may be set as the LongTerm reference frame. For example, as shown in FIG. 10, the frame image three frames before the previous (first) image rotation may be set as the LongTerm reference frame. In this case, P13 is encoded using P1 as the LongTerm reference frame. When processing the frame image of P14, P13 is used as a reference frame for encoding by inter-frame prediction.

Further, in the above description, it is assumed that the image rotation is 180°, but the image rotation may be another angle. That is, similar processing can be applied as long as the original image direction is restored by the second image rotation.

(Second embodiment)
In the second embodiment, encoding control in the case where a ceiling-suspended camera performs shooting by performing a periodic tilt operation will be described. In particular, when the camera is installed, the camera position information for rotating the image according to the periodic operation and the time information for the tilting operation to the camera position for rotating the image are set. Then, a mode of performing inter-frame prediction using a LongTerm reference frame based on the set initial information will be described. The functional configuration and hardware configuration of the image processing apparatus are almost the same as in the first embodiment. Therefore, the different parts will be mainly described below.

<Device operation>
The PTZ setting unit 201 receives from the user the setting of initial information according to the cyclic operation of the camera. The overall control unit 200 receives the initial information from the PTZ setting unit 201, and sets the settings according to the initial information in the image rotation determination unit 101. After that, the image rotation determination unit 101 calculates the image rotation position based on the initial information, and when it determines that the camera position has reached the image rotation position, notifies the encoding processing control unit 102 of the trigger information.

FIG. 11 is a diagram illustrating control of encoding processing according to the second embodiment. Here, the camera position reciprocates between (1) and (5) shown in FIG. The lower part of FIG. 11 shows which frame image is used as the reference frame to perform the encoding process by the inter-frame prediction at each camera position in such a reciprocating motion.

As shown in FIG. 11, while the camera periodically performs the tilt operation, the presence/absence of image rotation with respect to the frame image is changed every time the camera position reaches the position (3). Then, at that timing, the encoding process is performed on the frame image by inter-frame prediction using the LongTerm reference frame.

FIG. 12 is a diagram illustrating reference frames in each frame in the second embodiment. As shown in FIG. 12, as the initial setting information, the camera position is set according to the installation location of the camera (third column of the table). At the same time, the setting information of the reference frame (including the LongTerm reference frame) in each frame image (the second column of the table) and the camera position that rotates the frame image by 180° (the fourth column of the table) are set. In the following description, the period from the change in the presence/absence of 180° rotation of the frame image to the next rotation of the frame image by 180° will be described as T.

The image rotation determination unit 101 performs a determination process on whether to rotate the frame image based on the table information shown in FIG. 12 set by the overall control unit 200. As described above, the table information includes the given control content that defines the shooting direction at each time (each timing). The image rotation determination unit 101 starts counting the operation time at the same time when the shooting is started. Assuming that the operation time is Tcount, when the image pickup by the camera is started from the initial position/posture ((1) in FIG. 11), the camera position shown in (3) in FIG. 11 is obtained when Tcount=T/2. At this time, the image rotation determination unit 101 determines that the camera position requires image rotation, and notifies the encoding processing unit 102 of trigger information for rotating the image by 180°. As a result, the encoding process control unit 102 can perform the encoding process by the inter-frame prediction using the Long Term reference frame at the camera position where the image is rotated by 180°. Further, it becomes possible to store a frame image used as a Long Term reference frame in the future in the frame buffer. At this time, when it is determined that the camera position is (3), the image rotation determination unit 101 sets Tcount to the initial value 0 and restarts the process.

In the above description, the image rotation timing is detected based on the time from the start of the tilt operation (sixth column in the table). On the other hand, the image rotation timing may be detected based on the setting information (second column of the table) as to whether or not to use the LongTerm reference frame.

Further, in the above description, the table information according to the periodic operation of the camera is created and set at the time of installing the camera, but the table information may be updated during the operation of the camera. As a result, it is possible to deal with a case where the operating mode changes in the middle.

As described above, according to the second embodiment, when the ceiling-suspended surveillance camera performs a periodic operation, table information matched with the operation cycle of the camera is used. By using the table information, it is possible to obtain the same effect as that of the first embodiment.

(Third Embodiment)
In the third embodiment, processing when a surveillance camera having a digital pan/tilt/zoom (DPTZ) function is used will be described. Here, the DPTZ function is a function of generating a panned/tilted/zoomed image by performing a series of image processing such as trimming processing and rotation processing on the video obtained by the surveillance camera. For example, by using an ultra-wide-angle imaging optical system, it is possible to obtain an imaged image in a wide range of directions without using a movable part composed of a motor and gears.

The functional configuration of the image processing apparatus is almost the same as that of the first embodiment (FIG. 1), but instead of physically changing the orientation of the camera 300, the image processing unit 100 performs image processing to artificially pan, tilt, and tilt. Zoom in. Specifically, the overall control unit 200 provides the DPTZ setting information to the video processing unit 100. The DPTZ setting information is information that instructs the image processing unit 103 which image area to output. The video processing unit 100 outputs an encoded stream based on the camera information input from the camera 300 and the DPTZ setting information input from the overall control unit 200.

More specifically, the image processing unit 103 performs trimming processing and rotation processing on the frame image input from the camera 300 based on the DPTZ setting information. The image rotation determination unit 101 determines whether to perform image rotation based on the DPTZ setting information. Then, the encoding process control unit 102 controls the encoding process by the encoding unit 104 of the frame generated by the image processing unit 103 by the DPTZ function. Here, it is controlled to use the LongTerm reference frame as the reference frame used in the inter-frame prediction.

<Generation of frame image by DPTZ>
FIG. 13 is a diagram illustrating image processing based on the DPTZ setting information. An image 1200a shows a frame image output from the imaging unit 303, and an image 1200b shows a frame image generated by the image processing unit 103 based on the DPTZ setting information.

Specifically, an image 1200b is the result of performing the upper body region trimming process, the enlargement process, and the 180° rotation process on one of the three person images included in the image 1200a. A region 1201 shows a rectangular region (trimming region) set based on the DPTZ setting information in the image 1200a.

FIG. 14 is a diagram illustrating position coordinates of a rectangular area. Here, in the image 1200a, the left end of the image is set to coordinate 0 on the X axis, and the center in the vertical (up-down) direction of the image is set to coordinate 0 on the Y axis. A region 1301 is a region corresponding to the region 1201 and is a region defined by the four vertices A to D. Here, the four vertices are defined as A(x1, y1), B(x2, y1), C(x1, y2), D(x2, y2).

The image rotation determination unit 101 acquires the position coordinate information of the area 1301 based on the DPTZ setting information transmitted from the overall control unit 200. The image rotation determination unit 101 performs a determination process of whether to rotate the image with respect to the image subjected to the image processing, based on the acquired position coordinate information.

For example, whether or not to perform the image rotation on the image in the region 1301 is determined using the following determination formula (1).

|y1|-|y2| <0...(1)
That is, in the mathematical expression (1), it is determined whether the region 1301 includes more positive or negative regions in the Y-axis direction. When the formula (1) is satisfied (more negative regions are included in the Y-axis direction), the image rotation determination unit 101 determines to rotate the target rectangle by 180°, and sends the trigger information to the encoding processing control unit 102. Notice. On the other hand, the image rotation determination unit 101 determines that the rotation process is not necessary and does not notify the trigger information when the expression (1) is not satisfied (including more positive regions in the Y-axis direction).

Note that the determination using Equation (1) is an example, and the image rotation determination method is not limited to this.

As described above, according to the third embodiment, it is possible to determine the occurrence of image rotation even in the surveillance camera having the DPTZ function. Therefore, by using the LongTerm reference frame as the reference frame used in the inter-frame prediction at the timing when the image rotation occurs, it is possible to obtain the same effect as that of the first embodiment.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. It can also be realized by the processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Reference numeral 100 image processing unit; 101 image rotation determination unit; 102 encoding processing control unit; 103 image processing unit; 104 encoding unit; 200 overall control unit; 201 PTZ setting unit; 300 camera; 301 camera control unit; 302 lens; 303 Imaging unit

Claims (11)

Encoding means for encoding the frame images forming the moving image,
In the moving image, a determination unit that determines whether or not image rotation has occurred with respect to the preceding frame image,
Storage means for storing a frame image preceding the frame image determined to have image rotation by the determination means;
After the determination means determines that the image rotation has occurred, and when the determination means determines that the image rotation has occurred again in the moving image, the frame image stored in the storage means is used as the reference frame. Control means for controlling the encoding means so as to encode the frame image determined to have re-rotated by using inter-frame prediction;
An image processing apparatus comprising: The control unit determines that the image orientation of the frame image determined to have caused the image rotation again has become the same as the image orientation of the frame image stored in the storage unit due to the occurrence of the image rotation again. In this case, the encoding means is controlled to perform encoding of the frame image determined to have caused the image rotation again using inter-frame prediction using the frame image stored in the storage means as a reference frame. The image processing apparatus according to claim 1, wherein: An image pickup means for obtaining a moving image by image pickup;
Shooting direction control means for controlling the shooting direction of the imaging means,
An image processing unit that rotates a frame image forming the moving image acquired by the image pickup unit by a predetermined angle and outputs the frame image to the encoding unit.
Further has
The photographing direction control means is configured to provide direction information regarding a photographing direction of the image pickup means to the control means and the image processing means,
The image processing means, based on the direction information, determines whether to rotate the frame image corresponding to the direction information,
The control means determines, based on the direction information, whether or not to encode the frame image corresponding to the direction information using inter-frame prediction using the frame image stored in the storage means as a reference frame. The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. The imaging means is a ceiling-suspended camera installed on the ceiling of the passage,
The image processing apparatus according to claim 3, wherein the image capturing direction control unit controls the image capturing direction of the image capturing unit by a pan/tilt operation. The passage is a straight passage,
The photographing direction control means controls the photographing direction of the image pickup means along the linear passage by a tilt operation,
The image processing means determines to rotate the frame image corresponding to the direction information by 180° when the direction information indicates a vertically downward direction,
When the directional information indicates a vertically downward direction, the control means encodes a frame image corresponding to the directional information using inter-frame prediction using the frame image stored in the storage means as a reference frame. The image processing apparatus according to claim 4. The passage is a straight passage,
The photographing direction control means is configured to control the photographing direction of the image pickup means according to given control contents defining the photographing direction at each time,
The image processing means determines to rotate the frame image by 180° according to the given control content,
The image according to claim 4, wherein the control unit encodes a frame image according to the given control content using inter-frame prediction using a frame image stored in the storage unit as a reference frame. Processing equipment. The image processing apparatus according to claim 6, further comprising a setting unit that sets the given control content. The imaging means is a ceiling-suspended camera installed on the ceiling of the passage,
The image processing means is configured to further perform trimming processing and enlargement processing on the frame images forming the moving image acquired by the imaging means,
The image capturing method according to claim 3, wherein the image capturing direction control unit controls the image capturing direction in a pseudo manner by performing image processing by the image processing unit on the moving image acquired by the image capturing unit. Processing equipment. The passage is a straight passage,
The shooting direction control means controls the shooting direction in a pseudo manner along the linear path by changing a trimming area by the image processing means for the moving image,
The image processing means determines whether to rotate the frame image based on the position coordinates of the trimming area in the moving image,
The control means may encode the frame image using inter-frame prediction using the frame image stored in the storage means as a reference frame, based on the position coordinates of the trimming area in the moving image. The image processing apparatus according to claim 8. A control method in an image processing device having an encoding unit for encoding a frame image forming a moving image,
In the moving image, a determination step of determining whether or not image rotation has occurred with respect to the preceding frame image,
A storing step of storing a frame image preceding the frame image determined to have caused image rotation in the determining step in a storage unit;
When it is determined that the image rotation has occurred again in the moving image after it is determined that the image rotation has occurred in the determination step, inter-frame prediction using the frame image stored in the storage unit as a reference frame is performed. Using the control step of controlling the encoding unit to perform encoding of the frame image determined to have occurred the image rotation again,
A method for controlling an image processing apparatus, comprising: A program for causing a computer to function as each unit of the image processing apparatus according to claim 1.

Images