JP2010226594A - Image transmission device and imaging device mounting the same - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To improve transmission efficiency of a moving image in a region of interest. <P>SOLUTION: In an imaging device, a coding unit 204 calculates a prediction code amount of an entire image in the region of interest based on a code amount generated when coding the entire image in the region of interest. The coding unit 204 calculates the prediction code amount of a specific object included in the region of interest based on the code amount generated when coding the specific object included in the region of interest. A transmission unit 206 calculates a transmission rate of a wireless communication network. A control unit 207 controls a region-of-interest setting unit 202 and a region-of-interest processing unit 203 so as to change a ratio of the specific object in the region of interest on a frame image of a coding target according to the transmission rate, the prediction code amount of the entire image in the region of interest, and the prediction code amount of the specific object included in the region of interest. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image transmission apparatus for encoding and transmitting a moving image and an imaging apparatus using the same.

  In recent years, digital movie cameras capable of shooting moving images have become widespread. Digital movie cameras have improved image quality year by year, and those compatible with full HD (High Definiton) image quality have also been put into practical use. Accordingly, H.H. has high image compression efficiency. Digital movie cameras that compress and code moving images according to the H.264 / AVC standard have also been put into practical use.

  Among such digital movie cameras, there are those that extract and track object feature amounts such as the shape and color of a specific object. Using such a digital movie camera, for example, the user can shoot while tracking a child running in an athletic meet so as to be continuously within a predetermined region of interest. (For example, Patent Document 1)

JP-A-7-95597

  In addition, with the development of wireless communication technologies such as mobile phones and wireless local area networks (LANs), large amounts of data can be transmitted at high speed even in wireless communication networks. For this reason, even in a digital movie camera, it is demanded not only to record a captured moving image on a recording medium but also to transmit it via a wireless communication network.

  However, the transmission rate in the wireless communication network is easily affected by the propagation environment, and a sufficient transmission rate may not be ensured. On the other hand, when shooting while tracking a specific object, the user takes a moving image in the attention area including the specific object as a main viewing target. In many cases, it is not necessary to transmit a moving image of the entire screen.

  The present invention has been made in view of such a situation, and an object thereof is to provide an image transmission apparatus capable of efficiently transmitting a moving image in a region of interest and an imaging apparatus equipped with the image transmission apparatus.

  One embodiment of the present invention is an image transmission device. The image transmission device includes a setting unit that sets a region of interest on the image so as to include a specific object on the image, and an encoding unit that encodes an image in the region of interest and generates region-of-interest encoded data A transmission unit that transmits the region-of-interest encoded data, a transmission rate of the previous region-of-interest encoded data acquired by the transmission unit, the region-of-interest encoded data predicted by the encoding unit, and the specific object An image transmission device comprising: a processing unit that changes a ratio of the specific object in the region of interest according to a code amount.

  Further, the processing unit is configured such that the transmission rate is smaller than a first threshold, the code amount of the attention area encoded data is larger than a second threshold, and the code amount of the specific object is When larger than a 3rd threshold value, it is preferable to perform the process which reduces the ratio for which the said specific object accounts in the said attention area.

  Further, the processing unit is configured such that the transmission rate is smaller than a first threshold, the code amount of the attention area encoded data is larger than a second threshold, and the code amount of the specific object is When it is smaller than the third threshold value, it is preferable to execute a process of expanding the proportion of the specific object in the attention area.

  According to the present invention, it is possible to improve the transmission efficiency of moving images within a region of interest.

1 is a conceptual diagram illustrating a configuration of an imaging device 1 according to an embodiment of the present invention. Flow chart for changing the proportion of a specific object in a region of interest

  The outline will be described before the present invention is specifically described. Embodiments described herein relate generally to an imaging apparatus that can capture an image while tracking a specific object. When a specific object such as a person is detected in a frame image included in a moving image, the imaging apparatus sets a region of interest on the frame image (hereinafter also referred to as an entire region) so as to include the object. Further, the imaging apparatus extracts a feature amount (for example, the shape of a human face) from the object, and tracks the object. Then, the attention area is caused to follow the movement of the object so that the object continuously fits in the attention area.

  Furthermore, the imaging apparatus transmits a moving image in the region of interest in a wireless communication network (for example, a wireless LAN such as IEEE 802.11n). When data is transmitted through a wireless communication network, the transmission speed varies depending on the propagation environment, so that there may be a case where a moving image in the region of interest cannot be transmitted within a predetermined period.

  Therefore, in the imaging device according to the embodiment of the present invention, the transmission rate of the wireless communication network is acquired, and the code amount generated when the moving image in the attention area is encoded and the object are encoded. The amount of code occupied by the object in the attention area is variably controlled according to the transmission speed.

  Thereby, the code amount of the object included in the attention area can be adjusted according to the transmission speed, and the moving image in the attention area can be efficiently transmitted according to the propagation environment.

  FIG. 1 is a conceptual diagram illustrating a configuration of an imaging apparatus 1 according to an embodiment of the present invention. The imaging device 1 includes an imaging unit 10 and an image transmission device 20.

  The imaging unit 10 acquires a moving image and supplies it to the image transmission device 20. The imaging unit 10 is a solid-state imaging device such as a CCD (Charge Coupled Devices) sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, or a signal processing unit that changes an analog three primary color signal from the solid-state imaging device to a digital luminance signal and a color difference signal. including.

  The image transmission apparatus 20 uses the moving image acquired from the imaging unit 10 as, for example, H.264. According to the H.264 / AVC standard, encoded data (hereinafter also referred to as whole area encoded data) is generated by compression encoding and recorded on a recording medium. Further, the image transmission apparatus 20 generates encoded data (hereinafter, also referred to as attention area encoded data) by compressing and encoding the moving image in the attention area set on the moving image, via the wireless communication network. The image is output to an image receiving device (not shown).

  The image transmission apparatus 20 includes an object extraction unit 201, an attention area setting unit 202, an attention area processing unit 203, an encoding unit 204, a recording unit 205, a transmission unit 206, and a control unit 207.

  The object extraction unit 201 detects a specific object from the frame image included in the moving image received from the imaging unit 10. The specific object to be detected by the object extraction unit 201 can be designated by the user. Specific examples of the object include a person, a pet such as a dog or a cat, and a moving body such as a car or an airplane. In the following, for convenience of explanation, it is assumed that a person is detected as a specific object.

  The object extraction unit 201 identifies a person by detecting a face in the frame image. A torso area proportional to the size of the face area is set below the face area including the detected face. The face detection may be performed by a known method, and in the embodiment of the present invention, the edge detection method is used.

  The object extraction unit 201 tracks a person by searching for a color area similar to the color of the body area in the subsequent frame image. If the result of face detection in the subsequent frame image is taken into account, tracking accuracy can be improved.

  The attention area setting section 202 sets the attention area on the frame image so as to include the person detected by the object extraction section 201. The attention area may include a whole shape of a person and a rectangular shape including the surrounding area. The attention area may be set by the user, or the position information and the size information in the entire area of the person detected from the object extraction unit 201 may be received and automatically set.

  The attention area setting unit 202 detects the motion vector of the entire person from the tracking status of the object extraction unit 201, moves the attention area according to the motion vector, and causes the attention area to follow the movement of the person.

  Further, the attention area setting unit 202 executes processing for changing the ratio of the person in the attention area with respect to the attention area moving image according to the instruction of the control unit 207. For example, when the control unit 207 instructs to increase the proportion of the person in the attention area, the attention area setting unit 202 sets the attention area (hereinafter referred to as the first attention area) to include the target person. And an attention area (hereinafter also referred to as a second attention area) that includes the target person and is smaller than the first attention area is set based on the enlargement magnification designated by the control unit 207. The size of the second region of interest is smaller than the size of the first region of interest. For this reason, the proportion of the person in the second region of interest is greater than the proportion of the person in the first region of interest.

  On the other hand, when the control unit 207 instructs to reduce the proportion of the person in the attention area, the attention area setting unit 202 cancels the first attention area and sets the reduction magnification specified by the control unit 207 to the reduction magnification. Based on this, an attention area that includes the target person and is larger than the first attention area (hereinafter also referred to as a third attention area) is set. The size of the third region of interest is larger than the size of the first region of interest. For this reason, the proportion of the person in the third region of interest is smaller than the proportion of the person in the first region of interest.

  The attention area processing unit 203 receives position information, shape information, and the like of the attention area on the frame image from the attention area setting unit 202. With reference to these pieces of information, an image in the attention area is extracted from the frame image included in the moving image received from the imaging unit 10.

  As a result, the attention area processing unit 203 uses the image in the attention area extracted for each frame image as a unit image (hereinafter, also referred to as attention area unit image), and selects the attention area moving image in which the attention area unit images are continuous. Constitute.

  The attention area processing unit 203 includes the attention area moving image, position information and shape information of the attention area (hereinafter also referred to as first position information) on the frame image received from the attention area setting unit 202, and person position information. And shape information (hereinafter also referred to as second position information) are supplied to the encoding unit 204.

  The encoding unit 204 performs motion compensation processing, orthogonal transform processing, quantization processing, entropy encoding processing, and the like on the moving image received from the imaging unit 10. The entire area encoded data is generated by a compression encoding method compliant with the H.264 / AVC standard and supplied to the recording unit 205.

  The same processing is executed for the attention area moving image received from the attention area processing unit 203, and Region-of-interest encoded data is generated by a compression encoding method compliant with the H.264 / AVC standard and supplied to the transmission unit 206.

  Further, the encoding unit 204 refers to the first and second position information and the like for each of the entire image in the attention area and the person included in the attention area. The amount of code generated when a compression coding system compliant with the H.264 / AVC standard is applied is calculated. In general, moving images have a correlation in the time direction, and the amount of code when the encoding target frame image is encoded is close to the amount of code when the frame images preceding and following in the time direction are encoded. There are many. Therefore, the encoding unit 204 determines the entire image in the target region to be encoded and the code amount when the person included in the target region is encoded in the target region to be encoded next. And the predicted code amount of the person included in the region of interest are supplied to the control unit 207.

  In the above description, when the control unit 207 instructs to increase the proportion of the person in the attention area, the attention area setting unit 202 executes the process of making the area where the image should be extracted smaller than the attention area. did. On the other hand, when the control unit 207 instructs to reduce the proportion of the person in the attention area, the attention area setting unit 202 performs a process of making the area from which the image is to be extracted larger than the attention area.

  However, the present invention is not limited to this, and the attention area processing unit 203 may execute a process of changing the proportion of the person in the attention area. For example, if the control unit 207 instructs the attention area processing unit 203 to increase the proportion of the person in the attention area, the attention area processing unit 203 performs interpolation processing on the pixel data of the person. That is, the attention area processing unit 203 identifies a macroblock including person pixel data based on the position information and shape information of the person on the frame image received from the attention area setting unit 202. Based on the enlargement magnification designated by the control unit 207, a coefficient of an FIR (Finite Impulse Response) filter is calculated, and an interpolation filter having a predetermined passband characteristic is configured. An interpolation filter constructed while performing zero padding at predetermined intervals in the horizontal direction and the vertical direction is applied two-dimensionally to the specified macroblock pixel data to enlarge the shape of the person.

  When the control unit 207 instructs to reduce the proportion of the person in the attention area, the attention area processing block 203 performs a thinning process on the pixel data of the person. That is, the attention area processing unit 203 identifies a macroblock including person pixel data based on the position information and shape information of the person on the frame image received from the attention area setting unit 202. Based on the reduction ratio designated by the control unit 207, the coefficient of the FIR filter is calculated, and a thinning filter having a predetermined passband characteristic is configured. While the interpolation filter constructed in the horizontal direction and the vertical direction is applied two-dimensionally to the pixel data of the specified macroblock, the pixel data is thinned out at a predetermined interval to reduce the shape of the person.

  When the ratio of the person in the attention area to the attention area moving image is changed according to an instruction from the control section 207, the attention area processing section 203 displays the frame image on the frame image in addition to the first and second position information. The position information and shape information (hereinafter also referred to as third position information) of the person after the change is supplied to the encoding unit 204.

  At this time, the encoding unit 204 refers to the first to third position information and the like for each of the whole image in the attention area and the person included in the attention area. The amount of code generated when a compression coding system compliant with the H.264 / AVC standard is applied is calculated.

  Furthermore, when the attention area processing unit 203 performs the thinning process on the pixel data of the person, the encoding unit 204 performs the thinning process based on the second and third position information received from the attention area processing unit 203. A region surrounded by the boundary line along the shape of the person before the execution of the thinning process and the boundary line along the shape of the person after the thinning process is specified. Then, the encoding unit 204 performs compensation processing on the macroblock included in the identified region.

  In other words, the pixel data of the macroblock included in the specified area is predicted from the background pixel data of the person included in the same frame image and the background pixel data of the person included in the frame image that moves back and forth in the time direction. Then, the pixel data of the macro block included in the area specified by the predicted pixel data is compensated. Further, instead of compensating with the predicted pixel data, the pixel data of the macroblock included in the area specified to be displayed in a specific color (for example, blue) may be compensated. In addition, information related to the thinning process of a person (for example, a reduction ratio) is included in the region-of-interest encoded data and transmitted to the image receiving apparatus, and before the thinning process is executed on the image receiving apparatus side based on the information. You may expand to the shape of a person.

  The recording unit 205 records the entire area encoded data received from the encoding unit 204 on a recording medium such as an HDD (Hard Disk Drive), an optical disk, or a flash memory.

  The transmission unit 206 stores the attention area encoded data received from the encoding unit 204 in a transmission buffer (not shown). When the attention area encoded data stored in the transmission buffer reaches a predetermined transmission unit (for example, 1500 bytes), the attention area encoded data is read from the transmission buffer, converted into a data format, and transmitted to the wireless communication network. To do.

  In addition, the transmission unit 206 measures the number of bytes of attention area encoded data that can be transmitted within a predetermined period (for example, a moving image frame period), and calculates the transmission rate of the wireless communication network for each predetermined period. To the control unit 207.

  The control unit 207 controls the entire image transmission apparatus. In addition, the control unit 207 determines the entire image in the target region to be encoded received from the encoding unit 204, the predicted code amount of the person included in the target region, and the transmission speed of the wireless communication network received from the transmission unit 206. Based on this, it is determined whether or not the ratio of the person in the attention area should be changed.

  The control unit 207 performs a simulation or experiment on a threshold (hereinafter, also referred to as a first threshold) regarding the transmission speed of the wireless communication network and a threshold (hereinafter, referred to as a predicted code amount of the entire image in the region of interest). A threshold value (hereinafter also referred to as a third threshold value) relating to a predicted code amount of a person included in the attention area (hereinafter also referred to as a third threshold value). By comparing each of the prediction code amount of the entire image and the prediction code amount of the person included in the attention area with these threshold values, it is determined whether or not the ratio of the person in the attention area should be changed.

  That is, the control unit 207 has a transmission rate of the wireless communication network received from the transmission unit 207 smaller than the first threshold, and the predicted code amount of the entire image in the attention area received from the encoding unit 204 is the first. If the predicted code amount of the person included in the attention area is larger than the third threshold value, the instruction area processing unit 203 is instructed to reduce the proportion of the person in the attention area. To do.

  When the transmission rate is smaller than the first threshold value, the amount of information that can be transmitted to the wireless communication network within a predetermined period is also small. Therefore, it is preferable that the code amount of the attention area encoded data is as small as possible. In addition, when the predicted code amount of the person included in the attention area is larger than the third threshold value, the contribution of the person encoding to the code amount of the attention area encoded data excludes the person in the attention area. This is considered to be larger than the contribution of the region coding.

  Therefore, the control unit 207 determines that the predicted code amount of the entire image in the attention area is larger than the second threshold value, and the code amount of the attention area encoded data is too large compared to the transmission speed in the wireless communication network. In this case, if it is determined that the main factor is the encoding of the person included in the attention area, the proportion of the person in the attention area is reduced.

  If the proportion of the person in the region of interest decreases, the number of macroblocks including the pixel data of the person to be encoded decreases, and the size of the person's motion vector also decreases according to the proportion. For this reason, the amount of codes when a person is encoded can be reduced. Since the code amount of the person who greatly contributes to the code amount of the attention area encoded data can be reduced, the code amount of the attention area encoded data can be reduced as a result.

  On the other hand, the control unit 207 has a transmission rate of the wireless communication network received from the transmission unit 207 smaller than the first threshold value, and the predicted code amount of the entire image in the attention area received from the encoding unit 204 is the first. When the predicted code amount of the person included in the attention area is smaller than the third threshold value, the instruction area processing unit 203 is instructed to increase the proportion of the person in the attention area. To do.

  When the predicted code amount of the person included in the attention area is smaller than the third threshold value, the contribution of the person's encoding to the code amount of the attention area encoded data is that of the area excluding the person in the attention area. This is considered to be smaller than the contribution of encoding.

  Therefore, the control unit 207 determines that the predicted code amount of the entire image in the attention area is larger than the second threshold value, and the code amount of the attention area encoded data is too large compared to the transmission speed in the wireless communication network. In this case, when it is determined that the main factor is the coding of the area excluding the person in the attention area, the ratio of the person in the attention area is increased.

  When the proportion of the person in the attention area increases, the number of macroblocks included in the area excluding the person in the attention area decreases. For this reason, it is possible to reduce the amount of code when the area excluding the person in the attention area is encoded. Since the code amount of a region excluding a person who greatly contributes to the code amount of the attention area encoded data can be reduced, the code amount of the attention area encoded data can be reduced as a result.

  FIG. 2 is a flowchart showing a procedure for changing the proportion of the specific object in the attention area. The encoding unit 204 calculates a code amount generated by encoding the entire image in the attention area set on the frame image Fn to be encoded at the time Tn, and calculates the code amount of the entire image in the attention area. The prediction code amount is supplied to the control unit 207 (S10). In addition, the encoding unit 204 calculates a code amount generated by encoding the person included in the attention area set on the frame image Fn that is the encoding target at the time point Tn, and is included in the attention area. The predicted code amount of the person is supplied to the control unit 207 (S12).

  The transmission unit 206 refers to the remaining amount of the transmission buffer at the time Tn, calculates the transmission rate of the wireless communication network, and supplies it to the control unit 207 (S14).

  When the frame image Fn + 1 becomes the encoding target frame image at time Tn + 1, the control unit 207 compares the transmission rate with the first threshold value (S16). When the transmission rate is smaller than the first threshold value (Y in S16), the control unit 207 compares the predicted code amount of the entire image in the region of interest with the second threshold value (S18). When the predicted code amount of the entire image in the attention area is larger than the second threshold value (Y in S18), the control unit 207 sets the predicted code amount of the person included in the attention area as the third threshold value. Compare (S20). When the expected code amount of the person included in the attention area is larger than the third threshold value (Y in S20), the control unit 207 determines the person included in the attention area set on the frame image Fn + 1. The attention area processing unit 203 is instructed to increase the proportion of the attention area (S22). On the other hand, when the expected code amount of the person included in the attention area is smaller than the third threshold (N in S20), the control unit 207 displays the person included in the attention area set on the frame image Fn + 1. Is instructed to the attention area processing unit 203 so as to reduce the proportion of the attention area to the attention area (S24).

  According to such an embodiment of the present invention, the following operational effects can be enjoyed.

  (1) The ratio of the person included in the attention area to the attention area is changed based on the transmission speed of the wireless communication network, the prediction code amount of the entire image in the attention area, and the prediction code amount of the person included in the attention area. Therefore, it is possible to realize transmission of attention area encoded data that is optimal for the propagation environment of the wireless communication network.

  (2) Prediction codes of persons included in the attention area whose transmission speed of the wireless communication network is smaller than the first threshold value and whose prediction code amount of the entire image in the attention area is larger than the second threshold value When the amount is larger than the third threshold value, the ratio of the person included in the attention region to the attention region is reduced, so that the code amount can be adjusted without increasing the quantization scale or reducing the resolution. It is possible to realize transmission of attention area encoded data that is high quality and optimal for the propagation environment.

  (3) Predictive code of a person whose transmission speed of the wireless communication network is smaller than the first threshold and whose prediction code amount of the entire image in the attention area is larger than the second threshold and included in the attention area When the amount is smaller than the third threshold value, the ratio of the person included in the attention area to the attention area is increased, so that the code amount can be adjusted without increasing the quantization scale or decreasing the resolution. It is possible to realize transmission of attention area encoded data that is high quality and optimal for the propagation environment.

  Therefore, the transmission efficiency of attention area coding data can be improved.

  As mentioned above, although the form for implementing this invention has been demonstrated, this invention is not limited to the structure of this embodiment, It exists in the application range of this invention prescribed | regulated by the claim. Various modifications are possible as long as the functions of the configuration of the above-described embodiment can be achieved.

  For example, in the embodiment of the present invention, the image transmission device 20 is the H.264 standard. The compression encoding is performed according to the H.264 / AVC standard, but the compression encoding may be performed according to a standard such as MPEG-2 or MPEG-4.

DESCRIPTION OF SYMBOLS 201 Object extraction part, 202 Attention area setting part, 203 Attention area process part, 204 Encoding part, 205 Recording part, 206 Transmission part, 207 Control part.

Claims (4)

A setting unit for setting a region of interest on the image so as to include a specific object on the image;
An encoding unit that encodes an image in the region of interest to generate region-of-interest encoded data, and predicts a code amount of the region-of-interest encoded data and a code amount of the specific object;
A transmitter that transmits the attention area encoded data and obtains a transmission rate at the time of the attention area encoded data transmission;
The specific object occupies the attention area according to the transmission speed of the preceding attention area encoded data acquired by the transmission unit, the attention area encoded data predicted by the encoding section, and the code amount of the specific object. A processing unit for changing the ratio;
An image transmitting apparatus comprising:   The processing unit is configured such that the transmission rate is smaller than a first threshold, the code amount of the attention area encoded data is larger than a second threshold, and the code amount of the specific object is third. 2. The image transmission device according to claim 1, wherein when the threshold value is larger than the threshold value, a process of reducing a ratio of the specific object in the attention area is executed.   The processing unit is configured such that the transmission rate is smaller than a first threshold, the code amount of the attention area encoded data is larger than a second threshold, and the code amount of the specific object is third. 2. The image transmission device according to claim 1, wherein when the threshold value is smaller than the threshold value, a process of enlarging a ratio of the specific object in the attention area is executed. An imaging unit for acquiring a moving image;
The image transmission device according to any one of claims 1 to 3, which processes a moving image acquired by the imaging unit;
An imaging apparatus comprising:

Images