JP2020120254A - Distribution image generation method - Google Patents

Abstract

To provide an image free from discomfort for a user viewing the image while suppressing the amount of information when the image is distributed.SOLUTION: A distribution image generation method for generating a distribution image 52 distributed via a network communication line 3 includes: a step (s202) of acquiring a field angle from a distribution destination terminal of the distribution image 52; a step (s203) of acquiring an image generated by capturing at least all directions; and a generation step (s204) of generating a distribution image by extracting pixels from the acquired image. In the generation step, pixels are thinned out and extracted from the acquired image based on a function determined according to the acquired field angle.SELECTED DRAWING: Figure 3

Description

The present invention relates to a distribution image generation method.

There is known an image distribution system in which an image captured by a camera is received by a computer such as a mobile information terminal or a personal computer at a remote place away from the camera, and the received image is displayed on a display screen of the computer. Such an image distribution system is used, for example, as a monitoring system for monitoring the inside of a room that is away. In recent years, an image distribution system that has a camera for generating images of the entire sky and is capable of distributing a wider range of images has appeared.

For example, Patent Document 1 describes a video distribution device that panoramicly expands a panoramic image captured by a camera by an equidistant cylinder projection method and distributes the expanded panoramic image. In the video distribution device, by superimposing a portion of the user's gaze on a low-resolution omnidirectional image with a high-definition image and transmitting the image, the quality of the video provided to the user is improved, This makes it possible to reduce the amount of information when distributing video.

JP, 2016-15705, A

However, simply superimposing the high-resolution image and the low-resolution image on each other may cause a boundary due to the difference in resolution to appear in the image, which may give an uncomfortable feeling to a user who views the image.

In view of the above problems, it is an object of the present invention to provide a delivery image generation method that suppresses the amount of information when delivering an image and does not give a feeling of strangeness to a user who views the image. ..

In order to achieve the above object, the present invention is a distribution image generation method for generating a distribution image distributed via a network, the method comprising: acquiring an angle of view from a distribution destination terminal of the distribution image, at least in all directions. The method includes a step of acquiring a captured image generated and a generation step of generating a distribution image by extracting pixels from the acquired image, wherein the generation step is based on a function determined according to the acquired angle of view. Pixels are thinned out and extracted from the acquired image.

Further, the function is characterized in that the thinning interval increases as the distance from the center of the distribution image to the outside increases.

Further, the function defines the correspondence between the angle with respect to the user's line of sight seen from the center of the solid body to which the image is mapped and the distance from the center of the distribution image, and the angle when the distance is maximum. Is 180 degrees.

In addition, the function arranges the pixels on the solid within the angle of view inside a circular area included in the distribution image, and the pixels on the solid outside the angle of view of the circular area. It is characterized in that it is arranged to be arranged outside.

Further, the circular area is provided so as to be in contact with an edge of the distribution image.

Further, the function is defined such that an angle with respect to a line of sight of the user when the distance is a radius of a circle forming an outer shape of the circular region is the view angle.

According to the present invention, the distribution image to be distributed to the computer is generated such that the image quality of the image gradually decreases according to the function as it goes outside the peripheral portion. Therefore, while suppressing the amount of information when delivering the image, the user who views the image does not feel uncomfortable.

Schematic diagram of an image distribution system according to an embodiment of the present invention Block diagram of the image distribution system Flow chart of the image distribution system Diagram showing the image generation process in the generation process of the above flow diagram Diagram showing the correspondence between distribution images and virtual spheres Diagram showing the correspondence relationship when generating the low resolution part in the distribution image Diagram showing example functions

As shown in FIG. 1, the image distribution system 10 according to the present embodiment includes an image distribution device 12 and a mobile information terminal 14 connected to a network communication line 3. A peer-to-peer connection is established between the image distribution device 12 and the portable information terminal 14 by a signaling server (not shown), and they can communicate with each other.

The image distribution device 12 is a device that distributes an image through the network communication line 3, and includes a camera module 16 for photographing the surroundings. The camera module 16 is provided with two image sensors 18 and 19. These image sensors 18 and 19 output electric signals according to the intensities of the light formed on the rectangular light receiving surfaces 18a and 19a, respectively, and the light receiving surfaces 18a and 19a are opposite to each other. It is arranged as follows. Further, an optical component (fisheye lens) (not shown) for forming an image of a semi-circumference (image circle) is provided at a predetermined position so as to fit on the light receiving surfaces 18a, 19a of the image sensors 18, 19, respectively. The semi-periphery is a hemisphere with an infinite radius that is virtual about the light receiving surfaces 18a and 19a of the image sensors 18 and 19.

The camera module 16 further includes an analog-digital converter (not shown), and the analog signal output from each of the image sensors 18 and 19 is converted into a digital signal by the analog-digital converter. To be done. This digital signal is input to the CPU 20 of the image distribution device 12, and a captured image is generated. Accordingly, in the present embodiment, the two fisheye images formed on the image sensors 18 and 19, that is, the one fisheye including the one-direction semi-periphery image formed on one of the image sensors 18. Another fish-eye image including the image and the image of the other half of the sky formed on the other image sensor 19 is generated.

The image distribution device 12 includes a memory 22 in which an image distribution program is stored in advance, and a communication module 24 for connecting to the network communication line 3. As the image distribution program is executed by the CPU 20, the CPU 20 receives the angle-of-view information and attitude information described below from the portable information terminal 14 via the communication module 24 (FIG. 1), as shown in FIG. 26, the generation unit 28 that generates the distribution image 52 (FIG. 4) based on the digital signal acquired from the camera module 16, and the distribution unit 30 that distributes the distribution image 52 via the communication module 24.

Returning to FIG. 1, the portable information terminal 14 is typically a terminal such as a smartphone or a tablet, and in addition to the CPU 32 and the memory 34, a touch panel that functions as an input unit (not shown) that receives an input from a user. 36, a liquid crystal display 40 functioning as a display unit 38 (FIG. 2) for displaying information to the user, a communication module 42 for connecting to the network communication line 3, and an angular velocity for detecting the attitude of the mobile information terminal 14. The sensor 44 is provided.

An application for displaying the distribution image 52 acquired from the image distribution device 12 on the liquid crystal display 40 is stored in advance in the memory 34 of the mobile information terminal 14, and the CPU 32 executes the application, as shown in FIG. In addition, a transmitting unit 46 that transmits angle-of-view information and posture information described below to the image distribution device 12, a receiving unit 48 that receives the distribution image 52 from the image distribution device 12, a developing unit 50 that expands the received distribution image 52, and The display unit 38 functions to display the developed image on the liquid crystal display 40.

The image delivery flow in this embodiment will be described below with reference to FIG.

When the application is executed in the mobile information terminal 14, the mobile information terminal 14 executes a connection process (s101) for establishing a peer-to-Peer connection with the image distribution device 12.

When the communication with the image distribution device 12 is established by the connection process (s101), the portable information terminal 14 executes the acquisition process (s102) for acquiring the angle-of-view information and the posture information. The posture information is information indicating the posture of the portable information terminal 14 operated by the user, and typically, the roll angle (α) and the pitch angle determined based on the signal output from the angular velocity sensor 44. (Β) and yaw angle (γ). The angle-of-view information is information that indicates a range to be extracted from the captured image on the liquid crystal display 40 of the mobile information terminal 14 as a portion to be noticed by the user, and is used for the user's enlargement/reduction display operation on the touch panel 36. It is decided based on.

When the acquisition process (s102) is executed, the portable information terminal 14 executes a request process (s103) for requesting image distribution. In the request process (s103), the angle-of-view information and the posture information are transmitted to the image delivery device 12.

In the image distribution device 12, when the power is turned on and the connection processing (s201) is executed, the reception processing (s202) for receiving the angle-of-view information and the attitude information transmitted from the mobile information terminal 14 is executed.

When the image distribution device 12 receives the angle-of-view information and the posture information, the image distribution device 12 executes the photographing process (s203). The image capturing process (s203) is a process of capturing an image of a semi-periphery in two directions centering on the image sensors 18 and 19, and the CPU 20 controls the image sensors 18 and 19 and the analog-digital converter to perform analog- Two captured images are generated based on the digital signal input from the digital converter. Each captured image generated by the capturing process (s203) is a fisheye image. Specifically, it has a rectangular image area, and a circular semi-peripheral area including a semi-peripheral image formed on the light-receiving surfaces 18a, 19a of the image sensors 18, 19 in the area, and the semi-peripheral area. Information of the sky (black) generated around the area (four corners) is arranged.

When the image distribution device 12 executes the shooting process (s203), the image distribution device 12 executes the generation process (s204) for generating the distribution image 52 (FIG. 4B). The distribution image 52 is an image that is a source of the image 54 (FIG. 4A) that is developed and displayed on the mobile information terminal 14, and the pixels are partially extracted from the captured image based on the angle-of-view information and the posture information. It is generated by extracting it selectively. When the distribution image 52 is generated, the image distribution device 12 executes the distribution process (s206) and distributes the distribution image 52 to the mobile information terminal 14.

The mobile information terminal 14 executes a reception process (s104) for receiving the distribution image 52 from the image distribution device 12. When the distribution image 52 is received by the processing, the portable information terminal 14 executes the expansion processing (s105) for expanding the distribution image 52 into the image 54. When the expansion process (s105) is executed, the portable information terminal 14 executes the display process (s106) for displaying the image 54 on the liquid crystal display 40. After the display process (s106), each time the user changes the attitude of the mobile information terminal 14, the above processes (s102) to (s106) are executed by the mobile information terminal 14, and the processes (s202) to (s206). Is executed by the image distribution device 12.

Here, as shown in FIG. 4A, the image 54 developed and displayed on the portable information terminal 14 is configured such that the peripheral portion of the image has a lower image quality than the inner (center) portion thereof. There is. In other words, the inner portion of the image 54 that is viewed by the user has a higher image quality than the peripheral portion.

As shown in FIG. 4B, the distribution image 52 that is the source of such an image 54 is a rectangular image, and the portion displayed with the high image quality is based on the center of the distribution image 52. It spreads within a circle (inscribed circle) inscribed on four sides (each edge), and the above-mentioned low image quality portion spreads from the outer circumference of the circle to the four corners of the distribution image 52. Each pixel of such a distribution image 52 is extracted from the photographed image and arranged by a predetermined calculation by the CPU 20. In the present embodiment, a pixel corresponding to each pixel in the distribution image 52 is extracted from a sphere in which two captured images are virtually mapped (hereinafter referred to as a virtual sphere 56) based on angle-of-view information and posture information. Is calculated so that

Specifically, as shown in FIG. 4C, the first coordinate on the surface of the virtual sphere 56 corresponding to each pixel of the distribution image 52 is calculated according to the angle of view information by the correspondence equation (first calculation). Step 2), the second coordinate is calculated by applying the rotation formula including the posture information to the first coordinate (second calculating step), and the second coordinate is located at the second coordinate as shown in FIG. 4D. Pixel information is extracted.
For convenience of explanation, the coordinates of the distribution image 52 are shown by XY orthogonal coordinates with the center as the origin, as shown in FIG. 5B, and the horizontal direction (X axis) of the distribution image 52 is −1≦X. It takes a value of ≦1 and a value of −1≦Y≦1 in the vertical direction (Y axis). Further, the coordinates of the phantom sphere 56 are indicated by XYZ orthogonal coordinates whose origin is the center thereof, as shown in FIG. 5A, and the radius r of the phantom sphere 56 is 1.

The above-described first calculation step is a step of calculating the first coordinates for thinning and extracting pixels from the captured image based on a function that is determined according to the angle-of-view information, and the virtual coordinates corresponding to each pixel of the distribution image 52. A spherical coordinate calculation step of calculating spherical coordinates (r, θ, φ) of the sphere 56 based on a function, and an orthogonal coordinate calculation step of calculating orthogonal coordinates (x, y, z) corresponding to the calculated spherical coordinates. Including.

The spherical coordinate calculation step will be described by taking the point P of the distribution image as an example. The spherical coordinates of the point P′ on the phantom sphere 56 corresponding to the point P are represented by (r, θ _{p ′} , φ _{p ′} ), and the angle θ _{p ′} (angle with respect to the Z axis of the phantom sphere 56). And the angle φ _{p ′} (angle with respect to the X-axis) is obtained as follows. The radius r of the phantom sphere 56 is 1 as described above.

The angle θ _{p ′} is obtained by inputting the distance P _r from the origin to the point P in the XY orthogonal coordinates of the distribution image 52 to the function f(r) corresponding to the angle-of-view information. The distance P _r is obtained by the following correspondence formula based on the coordinate value (P _x , P _y ) of the point P.

As shown in FIG. 7, the function f(r) defines the relationship between the distance r and the angle θ, and is defined for each of a plurality of pieces of view angle information. For example, when the angle of view information is 30°, the function f(r) is determined so that θ becomes 30° when r=1 as shown in FIG. 7A. for the function, the angle theta _P'is obtained which corresponds to the distance P _r assignment has been the point P determined in Equation 1. Further, when the angle-of-view information is 90°, the function is determined so that θ becomes 90° when r=1 as shown in FIG. 7B. By substituting the distance P _r found in the _{equation 1,} the angle θ P ′ corresponding to the point P is found. That is, each function is defined so that the boundary between the high pixel portion and the low pixel portion in the distribution image 52 corresponds to the view angle information. The function may be a linear function as shown in FIG.

The angle φ _{p ′} is the same as φ _p in the XY orthogonal coordinates of the distribution image 52, and the φ _p is obtained by the following correspondence formula based on the coordinates (P _x , P _y ) of the point P.

Here, as shown in FIG. 6, when the angle φ is obtained for the pixels forming the low image quality portion, for example, the pixels on the circumference C, as in the above correspondence equation (Equation 2), Pixel information that is biased is extracted in which pixels located on the arc indicated by the broken line (broken arc) are not considered, and only pixels corresponding to the arc indicated by the alternate long and short dash line are considered. Therefore, in this embodiment, the pixels to be extracted are selected based on the ratio of the broken line arc to the circumference. Specifically, by uniformly arranging the points on the circumference C including the broken line portion on the alternate long and short dash line, it is possible to reduce the information amount of the distribution image 52 while realizing the extraction of pixel information with less deviation. ing. Therefore, for example, the pixel information corresponding to the pixel Q′ is extracted for the pixel Q on the circumference C. The corresponding equation for realizing such an even array is the following equation.

Here, φ _i is an angle for obtaining the ratio of the broken line arc to the circumference C.

When the spherical coordinates (1, θ, φ) for each pixel of the distribution image 52 are obtained as described above, the first coordinate (x ₁ , y ₁ for each pixel is calculated by the following conversion formula in the Cartesian coordinate calculation step. , Z ₁ ) is required.

Then, the second calculation step is executed. In the second calculation step, for each first coordinate, a second coordinate (x _{2, y} 2, _{z 2)} is obtained by applying the rotary including orientation information.

By the second calculation step, the pixels to be extracted in the phantom sphere 56 are specified. Specifically, as shown in the coordinate diagram 4(c) obtained in the first calculation step, when the second coordinate is obtained, the pixel information located at the coordinate is extracted, and the extracted pixel information corresponds to the extracted pixel information. Assigned to each pixel of the distribution image 52. In this way, the pixels on the virtual sphere 56 corresponding to the angle of view are extracted in the shape of a fisheye image within the inscribed circle that is the high-quality portion, and outside the angle of view outside the inscribed circle that is the low-quality portion. The distribution image 52 is generated by thinning and extracting the pixels on the virtual sphere 56.

The coordinates calculated in the first calculation step are to be shifted according to the posture information in the second calculation step. Therefore, the angle θ obtained in the first calculation step is calculated as an angle with respect to the z-axis, but finally, as shown in FIG. 4D, the angle θ corresponds to the posture information from the center of the virtual sphere 56. The angle with respect to the extending half line will be used as an index. Here, the half line extending corresponding to the posture information corresponds to the line of sight of the user seen from the center of the virtual sphere 56. Therefore, the angle θ indicates the angle with respect to the user's line of sight viewed from the center of the virtual sphere 56 onto which the captured image is mapped, and the function f(r) corresponds to the angle and the distance from the center of the distribution image. Will be determined.

Further, as shown in FIGS. 7A and 7B, the function f(r) determined according to the angle-of-view information is determined so that the increasing degree of θ increases as the value of r increases. There is. That is, the function f(r) is determined so that the interval (thinning interval) of the pixels to be extracted in the virtual sphere 56 increases from the center of the distribution image to the outside.

The portable information terminal 14 that has received the distribution image 52 executes the expansion process (s105) for expanding the distribution image 52 into the image 54. In the process, first, a development image is generated based on the distribution image 52. Pixels in the high image quality portion of the distribution image 52 are arranged in the center of the developed image, and pixels in the low image quality portion of the distribution image 52 are arranged around the high image quality portion. Wherein for each pixel of the low-quality portion is not directly arranged to locate and phi _Q'using Equation 4 above, this time of pixels arranged in a specified position, the phi _Q' Since the pixels are not continuously arranged on the circumference C including the interpolation, an interpolation process for interpolating between the pixels is performed. The interpolation process is not particularly limited, but, for example, pixel information approximate to each pixel is arranged between pixels on the same circumference.

When the above interpolation processing is executed, the development image is developed by using a known panoramic development method, and the image 54 is generated.

According to the image distribution system 10 of the present embodiment, in the distribution image 52 distributed from the image distribution device 12 to the mobile information terminal 14, the part most noticed by the user when expanded to the image 54 is generated with high image quality. The surrounding portion that is difficult to be noticed is generated by being compressed to a low image quality. As described above, by reducing the amount of image information, it is possible to provide a high-quality image to the user while realizing high-speed communication. It should be noted that the low image quality portion of the image 54 is the amount of change in the posture until the user receives the new distribution image 52 and the new image 54 is developed after the user changes the posture of the mobile information terminal 14. Only will be displayed on the liquid crystal display 40.

Further, the distribution image 52 is generated such that when it is developed as the image 54 by the portable information terminal 14, the image quality of the image gradually decreases as it goes outside the peripheral portion. Therefore, while suppressing the amount of information when delivering the image, the user who views the image does not feel uncomfortable.

Although the distribution image generation method according to the present invention has been described above based on the embodiment, the present invention is not limited to the above-described mode, and may be carried out in the following modes, for example. Absent.

<Modification>
(1) In the spherical coordinate calculation step of the above-described embodiment, a predetermined function (FIG. 7) associated with the angle-of-view information is used when the angle θ of the spherical coordinates is obtained. However, the function may be generated based on the acquired view angle information. For example, when a linear function or a quadratic function is used, the coefficient corresponding to the view angle information may be determined and the function may be generated. Further, when the exponential function is used, it suffices to determine the base corresponding to the view angle information and generate the function. In order to determine the coefficient and the bottom, only the correspondence between the value of the coefficient and the bottom and the view angle information may be determined in advance.

That is, the function arranges the pixels on the virtual sphere 56 within the view angle indicated by the view angle information inside the circle (inscribed circle) with the center of the distribution image as the base point, and arranges the pixels outside the view angle. The pixels on the sphere 56 are defined to be arranged outside the circle. At this time, the function is determined so that the angle θ becomes the angle-of-view information when the distance (r) takes the value of the radius of the circle (1 in the above embodiment). Such a function is used when the angle θ is 0 (origin) when the distance (r) is 0, and when the distance (r) is the maximum value (distance from the center of the distribution image to the corner of the distribution image). Typical examples are a Bezier curve and a spline curve that are defined so as to pass through a point where the angle θ is 180 and a point where the angle θ is the angle of view when the distance (r) is the radius of the circle.

(2) The mobile information terminal 14 included in the image distribution system 10 is not limited to one, and may be plural. Further, the personal computer is not limited to the portable information terminal 14, and may be a desktop type or notebook type personal computer. That is, it is only necessary that the computer receives the distribution image 52 from the image distribution device 12 and displays the image 54. When a personal computer is used, for example, information (angle-of-view information and distribution image 52) can be transmitted/received to/from the image distribution system 10 via a browser capable of Peer-to-Peer communication. Although the angular velocity sensor 44 is used in the portable information terminal 14 as the input unit for inputting the angle-of-view information, the angle-of-view information may be input by a screen operation using a mouse. The posture information is obtained by calculating a virtual posture based on the position coordinates in the image display unit of the browser designated by the mouse. The angle of view information is obtained based on the rotation operation of the mouse wheel.

(3) In the above embodiment, the image distribution device 12 includes the camera module 16 that captures the entire sky, but the camera module may include only one image sensor 18 that captures the semi-circumference. Absent. The image distribution device having such a camera module is attached to, for example, a ceiling in a room, used as a device for photographing the room and transmitting the state of the room as a distribution image 52. Then, when generating the distribution image 52, the captured image is mapped to the lower side of the virtual sphere 52, and the first coordinate calculating step and the second coordinate calculating step described above are executed.

(4) Further, in the above-described embodiment, the captured image generated by each of the image sensors 18 and 19 is a semi-perimeter image, but it may be an omnidirectional image including at least all azimuths. Here, the omnidirectional image is an image in which all directions in north, south, east, west, and west are imaged, in other words, an image in which a certain point is set as a base point and the periphery of the base point (360 degrees in the horizontal direction) is taken. Such an omnidirectional image may be an image obtained by removing the central portion of the photographed image output from the image sensor 18, or four camera modules having a photographing angle of view of 90 degrees are arranged in north, south, east, west, and The captured images obtained from the camera module may be arranged in the horizontal direction of the virtual sphere and mapped.

(5) In the above embodiment, the captured image generated by the camera module 16 is mapped to the virtual sphere 56, but the present invention is not limited to this mode, and the captured image captured by the camera module is mapped to the virtual cube. It doesn't matter. In the aspect, in the generation process, the first coordinate is obtained based on a predetermined correspondence equation that defines a correspondence relation between the position on the distribution image 52 and the position on the virtual cube, and the predetermined rotation formula is used for the first coordinate. The second coordinate is obtained by applying As a result, the pixel information in the virtual cube is extracted for each pixel of the distribution image 52. Similarly, the captured image may be mapped to the virtual cylinder.

(6) In the above embodiments and modifications, the case where the captured image is mapped to the virtual sphere 56, the virtual cube, or the like has been described as an example. However, the present invention is not limited to this mode, and the captured image is the virtual sphere, the virtual cube, or the like. The pixel information may be expanded in the memory 22 on the basis of a predetermined calculation formula on the assumption that the pixel information is mapped to. In this aspect, for each pixel of the distribution image 52, the first coordinate is obtained based on the above correspondence equation, the second coordinate is obtained based on the above rotation equation, and the pixel is read from the memory address corresponding to the second coordinate. Extract information.

The invention of the present application can be implemented in a mode in which various improvements, modifications, or variations are added based on the knowledge of those skilled in the art without departing from the spirit of the invention. Further, within the scope of producing the same action or effect, any of the matters specifying the invention may be replaced with another technique.

3... Network communication line 10... Image distribution system 12... Image distribution device 14... Portable information terminal (computer)
16... Camera module 28... Generation part 30... Delivery part 38... Display part 48... Receiving part 50... Development part 52... Delivery image 54... Image

Claims (6)

A distribution image generation method for generating a distribution image distributed via a network,
Obtaining an angle of view from a delivery destination terminal of the delivery image,
Acquiring an image generated by imaging at least all directions,
A generation step of generating a distribution image by extracting pixels from the acquired image,
Including
In the generating step, pixels are decimated and extracted from the acquired image based on a function that is determined according to the acquired angle of view. The distribution image generation method according to claim 1, wherein the function increases as the thinning interval increases from the center of the distribution image to the outside. The function defines a correspondence between an angle with respect to the line of sight of the user as viewed from the center of the solid to which the image is mapped and a distance from the center of the distribution image. When the distance is maximum, the angle is 180. The distribution image generation method according to claim 2 or claim 2, wherein the distribution image generation method is performed. The function arranges the three-dimensional pixels within the angle of view inside a circular area included in the distribution image, and the three-dimensional pixels outside the angle of view outside the circular area. The distribution image generation method according to claim 3, wherein the distribution image generation method is determined to be arranged. The distribution image generation method according to claim 4, wherein the circular area is provided so as to contact an edge of the distribution image. The function is defined such that an angle with respect to the line of sight of the user when the distance is a radius of a circle forming the outer shape of the circular region is the angle of view. The distribution image generation method described in.

Images