TWM625817U - Image simulation system with time sequence smoothness - Google Patents

Abstract

An image simulation system is provided, which includes an image capture circuit, a memory and a processor. The image capture circuit is configured to capture an image. The memory is configured to store multiple commands and a weather image. The processor is connected to the image capture circuit, and accesses the multiple commands to perform the following operations: generating a quadruple map and a classification parameter according to the image, where the classification parameter is related to a background and a foreground in the image; adjusting the quadruple map according to the classification parameters, and generate a guide mask according to the adjusted quadruple map, where the guide mask is configured to indicate the background and foreground in the image; and generating a special effects image according to the image, the guide mask, and the weather image, where The special effect image is configured to simulate weather corresponding to the weather image in the image.

Description

Image simulation system with timing smoothness

The present disclosure relates to an image processing technique, and more particularly, to a broken depth map correction system with temporal smoothness.

In recent years, thanks to the rapid development of deep learning in the field of computer vision, many complex problems that were difficult to describe with equations in the past can use convolutional neural networks to achieve good results. However, when it is necessary to generate weather effects on street view images, various problems will arise when the current technology is applied to street view images, resulting in unsatisfactory synthetic effects. Therefore, how to simulate a near-real weather effect on an image is a problem that those skilled in the art are eager to solve.

An aspect of the present disclosure discloses an image simulation system including an image capture circuit, a memory, and a processor. The image capturing circuit is used for capturing images. The memory is used to store a plurality of instructions and a fragmentation depth map. The processor is connected to the image capture circuit and accesses a plurality of instructions to perform the following operations: Generate a quadruplet map and classification parameters according to the image, wherein the classification parameters are related to the background and foreground in the image; adjust the quadruplet map according to the classification parameters, and generate a guide mask according to the adjusted quadruplet map, wherein the guide mask uses to indicate the background and foreground in the image; and generate a special effect image according to the image, the guide mask and the broken depth map, wherein the special effect image is used to simulate the weather corresponding to the broken depth map in the image.

Based on the above, the embodiment of the present disclosure can generate a guide mask according to the classification parameters of the image and the quadruplet map, and use the guide mask to optimize the depth map generated by the depth prediction to clearly distinguish the foreground and the background in the image, thereby preventing the The depth of the part close to the ground can be confused with the depth of the ground.

100: Video Simulation System

110: Image capture circuit

120: memory

130: Processor

S210~S230: Steps

PPM: Preprocessing Model

MGM: Mask Generation Model

TSSM: Time Series Smoothing Model

img:image

IMG': another image

SI: Semantic Segmentation Image

DI: Depth Map

GDI: Virtual Ground Map

GM: Guide Mask

SYI: special effects images

k: the kth secondary node

P_IMG: Probability Map

FIG. 1 is a block diagram of an image simulation system of the present disclosure.

FIG. 2 is a flowchart of the image simulation method of the present disclosure.

FIG. 3 is a schematic diagram of an image simulation method according to some embodiments of the present disclosure.

FIG. 4 is a schematic diagram of auxiliary nodes and classification labels according to some embodiments of the present disclosure.

FIG. 5 is a schematic diagram of a probability map corresponding to shadow probability according to some embodiments of the present disclosure.

Referring to FIG. 1, FIG. 1 is a block diagram of an image simulation system 100 of the present disclosure. In one embodiment, the image simulation system 100 includes an image capture circuit 110 , a memory 120 and a processor 130 . The image capturing circuit 110 is used for capturing images. The memory 120 is used to store a plurality of instructions and a fragmentation depth map. The processor 130 is connected to the image capture circuit 110 and the memory 120, and is used for accessing these instructions.

In some embodiments, the image simulation system 100 can be established by a computer, a server or a processing center. In some embodiments, the image capture circuit 110 may be a camera for capturing images or a camera capable of taking pictures continuously. In some embodiments, the processor 130 may be implemented by a processing unit, a central processing unit, or a computing unit. In some embodiments, the broken depth map may be a depth map corresponding to specific weather that may occur on the road. For example, a depth map of rain, fog, snow, or hail on a road.

In some embodiments, the image simulation system 100 is not limited to include the image capture circuit 110 , the memory 120 and the processor 130 , and the image simulation system 100 may further include other components required for operation and applications, for example, an image The simulation system 100 may further include an output interface (eg, a display panel for displaying information), an input interface (eg, a touch panel, a keyboard, a microphone, a scanner or a flash memory reader), and a communication circuit (eg, WiFi communication model, Bluetooth communication model, wireless telecommunication network communication model, etc.).

Referring to FIG. 2, FIG. 2 is a flowchart of the image simulation method of the present disclosure. The method of the embodiment shown in Fig. 2 is applicable to the image model of Fig. 1 system 100, but not limited thereto. For the sake of convenience and clarity, the following describes the detailed steps of the image simulation method shown in FIG. 2 with reference to both FIG. 1 and FIG.

In one embodiment, the image simulation method includes steps S210 - S230 , all of which can be performed by the processor 130 . First, in step S210, an image is captured, and a quad-connection map and a classification parameter are generated according to the image, wherein the classification parameter is related to the background and the foreground in the image.

In some embodiments, a semantic segmentation process may be performed on the image to generate a semantically segmented image, and a plurality of classification labels corresponding to a plurality of pixels of the image may be generated according to the semantically segmented image. Then, a histogram corresponding to a plurality of numerical categories can be generated according to the RGB values of the plurality of pixels, and classification parameters can be generated according to the plurality of numerical categories and a plurality of classification labels.

In some embodiments, the image may be semantically segmented to generate weights between multiple pixels in the image. Next, a plurality of pixels can be used as nodes in the quadruplet graph, and these weights correspond to the connections between the nodes in the quadruplet graph, wherein the nodes in the quadruplet graph correspond to the pixels in the image in sequence (For example, the nodes in the first column of the quadruplet graph correspond to the pixels in the first column in the image).

In some embodiments, a plurality of numerical categories can be used as a plurality of auxiliary nodes, and the plurality of auxiliary nodes can be connected to the corresponding nodes in the quadruplet graph to generate an adjusted quadruplet graph.

In some embodiments, the plurality of classification labels include foreground labels, background labels, and undetermined labels. In some embodiments, the minimum number may be selected from among the number of foreground labels and the number of undetermined labels corresponding to each of the plurality of numerical categories. Next, the respective minimum numbers of the plurality of numerical categories may be added to generate a minimum sum value, and a classification parameter may be generated based on the minimum sum value and the classification cost parameter.

Furthermore, in step S220, the quadruplet map is adjusted according to the classification parameter, and a guide mask is generated according to the adjusted quadruplet map, wherein the guide mask is used to indicate the background and the foreground in the image.

In some embodiments, the smoothing parameters may be generated from RGB values of each of the plurality of pixels and RGB values of surrounding pixels of each of the plurality of pixels. Next, the quadruplet map may be adjusted according to the smoothing parameter and the classification parameter of each of the plurality of pixels.

In some embodiments, the image can be converted from the RGB domain to the HSV domain to generate the HSV value of the image. Then, a distance calculation may be performed according to the HSV value of the image and the range of shadow values to generate shadow probabilities of a plurality of pixels of the image. Then, the classification parameter and the smoothing parameter can be adjusted according to the shadow probability of the plurality of pixels, and the plurality of classification labels and the quadruplet map can be adjusted according to the adjusted smoothing parameter and the adjusted classification parameter.

In some embodiments, the shade value range may include the value range of the hue channel, the value range of the saturation channel, and the value range of the lightness channel, wherein these value ranges may be the average value of shades obtained in the past, or given manually. The default value of the shadow, or a random value.

In some embodiments, a maximum-flow minimum segmentation operation may be performed on the quad-connection graph on the adjusted smoothing parameter and the adjusted classification parameter to adjust the classification labels corresponding to the pixels of the image, and according to the adjusted Multiple classification labels and quadruplet graphs generate guidance masks.

Furthermore, in step S230, a special effect image is generated according to the image, the guide mask and the broken depth map, wherein the special effect image is used to simulate the weather corresponding to the climate image in the image.

In some embodiments, a depth map may be generated from the image, and the weather corresponding to the broken depth map may be generated on the image using the guide mask and the depth map.

In some embodiments, another image of a frame preceding the image may be captured by the image capturing circuit 110 . In some embodiments, a depth map can be generated from the image, and a point cloud map can be generated from the depth map. Then, the virtual ground position corresponding to the horizon in the image can be identified from the point cloud image, and the guidance mask can be adjusted according to the virtual ground position. Then, timing parameters and edge parameters are generated according to the image, the other image, the adjusted guide mask, and the depth map, wherein the timing parameters are used to resolve the depth discontinuity between the image and the other image, and the edge parameters are used to enhance the Edge enhancement of the depth of an image as well as another image. Then, the depth map can be adjusted according to the timing parameters and the edge parameters, and a special effect image can be generated according to the adjusted depth map and the broken depth map.

In some embodiments, a conjugate gradient descent process can be performed to adjust the depth map according to timing parameters and edge parameters.

In some embodiments, a sparse point cloud image may be generated according to the broken depth map, and the sparse point cloud image and the image may be synthesized according to the adjusted depth map to generate a special effect image.

Through the above steps, the classification parameters can be adjusted to prevent the depth of the part close to the ground in the background from being confused with the depth of the ground, so as to use the adjusted classification parameters to generate a guide mask. In addition, smoothing parameters can be adjusted to prevent noise at the edges of objects in the image. In addition, the timing parameters and the edge parameters can be adjusted to solve the depth discontinuity between the image and another image of the previous frame and to enhance the edge enhancement of the depth of the image and the other image.

The above process is further described below by taking the actual model in the memory 120 as an example. Also referring to FIG. 3 , FIG. 3 is a schematic diagram of an image simulation method according to some embodiments of the present disclosure. In one embodiment, the memory 120 may further include a preprocessing model PPM, a mask generation model MGM, and a temporal smoothing model TSSM. The processor 130 can execute these models to perform the steps in Figure 2 above.

First, the preprocessing model PPM can preprocess the image IMG generated by the image capture circuit 110, wherein the image IMG belongs to RGB images, and the preprocessing includes semantic segmentation processing (eg, processing by UpperNet), depth prediction processing (eg, processing by UpperNet). by MegaDepth) and point cloud projection (eg by PCL).

Specifically, the preprocessing model PPM can perform semantic segmentation processing on the image IMG to generate a semantically segmented image SI, wherein the same objects in the semantically segmented image SI have the same weight. Next, the preprocessing model PPM A depth prediction process may be performed on the image IMG to generate a depth map DI. Next, the preprocessing model PPM may perform point cloud projection processing on the depth map DI to generate a virtual ground map GDI, wherein the virtual ground positions in the virtual ground map GDI have relatively high grayscale values. In this way, the preprocessing model PPM can input the semantic segmentation image SI and the depth map DI to the mask generation model MGM, and input the depth map DI and the virtual ground map GDI to the time series smoothing model TSSM.

Furthermore, the mask generation model MGM can identify the pixels belonging to the ground in the semantic segmentation image SI as the foreground, and identify the non-ground pixels as the background, and further identify the pixels at the junction of the foreground and the background as the undetermined area. . Then, the mask generation model MGM can segment the image SI according to the semantics to generate a quadruplet graph of the pixels in the image IMG, wherein the pixels in the image IMG respectively correspond to the nodes in the quadruplet graph, and the pixels in the quadruplet graph belong to the same object. Lines between pixels will have higher associated values.

Furthermore, the mask generation model MGM can generate histograms of 16 categories of red channel, green channel and blue channel according to the image IMG, wherein the size of the histogram is 16x16x16, and the value categories in the histogram are also 16x16x16. Then, the mask generation model MGM can connect the 16x16x16 auxiliary nodes corresponding to these numerical classes to the corresponding cells in the quadruplet graph. For example, the value range of a value class of the red channel is 0~30, and the node corresponding to this value range in the quadruplet graph can be connected to the auxiliary node of this value class.

Furthermore, the mask generation model MGM can calculate the sum of the classification costs (ie, classification parameters) of all auxiliary nodes using the following formula (1).

where E1 is the classification parameter, β k is the preset classification cost parameter of the kth auxiliary node, S is the number of pixels with foreground labels connected to each auxiliary node, and S' is the The number of pixels of the node of the background label, min(,) is the function of taking the minimum value, and N is the number of auxiliary nodes (for example, the above 16x16x16).

For example, take the kth secondary node as an example. Referring also to FIG. 4 , FIG. 4 is a schematic diagram of an auxiliary node and a classification label according to some embodiments of the present disclosure. In Figure 4, pixels with 1 connected node have background labels, pixels with 2 connected nodes have undetermined labels, and pixels with 4 nodes have foreground labels. Therefore, the minimum value of 1 and 4 can be multiplied by the above β to calculate the classification cost of the kth auxiliary node as β. Based on this, if the classification labels of all nodes in the kth auxiliary node have at most one label (foreground or background label), the classification cost can be minimized (ie, 0).

Referring back to FIG. 3, the mask generation model MGM can calculate the smoothing term of each pixel by the following formula (2), and use the sum of the smoothing terms as a smoothing parameter.

where E2 is the smoothing term of 1 pixel, Vp is the RGB value of 1 pixel, Vq is the RGB value of one of the pixels around this pixel, and σ is the statistical variance of all pixels. It is worth noting that from the above formula (2), it can be known that 4 smoothing terms can be calculated from 1 pixel, These smoothing terms can be added together as smoothing parameters, wherein these smoothing terms present a Gaussian distribution. The closer the color of two adjacent pixels is, the smaller the smoothing term, and vice versa, the larger the smoothing term. Using this smoothing parameter can greatly eliminate the occurrence of errors in the classification labels of pixels.

Furthermore, the mask generation model MGM can preset the shadow value range of the HSV domain for general shadows in the image. For example, the value range of the hue channel is set to 50~250, the value range of the saturation channel is set to 0~70, and the value range of the lightness channel is set to 25~100.

Furthermore, the mask generation model MGM can convert the image IMG from the RGB domain to the HSV domain to generate the HSV value of the image. Then, the mask generation model MGM can perform distance calculation according to the HSV value of the image and the range of shadow values to generate shadow probabilities of a plurality of pixels of the image IMG, and can generate a probability map according to the shadow probabilities.

For example, referring to FIG. 5, FIG. 5 is a schematic diagram of the probability map P_IMG corresponding to the shadow probability according to some embodiments of the present disclosure. As shown in FIG. 5 , the probability map P_IMG can be generated by assigning different shading probabilities to different grayscale values according to the shading probabilities of the above-mentioned pixels, wherein the area closer to white is more likely to be a shading.

Referring back to FIG. 3 , the mask generation model MGM can adjust the value of β k in the formula (1) according to the above-mentioned shadow probability, as shown in the following formula (3).

β k=1-ShadowProbk...(3)

where ShadowProbk is the shadow probability of each pixel corresponding to the kth auxiliary node. In other words, where the probability of shadowing is higher, it is less affected by the classification parameters, but only controlled by the above-mentioned smoothing term.

Furthermore, the mask generation model MGM can generate shadow parameters according to the shadow probability of the above-mentioned pixels, as shown in the following formula (4).

where E3 is the shadow parameter, ShadowProbp is the shadow probability of a pixel, a blurred pixel is a pixel with an ambiguous label and the surrounding pixels have a background label, and ShadowProbq is the shadow probability of the surrounding pixels of the pixel. It is worth noting that for other pixels, there are 4 pixels around the pixel, so 4 shadow parameters can be generated. Based on the above, the shadow parameter E3 will reduce the cost of the ground junction in the original semantic segmentation, so that the final junction falls on the edge of the semantic segmentation instead of the edge of the shadow.

Furthermore, the mask generation model MGM can adjust the above-mentioned smoothing parameter according to the shadow parameter of each pixel and a preset adjustment parameter, as shown in the following formula (5).

E2 ' =E2+w×E3...(5)

where E2' is the adjusted smoothing parameter, and w is the adjusted parameter. By adjusting the parameter w above, the relationship between the shadow and the color of the image IMG can be controlled.

Furthermore, the mask generation model MGM can perform the maximum flow minimum segmentation operation on E1, the adjusted smoothing parameter E2' of each pixel, a plurality of auxiliary nodes, and the associated values in the quadruplet graph to generate the adjusted classification label and adjust The four-connection diagram after. Next, the mask generation model MGM can generate a binary guide mask GM according to the adjusted classification labels and the adjusted quadruplet graph, wherein the position in the binary guide mask GM corresponds to the background label The value of can be set to 0, and the value of the position corresponding to the front background label in the binary guide mask GM can be set to 1. Thereby, the mask generation model MGM can input the binary guide mask GM to the temporal smoothing model TSSM.

Furthermore, the time series smoothing model TSSM can pre-store the image IMG and another image IMG' of the previous frame of the image IMG. The time series smoothing model TSSM can identify the virtual ground position corresponding to the horizon in the image IMG from the point cloud image, and adjust the binary guidance mask GM according to the virtual ground position, and then use the adjusted binary guidance mask GM to adjust the depth map. DI is filtered to produce a filtered depth map DI.

Furthermore, the time series smoothing model TSSM can generate the photometric parameters according to the following formula (6) according to another image IMG' of the previous frame of the image IMG.

min ʃ w(x)×∥O-warp(O ' )∥ ² dx...(6)

Where x is the position of each pixel in the image IMG, O is the depth value expected to be output by each pixel at position x, warp(,) is calculated from the original unprocessed video using the optical flow mapping function (DIS flow). The displacement of each pixel between the frame before and after (ie, warp(O') is calculated between the depth value O expected to be output by the pixel at position x and the depth value O' of the previous frame expected to be output by the pixel at x. displacement of the phase element), and w(x) is as shown in the following formula (7).

w(x)=λ×e ^{-∥V-warp(v ' )∥} ...(7)

where λ is a preset parameter, and V is the RGB value of the pixel at position x in the image IMG. at the pixel at position x if and the previous frame The more color gaps in the , the more likely it is at the intersection of objects, the less this needs to be smoothed.

Furthermore, when the continuous playback time of the continuous image is prolonged, the information projected from the previous frame will be continuously superimposed, causing the overall depth to approach a certain value. Therefore, the temporal smoothing model TSSM can adjust the photometric parameters to generate the temporal parameters as the following formula (8).

E3=min ʃ w(x)×∥O-warp(O ' )∥ ² +s×∥OP∥ ² dx...(8)

where E3 is the timing parameter, P is the depth value corresponding to position x in the depth map DI, and s is 0.1. This will make the difference between the expected depth value O and the depth value P in the depth map DI not too large, and the value of s will not deviate too much from the originally expected value.

Furthermore, the temporal smoothing model TSSM can perform gradient calculation on the depth value P corresponding to the position x in the depth map DI and the depth values of surrounding pixels to generate edge parameters as shown in the following formula (9).

E4=e×∥▽O-▽P∥ ² ...(9)

where ▽O is the gradient value of the depth value expected to be output by each pixel at position x, ▽P is the gradient value of the depth value corresponding to position x in the depth map DI, and e is as shown in the following formula (10).

It can be known from formula (10) that when there is a boundary around the pixel, e is 0, which means that the current value has nothing to do with the boundary, otherwise it is -1.

Furthermore, the timing smoothing model TSSM can add timing parameters E3 and edge parameters E4. Since each order after the addition is a square term, it can be regarded as a problem of the least squares method of a matrix. Thereby, the time series smoothing model TSSM can use the conjugate gradient descent process to iterate the depth values of the expected output, and adjust the depth values in the depth map DI to the depth values of the expected output.

Finally, the time series smoothing model TSSM can generate a sparse point cloud image according to the pre-stored broken depth map (for example, using dynamic recovery structure processing (Structure from Motion, SfM)), and according to the adjusted depth map The sparse point cloud image and the image IMG are processed. Synthesized to produce special effect image SYI.

To sum up, the image simulation system of the embodiment of the present disclosure can generate a guide mask according to the classification parameters of the image and the quadruple map, and use the guide mask to optimize the depth map generated by the depth prediction to clearly distinguish the foreground and background to prevent the depth of parts of the image near the ground from being confused with the depth of the ground. In addition, the timing parameters and the edge parameters can be used to smooth the depth and enhance the edge in time sequence, so as to prevent the depth in time sequence from flickering due to lack of continuity. In this way, a near-real weather effect can be effectively simulated on the image.

Although specific embodiments of the present disclosure have been disclosed with respect to the above-described embodiments, such embodiments are not intended to limit the present disclosure. Various substitutions and modifications can be made in the present disclosure by those of ordinary skill in the relevant art without departing from the principles and spirit of the present disclosure. Therefore, the protection scope of the present disclosure is determined by the appended claims.

100: Video Simulation System

110: Image capture circuit

120: memory

130: Processor

Claims (10)

An image simulation system includes: an image capture circuit for capturing an image; a memory for storing a plurality of commands and a fragmentation depth map; and a processor for connecting the image capture circuit and the memory body, and access the commands to perform the following operations: generate a quad-connected graph and a classification parameter according to the image, wherein the classification parameter is related to a background and a foreground in the image; adjust the quad-connected graph according to the classification parameter , and generate a guide mask according to the adjusted quadruplet map, wherein the guide mask is used to indicate the background and the foreground in the image; and generate a guide mask according to the image, the guide mask and the broken depth map A special effect image, wherein the special effect image is used to simulate the weather corresponding to the broken depth map in the image. The image simulation system according to claim 1, wherein the operation of generating the quadruplet graph and the classification parameter according to the image comprises: performing semantic segmentation processing on the image to generate a semantic segmentation image, and generating a separate image according to the semantic segmentation image. a plurality of classification labels corresponding to a plurality of pixels of the image; and generating a histogram corresponding to a plurality of numerical categories according to the RGB values of the pixels, and generating the classification parameters according to the numerical categories and the classification labels. The image simulation system of claim 2, wherein the classification labels include a foreground label, a background label, and an undetermined label, and the operation of generating the classification parameter according to the numerical categories and the classification labels includes: selecting a minimum number from among the number of the foreground labels and the number of undetermined labels corresponding to each of the numerical classes; and summing the minimum number of each of the numerical classes to generate a minimum sum value, The classification parameter is generated according to the minimum sum value and a classification cost parameter. The image simulation system as claimed in claim 2, wherein the operation of adjusting the quadruple graph according to the classification parameter comprises: according to the RGB value of each of the pixels and the surrounding pixels of the each of the pixels. The RGB values generate a smoothing parameter; and the quad-connected graph is adjusted according to the smoothing parameter and the classification parameter of each of the pixels. The image simulation system of claim 4, wherein the operation of adjusting the quadruple graph according to the classification parameter comprises: converting the image from the RGB domain to the HSV domain to generate the HSV value of the image; according to the HSV value of the image and A distance calculation is performed on a shadow value range to generate shadow probabilities of the pixels of the image; and the classification parameter and the smoothing parameter are adjusted according to the shadow probabilities of the pixels. number, and adjust the classification labels and the quadruplet graph according to the adjusted smoothing parameter and the adjusted classification parameter. The image simulation system according to claim 5, wherein the operation of adjusting the classification labels and the quadruple graph according to the adjusted smoothing parameter and the adjusted classification parameter includes: adjusting the adjusted smoothing parameter and adjusted Perform the maximum flow minimum segmentation operation on the quad-connected graph to adjust the classification labels corresponding to the pixels of the image, and generate the guidance mask according to the adjusted classification labels and the quad-connected graph cover. The image simulation system of claim 1, wherein the operation of generating the special effect image according to the image, the guide mask and the broken depth map comprises: generating a depth map according to the image, and using the guide mask and the depth The map produces weather on the imagery corresponding to the broken depth map. The image simulation system of claim 1, wherein the image capture circuit is further configured to capture another image one frame before the image, wherein the image is generated according to the image, the guide mask and the broken depth map The operation of the special effect image includes: generating a depth map according to the image, and generating a point cloud image according to the depth map; identifying a virtual ground corresponding to the horizon in the image from the point cloud image surface position, and adjust the guide mask according to the virtual ground position; generate a timing parameter and an edge parameter according to the image, the other image, the adjusted guide mask and the depth map, wherein the timing parameter is used for solving the depth discontinuity between the image and the other image, and the edge parameter is used to enhance the edge enhancement of the depth of the image and the other image; and adjusting the depth map according to the timing parameter and the edge parameter, and The special effect image is generated according to the adjusted depth map and the broken depth map. The image simulation system of claim 8, wherein the operation of adjusting the depth map according to the timing parameters and the edge parameters includes: performing conjugate gradient descent processing according to the timing parameters and the edge parameters to adjust the depth map. The image simulation system according to claim 8, wherein the operation of generating the special effect image according to the adjusted depth map and the broken depth map comprises: generating a sparse point cloud image according to the broken depth map, and according to the adjusted depth map The depth map synthesizes the sparse point cloud image with the image to generate the special effect image.

Images