Saliency for 3D Meshes

CS498dh Project

Figure 1. Saliency maps from automatically detected viewpoints and the original objects rendered (MATLAB's Gouraud shading) from the same angles. Left: most salient viewpoint. Right: Least salient viewpoint.

This code (GitHub link) is an implementation of the SIGGRAPH 2005 paper Mesh Saliency by Lee et al. It contains two parts:

  1. Saliency Map: Compute the saliency of every vertex location on the surface.
  2. Salient Viewpoint Selction: Estimate the most salient viewpoint by rendering the saliency map from multiple angles and choosing the view with most visible pixels.

Saliency Map

Lee et al.’s paper uses curvature-based features and Itti et al.’s [1998] center-surround mechanism. Other useful features to experiment with may include silhouettes, albedos, and so on, but not rotation-invariant features such as shading.

Figure 2. The first step is to compute the curvature values. The code uses uses compute_curvature.m from toolbox graph (with this patch).
Figure 3. From left to right: level 1 to 5. At each level, compute the Gaussian-weighted average of mean curvatures vertex with sizes \sigma_i and 2*\sigma_i then subtract the two. This code uses the same \sigma used in the Lee et al. paper.

In order to apply the filter operation in vertex coordinates, I computed the distance between every point and then then applied the gausian basis function. Accoring to MATLAB’s profiler, this part is the bottle neck, so it should be optimized first if the code is too slow. I simplified the meshes to have somewhere between 4000 and 15,000 faces using QSlim to for all the demos on this page.

D = pdist2(Mesh.v, Mesh.v);

weighted = exp(-0.5.*(D./sigma).^2);

weighted(D >= 2*sigma) = 0;

Figure 4. Multiply each level by (M-m)^2 where M is the global maximum and m is the mean of local maxima. Above is the result after adding them. This is also the same as in the paper. Because it was unclear how to compute the local maxima, I set the search window to be relative to \sigma_i.

Salient Viewpoint Selection

The 3D saliency map can be used to automatically select a viewpoint where the object is most salient. The paper uses a randomized method that starts at a random viewpoint then iteratively finds a local maximum. Instead, this implementation renders the saliency map from 60 different points (15 angles around the z axis for each of [2, 13, 26, 40] elevation. Because I used simplified meshes, this didn’t take very long.)

Figure 5. Some examples of the saliency maps rendered at different angles. The figure shows the raw saliency values, but the actual values used for scoring are raised to some integer exponents to avoid the large silhouettes always being preferred. This implementation does the rendering by creating an invisible MATLAB figure, rotating the object, and then converting the figure to image. This is slow, but still much faster than the Gaussian-weighted mean curvature computation step above.

Here are some successful and unsuccessful examples.

automatic viewpoint selection: successful examples

In each pair of figures, the left column is the most salient viewpoint and the right column is the least salient viewpoint, automatically selected. Computing each took around 20 seconds. Many of the 3D models are from 3D Warehouse.

Bad Examples