Matthew Gray

9.520 Problem Set 1 April 9, 1998

Problem 1

Part a

Assuming we set the RBF’s to interpolate around the points where we set the conditions (ie, x_{1} = 0, x_{2} = 3, x_{3} = 4), we get:

3c_{2} + 4c_{3} = 1
3c_{1} + c_{3} = 4
4c_{1} + c_{2} = 3

Therefore, c_{1} = 1, c_{2} = −1, and c_{3} = 1, and the resulting interpolation can be seen in figure 1.

Figure 1: One-dimensional RBF iterpolation of 3 points with 3 RBF’s

Part b

A modified set of radial basis functions of the form f(x) = σ_{i=1}^{N}y_{i}ϕ_{i}(x) can be found. One such set of basis functions ϕ_{i}(x) are:

ϕ_{1}(x) = |x|
ϕ_{2}(x) = |x − 3|

ϕ_{3}(x) = \frac{|x − 4| − 5|x − 3|}{3}

Problem 2

As σ → 0, the Gaussians G_{σ}(x − x_{i}) become “tall and skinny”. This causes f_{σ}(x) to become equivalent to nearest neighbor. This is because for any point closest to x_{j} and next closest to x_{k},

as σ → 0, \frac{e^{\frac{|x−xk|^{2}}{σ^{2}}}}{e^{\frac{|x−xk|^{2}}{σ_{2}}}} → ∞, which causes f_{σ}(x) → x_{j}.

As σ → ∞, the Gaussians become completely flat, yielding a constant f_{σ}(x) equal to the sample mean of the y_{i}.

Problem 3

Problem 4

The approximation error caused by projecting a dataset {x_{α}}_{α=1}^{N}, onto the first k principal components in the d dimensional space is

E(k) = \frac{1}{N} ∑_{α=1}^{N}(x_{α} − ∑_{i=1}^{k}(x_{α} · d_{i})d_{i})^{2}  (1)

This has the effect of removing the components of the (k + 1)th through dth eigenvectors. This means E(k) can be rewritten as:

E(k) = \frac{1}{N} ∑_{i=k+1}^{d} ∑_{α=1}^{N}((x_{α} · d_{i})d_{i})^{2}  (2)

and the inner most term can be combined with the \frac{1}{N} to get:

E(k) = ∑_{i=k+1}^{d} λ_{i}  (3)