FFT — The tough made simple.

Revision en5, by sidhant, 2016-03-11 10:26:18

#### Aim — To multiply 2n-degree polynomials in instead of the trivial O(n2)

I have poked around a lot of resources to understand FFT (fast fourier transform), but the math behind it would intimidate me and I would never really try to learn it. Finally last week I learned it from some pdfs and CLRS by building up an intuition of what is actually happening in the algorithm. Using this article I intend to clarify the concept to myself and bring all that I read under one article which would be simple to understand for others struggling with fft.

Let’s get started

Here A(x) and B(x) are polynomials of degree n - 1. Now we want to retrieve C(x) in

So our methodology would be this

1. Convert A(x) and B(x) from coefficient form to point value form. (FFT)
2. Now do the O(n) convulsion in point value form to obtain C(x) in point value form, i.e. basically C(x) = A(x) * B(x) in point value form.
3. Now convert C(x) from point value from to coefficient form (Inverse FFT).

Q) What is point value form ?
Ans) Well, a polynomial A(x) of degree n can be represented in its point value form like this A(x) = {(x0, y0), (x1, y1), (x2, y2), ..., (xn - 1, yn - 1)} , where yk = A(xk) and all the xk are distinct.
So basically the first element of the pair is the value of x for which we computed the function and second value in the pair is the value which is computed i.e A(xk).
Also the point value form and coefficient form has one-to-one correspondence i.e. for each point value form there is exactly one coefficient representation and vice — versa, If for k degree polynomial, k + 1 point value forms have been used at least.
Reason is simple, the point value form has n variables i.e, a0, a1, ..., an - 1 and n equations i.e. y0 = A(x0), y1 = A(x1), ..., yn - 1 = A(xn - 1) so only one solution is there.
Now using matrix multiplication the conversion from coefficient form to point value form for the polynomial can be shown like this

And the inverse, that is the conversion from point value form to coefficient form for the same polynomial can be shown as this

C(1) = A(1) * B(1) = 5 * 3 = 15, C(2) = A(2) * B(2) = 3 * 4 = 12, C(3) = A(3) * B(3) = 2 * 7 = 14

So C(x) = {(1, 15), (2, 12), (3, 14)} where degree of C(x) = degree of A(x) + degree of B(x) = 4
But we know that a polynomial of degree n - 1 requires n point value pairs, so 3 pairs of C(x) are not sufficient for determining C(x) uniquely as it is a polynomial of degree 4.
Therefore we need to calculate A(x) and B(x), for 2n point value pairs instead of n point value pairs so that C(x)’s point value form contains 2n pairs which would be sufficient to uniquely determine C(x) which would have a degree of 2(n - 1).

##### Now if we had performed this algorithm naively it would have gone on like this

Note — This is NOT the actual FFT algorithm but I would say that understanding this would layout framework to the real thing.
Note — This is actually DFT algorithm, ie. Discrete fourier transform.

1. We construct the point value form of A(x) and B(x) using x0, x1, ..., x2n - 1 which can be made using random distinct integers. So point value form of A(x) = {(x0, α0), (x1, α1), (x2, α2), ..., (x2n - 1, α2n - 1)} and B(x) = {(x0, β0), (x1, β1), (x2, β2), ..., (x2n - 1, β2n - 1)} - (1) Note — The x0, x1, ..., x2n - 1 should be same for $A(x) and$B(x). This conversion takes O(n2).
2. As C(x) = A(x) * B(x), then what would have been the point-value form of C(x) ?
If we plug in x0 to all 3 equations then we see that
C(x0) = A(x0) * B(x0)
C(x0) = α0 * β0
So C(x) in point value form will be C(x) = {(x0, α0 * β0), (x1, α1 * β1), (x2, α2 * β2), ..., (x2n - 1, α2n - 1 * β2n - 1)} - (2)
This is the convulsion, and it’s time complexity is O(n)
3. Now converting C(x) back from point value form to coefficient form can be represented by using the equation 2. Here calculating the inverse of the matrix requires LU decomposition or Lagrange’s Formula. I won’t be going into depth on how to do the inverse, as this wont be required in the REAL FFT. But we get to understand that using Lagrange’s Formula we would’ve been able to do this step in O(n2).

Note — Here the algorithm was performed wherein we used x0, x1, ..., x2n - 1 as ordinary real numbers, the FFT on the other hand uses roots of unity instead and we are able to optimize the O(n2) conversions from coefficient to point value form and vice versa to because of the special mathematical properties of roots of unity which allows us to use the divide and conquer approach. I would recommend to stop here and re-read the article till here until the algorithm is crystal clear as this is the raw concept of FFT.

##### A math primer on complex numbers and roots of unity would be a must now.

Q) What is a complex number ?
Answer — Quoting Wikipedia, “A complex number is a number that can be expressed in the form a + bi, where a and b are real numbers and i is the imaginary unit, that satisfies the equation i2 =  - 1. In this expression, a is the real part and b is the imaginary part of the complex number.” The argument of a complex number is equal to the magnitude of the vector from origin (0, 0) to (a, b), therefore arg(z) = a2 + b2 where z = a + bi.
Q) What are the roots of unity ? Answer — An nth root of unity, where n is a positive integer (i.e. n = 1,  2,  3, ...), is a number z satisfying the equation zn = 1. Fig(1) n = 2 Fig(2) n = 3 Fig(3) n = 4 Here the x axis is the real axis and the y axis is the imaginary axis.

Intuitively, we can see that the nth root of unity lies on the circle of radius 1 unit (as its argument is equal to 1) and they are symmetrically placed ie. they are the vertices of a n — sided regular polygon.

The n complex nth roots of unity can be represented as eik / n for k = 0, 1, ..., n - 1
Also Graphically see the roots of unity in a circle then this is quite intuitive.

If n = 4, then the 4 roots of unity would’ve been ei * 0 / n, ei * 1 / n, ei * 2 / n, ei * 3 / n = e1, ei / n, (ei / n)2, (ei / n)3 = (ei / n)0, (ei / n)1, (ei / n / )2, (ei / n)3 where n should be substituted by 4.
Now we notice that all the roots are actually power of ei / n. So we can now represent the n complex nth roots of unity by wn0, wn1, wn2, ..., wnn - 1, where wn = ei / n

Now let us prove some lemmas before proceeding further

Lemma 1 — For any integer n ≥ 0, k ≥ 0 and d ≥ 0, wdndk = wnk
Proof — wdndk = (ei / dn)dk = (ei / n)k = wnk

Lemma 2 — For any integer n ≥ 0, wnn / 2 = w2 =  - 1
Proof — wnn / 2 = w2 * (n / 2)n / 2 = wd * 2d * 1 where d = n / 2
wd * 2d * 1 = w21 — (Using Lemma 1)
w21 = eiπ = cos(π) + i * sin(π) =  - 1 + 0 =  - 1

Lemma 3 — If n > 0 is even, then the squares of the n complex $n$th roots of unity are the (n/2) complex (n/2)th roots of unity, formally (wnk)2 = (w2 * (n / 2)2k = wn / 2k

Proof — By using lemma 1 we have (wnk)2 = w2 * (n / 2)2k = wn / 2k, for any non-negative integer k. Note that if we square all the complex nth roots of unity, then we obtain each (n/2)th root of unity exactly twice since,

(Proved above)

Also, (wnk + n / 2)2 = wn2k + n = ei * k' / n, where k' = 2k + n

ei * k' / n = ei * (2k + n) / n = ei * (2k / n + 1) = e(2πi * 2k / n) + (2πi) = ei * 2k / n * ei = wn2k * (cos(2) + i * sin(2))

(Proved above)

Therefore, (wnk)2 = (wnk + n / 2)2 = wn / 2k

Lemma 4 — For any integer n ≥ 0, k ≥ 0, wnk + n / 2 =  - wnk
Proof — wnk + n / 2 = ei * (k + n / 2) / n = ei * (k / n + 1 / 2) = e(2πi * k / n) + (πi) = ei * k / n * eπi = wnk * (cos(π) + i * sin(π)) = wnk * ( - 1) =  - wnk

#### History

Revisions

Rev. Lang. By When Δ Comment
en19 sidhant 2016-12-15 15:43:01 68 Tiny change: 'y/48798)\n\nRefere' -br
en18 sidhant 2016-12-02 10:11:30 11
en17 sidhant 2016-12-02 10:08:00 6 Tiny change: '2k}*(cos(2) + i*sin(2)) = w_{n}' - (published)
en16 sidhant 2016-12-02 10:06:22 124 (saved to drafts)
en15 sidhant 2016-10-17 12:10:22 3897 Tiny change: 'n - 1} \\ w_n^{2} &' -
en14 sidhant 2016-03-12 13:12:08 269
en13 sidhant 2016-03-11 20:36:36 20 Typo - Convulsion is actually Convolution.
en12 sidhant 2016-03-11 17:34:46 0 (published)
en11 sidhant 2016-03-11 15:48:16 287 Tiny change: ' for $A(x) and$B(x).\nThis co' -
en10 sidhant 2016-03-11 12:22:09 182 Tiny change: ' A(x) * B(X)$\nH' -br en9 sidhant 2016-03-11 12:16:47 1195 Tiny change: '$w_{n} = $e^{2\pii\n}$ \n5' -br
en8 sidhant 2016-03-11 10:59:08 181 Tiny change: ' to RIGHT.\nIntuitiv' -br
en7 sidhant 2016-03-11 10:55:44 2737 Tiny change: 'm in $O(n*\logn)$ usi' -
en6 sidhant 2016-03-11 10:29:11 14 Tiny change: 'us_bi.png)\nQ) What ' -br\nQ) What '
en5 sidhant 2016-03-11 10:26:18 2616 Tiny change: '^{2\pii/n}\nNow let ' -
en4 sidhant 2016-03-11 09:14:22 2751 Tiny change: '$\n$\n\begin{pma' -br
en3 sidhant 2016-03-02 11:31:21 3540 Tiny change: '(x) * B(x)$\n$C' -br
en2 sidhant 2016-03-02 10:57:18 2132 Tiny change: 't started -\n\$\displa' -br
en1 sidhant 2016-03-02 10:41:33 1180 Initial revision (saved to drafts)