2:

if A is symmetric and positive-definite, then f(x) is minimized by Ax=b(if not symmetric, then 12(AT+A)x=b)

The Solution to Ax=b Minimizes the Quadratic Form:

Supporse A is symmetric, Let x be a point that satisfies Ax=b, and e be an error term, then:

If A is positive-definite, then the latter term is positive for any e≠0, therefore x minimizes f(x)

The fact that f(x) is a paraboloid is our best intuition of what it means for a matrix to be positive-definite. if A is negative-definite——the result of negatiing a positive-definite matrix. A could be singular, in which case no solution is unique; the set of solutions is a line or a hyperplane having a uniform vale for f. if A is none of the above, then x is a saddle point, and techniques like Steepest Descent and CG will likely fail. The value of b and c determine where the minimum point of paraboloid lies, but do not affect the paraboloid's shape.

Steepest Descent Method:

For the sake of most optimization problems have not closed-form solution, they usually tackles as iterative methods: Starting at some initial point x(0), then taking a series of steps x(1),x(2),..., until the stop condition is satisfied. So as Steepest Descent Method!
In Line search for setting step size, firstly obstains the step sieze α according to x(i+1)=x(i)+αr(i), where r(i)=b−Ax(i) is the residual in the iterative steps.
Totally, the method of Steepest Descent is:

The computational cost of Steepest Descent is dominated by matrx-vector products. There is a mathematical trick to avoid matrix-vector productation directly. See Equation 13 in An Introduction to the Conjugate Gradient Method Without the Agonizing Pain for detail.

The convergence rate is defined as:

The Method of Conjugate Directions:

Steepest Descent often finds itself taking steps in the same directions as earlier steps(remind the zigzag path, which appears because each gradient is orthogonal to the previous gradient). Conjugate Directions update each step via:

d(i) is a set of orthogonal search directions d(0),d(1),...,d(n−1). e(i+1)=ATx−ATx(i+1) is orthogonal to d(i), and that need never step in the same direction of d(i) again.
However, the solution x is unknown. The solution is to make the search directions A-orthogonal instead of orthogonal. Two vectors d(i) and d(j) are A-orthogonal/conjugate, if

The expression for α(i) turns to:

If the search vector were the residual, then this formula would be identical to Steepest Descent.

After n iterations, every component of the error term is cut away, and e(n)=0:

The A-orthogonal search directions set d(i) could be found by conjugate Gram-Schmidt process, see Equation 36 in An Introduction to the Conjugate Gradient Method Without the Agonizing Pain for detail.

The Method of Conjugate Gradients:

References

1. An Introduction to the Conjugate Gradient Method Without the Agonizing Pain
2. Conjugate Gradient Method
3. Nonlinear Conjugate Gradient Methods
4. Gram-Schmidt process example
5. Special Symbols