Chapter 2: Solving Linear Systems and Matrix Methods

This chapter introduces the fundamental problem of linear algebra: solving systems of linear equations. You'll learn two complementary ways to understand these systems—through rows and columns—and discover how to express them using matrices. The chapter builds toward the powerful elimination method that systematically transforms any solvable system into a form where the answer becomes obvious.

2.1 Vectors and Linear Equations

The Central Problem

Linear algebra begins with a practical question: how do we solve systems of linear equations? Let's start with something concrete—two equations in two unknowns:

We're looking for a point that satisfies both equations simultaneously.

Two Pictures of the Same System

There are two fundamentally different ways to visualize what's happening. Each reveals important insights.

The Row Picture

In the row picture, each equation represents a straight line in the plane. When you plot both lines, their intersection point is the solution. For our example, the lines meet at . This geometric view is intuitive—in 3D you'd see planes, and in higher dimensions you'd see hyperplanes.

Row Picture: Each equation (the dot product of row with unknown vector ) describes a line (in 2D) or plane (in 3D) or hyperplane. The solution is where all these objects meet.

The Column Picture

In the column picture, we rearrange the same system as a vector equation:

Now we're asking: what combination of the column vectors on the left produces the vector on the right? This is a linear combination problem. With and , we get exactly the right side:

Column Picture: The equation asks for a linear combination of the columns of that produces . We multiply the first column by , the second by , and so on, then add them up.

The column picture is less intuitive at first, but it's more powerful. It naturally extends to higher dimensions and reveals deep structure in the problem.

Basic Operations on Vectors

To work with vector equations, we need two operations:

Scalar Multiplication scales a vector by a number. Multiplying vector by 3 gives .

Vector Addition combines two vectors component by component. Adding and gives .

These two operations together produce linear combinations: expressions like .

Matrix Notation

When we collect the coefficients of our system into a rectangular array, we get the coefficient matrix:

This is a matrix (2 rows, 2 columns). The unknown vector is and the right-hand side is .

Matrix Equation: The system is a compact way to write all equations at once. When interpreted by rows, each equation is a dot product: . When interpreted by columns, we have a linear combination: .

Matrix Multiplication: Two Interpretations

Understanding matrix multiplication is crucial. There are two equally valid ways to compute :

By rows: Each component of comes from the dot product of that row with .

By columns: is a linear combination of the columns of , with coefficients from .

For our example with :

• Row interpretation: and .
• Column interpretation: .

Both give the same answer. The row interpretation is computational. The column interpretation is conceptual.

Three Dimensions and Beyond

With three equations and three unknowns, the row picture shows three planes in 3D space. They normally intersect at a single point (if a solution exists). The column picture asks whether the right-hand side can be written as a combination of three column vectors.

When we move to higher dimensions, the row picture becomes impossible to visualize, but the column picture's logic remains unchanged: we're still asking whether a vector lives in the space spanned by certain columns.

📝 Section Recap: A system of linear equations can be visualized in two ways. The row picture shows -dimensional hyperplanes meeting at a point. The column picture asks whether a vector is a linear combination of the columns of matrix . Matrix notation encodes the entire system compactly. Matrix-vector multiplication can be computed row-by-row (dot products) or column-by-column (linear combination)—both give the same result. This dual perspective is essential to understanding linear algebra.

2.2 The Idea of Elimination

The Elimination Method

The most practical way to solve linear systems is elimination. The goal is to transform the system into upper triangular form—where unknowns drop out one by one, and the last equation contains only the last unknown.

Let's see how this works on our original system:

Before elimination:

To eliminate from the second equation, we subtract 3 times the first equation from the second. Since the first equation has coefficient (which is 1), the multiplier is .

After elimination:

Now we solve by back substitution: from the second equation, . Substituting into the first: , so .

This process generalizes beautifully. For a system with equations and unknowns:

1. Use the first equation to eliminate the first unknown from all equations below it.
2. Use the new second equation to eliminate the second unknown from all equations below it.
3. Continue until you reach upper triangular form.
4. Solve by back substitution.

The Pivot and the Multiplier

The pivot is the entry we use to eliminate—the leading nonzero coefficient in the equation we're using as the "driver." In our example, the first pivot is 1 (the coefficient of in the first equation).

The multiplier is the factor by which we scale the pivot equation before subtracting. It's computed as:

In our example, the multiplier is .

Pivot and Multiplier: The pivot is the first nonzero entry in the row used for elimination. The multiplier is the ratio: (entry to eliminate) / (pivot). Pivots must be nonzero—if the pivot position contains zero, we exchange equations to move a nonzero entry into that position.

Three Equations in Three Unknowns

Consider:

Step 1: The first pivot is 2. Eliminate from equations 2 and 3.

• Multiplier for row 2: . Subtract 2 times row 1 from row 2.
• Multiplier for row 3: . Subtract -1 times row 1 from row 3 (i.e., add row 1 to row 3).

After Step 1:

Step 2: The second pivot is 1 (the coefficient of in the new row 2). Eliminate from row 3.

• Multiplier for row 3: . Subtract row 2 from row 3.

After Step 2 (upper triangular form):

Back substitution: From row 3: . From row 2: , so . From row 1: , so .

The solution is .

When Elimination Fails

Elimination is rock-solid when every pivot position contains a nonzero entry. But three things can go wrong:

1. No solution occurs when the equations are inconsistent. For example:

There's no way to satisfy the second equation. The row picture shows two parallel lines that never meet.

2. Infinitely many solutions occur when one equation is redundant (a multiple of another). For example:

The second equation places no constraint. Any point on the line is a solution. The value of is "free"—we can choose it arbitrarily, and adjusts to match.

3. Zero in the pivot position prevents immediate elimination, but we can often fix it with a row exchange. If row has a zero in the pivot position but a nonzero entry appears below it, we swap those rows.

Elimination that reaches a full set of pivots is called nonsingular, and it guarantees a unique solution.

📝 Section Recap: The elimination method transforms a system into upper triangular form through systematic row operations. The pivot is the entry used for elimination; the multiplier is the ratio of the entry to eliminate to the pivot. Back substitution then solves the triangular system efficiently. Elimination can fail if a pivot position contains zero (solved by row exchange), if the system is inconsistent (no solution), or if equations are redundant (infinitely many solutions). The geometric view: in the row picture, planes either meet at a point (unique solution), along a line (infinitely many), or don't meet (no solution). In the column picture, either lies uniquely as a combination of the columns, lies in the span with multiple combinations, or lies outside the span entirely.

2.3 Elimination Using Matrices

Matrix Representation of Elimination Steps

Each step of elimination can be represented by multiplying by a matrix. This transforms the system into an upper triangular system , where we can apply back substitution.

The elimination matrix performs a single elimination step: it subtracts times row from row . The matrix has the form:

• Start with the identity matrix .
• Change the entry from 0 to .

For example, to subtract 2 times row 1 from row 2:

When we multiply , we get a new matrix with eliminated from row 2.

The Power of Matrix Multiplication

Once we think in terms of matrices, elimination becomes elegant. If are the elimination matrices for successive steps, then:

where is the upper triangular result. And simultaneously:

This tells us that the overall transformation is a single matrix multiplication by the product .

Elimination as Matrix Multiplication: Each row operation can be expressed as left-multiplication by an elementary matrix. Stringing together all the elimination steps gives and , so we've transformed into the triangular system .

The Augmented Matrix

In practice, we don't track the elimination matrices separately. Instead, we work with the augmented matrix , which combines the coefficient matrix and right-hand side into one rectangular array:

When we perform row operations, both the left (matrix ) and right (vector ) sides are transformed together. This ensures that we're solving the correct system throughout.

Permutation Matrices and Row Exchanges

Sometimes we need to rearrange the order of equations to move a nonzero entry into the pivot position. A permutation matrix does exactly this.

For example, to exchange rows 2 and 3 of any matrix, multiply by:

Multiplying times a matrix swaps its rows 2 and 3.

Row Exchange Matrix: A permutation matrix is the identity matrix with rows and exchanged. Multiplying on the left exchanges rows and of any matrix.

Identity, Elementary, and Permutation Matrices

Three special matrices appear constantly in elimination:

Identity Matrix: The matrix has 1's on the diagonal and 0's elsewhere. for any vector .

Elementary Matrix or Elimination Matrix: is the identity with one additional nonzero entry in position . It subtracts times row from row . subtracts a multiple of one row from another.

Permutation Matrix: is the identity with rows and exchanged. swaps rows and of any matrix it multiplies.

Matrix Multiplication and Associativity

A key property is that matrix multiplication is associative:

This is why we can rearrange products of elimination matrices. If we first apply and then , the combined effect is —and the parentheses don't matter.

Warning: Matrix multiplication is NOT commutative. Usually . When subtracts 2 times row 1 from row 2, we write . If we multiply on the right instead (), the result is different—now columns are modified, not rows. To apply row operations, we always multiply on the left.

📝 Section Recap: Elimination can be expressed as matrix multiplication. Each row operation is an elementary matrix multiplying on the left: subtracting times row from row corresponds to . All elimination steps combine into a single matrix such that (upper triangular). We solve by computing , then using back substitution. The augmented matrix tracks both the coefficient matrix and right-hand side together. Permutation matrices exchange rows when needed to place nonzero pivots on the diagonal. Matrix multiplication is associative (so ) but not commutative (usually ).

2.4 Rules for Matrix Operations

Basic Facts About Matrices

A matrix is a rectangular array of numbers arranged in rows and columns. An matrix has rows and columns.

Matrix Addition: Matrices of the same size can be added entry-by-entry. We can scale any matrix by multiplying each entry by a constant . The zero matrix (all entries zero) plus any matrix gives that matrix back. Multiplying a matrix by reverses all signs.

Matrix addition works like vector addition—intuitive and component-wise. Matrix multiplication is trickier.

When Can We Multiply Matrices?

To multiply matrices and , the number of columns of must equal the number of rows of :

The resulting matrix has rows and columns.

Fundamental Law of Matrix Multiplication: For matrices to be compatible for multiplication: has columns, has rows, and has the same number of rows as and the same number of columns as . We can compute in three ways: (1) rows of times , (2) times columns of , or (3) columns of times rows of .

Computing Matrix Products

Let's compute where and .

By rows: Each row of is that row of times all of :

• Row 1 of :
• Row 2 of :

By columns: Each column of is times that column of :

• Column 1 of :
• Column 2 of :
• Column 3 of :

Both give:

Key Properties

Associative Law: . The order of operations doesn't matter; you can rearrange parentheses. This is crucial for elimination, where we compose multiple row operations.

Non-Commutativity: In general, . Even if both products are defined, they usually give different answers. This is fundamentally different from arithmetic with numbers.

Distributive Law: and . These work as you'd expect.

Block Multiplication: Matrices can be partitioned into blocks and multiplied as though the blocks were entries. If (two-column block matrix) and (two-row block matrix), then , provided the block dimensions are compatible.

An Important Warning

The identity matrix is special:

Identity Matrix: is the matrix with 1's on the diagonal and 0's elsewhere. For any matrix , we have and . It's the multiplicative identity for matrices, just as 1 is for numbers.

There are many matrices for which no inverse exists, making matrix algebra fundamentally different from arithmetic. A matrix that has an inverse is called invertible or nonsingular.

📝 Section Recap: Matrix addition is entry-wise and works only for matrices of the same size. Matrix multiplication requires the number of columns of the left matrix to equal the number of rows of the right matrix. The entry of is the dot product of row of with column of . Matrix multiplication can be computed row-by-row, column-by-column, or block-by-block. Multiplication is associative (allowing us to rearrange parentheses) but not commutative (usually ). The identity matrix satisfies and . Many matrices lack inverses, a key difference from arithmetic with numbers.

Key Concepts Summary

Two Views of Linear Systems:

• Row picture: equations give hyperplanes; the solution is where they meet.
• Column picture: Combine columns of to produce the vector .

Elimination Method:

• Transform into upper triangular form via row operations.
• Solve by back substitution, starting from the last equation.
• Pivots must be nonzero; exchange rows if needed.

Matrix Language:

• Each elimination step is a matrix multiplication by an elementary matrix.
• The augmented matrix keeps the system and right-hand side in sync.
• Permutation matrices exchange rows; the identity matrix is multiplicative identity.

Matrix Multiplication:

• requires columns of = rows of .
• Compute by rows ( row-by-row), by columns ( column-by-column), or by blocks.
• Multiplication is associative but not commutative; most matrices lack inverses.

Study Tips

1. Visualize both pictures: For 2D systems, sketch the lines. Understand that a line is the intersection of a row's equation with the plane.

2. Practice elimination by hand: Perform elimination on and systems until the process becomes automatic. Identify the pivots and multipliers.

3. Understand matrix notation: Rewrite each system as . Recognize the columns of and see how they combine to give .

4. Verify your solutions: After solving, substitute back into the original equations to confirm.

5. Connect rows and columns: For every statement about the row picture, there's a dual statement about the column picture. This duality is the heart of linear algebra.

6. Remember the failure modes: No solution (inconsistent system, parallel planes), infinitely many solutions (redundant equation, free variables), and zero pivot (exchange rows).

7. Use matrix products correctly: When solving using elimination matrices, remember that we multiply on the left: , not on the right.