## Fundamentals of Matrix Algebra, Third Edition Paperback – 2 November 2011

I highly advise picking up this title for any student about to enroll in a first course in linear algebra. had I had this book prior to my taking such a course I would've certainly labored over the small stuff quite a bit less and saved myself more time to think though the more conceptual aspects of the course. For LA, you MUST learn the fundamentals of matrix algebra cold like you know normal algebraic operators, and most professors give you about 3 weeks from day one to do so. After that the course takes off and will leave you behind.

I went through it in about 10 sittings after having taken a full linear algebra course and it did solidify a lot of the operational aspects to the course and helped really petrify some of the intuition gleaned. Hartman is a gifted math-text writer (and likely, teaching professor).

## Fundamentals of Matrix Algebra

This text deals with matrix algebra, as opposed to linear algebra. Without arguing semantics, I view matrix algebra as a subset of linear algebra, focused primarily on basic concepts and solution techniques. There is little formal development of theory and abstract concepts are avoided. This is akin to the master carpenter teaching his apprentice how to use a hammer, saw and plane before teaching how to make a cabinet.

This book is intended to be read. Each section starts with "AS YOU READ" questions that the reader should be able to answer after a careful reading of the section even if all the concepts of the section are not fully understood. I use these questions as a daily reading quiz for my students. The text is written in a conversational manner, hopefully resulting in a text that is easy (and even enjoyable) to read.

Many examples are given to illustrate concepts. When a concept is first learned, I try to demonstrate all the necessary steps so mastery can be obtained. Later, when this concept is now a tool to study another idea, certain steps are glossed over to focus on the new material at hand. I would suggest that technology be employed in a similar fashion.

## About

**APEX**: **A**ffordable **P**rint and **E**lectronic Te**X**tbooks

The traditional college textbook of today is **expensive**. Some may argue that these texts are worth their high price, much like some argue that a luxury car is worth its high price. However, when buying a car, one has a **choice**: if you want a luxury car, you can buy one if you can afford it. If you cannot afford it, you buy something less expensive.

Not so with textbooks. There are very few (any?) texts available through traditional publishers that are inexpensive, yet of high quality. However, with the proliferation of desktop publishing tools and print-on-demand services, alternatives have begun to appear.

(Alternatives *need* to appear as the current textbook model is unhealthy and likely unsustainable. Check out this NPR report which shows that textbook prices grew 82% between 2002 and 2012, how student spending on textbooks *dropped slightly* during that same period, meaning *students are buying fewer textbooks.* This Planet Money report also gives insight into the textbook market and is worth your 15 minutes of listening.)

The biggest remaining hurdle to textbook writing is time: it obviously takes a lot of time to write and produce a high-quality text. To address this issue, some math professors at Virginia Military Institute decided **collaboration** was the key: instead of having one or two authors doing all the work, what if many people worked together on writing a text? Each individual could specialize: one could write examples, another problem sets, another produce graphics, etc. The cost of time to any one person would be greatly reduced. These math professors decided to advertise this collaborative idea under the name of APEX, hoping to develop a consortium of like-minded individuals who worked together to change the math-textbook landscape.

The core values of this consortium are represented by the letters of APEX. Clearly, we are writing **textbooks**, though not limited to mathematics. The product must be **affordable**. (APEX Calculus is **free** in pdf form if you want a printed copy, you can print it yourself or buy a nice print copy through Amazon for about $15.) While there has been much to-do about how ebooks and tablets would revolutionize education, our experience is that many students still want something they can hold in their hand. And write in. And dog-ear. Hence we need to make **print** versions available. There is much to be done **electronic**ally, though, that can't be done in print. This exciting frontier is yet to be fully explored, though APEX Calculus has introduced at least one exciting feature - 3D graphics that can be manipulated in the .pdf!

To encourage collaboration, APEX Calculus is available as an open text. All source files are available on GitHub for others to monkey with, covered under a generous Creative Commons BY-NC license. Don't want a section? Take it out. Did we miss a section? Add one in.

Interested in writing an open-source text? Contact me!

**The APEX Calc Story**

The seeds of APEX Calculus were planted as I wrote *Fundamentals of Matrix Algebra* for my Matrix Algebra course and Troy wrote *An Introduction to MATLAB and Mathcad* for his students learning mathematical software. We realized that writing texts was rewarding personally and professionally and were amazed more people were not doing this.

Troy, a colleague Daniel Joseph and I conceived of the APEX model, wherein multiple people collaborate on writing a text and lightening everyone's load. I decided to lead the writing of the first APEX-model book, a text on Linear Algebra. Later, our focus shifted to writing a Multivariable Calculus book, enlisting the help of two others from other Virginia schools.

In the Fall of 2011, VMI offered grants to faculty to support projects that would significantly change education. With Troy's encouragement, I applied for grant money to buy course releases so I could devote significant time to writing. Many of my colleagues in the local MAA Section were interested in participating in the project, and I wrote the potential of their support in my grant application. To make the biggest impact, we decided that our text should be Calculus (and not just the multivariable portions).

I was awarded the grant and immediately began planning the text. I spent a lot of time determining how the book would look, including font choice, what the figures would look like and where they would be located, and a visual method for indicating where examples began & ended. Much of this was in line with features of traditional texts. Troy was intentional in thinking of something "new" we could do that other texts wouldn't, which led us to including the Notes space at the bottom of each page.

When I announced to my interested colleagues that I had received course releases to lead the project, most congratulated me and politely backed out. I understood why they couldn't participate: they were busy, just as I was. The big difference between us was that I had two course releases and they didn't.

In the Spring of 2012, I made great progress, writing chapters 1 through portions of 6. Troy helped by writing two sections, and Brian Heinold also contributed a few sections. Jennifer Bowen was editing the material, making suggestions and offering much appreciated compliments. In the Fall of 2012, VMI used these completed chapters to teach our Calc 1 course and I continued to write. By the Spring of 2013, I had completed all of the Calc 2 material and we were now teaching both Calc 1 and 2 with our newly completed text.

I had also been awarded one more course release. My dean and the VMI Board were impressed with how much had been accomplished and gave me the additional course release to help me finish. So during the Spring of 2013 I wrote much of the Calc 3 material. But not all. I continued to write over part of the summer, but went into the Fall of 2013 without a full Calc 3 text. We as a department were committed to using this text, as by this time our students had known no other calculus book. So I taught Calc 3 with 3 colleagues and wrote furiously, completing the text "just in time." My colleagues were incredibly patient and helpful during this time. Dimplekumar Chalishajar also contributed a few sections.

Somewhere in the midst of writing it became clear, without formal declaration, that this was "my" text. It was intended to be a collaborative effort, with many authors contributing and critiquing each other's work as we collectively worked toward completion of a text. (Lots of "c" words in that sentence.) This is not how it turned out. APEX Calculus was my book and the contributors were . contributing to my work. I edited, sometimes heavily, the sections they submitted without their care or concern. They wrote to help me out (and they did!), not to be officially known as a "coauthor."

APEX Calculus Version 1.0 was "released" for the Fall of 2013, consisting of chapters 1 - 8. During this academic year, I finished chapters 9 - 13 and also added material to the previously completed sections, most notably chapters 6 and 8. I also fixed lots of typos my colleagues found. In the Fall of 2014, Version 2.0 was released Version 3.0 was released June, 2015 Version 4.0 was released in May, 2018.

APEX Calculus is written by me, Gregory Hartman, Professor of Applied Mathematics at Virginia Military Institute. Contributions were made by Troy Siemers and Dimplekumar Chalishajar of VMI and Brian Heinold of Mount Saint Mary's University. Jennifer Bowen of the College of Wooster edited the text. It is copyrighted under the Creative Commons Attribution - Noncommercial (BY-NC) License.

**Thank you:** Thanks to Troy, Brian and Dimple for their contributions and to Jennifer for reading through so much material. Thanks to Troy, Lee Dewald, Dan Joseph, Meagan Herald, Bill Lowe and other faculty of VMI who have given me numerous suggestions and corrections based on their experience with teaching from the text. (Special thanks to Troy, Lee & Dan for teaching Calc III as I wrote the Calc III material.) Thanks to Randy Cone for encouraging his tutors of VMI's Open Math Lab to read through the text and check the solutions, and thanks to the tutors for spending their time doing so. A very special thanks to Kristi Brown and Paul Janiczek who took this opportunity far above & beyond what I expected, meticulously checking every solution and carefully reading every example. Their comments have been extraordinarily helpful. I am blessed to have so many people give of their time to make this book better.

## Fundamentals of Matrix Analysis with Applications

Providing comprehensive coverage of matrix theory from a geometric and physical perspective, *Fundamentals of Matrix Analysis with Applications* describes the functionality of matrices and their ability to quantify and analyze many practical applications. Written by a highly qualified author team, the book presents tools for matrix analysis and is illustrated with extensive examples and software implementations.

Beginning with a detailed exposition and review of the Gauss elimination method, the authors maintain readers&rsquo interest with refreshing discussions regarding the issues of operation counts, computer speed and precision, complex arithmetic formulations, parameterization of solutions, and the logical traps that dictate strict adherence to Gauss&rsquos instructions. The book heralds matrix formulation both as notational shorthand and as a quantifier of physical operations such as rotations, projections, reflections, and the Gauss reductions. Inverses and eigenvectors are visualized first in an operator context before being addressed computationally. Least squares theory is expounded in all its manifestations including optimization, orthogonality, computational accuracy, and even function theory. *Fundamentals of Matrix Analysis with Applications* also features:

- Novel approaches employed to explicate the QR, singular value, Schur, and Jordan decompositions and their applications
- Coverage of the role of the matrix exponential in the solution of linear systems of differential equations with constant coefficients
- Chapter-by-chapter summaries, review problems, technical writing exercises, select solutions, and group projects to aid comprehension of the presented concepts

*Fundamentals of Matrix Analysis with Applications* is an excellent textbook for undergraduate courses in linear algebra and matrix theory for students majoring in mathematics, engineering, and science. The book is also an accessible go-to reference for readers seeking clarification of the fine points of kinematics, circuit theory, control theory, computational statistics, and numerical algorithms.

## Test Summary/Review Sheets and Test Solutions

- Define these terms
**in your own words**: sample space , distribution function , the probability of ( E ) . - Describe what
**in your own words**what conditions 1 and 2 of Definition 1.2 are saying. - Describe what
**in your own words**what properties 1, 2, and 5 of Theorem 1.1 are saying. - What is a uniform distribution?
- With what issues do we need to be concerned when we have an infinite sample space?
- Reflection: Tell me at least one question you have, or at least one thing you found unclear, related to this section.
- Try these problems (include your work in the assignment): Exercises 1.2.4-5

- What is the general approach in Examples 2.2, 2.3, 2.4, and 2.6? (I.e., what do those examples have in common?)
- What is Bertrand's Paradox? How can this paradox be resolved?
- What is
**rnd**? - Write out pseudocode (or some other sort of solution outline) for Exercise 2.1.1.

- What is the difference between a density function and a distribution function?
- For a continuous random variable, what is the probability of a single outcome occurring?
- How does the book distinguish between some of the different uses of the word distribution ?
- What is the memoryless property of the exponential distribution function?
- Try this problem (include your work in the assignment): Exercise 2.2.4(b).

- Define what a permutation is
**in your own words**. - What is a fixed point in a permutation?
- What do we assume about mutual disjointedness when creating a tree diagram?
- What is the Birthday Problem ?
- What is a derangement and in what contexts/applications do they appear in this section?
- Reflection: Tell me at least one question you have, or at least one thing you found unclear, related to this section. ( Everything or nothing are not acceptable answers.)
- Try this problem (include your work in the assignment): Exercise 3.1.2.

- How are binomial coefficients related to Pascal's Triangle?
- Define Bernoulli Trials in your own words.
- Give an example (
**not**from the book) of a Bernoulli trials process. - Give one real life application of a Galton board.
- Reflection: Tell me at least one question you have, or at least one thing you found unclear, related to this section. ( Everything or nothing are not acceptable answers.)
- Try these problems (include your work in the assignment): Exercise 3.2.1(a)(b).

- What is a riffle shuffle?
- What is an
*a*-shuffle? - What is an unshuffle (sometimes called an inverse shuffle )?
- How is randomness measured in this section?
- According to the results in this section, about how many times must a deck be riffle-shuffled before it starts to get close to random?
- What is a faro shuffle? (You'll probably need to go outside of the book for this one. And by outside of the book I mean the internet or someone who took this class last year.)
- Reflection: Tell me at least one question you have, or at least one thing you found unclear, related to this section. ( Everything or nothing are not acceptable answers.)

**Bonus:** Learn how to do a faro shuffle by the end of the semester (show me by the last day of class) and get a 2% bonus on the final.

- In your own words, define conditional probability .
- Use this calculator to estimate your lifespan (you don't have to put in correct values if you don't want to):

https://www.ssa.gov/OACT/population/longevity.html

Give a conditional statement based on one of the rows in the results (the information isn't given in terms of probability, but you can still stay something like Given that [bla happens] I can expect [bla that follows]. - In your own words, explain why (P(F|E)=dfrac
) (i.e., explain how it was derived).

- What is a Bayes Probability ?
- What are mutually independent events ?
- What can Bayes' formula be used for? Give at least two applications.
- Try this problem (include your work in the assignment): Exercise 4.1.4(a),
**and try to do it using the conditional probability formula.**

- List the seven distribution functions discussed in this section.
- Give at least one application of
*each*of the seven distribution functions. - What is the difference between the binomial distribution and negative binomial distribution? (I.e., in what scenarios would we use one instead of the other?)
- What is the origin of the Benford distribution?
- Reflection: Tell me at least one question you have, or at least one thing you found unclear, related to this section. (Be specific, everything or nothing are not acceptable (and likely incorrect) answers.)
- Try these problems (include your work in the assignment): ExerciseS 5.1.7(a)(b) and 38(b).

- Give the formulas and a short description of the following densities: Continuous Uniform, Exponential, Gamma, and Normal. (These are the parts of the section on which we will be focusing.)
- To which discrete distribution is the exponential density function related?
- To which density function is the binomial distribution related?
- Reflection: Tell me at least one question you have, or at least one thing you found unclear, related to this section. (Be specific, everything or nothing are not acceptable (and likely incorrect) answers.)
- How is Theorem 5.1 used for simulations?

- In your own words define expected value .
- What does it mean for a series to converge absolutely? (You may need to look back at your calculus book.)
- How are they calculating ( E(X)approx -.0526 ) in Example 6.13?
- In your own words give an informal definition of variance .
- What is the variance of a sum of two independent variables?
- What is standard deviation , and how does it relate to expected value?
- Reflection: Tell me at least one question you have, or at least one thing you found unclear, related to this section. (Be specific, everything or nothing are not acceptable (and likely incorrect) answers.)

Here is a listing of some of contexts in which we find references made to probabilistic events in the Bible. Please take the time before class to read over these passeges:

- In worship: Leviticus 16:6-10.
- Dividing up the land: Numbers 26:52-56, 33:54, 34:13 Joshua 14:1-5, 18:8-10 1 Chronicles 6:54ff. Isaiah 34:16-17.
- In legal issues: Numbers 36:1-4 Proverbs 18:18.
- Determining Levitical duties: 1 Chronicles 24:5ff., 25:8ff., 26:13ff. Nehemiah 10:34 Obadiah 11 Luke 1:5-9.
- Determining guilt and in reference to judgement: Joshua 7:10-15 1 Samuel 14:36-46 (Urim and Thummim also referenced elsewhere) Jonah 1:7-10 Joel 3:1-3 Nahum 3:10.
- Choosing leaders: 1 Samuel 10:20-24.
- In the origin of the Feast of Purim: Esther 3:7 and Esther 9:20-32.
- At Jesus' death: Psalm 22:16-18 Matthew 27:35 Mark 15:24 Luke 23:34 John 19:23-24.
- To determine Judas' replacement: Acts 1:23-26.
- In reference to God's omniscience: Psalm 16:5-6 Proverbs 16:33.

- Give a short definition for each of these terms: vector, matrix, scalar, and transpose.
- What does the dot product measure?
- Is the cross product commutative?
- Is the cross product associative?
- If the dot product of two vectors is 0, then what is the angle between them? (Hint: See Theorem 2.4.)
- If the cross product of two vectors is 0, then what is the angle between them? (Hint: See Theorem 2.8.)
- State the two versions of the triangle inequality mentioned in these sections (one was for magnitudes and one was for dot products, not both are explicitly called the triangle inequality).
- Verify that both versions of the triangle inequality hold when applied to the vectors (mathbf
=langle 1,-2,3 angle) and (mathbf

=langle -4,5,6 angle).

- Visually, what does ( ext
_Q( extbf )) represent?

- The magnitute of the cross product tells us what about which geometric shape? (I.e., What is the shape and what does the cross product tell us about it?)

- How is (mathbf
imesmathbf

) oriented with respect to (mathbf

) and (mathbf

)?

- Is the cross product commutative?
- Is the cross product associative?
- What does it mean for a set to be closed under an operation?
- Are the vectors (mathbf
=langle 1,-2 angle) and (mathbf

=langle -2,5 angle) linearly independent? Give a short explanation for your answer.

- What does the Gram-Schmidt Orthogonalization process produce?

- Is (left(mathbf
^Tmathbf

ight)^T=mathbf

^Tmathbf

)? Explain/show work for your answer.

- Is matrix addition commutative?
- Is matrix addition associative?
- What are the three elementary row operations?

Watch at least one of these:

- or 1972 version or don't-click-if-the-80's-scare-you version. (first music video in space). .

- In your own words, define a Markov chain .
- What do the entries in the transition matrix represent?
- What is a regular Markov chain?
- What is the Fundamental Limit Theorem?
- What is the significance of the Fundamental Limit Theorem?
- Consider the transition matrix [A=egin
.1 & .4 & .5 .5 & 0 & .5 .4 & .6 & 0end .] What is the smallest power of this matrix that will yield a steady/independent state? You may use Mathematica to help with this, and you only need to look out to six decimal places (the default number displayed).

## The Four Fundamental Subspaces

Download the video from iTunes U or the Internet Archive.

OK, here is lecture ten in linear algebra. Two important things to do in this lecture.

One is to correct an error from lecture nine.

So the blackboard with that awful error is still with us.

And the second, the big thing to do is to tell you about the four subspaces that come with a matrix.

We've seen two subspaces, the column space and the null space.

First of all, and this is a great way to

OK. recap and correct the previous lecture -- so you remember I was just doing R^3. I couldn't have taken a simpler example than R^3. And I wrote down the standard basis.

The basis -- the obvious basis for the whole three dimensional space.

And then I wanted to make the point that there was nothing special, nothing about that basis that another basis couldn't have.

It could have linear independence, it could span a space.

There's lots of other bases.

So I started with these vectors, one one two and two two five, and those were independent.

And then I said three three seven wouldn't do, because three three seven is the sum of those.

So in my innocence, I put in three three eight.

I figured probably if three three seven is on the plane, is -- which I know, it's in the plane with these two, then probably three three eight sticks a little bit out of the plane and it's independent and it gives a basis.

But after class, to my sorrow, a student tells me, "Wait a minute, that ba- that third vector, three three eight, is not independent." And why did she say that?

She didn't actually take the time, didn't have to, to find w- w- what combination of this one and this one gives three three eight.

In other words, she looked ahead, because she said, wait a minute, if I look at that matrix, it's not invertible.

That third column can't be independent of the first two, because when I look at that matrix, it's got two identical rows.

Its rows are obviously dependent.

And that makes the columns dependent.

When I look at the matrix A that has those three columns, those three columns can't be independent because that matrix is not invertible because it's got two equal rows.

And today's lecture will reach the conclusion, the great conclusion, that connects the column space with the row space.

So those are -- the row space is now going to be another one of my fundamental subspaces.

The row space of this matrix, or of this one -- well, the row space of this one is OK, but the row space of this one, I'm looking at the rows of the matrix -- oh, anyway, I'll have two equal rows and the row space will be only two dimensional.

The rank of the matrix with these columns will only be two.

So only two of those columns, columns can be independent too.

The rows tell me something about the columns, in other words, something that I should have noticed and I didn't.

So now let me pin down these four fundamental subspaces.

So here are the four fundamental subspaces.

This is really the heart of this approach to linear algebra, to see these four subspaces, how they're related.

And now comes the row space, something new.

The row space, what's in that?

It's all combinations of the rows.

We want a space, so we have to take all combinations, and we start with the rows.

So the rows span the row space.

Are the rows a basis for the row space?

The rows are a basis for the row space when they're independent, but if they're dependent, as in this example, my error from last time, they're not -- those three rows are not a basis.

The row space wouldn't -- would only be two dimensional.

I only need two rows for a basis.

So the row space, now what's in it?

It's all combinations of the rows of A.

All combinations of the rows of A.

But I don't like working with row vectors.

All my vectors have been column vectors.

I'd like to stay with column vectors.

How can I get to column vectors out of these rows?

So if that's OK with you, I'm going to transpose the

matrix. I'm, I'm going to say all combinations of the columns of A transpose.

And that allows me to use the convenient notation, the column space of A transpose.

Nothing, no mathematics went on there.

We just got some vectors that were lying down to stand up.

But it means that we can use this column space of A transpose, that's telling me in a nice matrix notation what the row space is.

OK. And finally is another null space.

The fourth fundamental space will be the null space of A transpose.

The fourth guy is the null space of A transpose.

And of course my notation is N of A transpose.

That's the null space of A transpose.

Eh, we don't have a perfect name for this space as a -- connecting with A, but our usual name is the left null space, and I'll show you why in a moment.

So often I call this the -- just to write that word -- the left null space of A.

So just the way we have the row space of A and we switch it to the column space of A transpose, so we have this space of guys l- that I call the left null space of A, but the good notation is it's the null space of A transpose.

What, what big space are they in for -- when A is m by n?

In that case, the null space of A, what's in the null space of A?

Vectors with n components, solutions to A x equals zero.

So the null space of A is in R^n.

What's in the column space of A?

How many components dothose columns have?

m. So this column space is in R^m.

What about the column space of A transpose, which are just a disguised way of saying the rows of A?

The rows of A, in this three by six matrix, have six components, n components.

The column space is in R^n.

And the null space of A transpose, I see that this fourth space is already getting second, you know, second class citizen treatment and it doesn't deserve it.

It's, it should be there, it is there, and shouldn't be squeezed.

The null space of A transpose -- well, if the null space of A had vectors with n components, the null space of A transpose will be in R^m.

I want to draw a picture of the four spaces.

OK. Here are the four spaces.

OK, Let me put n dimensional space over on this side.

Then which were the subspaces in R^n?

The null space was and the row space was.

So here we have the -- can I make that picture of the row space?

And can I make this kind of picture of the null space?

That's just meant to be a sketch, to remind you that they're in this -- which you know, how -- what type of vectors are in it?

Vectors with n components.

Over here, inside, consisting of vectors with m components, is the column space and what I'm calling the null space of A transpose.

Those are the ones with m components.

To understand these spaces is our, is our job now.

Because by understanding those spaces, we know everything about this half of linear algebra.

What do I mean by understanding those spaces?

I would like to know a basis for those spaces.

For each one of those spaces, how would I create -- construct a basis?

What systematic way would produce a basis?

And what's their dimension?

OK. So for each of the four spaces, I have to answer those questions.

And then -- which has a somewhat long answer.

And what's the dimension, which is just a number, so it has a real short answer.

Can I give you the short answer first?

I shouldn't do it, but here it is.

I can tell you the dimension of the column space.

Let me start with this guy.

The dimension of the column space is the rank,

r. We actually got to that at the end of the last lecture, but only for an example.

So I really have to say, OK, what's going on there.

I should produce a basis and then I just look to see how many vectors I needed in that basis, and the answer will be r.

Actually, I'll do that, before I get on to the others.

What's a basis for the columns space?

We've done all the work of row reduction, identifying the pivot columns, the ones that have pivots, the ones that end up with pivots.

But now I -- the pivot columns I'm interested in are columns of A, the original A.

And those pivot columns, there are r of them.

So if I answer this question for the column space, the answer will be a basis is the pivot columns and the dimension is the rank r, and there are r pivot columns and everything great.

So that space we pretty well understand.

I probably have a little going back to see that -- to prove that this is a right answer, but you know it's the right answer.

Now let me look at the row space.

Shall I tell you the dimension of the row space?

Yes. Before we do even an example, let me tell you the dimension of the row space.

The row space and the column space have the same dimension.

The dimension of the column space of A transpose -- that's the row space -- is r.

That, that space is r dimensional.

That's the sort of insight that got used in this example.

If those -- are the three columns of a matrix -- let me make them the three columns of a matrix by just erasing some brackets.

OK, those are the three columns of a matrix.

The rank of that matrix, if I look at the columns, it wasn't obvious to me anyway.

But if I look at the rows, now it's obvious.

The row space of that matrix obviously is two dimensional, because I see a basis for the row space, this row and that row.

And of course, strictly speaking, I'm supposed to transpose those guys, make them stand up.

But the rank is two, and therefore the column space is two dimensional by this wonderful fact that the row space and column space have the same dimension.

And therefore there are only two pivot columns, not three, and, those, the three columns are dependent.

Now let me bury that error and talk about the row space.

Well, I'm going to give you the dimensions of all the spaces.

Because that's such a nice answer.

OK. So let me come back here.

So we have this great fact to establish, that the row space, its dimension is also the rank.

What about the null space?

What's a basis for the null space?

What's the dimension of the null space?

Let me, I'll put that answer up here for the null space.

Well, how have we constructed the null space?

We took the matrix A, we did those row operations to get it into a form U or, or even further.

We got it into the reduced form R.

And then we read off special solutions.

And every special solution came from a free variable.

And those special solutions are in the null space, and the great thing is they're a basis for it.

So for the null space, a basis will be the special solutions.

And there's one for every free variable, right?

For each free variable, we give that variable the value one, the other free variables zero.

We get the pivot variables, we get a vector in the -- we get a special solution.

So we get altogether n-r of them, because that's the number of free variables.

If we have r -- this is the dimension is r, is the number of pivot variables.

This is the number of free variables.

So the beauty is that those special solutions do form a basis and tell us immediately that the dimension of the null space is n -- I better write this well, because it's so nice -- n-r. And do you see the nice thing?

That the two dimensions in this n dimensional space, one subspace is r dimensional -- to be proved, that's the row space.

The other subspace is n-r dimensional, that's the null space.

And the two dimensions like together give n.

It's really copying the fact that we have n variables, r of them are pivot variables and n-r are free variables, and n altogether.

OK. And now what's the dimension of this poor misbegotten fourth subspace?

It's got to be m-r. The dimension of this left null space, left out practically, is m-r. Well, that's really just saying that this -- again, the sum of that plus that is m, and m is correct, it's the number of columns in A transpose.

A transpose is just as good a matrix as A.

It just happens to be n by m.

It happens to have m columns, so it will have m variables when I go to A x equals 0 and m of them, and r of them will be pivot variables and m-r will be free variables.

A transpose is as good a matrix as A.

It follows the same rule that the this plus the dimension -- this dimension plus this dimension adds up to the number of columns.

And over here, A transpose has m columns.

OK. OK. So I gave you the easy answer, the dimensions.

Now can I go back to check on a basis?

We would like to think that -- say the row space, because we've got a basis for the column space.

The pivot columns give a basis for the column space.

Now I'm asking you to look at the row space.

And I -- you could say, OK, I can produce a basis for the row space by transposing my matrix, making those columns, then doing elimination, row reduction, and checking out the pivot columns in this transposed matrix.

But that means you had to do all that row reduction on A transpose.

It ought to be possible, if we take a matrix A -- let me take the matrix -- maybe we had this matrix in the last lecture.

OK. That, that matrix was so easy.

We spotted its pivot columns, one and two, without actually doing row reduction.

But now let's do the job properly.

So I subtract this away from this to produce a zero.

Subtracting that away leaves me minus 1 -1 0, right?

And subtracting that from the last row, oh, well that's easy.

Now I've -- the first column is all set.

The second column I now see the pivot.

And I can clean up, if I -- actually,

OK. Why don't I make the pivot into a 1. I'll multiply that row through by by -1, and then I have 1 1. That was an elementary operation I'm allowed, multiply a row by a number.

And now I'll do elimination.

Two of those away from that will knock this guy out and make this into a 1. So that's now a 0 and that's a

I'm seeing the identity matrix here.

What happened to its row space -- well, what happened -- let me first ask, just because this is, is -- sometimes something does happen.

The column space of R is not the column space of A, right?

Because 1 1 1 is certainly in the column space of A and certainly not in the column space of R.

Those row operations preserve the row space.

So the row, so the column spaces are different.

Different column spaces, different column spaces.

But I believe that they have the same row space.

I believe that the row space of that matrix and the row space of this matrix are identical.

They have exactly the same vectors in them.

Those vectors are vectors with four components, right?

They're all combinations of those rows.

Or I believe you get the same thing by taking all combinations of these rows.

And if true, what's a basis?

What's a basis for the row space of R, and it'll be a basis for the row space of the original A, but it's obviously a basis for the row space of R.

What's a basis for the row space of that matrix?

So a basis for the row -- so a basis is, for the row space of A or of R, is, is the first R rows of R.

Sometimes it's true for A, but not necessarily.

But R, we definitely have a matrix here whose row space we can, we can identify.

The row space is spanned by the three rows, but if we want a basis we want independence.

The row space is also spanned by the first two rows.

This guy didn't contribute anything.

And of course over here this 1 2 3 1 in the bottom didn't contribute anything.

So this, here is a basis. 1 0 1 1 and 0 1 1 0. I believe those are in the row space.

I know they're independent.

Why are they in the row space?

Why are those two vectors in the row space?

Because all those operations we did, which started with these rows and took combinations of them -- I took this row minus this row, that gave me something that's still in the row space.

When I took a row minus a multiple of another row, I'm staying in the row space.

The row space is not changing.

My little basis for it is changing, and I've ended up with, sort of the best basis.

If the columns of the identity matrix are the best basis for R^3 or R^n, the rows of this matrix are the best basis for the row space.

Best in the sense of being as clean as I can make it.

Starting off with the identity and then finishing up with whatever has to be in there.

Do you see then that the dimension is r?

For sure, because we've got r pivots, r non-zero rows.

We've got the right number of vectors, r.

They're in the row space, they're independent.

They are a basis for the row space.

And we can even pin that down further.

How do I know that every row of A is a combination?

How do I know they span the row space?

Well, somebody says, I've got the right number of them, so they must.

But let me just say, how do I know that this row is a combination of these?

By just reversing the steps of row reduction.

If I just reverse the steps and go from A -- from R back to A, then what do I, what I doing?

I'm starting with these rows, I'm taking combinations of them.

After a couple of steps, undoing the subtractions that I did before, I'm back to these rows.

So these rows are combinations of those rows.

Those rows are combinations of those rows.

The two row spaces are the same.

And the natural basis is this guy.

Is that all right for the row space?

The row space is sitting there in R in its cleanest possible form.

OK. Now what about the fourth guy, the null space of A transpose?

First of all, why do I call that the left null space?

So let me save that and bring that down.

So the fourth space is the null space of A transpose.

So it has in it vectors, let me call them y, so that A transpose y equals 0. If A transpose y equals 0, then y is in the null space of A transpose, of course.

So this is a matrix times a column equaling zero.

And now, because I want y to sit on the left and I want A instead of A transpose, I'll just transpose that equation.

Can I just transpose that?

On the right, it makes the zero vector lie down.

And on the left, it's a product, A, A transpose times y.

If I take the transpose, then they come in opposite order, right?

So that's y transpose times A transpose transpose.

But nobody's going to leave it like that.

That's -- A transpose transpose is just A, of course.

When I transposed A transpose I got back to A.

Now do you see what I have now?

I have a row vector, y transpose, multiplying A, and multiplying from the left.

That's why I call it the left null space.

But by making it -- putting it on the left, I had to make it into a row instead of a column vector, and so my convention is I usually don't do that.

I usually stay with A transpose y equals 0. OK.

And you might ask, how do we get a basis -- or I might ask, how do we get a basis for this fourth space, this left null space?

OK. I'll do it in the example.

The left null space is not jumping out at me here.

I know which are the free variables -- the special solutions, but those are special solutions to A x equals zero, and now I'm looking at A transpose, and I'm not seeing it here.

So -- but somehow you feel that the work that you did which simplified A to R should have revealed the left null space too.

And it's slightly less immediate, but it's there.

So from A to R, I took some steps, and I guess I'm interested in what were those steps, or what were all of them together.

I don't -- I'm not interested in what particular ones they were.

I'm interested in what was the whole matrix that took me from A to R.

Do you remember Gauss-Jordan, where you tack on the identity matrix?

So I, I'll do it above, here.

So this is now, this is now the idea of -- I take the matrix A, which is m by n.

In Gauss-Jordan, when we saw him before -- A was a square invertible matrix and we were finding its inverse.

Now the matrix isn't square.

But I'll still tack on the identity matrix, and of course since these have length m it better be m by m.

And now I'll do the reduced row echelon form of this matrix.

The reduced row echelon form starts with these columns, starts with the first columns, works like mad, and produces R.

Of course, still that same size, m by n.

And then whatever it did to get R, something else is going to show up here.

It's whatever -- do you see that E is just going to contain a record of what we did?

We did whatever it took to get A to become R.

And at the same time, we were doing it to the identity matrix.

So we started with the identity matrix, we buzzed along.

So we took some -- all this row reduction amounted to multiplying on the left by some matrix, some series of elementary matrices that altogether gave us one matrix, and that matrix is E.

So all this row reduction stuff amounted to multiplying by E.

It certainly amounted to multiply it by something.

And that something took I to E, so that something was E.

So now look at the first part, E A is R.

All I've said is that the row reduction steps that we all know -- well, taking A to R, are in a, in some matrix, and I can find out what that matrix is by just tacking I on and seeing what comes out.

Let's just review the invertible square case.

Because I was interested in it in chapter two also.

When A was square and invertible, I took A I, I did row, row elimination.

And what was the R that came out?

So in chapter two, in chapter two, R was I.

The row the, the reduced row echelon form of a nice invertible square matrix is the identity.

So if R was I in that case, then E was -- then E was A inverse, because E A is I.

Good. That's, that was good and easy.

Now what I'm saying is that there still is an E.

It's not A inverse any more, because A is rectangular.

But there is still some matrix E that connected this to this -- oh, I should have figured out in advanced what it was.

I didn't -- I did those steps and sort of erased as I went -- and, I should have done them to the identity too.

I'll keep the identity matrix, like I'm supposed to do, and I'll do the same operations on it, and see what I end up

So I'm starting with the identity -- which I'll write in light, light enough, but -- OK.

I subtracted that row from that one and that row from that one.

OK, I'll do that to the identity.

So I subtract that first row from row two and row three.

Then I think I multiplied -- Do you remember?

I multiplied row two by minus one.

I subtracted two of row two away from row one.

Subtract two of this away from this.

That's minus one, two of these away leaves a plus 2 and 0. I believe that's E.

The way to check is to see, multiply that E by this A, just to see did I do it right.

So I believe E was -1 2 0, 1 -1 0, and -1 0 1. OK.

That's my E, that's my A, and that's R.

All I'm struggling to do is right, the reason I wanted this blasted E was so that I could figure out the left null space, not only its dimension, which I know -- actually, what is the dimension of the left null space?

What's the rank of the matrix?

And the dimension of the null -- of the left null space is supposed to be m-r. It's 3 -2, 1. I believe that the left null space is one dimensional.

There is one combination of those three rows that produces the zero row.

There's a basis -- a basis for the left null space has only got one vector in it.

It's here in the last row of E.

But we could have seen it earlier.

What combination of those rows gives the zero row? -1 of that plus one of that.

So a basis for the left null space of this matrix -- I'm looking for combinations of rows that give the zero row if I'm looking at the left null space.

For the null space, I'm looking at combinations of columns to get the zero column.

Now I'm looking at combinations of these three rows to get the zero row, and of course there is my zero row, and here is my vector that produced it. -1 of that row and one of that

OK. So in that example -- and actually in all examples, we have seen how to produce a basis for the left null space.

I won't ask you that all the time, because -- it didn't come out immediately from R.

We had to keep track of E for that left null space.

But at least it didn't require us to transpose the matrix and start all over again.

OK, those are the four subspaces.

The row space and the null space are in R^n.

Their dimensions add to n.

The column space and the left null space are in R^m, and their dimensions add to m.

So let me close these last minutes by pushing you a little bit more to a new type of vector space.

All our vector spaces, all the ones that we took seriously, have been subspaces of some real three or n dimensional space.

Now I'm going to write down another vector space, a new vector space.

Say all three by three matrices.

My matrices are the vectors.

You can put quotes around vectors.

Every three by three matrix is one of my vectors.

Now how I entitled to call those things vectors?

I mean, they look very much like matrices.

But they are vectors in my vector space because they obey

the rules. All I'm supposed to be able to do with vectors is add them -- I can add matrices -- I'm supposed to be able to multiply them by scalar numbers like seven -- well, I can multiply a matrix by And that -- and I can take combinations of matrices, I can take three of one matrix minus five of another

matrix. And those combinations, there's a zero matrix, the matrix that has all zeros in it.

If I add that to another matrix, it doesn't change it.

If I multiply a matrix by one it doesn't change it.

All those eight rules for a vector space that we never wrote down, all easily satisfied.

So now we have a different -- now of course you can say you can multiply those matrices.

For the moment, I'm only thinking of these matrices as forming a vector space -- so I only doing A plus B and c times A.

I'm not interested in A B for now.

The fact that I can multiply is not relevant to th- to a vector space.

OK. So I have three by three matrices.

What's -- tell me a subspace of this matrix space.

Let me call this matrix space M.

That's my matrix space, my space of all three by three matrices.

What about the upper triangular matrices?

All, all upper triangular matrices.

The intersection of two subspaces is supposed to be a subspace.

We gave a little effort to the proof of that fact.

If I look at the matrices that are in this subspace -- they're symmetric, and they're also in this subspace, they're upper triangular, what do they look like?

Well, if they're symmetric but they have zeros below the diagonal, they better have zeros above the diagonal, so the intersection would be diagonal matrices.

That's another subspace, smaller than those.

How can I use the word smaller?

Well, I'm now entitled to use the word smaller.

I mean, well, one way to say is, OK, these are contained in those.

These are contained in those.

But more precisely, I could give the dimension of these spaces.

So I could -- we can compute -- let's compute it next time, the dimension of all upper -- of the subspace of upper triangular three by three matrices.

The dimension of symmetric three by three matrices.

The dimension of diagonal three by three matrices.

Well, to produce dimension, that means I'm supposed to produce a basis, and then I just count how many vecto- how many I needed in the basis.

Let me give you the answer for this one.

The dimension of this -- say, this subspace, let me call it D, all diagonal matrices.

The dimension of this subspace is -- as I write you're working it out -- three.

Because here's a matrix in this -- it's a diagonal matrix.

Better make it diagonal, let me put a seven there.

That was not a very great choice, but it's three diagonal matrices, and I believe that they're a basis.

I believe that those three matrices are independent and I believe that any diagonal matrix is a combination of those three.

So they span the subspace of diagonal matrices.

It's like stretching the idea from R^n to R^(n by n), three by three.

But the -- we can still add, we can still multiply by numbers, and we just ignore the fact that we can multiply two matrices together.

## Solutions Manual to accompany Fundamentals of Matrix Analysis with Applications

Solutions Manual to accompany **Fundamentals of Matrix Analysis with Applications&mdash**an accessible and clear introduction to linear algebra with a focus on matrices and engineering applications.

#### Buy Set of 2 Items

**This item:** Solutions Manual to accompany Fundamentals of Matrix Analysis with Applications

## Fundamentals of Matrix Computations, 3rd Edition

Retaining the accessible and hands-on style of its predecessor, *Fundamentals of Matrix Computations, Third Edition* thoroughly details matrix computations and the accompanying theory alongside the author's useful insights. The book presents the most important algorithms of numerical linear algebra and helps readers to understand how the algorithms are developed and why they work.

Along with new and updated examples, the *Third Edition* features:

- A novel approach to Francis' QR algorithm that explains its properties without reference to the basic QR algorithm
- Application of classical Gram-Schmidt with reorthogonalization
- A revised approach to the derivation of the Golub-Reinsch SVD algorithm
- New coverage on solving product eigenvalue problems
- Expanded treatment of the Jacobi-Davidson method
- A new discussion on stopping criteria for iterative methods for solving linear equations

Throughout the book, numerous new and updated exercises—ranging from routine computations and verifications to challenging programming and proofs—are provided, allowing readers to immediately engage in applying the presented concepts. The new edition also incorporates MATLAB to solve real-world problems in electrical circuits, mass-spring systems, and simple partial differential equations, and an index of MATLAB terms assists readers with understanding the basic concepts related to the software.

*Fundamentals of Matrix Computations, Third Edition* is an excellent book for courses on matrix computations and applied numerical linear algebra at the upper-undergraduate and graduate level. The book is also a valuable resource for researchers and practitioners working in the fields of engineering and computer science who need to know how to solve problems involving matrix computations.

## A Matrix Algebra Approach to Artificial Intelligence

Matrix algebra plays an important role in many core artificial intelligence (AI) areas, including machine learning, neural networks, support vector machines (SVMs) and evolutionary computation. This book offers a comprehensive and in-depth discussion of matrix algebra theory and methods for these four core areas of AI, while also approaching AI from a theoretical matrix algebra perspective.

The book consists of two parts: the first discusses the fundamentals of matrix algebra in detail, while the second focuses on the applications of matrix algebra approaches in AI. Highlighting matrix algebra in graph-based learning and embedding, network embedding, convolutional neural networks and Pareto optimization theory, and discussing recent topics and advances, the book offers a valuable resource for scientists, engineers, and graduate students in various disciplines, including, but not limited to, computer science, mathematics and engineering.

**XIAN-DA ZHANG** is a Professor Emeritus at the Department of Automation, Tsinghua University, China. He was a Distinguished Professor at Xidian University, Xi’an, China, as part of the Ministry of Education of China and Cheung Kong Scholars Programme, from 1999 to 2002. His areas of research include intelligent signal and information processing, pattern recognition, machine learning and neural networks, evolutional computation, and correlated applied mathematics. He has published over 120 international journal and conference papers. The Japanese translation of his book “Linear Algebra in Signal Processing” (published in Chinese by Science Press, Beijing, in 1997) was published by Morikita Press, Tokyo, in 2008. He also authored the book “Matrix Analysis and Applications” (Cambridge University Press, UK, 2017).

“The book is of very high relevance for students, professors and researchers involved in artificial intelligence (AI), the work is also of very high relevance for the mathematics community in general since it addresses the importance of matrix algebra for the field of AI and how major approaches and state-of-the-art algorithms rely on matrix algebra.” (Carlos Pedro Gonçalves, zbMATH 1455.68010, 2021)