Reference notebooks

None of this is my work. Everything whose licence permits redistribution is reproduced here in full and unmodified, each page carrying its author, the work it comes from, its licence and a link back to the original; a runnable copy lives in references/vendor/ in the repo. Two sources ship no licence at all, which under GitHub's terms means all rights reserved, so those stay linked at the source and are fetched on demand by references/fetch.py.

Linear algebra

Vectors, matrices as transformations, eigendecomposition and SVD, drawn rather than just derived.

• Math: Linear Algebra

  Aurélien Géron · Hands-On Machine Learning, 3rd ed.

  The strongest standalone notebook of the set: every idea gets a matplotlib picture. Vectors and norms, addition as translation, matrices as linear transformations, determinants as signed area, inverses, rank and projections, all in numpy, with the notation written out beside each plot.

  VectorsMatrix transformationsDeterminantsInverse & ranknumpymatplotlib

  vendored Rendered in full below, runnable copy at references/vendor/handson-ml3/math_linear_algebra.ipynb

  Read: Math: Linear AlgebraUpstream notebookOpen in ColabRepoApache-2.0

• ML Foundations: Linear Algebra I & II

  Jon Krohn

  A dense notation reference. Scalars through tensors, norms, dot products, matrix multiplication, eigendecomposition and SVD, each written first in mathematical notation, then in numpy, PyTorch and TensorFlow side by side, which is useful when the same object has three names depending on which library you're reading.

  TensorsNormsEigendecompositionSVDPyTorchTensorFlow

  vendored Rendered in full below, runnable copy at references/vendor/ml-foundations/

  Read: Intro to Linear AlgebraRead: Linear Algebra II: Matrix OperationsLinear Algebra ILinear Algebra IIRepoMIT

• Land on Vector Spaces

  Engineers Code · Lorena Barba group, GWU

  Four notebooks that treat a matrix as a thing that moves space around. Animated transformations, change of basis, eigenvectors as the directions a transformation leaves pointing where they were, and SVD built up visually. The closest thing in notebook form to 3Blue1Brown's linear algebra series.

  Linear transformationsChange of basisEigenvectorsSVDAnimation

  vendored Rendered in full below, runnable copy at references/vendor/landlinear/

  Read: Transform all the vectorsRead: The matrix is everywhereRead: Eigenvectors for the winRead: Stick to the essentials: SVDNotebooksRepoCC BY 4.0 · BSD-3-Clause

• Computational Linear Algebra

  Rachel Thomas · fast.ai / USF MSDS

  Linear algebra taught backwards from its applications: topic modelling with NMF and SVD, background removal with robust PCA, PageRank via eigendecomposition, QR factorisation implemented and then profiled against LAPACK. numpy, scipy and scikit-learn throughout, with a video lecture per notebook.

  SVDNMFRobust PCAPageRankQRConditioning

  fetch Fetch locally: python references/fetch.py fastai-nla

  NotebooksCourse READMENo licence declared; fetched, not redistributed

• Mathematics for Machine Learning: tutorials

  Deisenroth, Faisal & Ong

  The companion notebooks to the MML book: PCA derived from orthogonal projections, linear regression from maximum likelihood, and a Gaussian mixture fitted with EM. This is where the linear algebra stops being abstract and starts being a model. Pairs with the free PDF of the book.

  PCAProjectionsLinear regressionGaussian mixtures

  fetch Fetch locally: python references/fetch.py mml-book

  PCA tutorialLinear regression tutorialBook (free PDF)No licence declared; fetched, not redistributed

Calculus

Limits, derivatives, gradients and the chain rule, with the plots that make each one obvious.

• Math: Differential Calculus

  Aurélien Géron · Hands-On Machine Learning, 3rd ed.

  Slope of a line, then the limit definition, then the derivative, with an animation of the secant collapsing onto the tangent rather than a paragraph asking you to imagine it. Runs through differentiability, the differentiation rules, higher-order derivatives, partial derivatives, gradients, Jacobians and Hessians, and proves each rule at the end.

  LimitsTangentsChain ruleGradientsJacobiansHessians

  vendored Rendered in full below, runnable copy at references/vendor/handson-ml3/math_differential_calculus.ipynb

  Read: Math: Differential CalculusUpstream notebookOpen in ColabRepoApache-2.0

• ML Foundations: Calculus I & II

  Jon Krohn

  Limits and the delta method, then the rules (product, quotient, chain), then partial derivatives and the gradient, and finally the same derivatives taken automatically by PyTorch and TensorFlow autodiff. The bridge from a derivative computed by hand to the one a framework hands back.

  LimitsChain rulePartial derivativesAutodiffGradients

  vendored Rendered in full below, runnable copy at references/vendor/ml-foundations/

  Read: Calculus I: Limits & DerivativesRead: Calculus II: Partial Derivatives & IntegralsCalculus ICalculus IIRepoMIT

• Machine Learning Refined: course notes

  Watt, Borhani & Katsaggelos

  Visualisation-first treatment of the calculus optimisation actually uses: first- and second-order Taylor approximations drawn on the surface they approximate, gradient descent and Newton's method animated on the same cost function. Leans towards building things from scratch, so read it for the pictures.

  Taylor seriesGradient descentNewton's methodCost surfacesAnimation

  vendored Rendered in full below, runnable copy at references/vendor/ml-refined/

  Read: The First-Order Optimality ConditionRead: The Geometry of First-Order Taylor SeriesRead: Gradient DescentRead: Quadratic FunctionsRead: Newton's MethodRead: Vectors and Vector OperationsRead: Vector and Matrix NormsNotesRepoCC BY-NC-SA 4.0

Both, in one place

Book-length references where the maths sits next to the models that use it.

• Dive into Deep Learning: maths chapters

  Zhang, Lipton, Li & Smola

  Every page is a runnable notebook and every chapter carries figures. The preliminaries cover linear algebra, calculus and automatic differentiation in the framework you'll actually use; the appendix goes further into the geometry of linear-algebraic operations, eigendecomposition, and single- and multivariable calculus. Notebooks download per page for PyTorch, TensorFlow, JAX or MXNet. Left linked rather than mirrored: the book is already a website, and its notebooks ship without saved outputs, so a copy here would read worse than the original.

  Linear algebraCalculusAutogradGeometryEigendecomposition

  fetch Fetch locally: python references/fetch.py d2l

  Linear algebraCalculusAutomatic differentiationGeometry & linear-algebraic opsMultivariable calculusCC BY-SA 4.0 (text) · MIT (code)

• The Matrix Calculus You Need For Deep Learning

  Terence Parr & Jeremy Howard

  An article rather than a notebook, but the notation reference for everything above. Jacobians, the vector chain rule and the element-wise binary operator rules, written in exactly the form they turn up in backpropagation derivations, with every symbol defined as it appears.

  JacobiansVector chain ruleNotationBackprop

  link Read at the source

  Read onlinearXiv PDFFree to read online