1 Linear Algebra
2 Scalars, Vectors, Matrices, and TensorsScalars: a single number (vs. objects) Ex) real-valued scalars, natural number scalar Lower-case 표기 Vectors: an array of numbers identifying points in space To access x1, x3, and x6, we define the set S = {1, 3, 6} and write xS - sign: the complement of a set Ex) x-1 : the vector containing all elements of x except for x1 Bold lower-case 표기
3 Scalars, Vectors, Matrices, and Tensorsupper-case variable names with bold typeface, such as A If a real-valued matrix A has a height of m and a width of n, then we say that Ai,j => i행 j열 위치의 element Ai,: => i행 A:,j => j열
4 Scalars, Vectors, Matrices, and TensorsTranspose (전치)
5 Scalars, Vectors, Matrices, and Tensorsan array of numbers arranged on a regular grid with a variable number of axes Notation: “A” with this typeface: A
6 Scalars, Vectors, Matrices, and TensorsVector vs. Matrix Addition matrix and matrix matrix and scalar matrix and vector Vector b가 matrix A의 각 row에 addition
7 Multiplying Matrices and VectorsMatrix product of matrices A and B A must have the same number of columns as B has rows. If A is of shape m × n and B is of shape n × p, then C is of shape m × p.
8 Multiplying Matrices and VectorsElement-wise product (or Hadamard product)
9 Multiplying Matrices and VectorsMatrix multiplication Distributive Law Associative Law Not commutative
10 Multiplying Matrices and VectorsDot product between two vectors Commutative Law Matrix transpose Vector commutative law 증명
11 Properties of Transpose
12 Linear Equation Systemis a
13 Linear Equation System
14 Matrix inversion Identity matrix I3 =
15 Matrix inversion How to solve the linear equation
16 Norms To measure the size of a vector PropertiesThe norm of a vector x measures the distance from the origin to the point x. Properties
17 Norms Euclidean norm when p=2 denoted asCommon to measure the size of a vector using the squared L2 norm as In many contexts, the squared L2 norm may be undesirable because it increases very slowly near the origin.
18 Norms L1 norm In several machine learning applications, it is important to discriminate between elements that are exactly zero and elements that are small but nonzero commonly used in machine learning when the difference betweenzero and nonzero elements is very important
19 Norms Max norm
20 Norms We want to measure the size of a matrix.In the context of deep learning, the most common way to do this is with the Frobenius norm:
21 Special Kinds of Matrices and VectorsDiagonal matrix A matrix D is diagonal if and only if Di,j= 0 for all i ≠ j. diag(v) : a square diagonal matrix whose diagonal entries are given by the entries of the vector v diag(v) x : to scale each element xi by vi Inversion
22 Special Kinds of Matrices and VectorsSymmetric matrix Ex) a matrix of distance measurements with Ai,j giving the distance from point i to point j
23 Special Kinds of Matrices and VectorsUnit vector
24 Special Kinds of Matrices and VectorsOrthogonal vectors Orthonormal vectors If the vectors are not only orthogonal but also have unit norm, we call them orthonormal.
25 Special Kinds of Matrices and VectorsAn orthogonal matrix A square matrix whose rows are mutually orthonormal and whose columns are mutually orthonormal Orthogonal matrix는 inverse matrix를 구하기가 쉬움
26 Eigendecomposition 비유: 정수 => 인수분해 EigendecompositionEx) 12 = 2*2*3 => 12는 5로 나눌 수 없음. 12의 배수는 3으로 나눌 수 있음 Eigendecomposition To decompose a matrix into a set of eigenvectors and eigenvalues eigenvector eigenvalue Eigenvector는 1개 이상 나올 수 있으며, 주어진 eigenvector에 대해서 해당 eigenvalue가 존재
27 Eigendecomposition
28 Eigendecomposition Suppose that a matrix A has n linearly independent eigenvectors, V={v(1), ,v(n)}, with corresponding eigenvalues λ ={λ1, , λn}.
29 Eigendecomposition Every real symmetric matrix can be decomposed into an expression using only real-valued eigenvectors and eigenvalues: Q is an orthogonal matrix composed of eigenvectors of A, and Λ is a diagonal matrix. The eigenvalue Λi,i is associated with the eigenvector in column i of Q, denoted as Q:,i
30 Eigendecomposition
31 Eigendecomposition u Au The effect of eigenvectors and eigenvaluea matrix A with two orthonormal eigenvectors, v(1) with eigenvalue λ1 and v(2) with eigenvalue λ2 u Au The effect of eigenvectors and eigenvalue
32 Eigendecomposition The matrix is singular if and only if any of the eigenvalues are zero. A matrix whose eigenvalues are all positive is called positive definite. A matrix whose eigenvalues are all positive or zero-valued is called positive semi-definite.
33 Singular Value DecompositionSVD : another way to factorize a matrix into singular values and singular values If a matrix is not square, the eigendecomposition is not defined, and we must use a singular value decomposition instead. A: m*n matrix, U: m*m matrix, D: m*n matrix, V: n*n matrix U, V: orthogonal matrix
34 Singular Value Decomposition
35 Moore-Penrose PseudoinverseMatrix inversion is not defined for matrices that are not square. 𝑨 −𝟏 =𝑩 =
36 Moore-Penrose PseudoinverseA+ 활용 linear eq. 풀기 =
37 Trace Operator
38 Determinant The determinant of a square matrix A: det(A)a function mapping matrices to real scalars equal to the product of all the eigenvalues of the matrix
39 Determinant
40 Principal Components Analysis (PCA)f(.) : encoding function that produces the code for an input g(.): decoding function that produces the reconstructed input given its code To keep the encoding problem easy, PCA constrains the columns of D to be orthogonal to each other
41 Principal Components Analysis (PCA)
42 Principal Components Analysis (PCA)To keep the encoding problem easy, PCA constrains the columns of D to be orthogonal to each other Using vector calculus
43 Principal Components Analysis (PCA)we can optimally encode x just using a matrix-vector operation. Then, we need to choose the encoding matrix D.
44 Principal Components Analysis (PCA)We must minimize the Frobenius norm of the matrix of errors computed over all dimensions and all points: When l=1 (i.e., D is a single vector, d) scalar
45 Principal Components Analysis (PCA)Scalar의 transpose는 자기 자신 Stacking all of the vectors describing the points
46 Principal Components Analysis (PCA)
47 Principal Components Analysis (PCA)