{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "(linear-regression)=\n", "# Multiple linear regression\n", "\n", "In practice, there usually more than one feature. Suppose that we have $n$ training samples and $d$ features. Given a training sample $\\boldsymbol x \\in\\mathbb R^d$, we predict the target variable as follows:\n", "\n", "```{math}\n", " :label: multiple-lin-reg\n", " y = w_0 + w_1 x_1 + \\ldots + w_d x_d = w_0 + \\sum\\limits_{j=1}^d w_j x_j\n", "```\n", "\n", "To fit the linear regression, minimize MSE or, equivalently, RSS:\n", "\n", "```{math}\n", " :label: multiple-lin-reg-rss\n", " RSS = \\sum\\limits_{i=1}^n \\Big(y_i - w_0 - \\sum\\limits_{j=1}^k w_j x_j\\Big)^2 \\to \\min\\limits_{\\boldsymbol w}.\n", "```\n", "\n", "## 2-d linear regression on Boston dataset\n", "\n", "{ref}`Here ` we trained a simple linear regression with a single feature (`lstat`). Now add one more feature: `age`." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
agelstatmedv
065.24.9824.0
178.99.1421.6
261.14.0334.7
345.82.9433.4
454.25.3336.2
\n", "
" ], "text/plain": [ " age lstat medv\n", "0 65.2 4.98 24.0\n", "1 78.9 9.14 21.6\n", "2 61.1 4.03 34.7\n", "3 45.8 2.94 33.4\n", "4 54.2 5.33 36.2" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import pandas as pd\n", "boston = pd.read_csv(\"../ISLP_datsets/Boston.csv\")\n", "boston[['age','lstat','medv']].head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now fit the linear regression model:" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "intercept: 33.2227605317929\n", "coefficients: [-1.03206856 0.03454434]\n", "r-score: 0.5512689379421003\n", "MSE: 37.88167709241267\n" ] } ], "source": [ "from sklearn.linear_model import LinearRegression\n", "import numpy as np\n", "\n", "X, y = boston[['lstat', 'age']], boston['medv']\n", "LR = LinearRegression()\n", "LR.fit(X, y)\n", "print(\"intercept:\", LR.intercept_)\n", "print(\"coefficients:\", LR.coef_)\n", "print(\"r-score:\", LR.score(X, y))\n", "print(\"MSE:\", np.mean((LR.predict(X) - y) ** 2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q**. Are these results better or worse than those of simple regression? Quadratic regression?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Analytical solution for best fit\n", "\n", "````{important}\n", "The intercept $w_0$ is a bit annoying since it has to be written separately all the time. For notational convenience introduce a fake feature $x_0 =1 $ for all samples. Than {eq}`multiple-lin-reg` can be rewritten as\n", "\n", "```{math}\n", " :label: multiple-lin-reg-inner\n", " \\sum\\limits_{j=0}^d w_j x_j = \\boldsymbol x^\\top \\boldsymbol w,\n", "```\n", "\n", "where $\\boldsymbol w$ is the vector of weights of the linear regression. \n", "````\n", "\n", "Taking into account this note, write the optimization task {eq}`multiple-lin-reg-rss` as\n", "\n", "```{math}\n", ":label: lin-reg-loss-opt\n", " \\mathcal L(\\boldsymbol w) = \\sum\\limits_{i=1}^n(y_i - \\boldsymbol x_i^\\top \\boldsymbol w)^2 \\to \\min\\limits_{\\boldsymbol w},\n", "```\n", "\n", "or\n", "\n", "$$\n", " \\mathcal L(\\boldsymbol w) = \\Vert\\boldsymbol {Xw} - \\boldsymbol y \\Vert_2^2 \\to \\min\\limits_{\\boldsymbol w}.\n", "$$\n", "\n", "Observe that\n", "\n", "$$\n", "\\Vert\\boldsymbol {Xw} - \\boldsymbol y \\Vert_2^2 = (\\boldsymbol{Xw} - \\boldsymbol y)^\\top(\\boldsymbol {Xw} - \\boldsymbol y) = \\boldsymbol w^\\top \\boldsymbol X^\\top \\boldsymbol X \\boldsymbol w - 2\\boldsymbol y^\\top \\boldsymbol{Xw} + \\boldsymbol y^\\top \\boldsymbol y.\n", "$$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now use some matrix calculus:\n", "\n", "* $\\nabla_{\\boldsymbol w} (\\boldsymbol y^\\top \\boldsymbol{Xw}) = \\boldsymbol X^\\top \\boldsymbol y$;\n", "* $\\nabla_{\\boldsymbol w}(\\boldsymbol w^\\top \\boldsymbol X^\\top \\boldsymbol X \\boldsymbol w) = 2\\boldsymbol X^\\top \\boldsymbol X\\boldsymbol w$.\n", "\n", "That's why the condition $\\nabla_{\\boldsymbol w}\\mathcal L(\\boldsymbol w)=\\boldsymbol 0$ implies that\n", "\n", "```{math}\n", " :label: lin-reg-grad\n", " \\boldsymbol X^\\top \\boldsymbol X\\boldsymbol w - \\boldsymbol X^\\top \\boldsymbol y = \\boldsymbol 0\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If the matrix $\\boldsymbol X$ has full column rank, then the square matrix $\\boldsymbol X^\\top \\boldsymbol X$ is invertible, and from {eq}`lin-reg-grad` we obtain that the optimal weights are\n", "\n", "```{math}\n", " :label: lin-reg-solution\n", " \\widehat{\\boldsymbol w} = (\\boldsymbol X^\\top \\boldsymbol X)^{-1} \\boldsymbol X^\\top \\boldsymbol y.\n", "```\n", "\n", "```{note}\n", "The matrix $\\boldsymbol X^\\dagger = (\\boldsymbol X^\\top \\boldsymbol X)^{-1} \\boldsymbol X^\\top$ is called **Moore-Penrose** {ref}`pseudo inverse `.\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q.** What is $\\boldsymbol X^\\dagger$ if $\\boldsymbol X$ is a square invertible matrix?\n", "\n", "W3sicXVlc3Rpb24iOiAiSG93IG1hbnkgYXJpdGhtZXRpYyBvcGVyYXRpb25zIGFyZSByZXF1aXJlZCB0byBjYWxjdWxhdGUgJFxcYm9sZHN5bWJvbCBYXlxcZGFnZ2VyJCBpZiAkXFxib2xkc3ltYm9sIFggXFxpbiBcXG1hdGhiYiBSXntuXFx0aW1lcyBkfSQiLCAidHlwZSI6ICJtYW55X2Nob2ljZSIsICJhbnN3ZXJzIjogW3siYW5zd2VyIjogIiRPKGReMykkIiwgImNvcnJlY3QiOiBmYWxzZSwgImZlZWRiYWNrIjogIlRoaXMgaXMgdGhlIGNvc3Qgb2YgaW52ZXJzaW9uIG9ubHkifSwgeyJhbnN3ZXIiOiAiJE9cXGJpZyhkXjIobitkKVxcYmlnKSQiLCAiY29ycmVjdCI6IHRydWUsICJmZWVkYmFjayI6ICJSaWdodCBvbiEgJGReMiBuJCB0byBjYWxjdWxhdGUgJFxcYm9sZHN5bWJvbCBYXlxcdG9wIFxcYm9sZHN5bWJvbCBYJCwgJGReMyQgdG8gaW52ZXJ0IGl0In0sIHsiYW5zd2VyIjogIiRPXFxiaWcoZChuXjIrZF4yKVxcYmlnKSQiLCAiY29ycmVjdCI6IGZhbHNlLCAiZmVlZGJhY2siOiAiTm9wZSwgJG5eMmQkIG5lZWRlZCB0byBjYWxjdWxhdGUgJFxcYm9sZHN5bWJvbCBYIFxcYm9sZHN5bWJvbCBYIF5cXHRvcCQsIG5vdCAkXFxib2xkc3ltYm9sIFheXFx0b3AgXFxib2xkc3ltYm9sIFgkIn0sIHsiYW5zd2VyIjogIiRPKGReMyArIG5eMykkIiwgImNvcnJlY3QiOiBmYWxzZSwgImZlZWRiYWNrIjogIk5vLCB0aGVyZSBpcyBub3RoaW5nIHRvIHNwZW5kICRPKG5eMykkIG9uIGl0In1dfV0=" ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "tags": [ "remove-input" ] }, "outputs": [ { "data": { "text/html": [ "
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/javascript": "var element=document.getElementById(\"pseudo_inv_complexity\");\n var questionsdUCgDlcWgnpd;\n try {\n questionsdUCgDlcWgnpd=JSON.parse(window.atob(element.innerHTML));\n } catch(err) {\n console.log(\"Fell into catch\");\n questionsdUCgDlcWgnpd = JSON.parse(element.innerHTML);\n }\n console.log(questionsdUCgDlcWgnpd);;\n // Make a random ID\nfunction makeid(length) {\n var result = [];\n var characters = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz';\n var charactersLength = characters.length;\n for (var i = 0; i < length; i++) {\n result.push(characters.charAt(Math.floor(Math.random() * charactersLength)));\n }\n return result.join('');\n}\n\n// Choose a random subset of an array. Can also be used to shuffle the array\nfunction getRandomSubarray(arr, size) {\n var shuffled = arr.slice(0), i = arr.length, temp, index;\n while (i--) {\n index = Math.floor((i + 1) * Math.random());\n temp = shuffled[index];\n shuffled[index] = shuffled[i];\n shuffled[i] = temp;\n }\n return shuffled.slice(0, size);\n}\n\nfunction printResponses(responsesContainer) {\n var responses=JSON.parse(responsesContainer.dataset.responses);\n var stringResponses='IMPORTANT!To preserve this answer sequence for submission, when you have finalized your answers:
  1. Copy the text in this cell below \"Answer String\"
  2. Double click on the cell directly below the Answer String, labeled \"Replace Me\"
  3. Select the whole \"Replace Me\" text
  4. Paste in your answer string and press shift-Enter.
  5. Save the notebook using the save icon or File->Save Notebook menu item



    6. Answer String:
    ';\n console.log(responses);\n responses.forEach((response, index) => {\n if (response) {\n console.log(index + ': ' + response);\n stringResponses+= index + ': ' + response +\"
    \";\n }\n });\n responsesContainer.innerHTML=stringResponses;\n}\nfunction check_mc() {\n var id = this.id.split('-')[0];\n //var response = this.id.split('-')[1];\n //console.log(response);\n //console.log(\"In check_mc(), id=\"+id);\n //console.log(event.srcElement.id) \n //console.log(event.srcElement.dataset.correct) \n //console.log(event.srcElement.dataset.feedback)\n\n var label = event.srcElement;\n //console.log(label, label.nodeName);\n var depth = 0;\n while ((label.nodeName != \"LABEL\") && (depth < 20)) {\n label = label.parentElement;\n console.log(depth, label);\n depth++;\n }\n\n\n\n var answers = label.parentElement.children;\n\n //console.log(answers);\n\n\n // Split behavior based on multiple choice vs many choice:\n var fb = document.getElementById(\"fb\" + id);\n\n\n\n\n if (fb.dataset.numcorrect == 1) {\n // What follows is for the saved responses stuff\n var outerContainer = fb.parentElement.parentElement;\n var responsesContainer = document.getElementById(\"responses\" + outerContainer.id);\n if (responsesContainer) {\n //console.log(responsesContainer);\n var response = label.firstChild.innerText;\n if (label.querySelector(\".QuizCode\")){\n response+= label.querySelector(\".QuizCode\").firstChild.innerText;\n }\n console.log(response);\n //console.log(document.getElementById(\"quizWrap\"+id));\n var qnum = document.getElementById(\"quizWrap\"+id).dataset.qnum;\n console.log(\"Question \" + qnum);\n //console.log(id, \", got numcorrect=\",fb.dataset.numcorrect);\n var responses=JSON.parse(responsesContainer.dataset.responses);\n console.log(responses);\n responses[qnum]= response;\n responsesContainer.setAttribute('data-responses', JSON.stringify(responses));\n printResponses(responsesContainer);\n }\n // End code to preserve responses\n \n for (var i = 0; i < answers.length; i++) {\n var child = answers[i];\n //console.log(child);\n child.className = \"MCButton\";\n }\n\n\n\n if (label.dataset.correct == \"true\") {\n // console.log(\"Correct action\");\n if (\"feedback\" in label.dataset) {\n fb.textContent = jaxify(label.dataset.feedback);\n } else {\n fb.textContent = \"Correct!\";\n }\n label.classList.add(\"correctButton\");\n\n fb.className = \"Feedback\";\n fb.classList.add(\"correct\");\n\n } else {\n if (\"feedback\" in label.dataset) {\n fb.textContent = jaxify(label.dataset.feedback);\n } else {\n fb.textContent = \"Incorrect -- try again.\";\n }\n //console.log(\"Error action\");\n label.classList.add(\"incorrectButton\");\n fb.className = \"Feedback\";\n fb.classList.add(\"incorrect\");\n }\n }\n else {\n var reset = false;\n var feedback;\n if (label.dataset.correct == \"true\") {\n if (\"feedback\" in label.dataset) {\n feedback = jaxify(label.dataset.feedback);\n } else {\n feedback = \"Correct!\";\n }\n if (label.dataset.answered <= 0) {\n if (fb.dataset.answeredcorrect < 0) {\n fb.dataset.answeredcorrect = 1;\n reset = true;\n } else {\n fb.dataset.answeredcorrect++;\n }\n if (reset) {\n for (var i = 0; i < answers.length; i++) {\n var child = answers[i];\n child.className = \"MCButton\";\n child.dataset.answered = 0;\n }\n }\n label.classList.add(\"correctButton\");\n label.dataset.answered = 1;\n fb.className = \"Feedback\";\n fb.classList.add(\"correct\");\n\n }\n } else {\n if (\"feedback\" in label.dataset) {\n feedback = jaxify(label.dataset.feedback);\n } else {\n feedback = \"Incorrect -- try again.\";\n }\n if (fb.dataset.answeredcorrect > 0) {\n fb.dataset.answeredcorrect = -1;\n reset = true;\n } else {\n fb.dataset.answeredcorrect--;\n }\n\n if (reset) {\n for (var i = 0; i < answers.length; i++) {\n var child = answers[i];\n child.className = \"MCButton\";\n child.dataset.answered = 0;\n }\n }\n label.classList.add(\"incorrectButton\");\n fb.className = \"Feedback\";\n fb.classList.add(\"incorrect\");\n }\n // What follows is for the saved responses stuff\n var outerContainer = fb.parentElement.parentElement;\n var responsesContainer = document.getElementById(\"responses\" + outerContainer.id);\n if (responsesContainer) {\n //console.log(responsesContainer);\n var response = label.firstChild.innerText;\n if (label.querySelector(\".QuizCode\")){\n response+= label.querySelector(\".QuizCode\").firstChild.innerText;\n }\n console.log(response);\n //console.log(document.getElementById(\"quizWrap\"+id));\n var qnum = document.getElementById(\"quizWrap\"+id).dataset.qnum;\n console.log(\"Question \" + qnum);\n //console.log(id, \", got numcorrect=\",fb.dataset.numcorrect);\n var responses=JSON.parse(responsesContainer.dataset.responses);\n if (label.dataset.correct == \"true\") {\n if (typeof(responses[qnum]) == \"object\"){\n if (!responses[qnum].includes(response))\n responses[qnum].push(response);\n } else{\n responses[qnum]= [ response ];\n }\n } else {\n responses[qnum]= response;\n }\n console.log(responses);\n responsesContainer.setAttribute('data-responses', JSON.stringify(responses));\n printResponses(responsesContainer);\n }\n // End save responses stuff\n\n\n\n var numcorrect = fb.dataset.numcorrect;\n var answeredcorrect = fb.dataset.answeredcorrect;\n if (answeredcorrect >= 0) {\n fb.textContent = feedback + \" [\" + answeredcorrect + \"/\" + numcorrect + \"]\";\n } else {\n fb.textContent = feedback + \" [\" + 0 + \"/\" + numcorrect + \"]\";\n }\n\n\n }\n\n if (typeof MathJax != 'undefined') {\n var version = MathJax.version;\n console.log('MathJax version', version);\n if (version[0] == \"2\") {\n MathJax.Hub.Queue([\"Typeset\", MathJax.Hub]);\n } else if (version[0] == \"3\") {\n MathJax.typeset([fb]);\n }\n } else {\n console.log('MathJax not detected');\n }\n\n}\n\nfunction make_mc(qa, shuffle_answers, outerqDiv, qDiv, aDiv, id) {\n var shuffled;\n if (shuffle_answers == \"True\") {\n //console.log(shuffle_answers+\" read as true\");\n shuffled = getRandomSubarray(qa.answers, qa.answers.length);\n } else {\n //console.log(shuffle_answers+\" read as false\");\n shuffled = qa.answers;\n }\n\n\n var num_correct = 0;\n\n\n\n shuffled.forEach((item, index, ans_array) => {\n //console.log(answer);\n\n // Make input element\n var inp = document.createElement(\"input\");\n inp.type = \"radio\";\n inp.id = \"quizo\" + id + index;\n inp.style = \"display:none;\";\n aDiv.append(inp);\n\n //Make label for input element\n var lab = document.createElement(\"label\");\n lab.className = \"MCButton\";\n lab.id = id + '-' + index;\n lab.onclick = check_mc;\n var aSpan = document.createElement('span');\n aSpan.classsName = \"\";\n //qDiv.id=\"quizQn\"+id+index;\n if (\"answer\" in item) {\n aSpan.innerHTML = jaxify(item.answer);\n //aSpan.innerHTML=item.answer;\n }\n lab.append(aSpan);\n\n // Create div for code inside question\n var codeSpan;\n if (\"code\" in item) {\n codeSpan = document.createElement('span');\n codeSpan.id = \"code\" + id + index;\n codeSpan.className = \"QuizCode\";\n var codePre = document.createElement('pre');\n codeSpan.append(codePre);\n var codeCode = document.createElement('code');\n codePre.append(codeCode);\n codeCode.innerHTML = item.code;\n lab.append(codeSpan);\n //console.log(codeSpan);\n }\n\n //lab.textContent=item.answer;\n\n // Set the data attributes for the answer\n lab.setAttribute('data-correct', item.correct);\n if (item.correct) {\n num_correct++;\n }\n if (\"feedback\" in item) {\n lab.setAttribute('data-feedback', item.feedback);\n }\n lab.setAttribute('data-answered', 0);\n\n aDiv.append(lab);\n\n });\n\n if (num_correct > 1) {\n outerqDiv.className = \"ManyChoiceQn\";\n } else {\n outerqDiv.className = \"MultipleChoiceQn\";\n }\n\n return num_correct;\n\n}\nfunction check_numeric(ths, event) {\n\n if (event.keyCode === 13) {\n ths.blur();\n\n var id = ths.id.split('-')[0];\n\n var submission = ths.value;\n if (submission.indexOf('/') != -1) {\n var sub_parts = submission.split('/');\n //console.log(sub_parts);\n submission = sub_parts[0] / sub_parts[1];\n }\n //console.log(\"Reader entered\", submission);\n\n if (\"precision\" in ths.dataset) {\n var precision = ths.dataset.precision;\n // console.log(\"1:\", submission)\n submission = Math.round((1 * submission + Number.EPSILON) * 10 ** precision) / 10 ** precision;\n // console.log(\"Rounded to \", submission, \" precision=\", precision );\n }\n\n\n //console.log(\"In check_numeric(), id=\"+id);\n //console.log(event.srcElement.id) \n //console.log(event.srcElement.dataset.feedback)\n\n var fb = document.getElementById(\"fb\" + id);\n fb.style.display = \"none\";\n fb.textContent = \"Incorrect -- try again.\";\n\n var answers = JSON.parse(ths.dataset.answers);\n //console.log(answers);\n\n var defaultFB = \"\";\n var correct;\n var done = false;\n answers.every(answer => {\n //console.log(answer.type);\n\n correct = false;\n // if (answer.type==\"value\"){\n if ('value' in answer) {\n if (submission == answer.value) {\n if (\"feedback\" in answer) {\n fb.textContent = jaxify(answer.feedback);\n } else {\n fb.textContent = jaxify(\"Correct\");\n }\n correct = answer.correct;\n //console.log(answer.correct);\n done = true;\n }\n // } else if (answer.type==\"range\") {\n } else if ('range' in answer) {\n //console.log(answer.range);\n if ((submission >= answer.range[0]) && (submission < answer.range[1])) {\n fb.textContent = jaxify(answer.feedback);\n correct = answer.correct;\n //console.log(answer.correct);\n done = true;\n }\n } else if (answer.type == \"default\") {\n defaultFB = answer.feedback;\n }\n if (done) {\n return false; // Break out of loop if this has been marked correct\n } else {\n return true; // Keep looking for case that includes this as a correct answer\n }\n });\n\n if ((!done) && (defaultFB != \"\")) {\n fb.innerHTML = jaxify(defaultFB);\n //console.log(\"Default feedback\", defaultFB);\n }\n\n fb.style.display = \"block\";\n if (correct) {\n ths.className = \"Input-text\";\n ths.classList.add(\"correctButton\");\n fb.className = \"Feedback\";\n fb.classList.add(\"correct\");\n } else {\n ths.className = \"Input-text\";\n ths.classList.add(\"incorrectButton\");\n fb.className = \"Feedback\";\n fb.classList.add(\"incorrect\");\n }\n\n // What follows is for the saved responses stuff\n var outerContainer = fb.parentElement.parentElement;\n var responsesContainer = document.getElementById(\"responses\" + outerContainer.id);\n if (responsesContainer) {\n console.log(submission);\n var qnum = document.getElementById(\"quizWrap\"+id).dataset.qnum;\n //console.log(\"Question \" + qnum);\n //console.log(id, \", got numcorrect=\",fb.dataset.numcorrect);\n var responses=JSON.parse(responsesContainer.dataset.responses);\n console.log(responses);\n if (submission == ths.value){\n responses[qnum]= submission;\n } else {\n responses[qnum]= ths.value + \"(\" + submission +\")\";\n }\n responsesContainer.setAttribute('data-responses', JSON.stringify(responses));\n printResponses(responsesContainer);\n }\n // End code to preserve responses\n\n if (typeof MathJax != 'undefined') {\n var version = MathJax.version;\n console.log('MathJax version', version);\n if (version[0] == \"2\") {\n MathJax.Hub.Queue([\"Typeset\", MathJax.Hub]);\n } else if (version[0] == \"3\") {\n MathJax.typeset([fb]);\n }\n } else {\n console.log('MathJax not detected');\n }\n return false;\n }\n\n}\n\nfunction isValid(el, charC) {\n //console.log(\"Input char: \", charC);\n if (charC == 46) {\n if (el.value.indexOf('.') === -1) {\n return true;\n } else if (el.value.indexOf('/') != -1) {\n var parts = el.value.split('/');\n if (parts[1].indexOf('.') === -1) {\n return true;\n }\n }\n else {\n return false;\n }\n } else if (charC == 47) {\n if (el.value.indexOf('/') === -1) {\n if ((el.value != \"\") && (el.value != \".\")) {\n return true;\n } else {\n return false;\n }\n } else {\n return false;\n }\n } else if (charC == 45) {\n var edex = el.value.indexOf('e');\n if (edex == -1) {\n edex = el.value.indexOf('E');\n }\n\n if (el.value == \"\") {\n return true;\n } else if (edex == (el.value.length - 1)) { // If just after e or E\n return true;\n } else {\n return false;\n }\n } else if (charC == 101) { // \"e\"\n if ((el.value.indexOf('e') === -1) && (el.value.indexOf('E') === -1) && (el.value.indexOf('/') == -1)) {\n // Prev symbol must be digit or decimal point:\n if (el.value.slice(-1).search(/\\d/) >= 0) {\n return true;\n } else if (el.value.slice(-1).search(/\\./) >= 0) {\n return true;\n } else {\n return false;\n }\n } else {\n return false;\n }\n } else {\n if (charC > 31 && (charC < 48 || charC > 57))\n return false;\n }\n return true;\n}\n\nfunction numeric_keypress(evnt) {\n var charC = (evnt.which) ? evnt.which : evnt.keyCode;\n\n if (charC == 13) {\n check_numeric(this, evnt);\n } else {\n return isValid(this, charC);\n }\n}\n\n\n\n\n\nfunction make_numeric(qa, outerqDiv, qDiv, aDiv, id) {\n\n\n\n //console.log(answer);\n\n\n outerqDiv.className = \"NumericQn\";\n aDiv.style.display = 'block';\n\n var lab = document.createElement(\"label\");\n lab.className = \"InpLabel\";\n lab.textContent = \"Type numeric answer here:\";\n aDiv.append(lab);\n\n var inp = document.createElement(\"input\");\n inp.type = \"text\";\n //inp.id=\"input-\"+id;\n inp.id = id + \"-0\";\n inp.className = \"Input-text\";\n inp.setAttribute('data-answers', JSON.stringify(qa.answers));\n if (\"precision\" in qa) {\n inp.setAttribute('data-precision', qa.precision);\n }\n aDiv.append(inp);\n //console.log(inp);\n\n //inp.addEventListener(\"keypress\", check_numeric);\n //inp.addEventListener(\"keypress\", numeric_keypress);\n /*\n inp.addEventListener(\"keypress\", function(event) {\n return numeric_keypress(this, event);\n }\n );\n */\n //inp.onkeypress=\"return numeric_keypress(this, event)\";\n inp.onkeypress = numeric_keypress;\n inp.onpaste = event => false;\n\n inp.addEventListener(\"focus\", function (event) {\n this.value = \"\";\n return false;\n }\n );\n\n\n}\nfunction jaxify(string) {\n var mystring = string;\n\n var count = 0;\n var loc = mystring.search(/([^\\\\]|^)(\\$)/);\n\n var count2 = 0;\n var loc2 = mystring.search(/([^\\\\]|^)(\\$\\$)/);\n\n //console.log(loc);\n\n while ((loc >= 0) || (loc2 >= 0)) {\n\n /* Have to replace all the double $$ first with current implementation */\n if (loc2 >= 0) {\n if (count2 % 2 == 0) {\n mystring = mystring.replace(/([^\\\\]|^)(\\$\\$)/, \"$1\\\\[\");\n } else {\n mystring = mystring.replace(/([^\\\\]|^)(\\$\\$)/, \"$1\\\\]\");\n }\n count2++;\n } else {\n if (count % 2 == 0) {\n mystring = mystring.replace(/([^\\\\]|^)(\\$)/, \"$1\\\\(\");\n } else {\n mystring = mystring.replace(/([^\\\\]|^)(\\$)/, \"$1\\\\)\");\n }\n count++;\n }\n loc = mystring.search(/([^\\\\]|^)(\\$)/);\n loc2 = mystring.search(/([^\\\\]|^)(\\$\\$)/);\n //console.log(mystring,\", loc:\",loc,\", loc2:\",loc2);\n }\n\n //console.log(mystring);\n return mystring;\n}\n\n\nfunction show_questions(json, mydiv) {\n console.log('show_questions');\n //var mydiv=document.getElementById(myid);\n var shuffle_questions = mydiv.dataset.shufflequestions;\n var num_questions = mydiv.dataset.numquestions;\n var shuffle_answers = mydiv.dataset.shuffleanswers;\n var max_width = mydiv.dataset.maxwidth;\n\n if (num_questions > json.length) {\n num_questions = json.length;\n }\n\n var questions;\n if ((num_questions < json.length) || (shuffle_questions == \"True\")) {\n //console.log(num_questions+\",\"+json.length);\n questions = getRandomSubarray(json, num_questions);\n } else {\n questions = json;\n }\n\n //console.log(\"SQ: \"+shuffle_questions+\", NQ: \" + num_questions + \", SA: \", shuffle_answers);\n\n // Iterate over questions\n questions.forEach((qa, index, array) => {\n //console.log(qa.question); \n\n var id = makeid(8);\n //console.log(id);\n\n\n // Create Div to contain question and answers\n var iDiv = document.createElement('div');\n //iDiv.id = 'quizWrap' + id + index;\n iDiv.id = 'quizWrap' + id;\n iDiv.className = 'Quiz';\n iDiv.setAttribute('data-qnum', index);\n iDiv.style.maxWidth =max_width+\"px\";\n mydiv.appendChild(iDiv);\n // iDiv.innerHTML=qa.question;\n \n var outerqDiv = document.createElement('div');\n outerqDiv.id = \"OuterquizQn\" + id + index;\n // Create div to contain question part\n var qDiv = document.createElement('div');\n qDiv.id = \"quizQn\" + id + index;\n \n if (qa.question) {\n iDiv.append(outerqDiv);\n\n //qDiv.textContent=qa.question;\n qDiv.innerHTML = jaxify(qa.question);\n outerqDiv.append(qDiv);\n }\n\n // Create div for code inside question\n var codeDiv;\n if (\"code\" in qa) {\n codeDiv = document.createElement('div');\n codeDiv.id = \"code\" + id + index;\n codeDiv.className = \"QuizCode\";\n var codePre = document.createElement('pre');\n codeDiv.append(codePre);\n var codeCode = document.createElement('code');\n codePre.append(codeCode);\n codeCode.innerHTML = qa.code;\n outerqDiv.append(codeDiv);\n //console.log(codeDiv);\n }\n\n\n // Create div to contain answer part\n var aDiv = document.createElement('div');\n aDiv.id = \"quizAns\" + id + index;\n aDiv.className = 'Answer';\n iDiv.append(aDiv);\n\n //console.log(qa.type);\n\n var num_correct;\n if ((qa.type == \"multiple_choice\") || (qa.type == \"many_choice\") ) {\n num_correct = make_mc(qa, shuffle_answers, outerqDiv, qDiv, aDiv, id);\n if (\"answer_cols\" in qa) {\n //aDiv.style.gridTemplateColumns = 'auto '.repeat(qa.answer_cols);\n aDiv.style.gridTemplateColumns = 'repeat(' + qa.answer_cols + ', 1fr)';\n }\n } else if (qa.type == \"numeric\") {\n //console.log(\"numeric\");\n make_numeric(qa, outerqDiv, qDiv, aDiv, id);\n }\n\n\n //Make div for feedback\n var fb = document.createElement(\"div\");\n fb.id = \"fb\" + id;\n //fb.style=\"font-size: 20px;text-align:center;\";\n fb.className = \"Feedback\";\n fb.setAttribute(\"data-answeredcorrect\", 0);\n fb.setAttribute(\"data-numcorrect\", num_correct);\n iDiv.append(fb);\n\n\n });\n var preserveResponses = mydiv.dataset.preserveresponses;\n console.log(preserveResponses);\n console.log(preserveResponses == \"true\");\n if (preserveResponses == \"true\") {\n console.log(preserveResponses);\n // Create Div to contain record of answers\n var iDiv = document.createElement('div');\n iDiv.id = 'responses' + mydiv.id;\n iDiv.className = 'JCResponses';\n // Create a place to store responses as an empty array\n iDiv.setAttribute('data-responses', '[]');\n\n // Dummy Text\n iDiv.innerHTML=\"Select your answers and then follow the directions that will appear here.\"\n //iDiv.className = 'Quiz';\n mydiv.appendChild(iDiv);\n }\n//console.log(\"At end of show_questions\");\n if (typeof MathJax != 'undefined') {\n console.log(\"MathJax version\", MathJax.version);\n var version = MathJax.version;\n setTimeout(function(){\n var version = MathJax.version;\n console.log('After sleep, MathJax version', version);\n if (version[0] == \"2\") {\n MathJax.Hub.Queue([\"Typeset\", MathJax.Hub]);\n } else if (version[0] == \"3\") {\n MathJax.typeset([mydiv]);\n }\n }, 500);\nif (typeof version == 'undefined') {\n } else\n {\n if (version[0] == \"2\") {\n MathJax.Hub.Queue([\"Typeset\", MathJax.Hub]);\n } else if (version[0] == \"3\") {\n MathJax.typeset([mydiv]);\n } else {\n console.log(\"MathJax not found\");\n }\n }\n }\n return false;\n}\n/* This is to handle asynchrony issues in loading Jupyter notebooks\n where the quiz has been previously run. The Javascript was generally\n being run before the div was added to the DOM. I tried to do this\n more elegantly using Mutation Observer, but I didn't get it to work.\n\n Someone more knowledgeable could make this better ;-) */\n\n function try_show() {\n if(document.getElementById(\"dUCgDlcWgnpd\")) {\n show_questions(questionsdUCgDlcWgnpd, dUCgDlcWgnpd); \n } else {\n setTimeout(try_show, 200);\n }\n };\n \n {\n // console.log(element);\n\n //console.log(\"dUCgDlcWgnpd\");\n // console.log(document.getElementById(\"dUCgDlcWgnpd\"));\n\n try_show();\n }\n ", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from jupyterquiz import display_quiz\n", "display_quiz(\"#pseudo_inv_complexity\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "(computational-issues)=\n", "## Computational issues\n", "\n", "* **Multicollinearity**. if there are highly correlated or even linearly dependent features in the design matrix $\\boldsymbol X$, then the matrix $\\boldsymbol X^\\top \\boldsymbol X$ is nearly singular. One can cope with multicollinearity by feature selection.\n", "\n", "* **Curse of dimensionality**. If $d \\gg 1$, it is too computationally expensive to invert $\\boldsymbol X^\\top \\boldsymbol X$. In this case instead analytical solution {eq}`lin-reg-solution` numerical methods such as gradient descent are used.\n", "\n", "* **Ill-conditioning**. In practice the matrix $\\boldsymbol X^\\top \\boldsymbol X$ has large {ref}`condition-number`. This implies the numerical instability of the solution {eq}`lin-reg-solution`.\n", "\n", " " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 12-d linear regression on Boston dataset\n", "\n", "Take all $12$ features and train multiple regression!" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "condition number: 137383410.3557865\n", "intercept: 41.61727017593546\n", "coefficients: [-1.21388618e-01 4.69634633e-02 1.34676947e-02 2.83999338e+00\n", " -1.87580220e+01 3.65811904e+00 3.61071055e-03 -1.49075365e+00\n", " 2.89404521e-01 -1.26819813e-02 -9.37532900e-01 -5.52019101e-01]\n", "MSE: 22.42968143948993\n" ] } ], "source": [ "target = boston['medv']\n", "train = boston.drop(['medv', \"Unnamed: 0\"], axis=1)\n", "X_train = np.hstack([np.ones(506)[:, None], train])\n", "X_T_X = X_train.T.dot(X_train)\n", "print(\"condition number:\", np.linalg.cond(X_T_X))\n", "w = np.linalg.inv(X_T_X).dot(X_train.T).dot(target)\n", "print(\"intercept:\", w[0])\n", "print(\"coefficients:\", w[1:])\n", "print(\"MSE:\", np.mean((X_train.dot(w) - target)**2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Do the same thing with `sklearn`:" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "intercept: 41.61727017595457\n", "coefficients: [-1.21388618e-01 4.69634633e-02 1.34676947e-02 2.83999338e+00\n", " -1.87580220e+01 3.65811904e+00 3.61071055e-03 -1.49075365e+00\n", " 2.89404521e-01 -1.26819813e-02 -9.37532900e-01 -5.52019101e-01]\n", "r-score: 0.7343070437613075\n", "MSE: 22.429681439489933\n" ] } ], "source": [ "LR = LinearRegression()\n", "LR.fit(train, target)\n", "print(\"intercept:\", LR.intercept_)\n", "print(\"coefficients:\", LR.coef_)\n", "print(\"r-score:\", LR.score(train, target))\n", "print(\"MSE:\", np.mean((LR.predict(train) - target) ** 2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here we see that the improvement of MSE and $R^2$-score to those of {ref}`1-d ` and {ref}`quadratic ` regression." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Connection to polynomial regression\n", "\n", "Note that the polynomial regression {eq}`poly-reg` is a special case of multiple regression. If the polynomial degree is $m$, then {eq}`poly-reg` can be written in the form {eq}`multiple-lin-reg-inner` with feature matrix\n", "\n", "```{math}\n", ":label: poly-feature-matrix\n", " \\boldsymbol X = [\\boldsymbol 1 \\;\\boldsymbol x\\;\\boldsymbol x^2 \\; \\ldots\\;\\boldsymbol x^m] = \\begin{pmatrix}\n", " 1 & x_1 & x_1^2 & \\ldots & x_1^m \\\\\n", " 1 & x_2 & x_2^2 & \\ldots & x_2^m \\\\\n", " \\vdots & \\vdots & \\vdots & \\ddots & \\vdots \\\\\n", " 1 & x_n & x_n^2 & \\ldots & x_n^m \\\\\n", " \\end{pmatrix}\n", "```\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exercises\n", "\n", "1. Prove that matrix $\\boldsymbol X$ from {eq}`poly-feature-matrix` has full column rank if $m \\leqslant n$ and $x_i \\ne x_j$ when $i\\ne j$." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```{admonition} TODO\n", ":class: warning\n", "\n", "* Create references to matrix calculus chapters\n", "* Develop the idea of connection to polynomial regression\n", "* Add some simple quizzes\n", "* Add more datasets\n", "* Show the computational issues on real and/or simulated datasets\n", "```" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.4" }, "orig_nbformat": 4 }, "nbformat": 4, "nbformat_minor": 2 }