{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "t9lgYi7EPK2m" }, "source": [ "# 3 - Item Response Theory with Stan\n", "\n", "[![View filled on Github](https://img.shields.io/static/v1.svg?logo=github&label=Repo&message=View%20On%20Github&color=lightgrey)](https://github.com/annabavaresco/ancm2024/blob/main/docs/week_3/3_IRT_Stan.ipynb)\n", "[![View filled in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/annabavaresco/ancm2024/blob/main/docs/week_3/3_IRT_Stan.ipynb)" ] }, { "cell_type": "markdown", "metadata": { "id": "z_sHGGTNPQcx" }, "source": [ "In this lab, you will explore item response theory and Bayesian modelling with the Stan language." ] }, { "cell_type": "markdown", "metadata": { "id": "QgZsGI6AOvoO" }, "source": [ "## Setup" ] }, { "cell_type": "markdown", "metadata": { "id": "TucqWWFKNWed" }, "source": [ "First, you need to install Stan. This may take several minutes :))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "HyCe6fys4bBM", "outputId": "57dd88a3-b216-457a-cc52-57cae76070d6" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "CmdStan install directory: /root/.cmdstan\n", "Installing CmdStan version: 2.35.0\n", "Downloading CmdStan version 2.35.0\n", "Download successful, file: /tmp/tmpcn44248l\n", "Extracting distribution\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "DEBUG:cmdstanpy:cmd: make build -j1\n", "cwd: None\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Unpacked download as cmdstan-2.35.0\n", "Building version cmdstan-2.35.0, may take several minutes, depending on your system.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "DEBUG:cmdstanpy:cmd: make examples/bernoulli/bernoulli\n", "cwd: None\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Installed cmdstan-2.35.0\n", "Test model compilation\n" ] }, { "data": { "text/plain": [ "True" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import numpy as np\n", "import pandas as pd\n", "# Colab setup (courtesy of Justin Bois)\n", "# N.B. This cell may take several minutes to complete (3 mins on the instructor's machine)\n", "import os, sys, subprocess\n", "cmd = \"pip install --upgrade iqplot bebi103 arviz cmdstanpy watermark\"\n", "process = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE)\n", "stdout, stderr = process.communicate()\n", "import cmdstanpy; cmdstanpy.install_cmdstan()" ] }, { "cell_type": "markdown", "metadata": { "id": "HcDGuqKbJXDC" }, "source": [ "Next, you need to download the data and Stan template [here](https://drive.google.com/file/d/1BBeL2BtfTIBqMlJFTC_OZTUdT_pt9mpR/view?usp=share_link). Save it to your own Google Drive as in previous labs, and then mount your drive." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "XLNJTp_w4600", "outputId": "05b9bb76-98a0-4ccf-c30a-64dadade2a6a" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Mounted at /content/drive\n" ] } ], "source": [ "from google.colab import drive\n", "drive.mount('/content/drive')" ] }, { "cell_type": "markdown", "metadata": { "id": "Z2ObpzjxClnA" }, "source": [ "Unzip the files into a folder (you will be able to find this folder if you click the folder icon in your left sidebar):" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "K3zIlfAh-t9o" }, "outputs": [], "source": [ "!unzip -qq '/content/drive/MyDrive/irt4ancm.zip'" ] }, { "cell_type": "markdown", "metadata": { "id": "J2wCXdewOSUh" }, "source": [ "The following cell prints a list of all of the segments used in the experiment, so that you can find and listen to the results. All of the audio was extracted from the official YouTube videos of the Eurovision Song Contest finals." ] }, { "cell_type": "markdown", "metadata": { "id": "xeY9z78vO0Ct" }, "source": [ "## Background" ] }, { "cell_type": "markdown", "metadata": { "id": "VMgzZvvTO3Bm" }, "source": [ "The data in this lab come from the Eurovision Song Contest edition of the Hooked on Music experiment. You can try the experiment [here](https://app.amsterdammusiclab.nl/eurovision_2021). In this experiment, people were presented with segments from Eurovision songs and were asked if they'd ever heard the song. If their answer was 'yes', the song was muted for a few seconds and then went back on. In some trials, the song resumed at the right point. In others, it resumed a bit earlier or later. Participants were then alsp asked whether the second segment was the 'right' continuation for the first or not. \n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 576 }, "id": "ptYoGCMl7EI3", "outputId": "db6af206-e3a4-4d36-8e25-cd43b6408ead" }, "outputs": [ { "data": { "application/vnd.google.colaboratory.intrinsic+json": { "summary": "{\n \"name\": \"segment_df\",\n \"rows\": 437,\n \"fields\": [\n {\n \"column\": \"segment\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 126,\n \"min\": 1,\n \"max\": 437,\n \"num_unique_values\": 437,\n \"samples\": [\n 396,\n 79,\n 279\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"song\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 23,\n \"min\": 1,\n \"max\": 77,\n \"num_unique_values\": 77,\n \"samples\": [\n 5,\n 36,\n 11\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"country\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 40,\n \"samples\": [\n \"Portugal\",\n \"Serbia\",\n \"Azerbaijan\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"year\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 1,\n \"min\": 2016,\n \"max\": 2019,\n \"num_unique_values\": 4,\n \"samples\": [\n 2017,\n 2019,\n 2016\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"artist\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 76,\n \"samples\": [\n \"Frans\",\n \"Naviband\",\n \"Douwe Bob\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"title\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 77,\n \"samples\": [\n \"If I Were Sorry\",\n \"Historyja Majho Zyccia\",\n \"Slow Down\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"start_position\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 49.279014339093685,\n \"min\": 0.0,\n \"max\": 166.12,\n \"num_unique_values\": 403,\n \"samples\": [\n 27.098,\n 67.223,\n 147.616\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"segment_type\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 6,\n \"samples\": [\n \"i\",\n \"v\",\n \"o\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}", "type": "dataframe", "variable_name": "segment_df" }, "text/html": [ "\n", "
\n", "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
songcountryyearartisttitlestart_positionsegment_type
segment
11Ukraine2016Jamala19440.000i
21Ukraine2016Jamala19447.925v
31Ukraine2016Jamala194439.500c
41Ukraine2016Jamala194472.043v
51Ukraine2016Jamala1944132.559b
........................
43369Czechia2019Lake MalawiFriend of a Friend78.128v
43470Denmark2019LeonoraLove Is Forever61.508v
43571Cyprus2019TamtaReplay66.212v
43673Slovenia2019Zala Kralj & Gašper ŠantlSebi70.698v
43775Serbia2019Nevena BožovićKruna106.544v
\n", "

437 rows × 7 columns

\n", "
\n", "
\n", "\n", "
\n", " \n", "\n", " \n", "\n", " \n", "
\n", "\n", "\n", "
\n", " \n", "\n", "\n", "\n", " \n", "
\n", "\n", "
\n", " \n", " \n", " \n", "
\n", "\n", "
\n", "
\n" ], "text/plain": [ " song country year artist title \\\n", "segment \n", "1 1 Ukraine 2016 Jamala 1944 \n", "2 1 Ukraine 2016 Jamala 1944 \n", "3 1 Ukraine 2016 Jamala 1944 \n", "4 1 Ukraine 2016 Jamala 1944 \n", "5 1 Ukraine 2016 Jamala 1944 \n", "... ... ... ... ... ... \n", "433 69 Czechia 2019 Lake Malawi Friend of a Friend \n", "434 70 Denmark 2019 Leonora Love Is Forever \n", "435 71 Cyprus 2019 Tamta Replay \n", "436 73 Slovenia 2019 Zala Kralj & Gašper Šantl Sebi \n", "437 75 Serbia 2019 Nevena Božović Kruna \n", "\n", " start_position segment_type \n", "segment \n", "1 0.000 i \n", "2 7.925 v \n", "3 39.500 c \n", "4 72.043 v \n", "5 132.559 b \n", "... ... ... \n", "433 78.128 v \n", "434 61.508 v \n", "435 66.212 v \n", "436 70.698 v \n", "437 106.544 v \n", "\n", "[437 rows x 7 columns]" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "segment_df = pd.read_csv('irt4ancm/segment_list.csv')\n", "segment_df = segment_df.set_index('segment')\n", "segment_df" ] }, { "cell_type": "markdown", "metadata": { "id": "u9gTwgHqPDFQ" }, "source": [ "## Lab" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Open the `irt4ancm.stan` file in the right-hand pane. You will make any adjstments to your model there. Here is a breakdown of what the main code blocks from `irt4ancm.stan` are doing:\n", "\n", "```\n", "data {\n", " int M; // number of observations\n", " int N; // number of participants\n", " int I; // number of song segments\n", " int J; // number of songs\n", " array[M] int is_recognised; // was the segment recognised?\n", " array[M] int is_verified; // was the segment verified correctly?\n", " array[M] int participant; // participant number\n", " array[M] int seg; // segment number\n", " array[M] int song; // segment number\n", " array[M] int continuation_correctness; // did the verification segment restart in the correct place?\n", " vector[M] recognition_time; // how long did it take to recognise the segment?\n", " matrix[I,10] audio_features; // audio features for each segment\n", " vector[N] sophistication; // Goldsmith's music sophistication for each participant\n", "}\n", "``` \n", "This block is simply defining the variables corresponding to the data we're going to fit the model on.\n", "\n", "```\n", "parameters {\n", " real mu_delta;\n", " real sigma_theta; // participant prior SD\n", " real sigma_delta; // difficulty prior SD\n", " vector[N] theta; // participant abilities\n", " vector[I] delta; // segment difficulties\n", "}\n", "```\n", "\n", "Here is where you declare your parameters. Any parameter that you plan to include in the model needs to be specified here. In this case, we're starting off with a Rasch (1PL) model, where the only parameters are $\\theta$ (the participant's ability) and $\\delta$ (the difficulty of the segment). As you experiment with more complex models (2PL, 3PL, and 4PL), will need to add more parameters. For each parameter, we should also specify what is our hypothesis about its distribution. This is what the following block (included inside `model`) does:\n", "\n", "```\n", " // Hyperpriors\n", " mu_delta ~ std_normal();\n", " sigma_theta ~ std_normal(); // Stan automatically cuts off the negative values\n", " sigma_delta ~ std_normal(); // Stan automatically cuts off the negative values\n", "\n", " // Priors\n", " theta ~ normal(0, sigma_theta);\n", " delta ~ normal(mu_delta, sigma_delta);\n", "```\n", "\n", "How do we decide on which distribution to choose for each parameter? Unless you have a specific educated guess about it, starting with a normal distribution is usually a good choice. Let's now move to the core part of the code, where our model is atually defined:\n", "\n", "```\n", "is_verified[m] ~ bernoulli_logit(theta[participant[m]] - delta[seg[m]]);\n", "```\n", "\n", "This line essentially means that the variable you are trying to model (`is_verified`) is distributed as (`~`) a bernoulli logit distribution with probability $\\theta-\\delta$. When editing the `irt4ancm.stan` file, you might actually want to start from this line. You can choose to either predict the `is_verified` or the `is_recognised` variable. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "CdvPWFY6P8Cd" }, "outputs": [], "source": [ "from cmdstanpy import CmdStanModel" ] }, { "cell_type": "markdown", "metadata": { "id": "HBuo286zVi5h" }, "source": [ "As we've just seen, the 'recipe' for our model is defined in the stan file. Now we will actually compile the model. Every time you change the model, you will need to save the sta file and recompile it by running the cell below.\n", "\n", "💡 Tip: Colab will probably save your edits to the stan file automatically. However, be aware that the saved changes may be available only within the session. In other words, when the notebook runtime is disconnected, you may lose the edits you make to your stan file. To make sure you don't lose your progress, you may want to use one of the following strategies:\n", "\n", "1. Unzip the `irt4ancm.zip` folder directly in GG Drive and make sure you edit the stan file saved to your Drive\n", "\n", "2. Alternatively, you can simply download the stan file before closing Colab. You can do that by simply hovering on the stan file on the left pane > clicking on the 3 dots > selecting 'download'." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "liUBG4FrPXdv", "outputId": "2a903e19-8a36-4056-ef4b-c84a876d1b98" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "13:02:37 - cmdstanpy - WARNING - CmdStanModel(model_name=...) is deprecated and will be removed in the next major version.\n", "WARNING:cmdstanpy:CmdStanModel(model_name=...) is deprecated and will be removed in the next major version.\n", "13:02:37 - cmdstanpy - INFO - compiling stan file /content/irt4ancm/irt4ancm.stan to exe file /content/irt4ancm/irt4ancm\n", "INFO:cmdstanpy:compiling stan file /content/irt4ancm/irt4ancm.stan to exe file /content/irt4ancm/irt4ancm\n", "DEBUG:cmdstanpy:cmd: make STANCFLAGS+=--filename-in-msg=irt4ancm.stan /content/irt4ancm/irt4ancm\n", "cwd: /root/.cmdstan/cmdstan-2.35.0\n", "DEBUG:cmdstanpy:Console output:\n", "\n", "--- Translating Stan model to C++ code ---\n", "bin/stanc --filename-in-msg=irt4ancm.stan --o=/content/irt4ancm/irt4ancm.hpp /content/irt4ancm/irt4ancm.stan\n", "\n", "--- Compiling C++ code ---\n", "g++ -std=c++17 -pthread -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes -Wno-class-memaccess -I stan/lib/stan_math/lib/tbb_2020.3/include -O3 -I src -I stan/src -I stan/lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.4.0 -I stan/lib/stan_math/lib/boost_1.84.0 -I stan/lib/stan_math/lib/sundials_6.1.1/include -I stan/lib/stan_math/lib/sundials_6.1.1/src/sundials -DBOOST_DISABLE_ASSERTS -c -Wno-ignored-attributes -x c++ -o /content/irt4ancm/irt4ancm.o /content/irt4ancm/irt4ancm.hpp\n", "\n", "--- Linking model ---\n", "g++ -std=c++17 -pthread -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes -Wno-class-memaccess -I stan/lib/stan_math/lib/tbb_2020.3/include -O3 -I src -I stan/src -I stan/lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.4.0 -I stan/lib/stan_math/lib/boost_1.84.0 -I stan/lib/stan_math/lib/sundials_6.1.1/include -I stan/lib/stan_math/lib/sundials_6.1.1/src/sundials -DBOOST_DISABLE_ASSERTS -Wl,-L,\"/root/.cmdstan/cmdstan-2.35.0/stan/lib/stan_math/lib/tbb\" -Wl,-rpath,\"/root/.cmdstan/cmdstan-2.35.0/stan/lib/stan_math/lib/tbb\" /content/irt4ancm/irt4ancm.o src/cmdstan/main.o -ltbb stan/lib/stan_math/lib/sundials_6.1.1/lib/libsundials_nvecserial.a stan/lib/stan_math/lib/sundials_6.1.1/lib/libsundials_cvodes.a stan/lib/stan_math/lib/sundials_6.1.1/lib/libsundials_idas.a stan/lib/stan_math/lib/sundials_6.1.1/lib/libsundials_kinsol.a stan/lib/stan_math/lib/tbb/libtbb.so.2 -o /content/irt4ancm/irt4ancm\n", "rm /content/irt4ancm/irt4ancm.o /content/irt4ancm/irt4ancm.hpp\n", "\n", "13:02:49 - cmdstanpy - INFO - compiled model executable: /content/irt4ancm/irt4ancm\n", "INFO:cmdstanpy:compiled model executable: /content/irt4ancm/irt4ancm\n" ] } ], "source": [ "model = CmdStanModel(model_name=\"irt4ancm\", stan_file=\"irt4ancm/irt4ancm.stan\")" ] }, { "cell_type": "markdown", "metadata": { "id": "ariXufvLVm0A" }, "source": [ "We fit the model here using `all_plays.json`, which contains a complete set of data. You may find it more interesting to explore `rec_only.json` as an alternative, which contains only plays where the participant claimed to recognise the segment." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 1000, "referenced_widgets": [ "ae4c402e37794df6871be4d64fdab4af", "6f98b9dacb814aa4ad8a7f59c4e69400", "e07a0b24f3f142fc9733f19b3aaa78fb", "c867f146a1924263ae7e9001e6e8fb68", "0553b81ee6ac4ef5a48a139bb54d96ad", "6b88027e796b4183aa0fa13933913645", "b1fcba53f8a342f48fb0876fd4e8bae9", "898f15ebf53543aa8984937ba8bfda69", "e0b4c592b3ad4be58ac5b915a2088bd6", "2318b5395f1f4d21ae94f04e372577b7", "6cd9897f424344b99da25f95602e5aa5", "7440bb5f786040f0af16185279da23af", "10b260f1a5c74ee887c372296478a48c", "cc8db68771f3498da873f2dbb0f686b7", "6ccd15b0016a4245acd6da22faf6f038", "e3f58e5158e149e199b6eb4c22f3c6d4", "b4bfcc8fd40642dbb792aa4e1dd946bc", "86a4331aa62e4e94b9bda845a9d4c4d0", "7be3ae70869242a5ba35b963c2fad86e", "e87561d8806246f59a668a76a54cb63c", "4d4ef945113f4590a80eca2bf2ef36b9", "991b0f118a4442409a0cb31af9ace12d", "ef3b5623fa3d4b559b2120cc07033dda", "3a89930f6a774939803bce07f4bb20af", "2dc41fa030b548748037fe6306b0ceca", "704f2afcc3854fe9b61296a7aa41b7aa", "419233e503434179846e6530b5f95345", "b1322cd858c5428fb0cc5eca303e2515", "9fd326a2468c4e7cb2e31f6033b4dcd7", "fb5bba5506f5472f880ae86943d5f284", "f1069348739341cf8031c36a50bfabb1", "f9b992ab76214c0c8e65098823311f6f", "af4285d60e484786b969054623ba0c14", "f7073f38455740e99a448ca3866b6d64", "d2eb4580992f456fb7ace05caf78c2ba", "287ab4796c61481dae9b1cea01907c25", "3a37861fc35348618fc5e9946fa26ca5", "65fc914d6e0d432cb5e7a3c4a9403d0d", "3074484d3fe248039f0a6aeb322f4dbd", "a324bebf97d2424286d066b36091ebc1", "66721d5beac74c48ac041732fa0867de", "f1f827b09dc54fdf933335abaa671948", "9483d615ad0a4d96bed8fd4995d6a519", "90c4c9d334df4187beffef1305d6022e" ] }, "id": "_njDAhoqQmo7", "outputId": "d9bef61e-83b8-4ee7-f523-8df4455dd4d3" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "DEBUG:cmdstanpy:cmd: /content/irt4ancm/irt4ancm info\n", "cwd: None\n", "13:02:53 - cmdstanpy - INFO - CmdStan start processing\n", "INFO:cmdstanpy:CmdStan start processing\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "ae4c402e37794df6871be4d64fdab4af", "version_major": 2, "version_minor": 0 }, "text/plain": [ "chain 1 | | 00:00 Status" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "7440bb5f786040f0af16185279da23af", "version_major": 2, "version_minor": 0 }, "text/plain": [ "chain 2 | | 00:00 Status" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "ef3b5623fa3d4b559b2120cc07033dda", "version_major": 2, "version_minor": 0 }, "text/plain": [ "chain 3 | | 00:00 Status" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "f7073f38455740e99a448ca3866b6d64", "version_major": 2, "version_minor": 0 }, "text/plain": [ "chain 4 | | 00:00 Status" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stderr", "output_type": "stream", "text": [ "DEBUG:cmdstanpy:idx 0\n", "DEBUG:cmdstanpy:running CmdStan, num_threads: 1\n", "DEBUG:cmdstanpy:CmdStan args: ['/content/irt4ancm/irt4ancm', 'id=1', 'random', 'seed=84057', 'data', 'file=irt4ancm/all_plays.json', 'output', 'file=/tmp/tmpmu1ubl7v/irt4ancmlqtbjyc1/irt4ancm-20240906130253_1.csv', 'method=sample', 'algorithm=hmc', 'adapt', 'engaged=1']\n", "DEBUG:cmdstanpy:idx 1\n", "DEBUG:cmdstanpy:running CmdStan, num_threads: 1\n", "DEBUG:cmdstanpy:CmdStan args: ['/content/irt4ancm/irt4ancm', 'id=2', 'random', 'seed=84057', 'data', 'file=irt4ancm/all_plays.json', 'output', 'file=/tmp/tmpmu1ubl7v/irt4ancmlqtbjyc1/irt4ancm-20240906130253_2.csv', 'method=sample', 'algorithm=hmc', 'adapt', 'engaged=1']\n", "DEBUG:cmdstanpy:idx 2\n", "DEBUG:cmdstanpy:running CmdStan, num_threads: 1\n", "DEBUG:cmdstanpy:CmdStan args: ['/content/irt4ancm/irt4ancm', 'id=3', 'random', 'seed=84057', 'data', 'file=irt4ancm/all_plays.json', 'output', 'file=/tmp/tmpmu1ubl7v/irt4ancmlqtbjyc1/irt4ancm-20240906130253_3.csv', 'method=sample', 'algorithm=hmc', 'adapt', 'engaged=1']\n", "DEBUG:cmdstanpy:idx 3\n", "DEBUG:cmdstanpy:running CmdStan, num_threads: 1\n", "DEBUG:cmdstanpy:CmdStan args: ['/content/irt4ancm/irt4ancm', 'id=4', 'random', 'seed=84057', 'data', 'file=irt4ancm/all_plays.json', 'output', 'file=/tmp/tmpmu1ubl7v/irt4ancmlqtbjyc1/irt4ancm-20240906130253_4.csv', 'method=sample', 'algorithm=hmc', 'adapt', 'engaged=1']\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " " ] }, { "name": "stderr", "output_type": "stream", "text": [ "13:05:30 - cmdstanpy - INFO - CmdStan done processing.\n", "INFO:cmdstanpy:CmdStan done processing.\n", "DEBUG:cmdstanpy:runset\n", "RunSet: chains=4, chain_ids=[1, 2, 3, 4], num_processes=4\n", " cmd (chain 1):\n", "\t['/content/irt4ancm/irt4ancm', 'id=1', 'random', 'seed=84057', 'data', 'file=irt4ancm/all_plays.json', 'output', 'file=/tmp/tmpmu1ubl7v/irt4ancmlqtbjyc1/irt4ancm-20240906130253_1.csv', 'method=sample', 'algorithm=hmc', 'adapt', 'engaged=1']\n", " retcodes=[0, 0, 0, 0]\n", " per-chain output files (showing chain 1 only):\n", " csv_file:\n", "\t/tmp/tmpmu1ubl7v/irt4ancmlqtbjyc1/irt4ancm-20240906130253_1.csv\n", " console_msgs (if any):\n", "\t/tmp/tmpmu1ubl7v/irt4ancmlqtbjyc1/irt4ancm-20240906130253_0-stdout.txt\n", "DEBUG:cmdstanpy:Chain 1 console:\n", "method = sample (Default)\n", " sample\n", " num_samples = 1000 (Default)\n", " num_warmup = 1000 (Default)\n", " save_warmup = false (Default)\n", " thin = 1 (Default)\n", " adapt\n", " engaged = true (Default)\n", " gamma = 0.05 (Default)\n", " delta = 0.8 (Default)\n", " kappa = 0.75 (Default)\n", " t0 = 10 (Default)\n", " init_buffer = 75 (Default)\n", " term_buffer = 50 (Default)\n", " window = 25 (Default)\n", " save_metric = false (Default)\n", " algorithm = hmc (Default)\n", " hmc\n", " engine = nuts (Default)\n", " nuts\n", " max_depth = 10 (Default)\n", " metric = diag_e (Default)\n", " metric_file = (Default)\n", " stepsize = 1 (Default)\n", " stepsize_jitter = 0 (Default)\n", " num_chains = 1 (Default)\n", "id = 1 (Default)\n", "data\n", " file = irt4ancm/all_plays.json\n", "init = 2 (Default)\n", "random\n", " seed = 84057\n", "output\n", " file = /tmp/tmpmu1ubl7v/irt4ancmlqtbjyc1/irt4ancm-20240906130253_1.csv\n", " diagnostic_file = (Default)\n", " refresh = 100 (Default)\n", " sig_figs = -1 (Default)\n", " profile_file = profile.csv (Default)\n", " save_cmdstan_config = false (Default)\n", "num_threads = 1 (Default)\n", "\n", "\n", "Gradient evaluation took 0.007894 seconds\n", "1000 transitions using 10 leapfrog steps per transition would take 78.94 seconds.\n", "Adjust your expectations accordingly!\n", "\n", "\n", "Iteration: 1 / 2000 [ 0%] (Warmup)\n", "Informational Message: The current Metropolis proposal is about to be rejected because of the following issue:\n", "Exception: normal_lpdf: Scale parameter is 0, but must be positive! (in 'irt4ancm.stan', line 33, column 2 to column 40)\n", "If this warning occurs sporadically, such as for highly constrained variable types like covariance matrices, then the sampler is fine,\n", "but if this warning occurs often then your model may be either severely ill-conditioned or misspecified.\n", "\n", "Informational Message: The current Metropolis proposal is about to be rejected because of the following issue:\n", "Exception: normal_lpdf: Scale parameter is 0, but must be positive! (in 'irt4ancm.stan', line 33, column 2 to column 40)\n", "If this warning occurs sporadically, such as for highly constrained variable types like covariance matrices, then the sampler is fine,\n", "but if this warning occurs often then your model may be either severely ill-conditioned or misspecified.\n", "\n", "Iteration: 100 / 2000 [ 5%] (Warmup)\n", "Iteration: 200 / 2000 [ 10%] (Warmup)\n", "Iteration: 300 / 2000 [ 15%] (Warmup)\n", "Iteration: 400 / 2000 [ 20%] (Warmup)\n", "Iteration: 500 / 2000 [ 25%] (Warmup)\n", "Iteration: 600 / 2000 [ 30%] (Warmup)\n", "Iteration: 700 / 2000 [ 35%] (Warmup)\n", "Iteration: 800 / 2000 [ 40%] (Warmup)\n", "Iteration: 900 / 2000 [ 45%] (Warmup)\n", "Iteration: 1000 / 2000 [ 50%] (Warmup)\n", "Iteration: 1001 / 2000 [ 50%] (Sampling)\n", "Iteration: 1100 / 2000 [ 55%] (Sampling)\n", "Iteration: 1200 / 2000 [ 60%] (Sampling)\n", "Iteration: 1300 / 2000 [ 65%] (Sampling)\n", "Iteration: 1400 / 2000 [ 70%] (Sampling)\n", "Iteration: 1500 / 2000 [ 75%] (Sampling)\n", "Iteration: 1600 / 2000 [ 80%] (Sampling)\n", "Iteration: 1700 / 2000 [ 85%] (Sampling)\n", "Iteration: 1800 / 2000 [ 90%] (Sampling)\n", "Iteration: 1900 / 2000 [ 95%] (Sampling)\n", "Iteration: 2000 / 2000 [100%] (Sampling)\n", "\n", " Elapsed Time: 48.617 seconds (Warm-up)\n", " 29.932 seconds (Sampling)\n", " 78.549 seconds (Total)\n", "\n", "\n", "13:05:30 - cmdstanpy - WARNING - Non-fatal error during sampling:\n", "Exception: normal_lpdf: Scale parameter is 0, but must be positive! (in 'irt4ancm.stan', line 33, column 2 to column 40)\n", "\tException: normal_lpdf: Scale parameter is 0, but must be positive! (in 'irt4ancm.stan', line 33, column 2 to column 40)\n", "Exception: normal_lpdf: Scale parameter is 0, but must be positive! (in 'irt4ancm.stan', line 32, column 2 to column 33)\n", "\tException: normal_lpdf: Scale parameter is 0, but must be positive! (in 'irt4ancm.stan', line 32, column 2 to column 33)\n", "Exception: normal_lpdf: Scale parameter is 0, but must be positive! (in 'irt4ancm.stan', line 32, column 2 to column 33)\n", "\tException: normal_lpdf: Scale parameter is 0, but must be positive! (in 'irt4ancm.stan', line 32, column 2 to column 33)\n", "Exception: normal_lpdf: Scale parameter is 0, but must be positive! (in 'irt4ancm.stan', line 33, column 2 to column 40)\n", "\tException: normal_lpdf: Scale parameter is 0, but must be positive! (in 'irt4ancm.stan', line 33, column 2 to column 40)\n", "Consider re-running with show_console=True if the above output is unclear!\n", "WARNING:cmdstanpy:Non-fatal error during sampling:\n", "Exception: normal_lpdf: Scale parameter is 0, but must be positive! (in 'irt4ancm.stan', line 33, column 2 to column 40)\n", "\tException: normal_lpdf: Scale parameter is 0, but must be positive! (in 'irt4ancm.stan', line 33, column 2 to column 40)\n", "Exception: normal_lpdf: Scale parameter is 0, but must be positive! (in 'irt4ancm.stan', line 32, column 2 to column 33)\n", "\tException: normal_lpdf: Scale parameter is 0, but must be positive! (in 'irt4ancm.stan', line 32, column 2 to column 33)\n", "Exception: normal_lpdf: Scale parameter is 0, but must be positive! (in 'irt4ancm.stan', line 32, column 2 to column 33)\n", "\tException: normal_lpdf: Scale parameter is 0, but must be positive! (in 'irt4ancm.stan', line 32, column 2 to column 33)\n", "Exception: normal_lpdf: Scale parameter is 0, but must be positive! (in 'irt4ancm.stan', line 33, column 2 to column 40)\n", "\tException: normal_lpdf: Scale parameter is 0, but must be positive! (in 'irt4ancm.stan', line 33, column 2 to column 40)\n", "Consider re-running with show_console=True if the above output is unclear!\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n" ] } ], "source": [ "fit = model.sample(data=\"irt4ancm/all_plays.json\")" ] }, { "cell_type": "markdown", "metadata": { "id": "GKrDuu0vWKuB" }, "source": [ "Stan has a handy set of diagnostics that can warn you of any problems with your model fit. For the purposes of this lab, you will probably not have time to fix any problems, but you can report on them in the assignment." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "cmMMWpG2WHUC", "outputId": "324c33c9-95e7-4c64-bbbf-cc254ed1f6da" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "DEBUG:cmdstanpy:cmd: /root/.cmdstan/cmdstan-2.35.0/bin/diagnose /tmp/tmpmu1ubl7v/irt4ancmlqtbjyc1/irt4ancm-20240906130253_1.csv /tmp/tmpmu1ubl7v/irt4ancmlqtbjyc1/irt4ancm-20240906130253_2.csv /tmp/tmpmu1ubl7v/irt4ancmlqtbjyc1/irt4ancm-20240906130253_3.csv /tmp/tmpmu1ubl7v/irt4ancmlqtbjyc1/irt4ancm-20240906130253_4.csv\n", "cwd: None\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Processing csv files: /tmp/tmpmu1ubl7v/irt4ancmlqtbjyc1/irt4ancm-20240906130253_1.csv, /tmp/tmpmu1ubl7v/irt4ancmlqtbjyc1/irt4ancm-20240906130253_2.csv, /tmp/tmpmu1ubl7v/irt4ancmlqtbjyc1/irt4ancm-20240906130253_3.csv, /tmp/tmpmu1ubl7v/irt4ancmlqtbjyc1/irt4ancm-20240906130253_4.csv\n", "\n", "Checking sampler transitions treedepth.\n", "Treedepth satisfactory for all transitions.\n", "\n", "Checking sampler transitions for divergences.\n", "No divergent transitions found.\n", "\n", "Checking E-BFMI - sampler transitions HMC potential energy.\n", "E-BFMI satisfactory.\n", "\n", "Effective sample size satisfactory.\n", "\n", "Split R-hat values satisfactory all parameters.\n", "\n", "Processing complete, no problems detected.\n", "\n" ] } ], "source": [ "print(fit.diagnose())" ] }, { "cell_type": "markdown", "metadata": { "id": "TmreiQakV-Df" }, "source": [ "If the model is (mostly) problem-free, you can look at a summary of the parameter values. Remember that we get not a specific value but rather a whole distribution on each parameter. Stan reports means, standard error and deviation, and (most popular in the literature) 5%/50%/95% quantiles.\n", "\n", "The final three columns are convergence statistics. As a (very) rough rule of thumb, you want `N_eff` to be above 400 and `R_hat` to be less than 1.05." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 478 }, "id": "yvaxnrHST_d-", "outputId": "a3620aa1-8fc4-47f9-901f-01d8bee6e356" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "DEBUG:cmdstanpy:cmd: /root/.cmdstan/cmdstan-2.35.0/bin/stansummary --percentiles= 5,50,95 --sig_figs=6 --csv_filename=/tmp/tmpmu1ubl7v/stansummary-irt4ancm-rukd60vu.csv /tmp/tmpmu1ubl7v/irt4ancmlqtbjyc1/irt4ancm-20240906130253_1.csv /tmp/tmpmu1ubl7v/irt4ancmlqtbjyc1/irt4ancm-20240906130253_2.csv /tmp/tmpmu1ubl7v/irt4ancmlqtbjyc1/irt4ancm-20240906130253_3.csv /tmp/tmpmu1ubl7v/irt4ancmlqtbjyc1/irt4ancm-20240906130253_4.csv\n", "cwd: None\n" ] }, { "data": { "application/vnd.google.colaboratory.intrinsic+json": { "summary": "{\n \"name\": \"fit\",\n \"rows\": 936,\n \"fields\": [\n {\n \"column\": \"Mean\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 180.6264144254607,\n \"min\": -5525.36,\n \"max\": 4.48854,\n \"num_unique_values\": 934,\n \"samples\": [\n 1.09489,\n -0.239381,\n 1.52827\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"MCSE\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 0.02903419337143348,\n \"min\": 0.00141501,\n \"max\": 0.895102,\n \"num_unique_values\": 936,\n \"samples\": [\n 0.00889385,\n 0.0110754,\n 0.00800594\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"StdDev\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 0.8624207228137647,\n \"min\": 0.0444323,\n \"max\": 26.4279,\n \"num_unique_values\": 933,\n \"samples\": [\n 0.537039,\n 0.588412,\n 0.397121\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"5%\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 182.01731763061633,\n \"min\": -5568.77,\n \"max\": 3.23778,\n \"num_unique_values\": 936,\n \"samples\": [\n 0.508885,\n -1.2605,\n 0.969572\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"50%\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 180.61740849371978,\n \"min\": -5525.08,\n \"max\": 4.43376,\n \"num_unique_values\": 934,\n \"samples\": [\n 1.08922,\n -0.205475,\n 1.52513\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"95%\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 179.24128072093868,\n \"min\": -5482.13,\n \"max\": 5.96399,\n \"num_unique_values\": 935,\n \"samples\": [\n 1.67643,\n 0.693752,\n 2.10246\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"N_Eff\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 699.5391957000471,\n \"min\": 120.208,\n \"max\": 4661.17,\n \"num_unique_values\": 934,\n \"samples\": [\n 1628.64,\n 2822.57,\n 1873.02\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"N_Eff/s\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 5.919269491039114,\n \"min\": 1.01716,\n \"max\": 39.4412,\n \"num_unique_values\": 934,\n \"samples\": [\n 13.781,\n 23.8837,\n 15.8489\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"R_hat\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 0.0014965914729457167,\n \"min\": 0.99908,\n \"max\": 1.03309,\n \"num_unique_values\": 451,\n \"samples\": [\n 1.00115,\n 1.0005,\n 0.999954\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}", "type": "dataframe" }, "text/html": [ "\n", "
\n", "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
MeanMCSEStdDev5%50%95%N_EffN_Eff/sR_hat
lp__-5525.3600000.89510226.427900-5568.770000-5525.080000-5482.130000871.7297.376281.003680
mu_delta1.2597200.0087970.0964471.1020201.2603501.420080120.2081.017161.033090
sigma_theta1.6732000.0025290.0742051.5520201.6709001.798820861.0497.285911.003320
sigma_delta0.8138070.0014150.0444320.7435470.8122730.889880986.0078.343261.001050
theta[1]-1.8696700.0138520.785863-3.246340-1.820170-0.6920693218.81027.236500.999585
..............................
delta[433]0.7398240.0078180.3674950.1445540.7392321.3430202209.49018.696001.001850
delta[434]1.2236000.0080580.3796330.6064061.2126501.8604002219.48018.780501.000750
delta[435]1.0115300.0082520.3844060.3820541.0060301.6504002169.80018.360101.003760
delta[436]1.6203900.0081850.3802851.0052001.6103602.2492802158.78018.266901.003100
delta[437]2.0330200.0095910.4223291.3743802.0267302.7503401938.97016.406901.001530
\n", "

936 rows × 9 columns

\n", "
\n", "
\n", "\n", "
\n", " \n", "\n", " \n", "\n", " \n", "
\n", "\n", "\n", "
\n", " \n", "\n", "\n", "\n", " \n", "
\n", "\n", "
\n", "
\n" ], "text/plain": [ " Mean MCSE StdDev 5% 50% \\\n", "lp__ -5525.360000 0.895102 26.427900 -5568.770000 -5525.080000 \n", "mu_delta 1.259720 0.008797 0.096447 1.102020 1.260350 \n", "sigma_theta 1.673200 0.002529 0.074205 1.552020 1.670900 \n", "sigma_delta 0.813807 0.001415 0.044432 0.743547 0.812273 \n", "theta[1] -1.869670 0.013852 0.785863 -3.246340 -1.820170 \n", "... ... ... ... ... ... \n", "delta[433] 0.739824 0.007818 0.367495 0.144554 0.739232 \n", "delta[434] 1.223600 0.008058 0.379633 0.606406 1.212650 \n", "delta[435] 1.011530 0.008252 0.384406 0.382054 1.006030 \n", "delta[436] 1.620390 0.008185 0.380285 1.005200 1.610360 \n", "delta[437] 2.033020 0.009591 0.422329 1.374380 2.026730 \n", "\n", " 95% N_Eff N_Eff/s R_hat \n", "lp__ -5482.130000 871.729 7.37628 1.003680 \n", "mu_delta 1.420080 120.208 1.01716 1.033090 \n", "sigma_theta 1.798820 861.049 7.28591 1.003320 \n", "sigma_delta 0.889880 986.007 8.34326 1.001050 \n", "theta[1] -0.692069 3218.810 27.23650 0.999585 \n", "... ... ... ... ... \n", "delta[433] 1.343020 2209.490 18.69600 1.001850 \n", "delta[434] 1.860400 2219.480 18.78050 1.000750 \n", "delta[435] 1.650400 2169.800 18.36010 1.003760 \n", "delta[436] 2.249280 2158.780 18.26690 1.003100 \n", "delta[437] 2.750340 1938.970 16.40690 1.001530 \n", "\n", "[936 rows x 9 columns]" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "fit.summary()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You may want to rearrange the parameter estimates in a numpy array (or even a pandas dataframe, when your model has more parameters), so that you can look at the $\\delta$ estimates for each segment." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "deltas = fit.stan_variables()['delta'].mean(axis=0)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "By simply sorting the array, you can then identify the segments corresponding to the higher/lower $\\delta$ values." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# segments ids corresponding to lower deltas\n", "print(np.argsort(deltas)[:10])\n", "\n", "# segments ids corresponding to higher deltas\n", "print(np.flip(np.argsort(deltas))[:10])" ] }, { "cell_type": "markdown", "metadata": { "id": "aiqRafBuW4ek" }, "source": [ "Edit `irt4ancn.stan` to try different models. Ask Ashley for help with the syntax! Handy distributions include:\n", "\n", "* `~ std_normal()` for a standard normal (or half-normal) distribution\n", "* `~ normal(mu, sigma)` for a normal distribution with specified mean and standard deviation\n", "* `~ lognormal(mu, sigma)` for a log-normal distribution (handy for discrimination parameters)\n", "* `~ bernoulli(p)` for a Bernoulli distribution parameterised by the probability of success\n", "* `~ bernoulli_logit(z)` for a Bernoulli distribution parameterised by the inverse logistic function of the probability of success.\n", "\n" ] }, { "cell_type": "markdown", "metadata": { "id": "sNDqrfjzX5ud" }, "source": [ "The full 4PL IRT model looks like this:\n", "\n", "$\\mathrm{P}[x_{ni} = 1] = \\gamma_i + (\\zeta_i - \\gamma_i) \\frac{\\mathrm{e}^{\\alpha_i(\\theta_n - \\delta_i)}}{1 + \\mathrm{e}^{\\alpha_i(\\theta_n - \\delta_i)}}$\n", "\n", "* For the 3PL, $\\zeta$ is fixed to 1.\n", "* For the 2PL, $\\zeta$ is fixed to 1 and $\\gamma$ is fixed to 0.\n", "* For the 1PL (Rasch model), $\\zeta$ is fixed to 1, $\\gamma$ is fixed to 0, and $\\alpha$ is fixed to 1.\n", "\n", "Don't forget to add priors as you add more parameters!\n", "\n", "**WARNING**: In the 2PL, 3PL, and 4PL, $\\theta$ needs to be distributed as a standard normal distribution and there can be no hyper-parameter $\\sigma_\\theta$. Otherwise, the model is not identified, and Stan will run into many problems while sampling.\n" ] }, { "cell_type": "markdown", "metadata": { "id": "s4NLzTW_PGaR" }, "source": [ "### **ASSIGNMENT**" ] }, { "cell_type": "markdown", "metadata": { "id": "iVz5tgVATDQl" }, "source": [ "1. Explore 1-, 2-, 3- and 4-parameter IRT models for the Hooked on Music data according to the template. Which segments are most difficult? Which are easiest? Most/least discriminating? Are the guessing parameters what you would expect? (Tip: if you're struggling with the syntax, here is the [IRT section](https://mc-stan.org/docs/stan-users-guide/regression.html#item-response-models.section) of the Stan Users Guide)\n", "2. Explore an alternative data model (e.g., using `rec_only.json` or focussing on `is_recognised` instead of `is_verified`), again with 1-, 2-, 3- and 4-parameter IRT models. How do your results compare to what you found in Step 1?\n", "\n", "\n", "Write a short report (less than one page) summarising your findings and (to the extent you can) any musical explanations or surprising findings based on what you can hear in the songs.\n", "\n", "⚠️ Please make sure you include your `irt4ancn.stan` file(s) in the assignment submission, as well as the notebook and the PDF with your report." ] }, { "cell_type": "markdown", "metadata": { "id": "shazGclJUbsj" }, "source": [ "## Additional Resources" ] }, { "cell_type": "markdown", "metadata": { "id": "XVxE_YCwUex8" }, "source": [ "\n", "\n", "* The `cmdstanpy` [documentation](https://cmdstanpy.readthedocs.io/en/v1.0.7/)\n", "* The Stan [User’s Guide](https://mc-stan.org/docs/stan-users-guide/index.html])\n", "\n" ] } ], "metadata": { "colab": { "provenance": [] }, "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "name": "python" }, "widgets": { "application/vnd.jupyter.widget-state+json": { "0553b81ee6ac4ef5a48a139bb54d96ad": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "10b260f1a5c74ee887c372296478a48c": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_b4bfcc8fd40642dbb792aa4e1dd946bc", "placeholder": "​", "style": "IPY_MODEL_86a4331aa62e4e94b9bda845a9d4c4d0", "value": "chain 2 " } }, "2318b5395f1f4d21ae94f04e372577b7": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "287ab4796c61481dae9b1cea01907c25": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_66721d5beac74c48ac041732fa0867de", "max": 22, "min": 0, "orientation": "horizontal", "style": "IPY_MODEL_f1f827b09dc54fdf933335abaa671948", "value": 22 } }, "2dc41fa030b548748037fe6306b0ceca": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_fb5bba5506f5472f880ae86943d5f284", "max": 22, "min": 0, "orientation": "horizontal", "style": "IPY_MODEL_f1069348739341cf8031c36a50bfabb1", "value": 22 } }, "3074484d3fe248039f0a6aeb322f4dbd": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "3a37861fc35348618fc5e9946fa26ca5": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_9483d615ad0a4d96bed8fd4995d6a519", "placeholder": "​", "style": "IPY_MODEL_90c4c9d334df4187beffef1305d6022e", "value": " 02:37 Sampling completed" } }, "3a89930f6a774939803bce07f4bb20af": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_b1322cd858c5428fb0cc5eca303e2515", "placeholder": "​", "style": "IPY_MODEL_9fd326a2468c4e7cb2e31f6033b4dcd7", "value": "chain 3 " } }, "419233e503434179846e6530b5f95345": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "4d4ef945113f4590a80eca2bf2ef36b9": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "65fc914d6e0d432cb5e7a3c4a9403d0d": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "66721d5beac74c48ac041732fa0867de": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "6b88027e796b4183aa0fa13933913645": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "6ccd15b0016a4245acd6da22faf6f038": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_4d4ef945113f4590a80eca2bf2ef36b9", "placeholder": "​", "style": "IPY_MODEL_991b0f118a4442409a0cb31af9ace12d", "value": " 02:37 Sampling completed" } }, "6cd9897f424344b99da25f95602e5aa5": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "6f98b9dacb814aa4ad8a7f59c4e69400": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_6b88027e796b4183aa0fa13933913645", "placeholder": "​", "style": "IPY_MODEL_b1fcba53f8a342f48fb0876fd4e8bae9", "value": "chain 1 " } }, "704f2afcc3854fe9b61296a7aa41b7aa": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_f9b992ab76214c0c8e65098823311f6f", "placeholder": "​", "style": "IPY_MODEL_af4285d60e484786b969054623ba0c14", "value": " 02:37 Sampling completed" } }, "7440bb5f786040f0af16185279da23af": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": [ "IPY_MODEL_10b260f1a5c74ee887c372296478a48c", "IPY_MODEL_cc8db68771f3498da873f2dbb0f686b7", "IPY_MODEL_6ccd15b0016a4245acd6da22faf6f038" ], "layout": "IPY_MODEL_e3f58e5158e149e199b6eb4c22f3c6d4" } }, "7be3ae70869242a5ba35b963c2fad86e": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "86a4331aa62e4e94b9bda845a9d4c4d0": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "898f15ebf53543aa8984937ba8bfda69": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "90c4c9d334df4187beffef1305d6022e": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "9483d615ad0a4d96bed8fd4995d6a519": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "991b0f118a4442409a0cb31af9ace12d": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "9fd326a2468c4e7cb2e31f6033b4dcd7": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "a324bebf97d2424286d066b36091ebc1": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "ae4c402e37794df6871be4d64fdab4af": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": [ "IPY_MODEL_6f98b9dacb814aa4ad8a7f59c4e69400", "IPY_MODEL_e07a0b24f3f142fc9733f19b3aaa78fb", "IPY_MODEL_c867f146a1924263ae7e9001e6e8fb68" ], "layout": "IPY_MODEL_0553b81ee6ac4ef5a48a139bb54d96ad" } }, "af4285d60e484786b969054623ba0c14": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "b1322cd858c5428fb0cc5eca303e2515": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "b1fcba53f8a342f48fb0876fd4e8bae9": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "b4bfcc8fd40642dbb792aa4e1dd946bc": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "c867f146a1924263ae7e9001e6e8fb68": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_2318b5395f1f4d21ae94f04e372577b7", "placeholder": "​", "style": "IPY_MODEL_6cd9897f424344b99da25f95602e5aa5", "value": " 02:37 Sampling completed" } }, "cc8db68771f3498da873f2dbb0f686b7": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_7be3ae70869242a5ba35b963c2fad86e", "max": 22, "min": 0, "orientation": "horizontal", "style": "IPY_MODEL_e87561d8806246f59a668a76a54cb63c", "value": 22 } }, "d2eb4580992f456fb7ace05caf78c2ba": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_3074484d3fe248039f0a6aeb322f4dbd", "placeholder": "​", "style": "IPY_MODEL_a324bebf97d2424286d066b36091ebc1", "value": "chain 4 " } }, "e07a0b24f3f142fc9733f19b3aaa78fb": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_898f15ebf53543aa8984937ba8bfda69", "max": 22, "min": 0, "orientation": "horizontal", "style": "IPY_MODEL_e0b4c592b3ad4be58ac5b915a2088bd6", "value": 22 } }, "e0b4c592b3ad4be58ac5b915a2088bd6": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": "blue", "description_width": "" } }, "e3f58e5158e149e199b6eb4c22f3c6d4": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "e87561d8806246f59a668a76a54cb63c": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": "blue", "description_width": "" } }, "ef3b5623fa3d4b559b2120cc07033dda": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": [ "IPY_MODEL_3a89930f6a774939803bce07f4bb20af", "IPY_MODEL_2dc41fa030b548748037fe6306b0ceca", "IPY_MODEL_704f2afcc3854fe9b61296a7aa41b7aa" ], "layout": "IPY_MODEL_419233e503434179846e6530b5f95345" } }, "f1069348739341cf8031c36a50bfabb1": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": "blue", "description_width": "" } }, "f1f827b09dc54fdf933335abaa671948": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": "blue", "description_width": "" } }, "f7073f38455740e99a448ca3866b6d64": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": [ "IPY_MODEL_d2eb4580992f456fb7ace05caf78c2ba", "IPY_MODEL_287ab4796c61481dae9b1cea01907c25", "IPY_MODEL_3a37861fc35348618fc5e9946fa26ca5" ], "layout": "IPY_MODEL_65fc914d6e0d432cb5e7a3c4a9403d0d" } }, "f9b992ab76214c0c8e65098823311f6f": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "fb5bba5506f5472f880ae86943d5f284": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } } } } }, "nbformat": 4, "nbformat_minor": 0 }