Show loading options for DataFrame

machine-learning/DataSources.ipynb

@@ -0,0 +1,253 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Loading data \n",
+    "\n",
+    "Getting data into the DataFrame is the most important step. The DataFrame itself supports [loading from a csv](https://docs.microsoft.com/en-us/dotnet/api/microsoft.data.analysis.dataframe.loadcsvfromstring?view=ml-dotnet-preview#microsoft-data-analysis-dataframe-loadcsvfromstring(system-string-system-char-system-boolean-system-string()-system-type()-system-int64-system-int32-system-boolean)). Not all data is already in a csv file. There is the option to convert from an IDataView into a DataFrame. ML.NET supports loading from a few different sources into an IDataView. See docs [here](https://docs.microsoft.com/en-us/dotnet/machine-learning/how-to-guides/load-data-ml-net). \n",
+    "\n",
+    "If you run into issue, please file them in our [Github repo](https://github.com/dotnet/machinelearning/issues). If possible, please include the problem data set. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "dotnet_interactive": {
+     "language": "csharp"
+    },
+    "vscode": {
+     "languageId": "dotnet-interactive.csharp"
+    }
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div><div></div><div></div><div><strong>Installed Packages</strong><ul><li><span>DataView.InteractiveExtension, 1.0.45</span></li><li><span>Microsoft.Data.Analysis, 0.19.1</span></li><li><span>Microsoft.ML, 1.7.1</span></li></ul></div></div>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/markdown": [
+       "Loading extensions from `DataView.InteractiveExtension.dll`"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/markdown": [
+       "Loading extensions from `Microsoft.Data.Analysis.Interactive.dll`"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/markdown": [
+       "Added support IDataView to kernel .NET."
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "// load extension to get data frame api, visualization and formatting\n",
+    "\n",
+    "#r \"nuget: Microsoft.Data.Analysis, 0.19.1\"\n",
+    "#r \"nuget: DataView.InteractiveExtension, 1.0.45\"\n",
+    "#r \"nuget: Microsoft.ML\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Directly from CSV\n",
+    "We can easily load our data directly from a CSV into the DataFrame. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "dotnet_interactive": {
+     "language": "csharp"
+    },
+    "vscode": {
+     "languageId": "dotnet-interactive.csharp"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "var csvFilePath = @\"data/usa_hockey.csv\";"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "dotnet_interactive": {
+     "language": "csharp"
+    },
+    "vscode": {
+     "languageId": "dotnet-interactive.csharp"
+    }
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<table id=\"table_637886533772330844\"><thead><tr><th><i>index</i></th><th>Birthday</th><th>Nat</th><th>Height</th><th>Weight</th><th>DraftYear</th><th>OverallDraft</th><th>Hand</th><th>Last Name</th><th>First Name</th><th>Position</th><th>Team</th><th>GamesPlayed</th><th>Goals</th><th>Assists</th><th>Points</th><th>PIM</th><th>Shifts</th><th>TimeOnIce</th></tr></thead><tbody><tr><td><i><div class=\"dni-plaintext\">0</div></i></td><td>88-16-04</td><td>USA</td><td><div class=\"dni-plaintext\">72</div></td><td><div class=\"dni-plaintext\">218</div></td><td><div class=\"dni-plaintext\">2006</div></td><td><div class=\"dni-plaintext\">7</div></td><td>R</td><td>Okposo</td><td>Kyle</td><td>RW</td><td>BUF</td><td><div class=\"dni-plaintext\">65</div></td><td><div class=\"dni-plaintext\">19</div></td><td><div class=\"dni-plaintext\">26</div></td><td><div class=\"dni-plaintext\">45</div></td><td><div class=\"dni-plaintext\">24</div></td><td><div class=\"dni-plaintext\">1443</div></td><td><div class=\"dni-plaintext\">73983</div></td></tr><tr><td><i><div class=\"dni-plaintext\">1</div></i></td><td>90-08-10</td><td>USA</td><td><div class=\"dni-plaintext\">76</div></td><td><div class=\"dni-plaintext\">210</div></td><td><div class=\"dni-plaintext\">2009</div></td><td><div class=\"dni-plaintext\">114</div></td><td>L</td><td>Helgeson</td><td>Seth</td><td>D</td><td>N.J</td><td><div class=\"dni-plaintext\">9</div></td><td><div class=\"dni-plaintext\">1</div></td><td><div class=\"dni-plaintext\">0</div></td><td><div class=\"dni-plaintext\">1</div></td><td><div class=\"dni-plaintext\">15</div></td><td><div class=\"dni-plaintext\">177</div></td><td><div class=\"dni-plaintext\">7273</div></td></tr><tr><td><i><div class=\"dni-plaintext\">2</div></i></td><td>96-26-11</td><td>USA</td><td><div class=\"dni-plaintext\">77</div></td><td><div class=\"dni-plaintext\">203</div></td><td><div class=\"dni-plaintext\">2015</div></td><td><div class=\"dni-plaintext\">37</div></td><td>R</td><td>Carlo</td><td>Brandon</td><td>D</td><td>BOS</td><td><div class=\"dni-plaintext\">82</div></td><td><div class=\"dni-plaintext\">6</div></td><td><div class=\"dni-plaintext\">10</div></td><td><div class=\"dni-plaintext\">16</div></td><td><div class=\"dni-plaintext\">59</div></td><td><div class=\"dni-plaintext\">2080</div></td><td><div class=\"dni-plaintext\">102414</div></td></tr><tr><td><i><div class=\"dni-plaintext\">3</div></i></td><td>90-16-11</td><td>USA</td><td><div class=\"dni-plaintext\">74</div></td><td><div class=\"dni-plaintext\">219</div></td><td><div class=\"dni-plaintext\">&lt;null&gt;</div></td><td><div class=\"dni-plaintext\">&lt;null&gt;</div></td><td>L</td><td>Schaller</td><td>Tim</td><td>C</td><td>BOS</td><td><div class=\"dni-plaintext\">59</div></td><td><div class=\"dni-plaintext\">7</div></td><td><div class=\"dni-plaintext\">7</div></td><td><div class=\"dni-plaintext\">14</div></td><td><div class=\"dni-plaintext\">23</div></td><td><div class=\"dni-plaintext\">1035</div></td><td><div class=\"dni-plaintext\">43436</div></td></tr><tr><td><i><div class=\"dni-plaintext\">4</div></i></td><td>92-20-03</td><td>USA</td><td><div class=\"dni-plaintext\">72</div></td><td><div class=\"dni-plaintext\">215</div></td><td><div class=\"dni-plaintext\">2010</div></td><td><div class=\"dni-plaintext\">37</div></td><td>R</td><td>Faulk</td><td>Justin</td><td>D</td><td>CAR</td><td><div class=\"dni-plaintext\">75</div></td><td><div class=\"dni-plaintext\">17</div></td><td><div class=\"dni-plaintext\">20</div></td><td><div class=\"dni-plaintext\">37</div></td><td><div class=\"dni-plaintext\">32</div></td><td><div class=\"dni-plaintext\">1987</div></td><td><div class=\"dni-plaintext\">104133</div></td></tr><tr><td><i><div class=\"dni-plaintext\">5</div></i></td><td>94-01-05</td><td>USA</td><td><div class=\"dni-plaintext\">74</div></td><td><div class=\"dni-plaintext\">205</div></td><td><div class=\"dni-plaintext\">2012</div></td><td><div class=\"dni-plaintext\">120</div></td><td>L</td><td>Slavin</td><td>Jaccob</td><td>D</td><td>CAR</td><td><div class=\"dni-plaintext\">82</div></td><td><div class=\"dni-plaintext\">5</div></td><td><div class=\"dni-plaintext\">29</div></td><td><div class=\"dni-plaintext\">34</div></td><td><div class=\"dni-plaintext\">12</div></td><td><div class=\"dni-plaintext\">2135</div></td><td><div class=\"dni-plaintext\">115316</div></td></tr><tr><td><i><div class=\"dni-plaintext\">6</div></i></td><td>90-20-06</td><td>USA</td><td><div class=\"dni-plaintext\">75</div></td><td><div class=\"dni-plaintext\">221</div></td><td><div class=\"dni-plaintext\">2008</div></td><td><div class=\"dni-plaintext\">128</div></td><td>R</td><td>Pateryn</td><td>Greg</td><td>D</td><td>DAL/MTL</td><td><div class=\"dni-plaintext\">36</div></td><td><div class=\"dni-plaintext\">1</div></td><td><div class=\"dni-plaintext\">8</div></td><td><div class=\"dni-plaintext\">9</div></td><td><div class=\"dni-plaintext\">10</div></td><td><div class=\"dni-plaintext\">720</div></td><td><div class=\"dni-plaintext\">33312</div></td></tr><tr><td><i><div class=\"dni-plaintext\">7</div></i></td><td>90-27-05</td><td>USA</td><td><div class=\"dni-plaintext\">74</div></td><td><div class=\"dni-plaintext\">196</div></td><td><div class=\"dni-plaintext\">2009</div></td><td><div class=\"dni-plaintext\">198</div></td><td>R</td><td>Dowd</td><td>Nic</td><td>C</td><td>L.A</td><td><div class=\"dni-plaintext\">70</div></td><td><div class=\"dni-plaintext\">6</div></td><td><div class=\"dni-plaintext\">16</div></td><td><div class=\"dni-plaintext\">22</div></td><td><div class=\"dni-plaintext\">25</div></td><td><div class=\"dni-plaintext\">1230</div></td><td><div class=\"dni-plaintext\">52314</div></td></tr><tr><td><i><div class=\"dni-plaintext\">8</div></i></td><td>90-16-07</td><td>USA</td><td><div class=\"dni-plaintext\">75</div></td><td><div class=\"dni-plaintext\">221</div></td><td><div class=\"dni-plaintext\">&lt;null&gt;</div></td><td><div class=\"dni-plaintext\">&lt;null&gt;</div></td><td>L</td><td>Lashoff</td><td>Brian</td><td>D</td><td>DET</td><td><div class=\"dni-plaintext\">5</div></td><td><div class=\"dni-plaintext\">0</div></td><td><div class=\"dni-plaintext\">0</div></td><td><div class=\"dni-plaintext\">0</div></td><td><div class=\"dni-plaintext\">0</div></td><td><div class=\"dni-plaintext\">93</div></td><td><div class=\"dni-plaintext\">3754</div></td></tr><tr><td><i><div class=\"dni-plaintext\">9</div></i></td><td>86-09-08</td><td>USA</td><td><div class=\"dni-plaintext\">71</div></td><td><div class=\"dni-plaintext\">197</div></td><td><div class=\"dni-plaintext\">&lt;null&gt;</div></td><td><div class=\"dni-plaintext\">&lt;null&gt;</div></td><td>R</td><td>Cannone</td><td>Patrick</td><td>C</td><td>MIN</td><td><div class=\"dni-plaintext\">3</div></td><td><div class=\"dni-plaintext\">0</div></td><td><div class=\"dni-plaintext\">0</div></td><td><div class=\"dni-plaintext\">0</div></td><td><div class=\"dni-plaintext\">0</div></td><td><div class=\"dni-plaintext\">35</div></td><td><div class=\"dni-plaintext\">1419</div></td></tr></tbody></table>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "using Microsoft.Data.Analysis;\n",
+    "\n",
+    "var csvDataFrame = DataFrame.LoadCsv(csvFilePath);\n",
+    "csvDataFrame"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## ML.NET IDataView Loader\n",
+    "You may want to load from a different data source. ML.NET supports many different data souces, and we can convert an IDataView into a DataFrame. Find out more about IDataViews [here](https://github.com/dotnet/machinelearning/blob/main/docs/code/IDataViewDesignPrinciples.md). "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "dotnet_interactive": {
+     "language": "csharp"
+    },
+    "vscode": {
+     "languageId": "dotnet-interactive.csharp"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "using Microsoft.ML.Data; \n",
+    "\n",
+    "public class SalaryData\n",
+    "{\n",
+    "    [LoadColumn(0)]\n",
+    "    public float Salary { get; set; }\n",
+    "\n",
+    "    [LoadColumn(1)]\n",
+    "    public string Name { get; set; }\n",
+    "}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### From File"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "dotnet_interactive": {
+     "language": "csharp"
+    },
+    "vscode": {
+     "languageId": "dotnet-interactive.csharp"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "using Microsoft.ML;\n",
+    "using Microsoft.ML.Data;\n",
+    "using System;\n",
+    "using System.Collections.Generic;\n",
+    "using System.Linq;\n",
+    "\n",
+    "//Create MLContext\n",
+    "MLContext mlContext = new MLContext();\n",
+    "\n",
+    "//Load Data\n",
+    "IDataView data = mlContext.Data.LoadFromTextFile<SalaryData>(\"data/playerSalary.csv\", separatorChar: ',', hasHeader: true);\n",
+    "var df = data.ToDataFrame();\n",
+    "\n",
+    "df"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### From JSON"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "dotnet_interactive": {
+     "language": "csharp"
+    },
+    "vscode": {
+     "languageId": "dotnet-interactive.csharp"
+    }
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<table id=\"table_637886552733619009\"><thead><tr><th><i>index</i></th><th>Salary</th><th>Name</th></tr></thead><tbody><tr><td><i><div class=\"dni-plaintext\">0</div></i></td><td><div class=\"dni-plaintext\">3000000</div></td><td>Adam Larsson</td></tr><tr><td><i><div class=\"dni-plaintext\">1</div></i></td><td><div class=\"dni-plaintext\">1600000</div></td><td>Andrej Sustr</td></tr><tr><td><i><div class=\"dni-plaintext\">2</div></i></td><td><div class=\"dni-plaintext\">2200000</div></td><td>Antoine Roussel</td></tr><tr><td><i><div class=\"dni-plaintext\">3</div></i></td><td><div class=\"dni-plaintext\">950000</div></td><td>Anton Rodin</td></tr></tbody></table>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "using Newtonsoft.Json;\n",
+    "using System.IO;\n",
+    "\n",
+    "// Load the json file into an ennumerable, then into the data view from the ennumerable. \n",
+    "var accounts = JsonConvert.DeserializeObject<List<SalaryData>>(File.ReadAllText(@\"data\\playerSalary.json\"));\n",
+    "IDataView dataView = mlContext.Data.LoadFromEnumerable<SalaryData>(accounts);\n",
+    "\n",
+    "// Convert to DataFrame\n",
+    "var jsonDataFrame = dataView.ToDataFrame(); \n",
+    "\n",
+    "jsonDataFrame"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": ".NET (C#)",
+   "language": "C#",
+   "name": ".net-csharp"
+  },
+  "language_info": {
+   "file_extension": ".cs",
+   "mimetype": "text/x-csharp",
+   "name": "C#",
+   "pygments_lexer": "csharp",
+   "version": "8.0"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

machine-learning/data/playerSalary.json

@@ -0,0 +1,18 @@
+[
+{
+    "Salary": 3000000,
+    "Name": "Adam Larsson"
+},
+{
+   "Salary": 1600000,
+   "Name": "Andrej Sustr"
+}, 
+{
+   "Salary": 2200000,
+   "Name": "Antoine Roussel"
+}, 
+{
+   "Salary": 950000,
+   "Name": "Anton Rodin"
+}
+]