\n",
" SELECT columnnames\n",
" FROM tablename\n",
" WHERE conditions\n",
"\n",
"To make sure all of the abundances are valid, required that each abundance be larger than -10 (bad values are flagged as -999). \n",
"\n",
"While I haven't left you to fill in the query, please take a good look at it to make sure you see the possibilities."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"88346\n"
]
}
],
"source": [
"from astroquery.sdss import SDSS\n",
"sql=\" SELECT FE_H, MG_FE, O_FE, SI_FE, AL_FE, MN_FE, NI_FE, apogeeDistMass.age, apogeeStar.ra, apogeeStar.dec \\\n",
" FROM aspcapStar \\\n",
" JOIN apogeeStar on apogeeStar.apstar_id = aspcapStar.apstar_id \\\n",
" JOIN apogeeDistMass on apogeeDistMass.apstar_id = apogeeStar.apstar_id \\\n",
" WHERE aspcapStar.LOGG>1 AND aspcapStar.LOGG<2 AND FE_H>-10 AND MG_FE>-10 AND O_FE>-10 AND SI_FE>-10 \\\n",
" AND AL_FE>-10 AND MN_FE>-10 AND NI_FE>-10 \\\n",
" \"\n",
"dr17=SDSS.query_sql(sql, data_release=17) \n",
"print(len(dr17))"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"ename": "ValueError",
"evalue": "Some errors were detected !\n Line #4 (got 6 columns instead of 1)",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mValueError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m \n",
"I encourage you to follow the links for each method and read a bit about them to try to get some rough feeling for the methodologies involved."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from sklearn import manifold"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"All methods take an n_components= keyword, which we will set to 2, to just get the first two components for visualization."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"n_components=2"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Several of the methods take a n_neighbors= keyword, which specifies the number of nearby points over which one wants the method to preserve local distances. We'll use a default value of 10, but you can experiment with this!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"n_neighbors=10"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"OK, first we will try local linear embedding (LLE). "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"LLE = manifold.LocallyLinearEmbedding(n_neighbors=n_neighbors,n_components=n_components,eigen_solver=\"auto\")\n",
"Y = LLE.fit_transform(X)\n",
"\n",
"plot(Y,color_codes)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"What are these plots of? What do you think about the ability of this projection to separate the data in interesting ways?\n",
"
\n",
"To facilitate this, here's a function that will take some array of projected coordinates, and make a series of plots color coded by whateven quantities you supply."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def plot(Y,color_codes) :\n",
" \"\"\" Make a series of plots of a projection, plotting the first component against the second\n",
" Color-code these plots by whatever arrays are passed in colors, which must be a list\n",
" \"\"\"\n",
" nx=len(color_codes)\n",
" fig = plt.figure(figsize=(15, 8))\n",
" ax=fig.subplots(1,nx)\n",
" for i,color in enumerate(color_codes) :\n",
" ax[i].scatter(Y[:,0],Y[:,1],c=color)\n",
" plt.show()\n",
" \n",
"# set some quantities to use for the color coding\n",
"fe_h=X[:,0]\n",
"mg_fe=X[:,2]\n",
"al_fe=X[:,3]\n",
"age=dr17['age'][::nsamp]\n",
"color_codes=[fe_h,mg_fe,al_fe,age]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"First we'll just try PCA, just looking at the first two components."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from sklearn.decomposition import PCA\n",
"#do the PCA decomposition\n",
"pca= #instantiate a PCA object with n_components=2\n",
"X_projected= # do the decomposition and get the transformed coordinates with fit_transform()\n",
"\n",
"plot(X_projected,color_codes)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Just to make sure you're understanding the code, describe what these are plots of:\n",
"
ANSWER HERE: "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"OK, now we're going to try several manifold learning routines from scikit-learn manifold learning. These all are used in the same way, just like the PCA() routine:\n",
"\n",
"
\n",
"That's it! Then we'll use our plot() routine to plot the projections\n",
"
ANSWER HERE: "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"OK, now we can try Isomap() . See if you can fill in the instatiation and the fit_transform() call"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"isomap= # fill in the instantiation\n",
"Y= # get projected coordinates\n",
"\n",
"plot(Y,color_codes)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"What do you think about the ability of this projection to separate the data in interesting ways?\n",
"
ANSWER HERE: "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now try Multi-dimensional scaling (MDS) . You may need to specify max_iter= and n_init="
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"mds= # fill in the instantiation\n",
"Y= # get the projected coordinates\n",
"\n",
"plot(Y,color_codes)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"What do you think about the ability of this projection to separate the data in interesting ways?\n",
"
ANSWER HERE: "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Finally, we have t-SNE . This one has more tuning parameters, including perplexity and early-exaggeration. See this page for a more extensive discussion : good reading while you are waiting for this one to run!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"tsne= # fill in the instantiation\n",
"Y= # get projected coordinates\n",
"\n",
"plot(Y,color_codes)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here we specify some values for perplexity and early_exaggeration.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"tsne=manifold.TSNE(n_components=2,perplexity=30,early_exaggeration=800)\n",
"\n",
"Y=tsne.fit_transform(X)\n",
"\n",
"plot(Y,color_codes)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"That one is intriguing, isn't it? Don't you want to know what those distinct clouds of points might be? How might you track that down?\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"What do you think about the ability of this projection to separate the data in interesting ways?\n",
"
ANSWER HERE: "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For fun, here's some code I borrowed from the source for plots at scikit-learn, that lets you declare all of the methods you want try, and loop over them, runinng them, timing them, and making plots!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from sklearn import manifold, datasets\n",
"from functools import partial\n",
"from collections import OrderedDict\n",
"from time import time\n",
"from matplotlib.ticker import NullFormatter\n",
"\n",
"n_neighbors = 10\n",
"n_components = 2\n",
"\n",
"fig = plt.figure(figsize=(15, 16))\n",
"fig.suptitle(\n",
" \"Manifold Learning with %i points, %i neighbors\" % (1000, n_neighbors), fontsize=14\n",
")\n",
"\n",
"color=X[:,0]\n",
"color2=X[:,3]\n",
"\n",
"# Set-up manifold methods\n",
"LLE = partial(\n",
" manifold.LocallyLinearEmbedding,\n",
" n_neighbors=n_neighbors,\n",
" n_components=n_components,\n",
" eigen_solver=\"auto\",\n",
")\n",
"\n",
"methods = OrderedDict()\n",
"methods[\"LLE\"] = LLE(method=\"standard\")\n",
"#methods[\"LTSA\"] = LLE(method=\"ltsa\")\n",
"#methods[\"Hessian LLE\"] = LLE(method=\"hessian\")\n",
"#methods[\"Modified LLE\"] = LLE(method=\"modified\")\n",
"methods[\"Isomap\"] = manifold.Isomap(n_neighbors=n_neighbors, n_components=n_components)\n",
"methods[\"MDS\"] = manifold.MDS(n_components, max_iter=100, n_init=1)\n",
"#methods[\"SE\"] = manifold.SpectralEmbedding(n_components=n_components, n_neighbors=n_neighbors)\n",
"methods[\"t-SNE\"] = manifold.TSNE(n_components=n_components, init=\"pca\", random_state=0)\n",
"\n",
"# Plot results\n",
"for i, (label, method) in enumerate(methods.items()):\n",
" t0 = time()\n",
" Y = method.fit_transform(X)\n",
" t1 = time()\n",
" print(\"%s: %.2g sec\" % (label, t1 - t0))\n",
" \n",
" nx = len(color_codes)\n",
" for ix, color in enumerate(color_codes) :\n",
" ax = fig.add_subplot(5, nx, i*nx + ix + 1)\n",
" ax.scatter(Y[:, 0], Y[:, 1], c=color, cmap=plt.cm.Spectral)\n",
" ax.set_title(\"%s (%.2g sec)\" % (label, t1 - t0))\n",
" ax.xaxis.set_major_formatter(NullFormatter())\n",
" ax.yaxis.set_major_formatter(NullFormatter())\n",
" ax.axis(\"tight\")\n",
" "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Discuss what you think about the different methods, commenting on their potential for highlighting interesting features in the data. Do you think they have any value compared to just looking at the original data?\n",
"
ANSWER HERE: "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.11"
}
},
"nbformat": 4,
"nbformat_minor": 1
}