Week of April 25th

What I did

Sunday: Thesis chapter \(\LaTeX\) writing
Monday: Thesis chapter \(\LaTeX\) writing
Tuesday: More writing, but I also redid some figures comparing simulation to experimental data.
Wednesday: Finished revising thesis chapter two to the point that it was good enough to send along. Attempted to install tensorflow, but couldn’t get the GPU working. Will do tomorrow.
Thursday: Finished up the multi-moment-gausianator.py script and generated all the training data for my NN. Finished setting up an anaconda venv for Tensorflow 2.4.0.
Friday: Got data loaded into tensorflow working. Currently loads all the csvs into a stacked csv with the index as the MSD of the csv (xanes spectrum). Might change this to be tensors or np arrays.

What I learned

Tuesday: By including some disorder in the simulated bulk spectrum I made a slightly better representation of the experiment vs simulation comparison. The problem is that one of the simulation peaks doesn’t cross in the same place that they very clearly cross in the experimental data. Could just be that the size difference is coded in the third peak.
Wednesday: The problem was that anaconda install tensorflow-gpu didn’t work, and when I did a pip install tensorflow-gpu; pip install cudnn; pip install cuda-toolkit, I didn’t specify exact versions, and if you just install all the most recent ones, it turns out they’re not compatible. I fixed it by downgrading to tf 2.4.0 and cudnn 8.0.
Thursday: Loading lots of csv files into tensorflow is trickier than I thought, especially since I encoded the labels in the header. I also encoded the labels in a separate csv file, so maybe I’ll just load the data and fix the labels after? Or alternatively, I could write my own generator function. But it would be nice to use tensorflow’s built in dataloader funcitons.
Friday: Leaned some good bash shortcuts

sudo !! # runs previous command but with sudo prepended to it
ctr+k   # kills all words to right of cursor 
ctr+u   # deletes all words to left of curso
ctr+y   # pastes back what was cut by either of the previous two commands

Saturday: Found a bug where there were lots of weighted spectra with separate skewnorm params that all had the same MSD. I realized the ones with identical MSD’s were all centered at 0. Temporary fix is to just no center any skews at zero. I’ve posted it here because I’m going to end up altering this file and it might be useful again in the future for troubleshooting.

file_name	avg_MSD	mean	sigma	skew	kurtosis	mean_param	std_param	skewness_param
0.csv	0.075545028	-0.270415116	0.056050971	-0.850965013	0.705345255	-0.2	0.09	-5
1.csv	0.074995124	-0.26904666	0.057728318	-0.734386597	0.57954318	-0.2	0.09	-3.5
2.csv	0.073058968	-0.264228468	0.063045253	-0.453825564	0.305050273	-0.2	0.09	-2
3.csv	0.060200486	-0.232114234	0.084075418	-0.023919331	0.006028161	-0.2	0.09	-0.5
4.csv	0.027788172	-0.149222937	0.074308074	0.136948767	0.061744315	-0.2	0.09	1
5.csv	0.02143058	-0.13332645	0.0604536	0.575781421	0.418982939	-0.2	0.09	2.5
6.csv	0.020233778	-0.130334445	0.056979913	0.784426755	0.632784755	-0.2	0.09	4
7.csv	0.019839474	-0.129348685	0.055752952	0.873414847	0.73026458	-0.2	0.09	5.5
8.csv	0.008099909	-0.070415116	0.056050971	-0.850965013	0.705345255	0	0.09	-5
9.csv	0.008099909	-0.06904666	0.057728318	-0.734386597	0.57954318	0	0.09	-3.5
10.csv	0.008099909	-0.064228468	0.063045253	-0.453825564	0.305050273	0	0.09	-2
11.csv	0.008099909	-0.032114234	0.084075418	-0.023919331	0.006028161	0	0.09	-0.5
12.csv	0.008099909	0.050777063	0.074308074	0.136948767	0.061744315	0	0.09	1
13.csv	0.008099909	0.06667355	0.0604536	0.575781421	0.418982939	0	0.09	2.5
14.csv	0.008099909	0.069665555	0.056979913	0.784426755	0.632784755	0	0.09	4
15.csv	0.008099909	0.070651315	0.055752952	0.873414847	0.73026458	0	0.09	5.5
16.csv	0.019933953	0.129584884	0.056050971	-0.850965013	0.705345255	0.2	0.09	-5
17.csv	0.020481336	0.13095334	0.057728318	-0.734386597	0.57954318	0.2	0.09	-3.5
18.csv	0.022408613	0.135771532	0.063045253	-0.453825564	0.305050273	0.2	0.09	-2
19.csv	0.03520122	0.167885766	0.084075418	-0.023919331	0.006028161	0.2	0.09	-0.5
20.csv	0.067654418	0.250777063	0.074308074	0.136948767	0.061744315	0.2	0.09	1
21.csv	0.074041506	0.26667355	0.0604536	0.575781421	0.418982939	0.2	0.09	2.5
22.csv	0.075243822	0.269665555	0.056979913	0.784426755	0.632784755	0.2	0.09	4
23.csv	0.075639942	0.270651315	0.055752952	0.873414847	0.73026458	0.2	0.09	5.5

What I will do next

Add mean=0 centered gaussian to training data
preprocess training data
Keep thinking about NN architecture and build.