Jeremy Thaller

He/Him

Week of May 3rd

What I did

  • Tried to work on the selenium-based automated NY Times crossword retriever
  • Created this Github Pages website
  • Started and finished the project Facebook friend identifier
  • Continued work on the Code Academy course: Build Chatbots with Python

What I learned

  • JSON files are super nice to deal with in Python. Man, I worked way harder than I had to with the Facebook message counter project. JSON files are literally python dictionaries. If you run:
with open(filename.json, "r") as read_file:
    data = json.load(read_file)

data is of type dictionary. It’s wonderful.

  • With this project, I created a file called preprocessing.py, where I intended to clean up the chat data. I soon realized, however, that much of the standard preprocessing would remove the idiosyncrasies that distinguish each person’s chat/writing style. I ended up skipping the preprocessing script, and the NN still made quite good predictions.
    • Stemming and lemification may be useful/necessary for bag-of-words language and topic modelling, but for message sender identification (classifier), only basic preprocessing seems necessary – i.e. removing links and possibly emojis.

What I will do next

  • The Facebook friend identifier program worked quite well at predicting messages sent from myself or Mike, however it often struggled to identify Rohan. I have a few theories as to why.
    1. Perhaps there is actually too much training data for Rohan (opposite of data sparsity). The Rohan training data dates all the way back to 2015. It’s possible his writing/language style has change dramatically since then. This is important because all my testing messages are recent. Furthermore, Rohan and I talked much more in 2015 than we do now. Thus, his training model could be skewed because of a higher density of outdated chat data. Note: my training data would not be skewed because I also get to add training data from chats with my other friends – i.e. Rohan only gets his training data from the Rohan-Jeremy chat JSON; I get data from that file and the Mike-Jeremy and Thomas-Jeremy files. Solution: Remove rohan_json3 from the training data.
    2. It’s possible that links and emoji’s are screwing up Rohan’s data. We changed the thumbs up button (you know, the one that grows when you hold it down longer) to be a rocket. As far as I can tell, this becomes a very strange value when parsed. After all, each rocket must contain the data to represent its unique size. Solution: *Change the preprocessing file to only remove emojis, or standardized all the rocket ones to the same string.
  • The next logical step in this project would be to provide a group chat to the NN and have it guess who each person in the chat is. For example, replace each name label with something like “mystery person one’’ etc. and have the trained NN predict the identify of each mystery person. All the hard parts for this are already written.

2022

Back to Top ↑

2021

Color Theory: Triadic

less than 1 minute read

I’ve recently added somemissing .SCSS files tomy websites github repository, which will now let write my own theme. I’ve taken inspiration from flower arrang...

Week of August 9th

less than 1 minute read

What I did: Monday: Kaggle work with giant spotify dataset Tuesday: Added use of the Spotify API, spotipy to the notebook. Weds - Sat: Added recommend...

Week of Aug 2nd

less than 1 minute read

I created a really nice template for ML projects that I hope BNL will continue to use for a long time.

Week of June 14th

1 minute read

What I did Monday: Fixed MSD Values of particle averaged structure. I was double counting some bonds before. Tuesday: Neural Network now predicts 5 feat...

Week of June 6th

1 minute read

What I did Sunday: Trained the neural network on the skew-norm dataset and spent some time playing around with the architecture and hyperparameters. Mon...

Week of May 17th

1 minute read

What I did Monday: Energy mesh interpolation codes Tuesday - Thursday: Felt dead from the seconds shot of the Moderna covid vaccine. Didn’t do much. I’v...

Week of May 10th

2 minute read

What I did Monday: Finished script to generate thousands of real-disordered structures. Tuesday: Wrote a script to loop through the results from the sim...

Week of May 2nd

1 minute read

What I did Monday: NN data-loading Tuesday: NN pre-processing Wednesday: NN pause. I need to test that my gaussian averaging method is valid enough to...

Week of April 25th

4 minute read

What I did Sunday: Thesis chapter \(\LaTeX\) writing Monday: Thesis chapter \(\LaTeX\) writing Tuesday: More writing, but I also redid some figures co...

Week of April 4th

1 minute read

Website building, mathematica rendering, and lots of seaborn manipulation

Week of March 29th

2 minute read

What I did Monday: Got FEFF to work. Read paper: “Probing Atomic Distributions in Mono- and Bimetallic Nanoparticles by Supervised Machine Learning” Nano ...

Week of March 22nd

1 minute read

What I did Monday: Train ride back to Boston. Wrote python script to generate .inp files with different radially dispalced atoms Tuesday: Cleaned up the...

Week of March 15th

less than 1 minute read

What I did Monday: Worked on jupyter notebook. See plotly plot from last week Tuesday: Feff9 on linux debugging all day conference call. Wednesday: Pr...

Week of March 8th

2 minute read

What I did Sunday: XAFS Thesis writing Monday: Traveling Tuesday: FEFF Codes and disorder in XANES paper reading Wednesday: 2010 Phys Review B, “Eff...

Week of March 1st

1 minute read

This blog will now partly shift from what I learned as a data scientist to what I’ve learned as a scientist, to keep track of my thesis progress. Of course s...

Paygap

less than 1 minute read

<!– — title: “On the Importance of Data Literacy in Decision Making” date: ‘2021-02-10’ categories: miscellaneous tags: statistics

Week of January 10th

1 minute read

I’m getting close to my final exams so most of my time will go towards studying. Accordingly, expect a hiatus from blog postings.

Back to Top ↑

2020

Week of December 27th

3 minute read

Bayesian analysis of Amazon reviews in R and the most important functions in Pandas

Week of December 13th

1 minute read

GRU’s, strings, and batch normalization. I didn’t post anything last week because I was busy with grad school work and traveling – I flew back home to New Yo...

Entropy, Information, and Guess Who

5 minute read

Decision trees are one of the first data structures you learn in computer science (graphically, in practice you might learn them as a linked list), but you m...

Weeks of October 6th

5 minute read

Note, last week is missing. Most of what I did is included in the prior week, and the rest is included here. It wasn’t enough to merit its own week, because ...

Weeks of August 30th and September 6th

1 minute read

I’ve decided to combine my weekly’s for these two weeks because I’ve had to spend a lot of time traveling (Germany->Strasbourg->Paris->Montpellier) ...

The Dance of Forward Pass and Backprop

5 minute read

In order to implement a neural network, a solid understanding of Linear Algebra is needed for the forward pass, whereas vector calculus is needed for backpro...

WSL2: VM’s in Windows are now Obsolete

1 minute read

In my previous miscellaneous post, I wrote about why I gave up on WSL1 and switched to running ubuntu in a virtual machine. After looking into WSL2 more, I c...

To VM or Dual Boot

2 minute read

Update: Using a virtual machine has not been as smooth as I expected. I’ve ended up with some weird problems after a few days. For example: sometimes my trac...

First Post: Format of this blog

less than 1 minute read

This is my first post, and I would like to explain my plan for this blog. During my summer work with my undergraduate thesis advisor, I would have a weekly m...

Back to Top ↑