---

A Personal Data Architecture for Fun and Profit

I like doing amateur data analysis, so I’ve been capturing my personal data for the last couple of years and exploring it. Using free (as in beer) tools for getting data, processing it and generating reports, I’ve become more productive, healthier, and wealthier.

My Tools

I’ve been using Google Cloud Platform to store my data, process and visualize it. The free tier of services is pretty broad and has suited my needs well.

Looker Studio/Google Data Studio

Looker Studio is a data visualization/business intelligence tool. It’s free to use, and is very powerful. I was previously using Tableau, but it’s too expensive for a license, and the free version requires publicly exposing your data to the internet. For obvious reasons, I wouldn’t want to put my finance data on the public internet. It has connectors to CSV files, Google Cloud Storage, and many popular databases.

Google Cloud Storage

Generally, my pattern is to generate CSV files and send them to google storage. Allegedly I’m billed 1¢ a month for this service, but I’ve yet to have my credit card billed for the services.

Google Sheets

A simple way to store data. Looker has a connector that makes it easy to grab data from a spreadsheet and visualize it.

Google Big Query

A column-store database suitable for analytics. For my purposes, using GBQ is overkill, but I use GBQ professionally at work and wanted to use it for one of my projects. I also enjoy using the public data sets Google has made available for use. At 1TB of free data processing, I find I only use a fraction of the free tier for my needs.

The Data Sets

Finances

What am I spending my money on? How are my investments performing? What percentage of my income do I invest? Having this information encourages me to invest more of my money, check if I’m spending excessively and meeting my financial goals.

Plaid

I use Plaid to get my spending, balances and investments. Its development tier is free to use, and suits my use case well. Once I’ve downloaded my transactions for the day, I write them to a CSV and use an awk script to categorize them.

Weight and Diet

Tracking how many calories I eat per day and how it impacts my weight. I currently collect the data manually in a google spreadsheet. I’d like to find ways to automate this, but I’d still have to record what I eat at some point.

Time Tracking

What do I spend my time on? How much time do I work vs spend on social media? How much time do I spend in meetings? I find having this data set helps me cut down on social media use – at one point I was spending 10 hours a week on reddit alone. This gives me more time to do productive things, like write this blog post. It also gives me a realistic sense of how much time I have and prioritize the goals that are most important to me.

I use the application Time Sink to record how I spend my time on my Mac. For $9.99 it’s quite a good app, and has an option to export reports as a CSV periodically. I generate a CSV every day and upload it to Google Cloud Storage.

For my iOS devices, I’ve been manually transcribing the information from the Screen Time interface. I’d love to find a way to automate it, but I haven’t found a good way to capture the data and send it to Cloud Storage.

I’d like to track the time I spend offline as well. For example, I track the time I spend reading books on my devices, but don’t track the time I spend reading physical books. I’ve thought of using a timer and collecting that information manually, but don’t know if I’d remember to do it.

Future Steps

Here’s a few ideas on new data projects I could take on:

Mood

I’d like to set up mood tracking for my mental health, and correlate it to how I spend my time. Currently I use the app Daylio to track my mood - it allows for manual export of it’s data to CSV, so its ideal for data analysis.

OmniFocus

I used the app OmniFocus to track the tasks I need to do each day. It archives the completed tasks, and allows for easy export. I could try to see how much I actually accomplish each day, what I realistically don’t have time for, and how my screen time tracks with accomplishments.

---

<< Previous Next >>