I knew when I met you that an adventure was going to happen.
Friday, January 31st was my last day at Heroku. I joined back in June of 2016 and my time there has been a period of intense challenge, growth, failure, triumph, and self-discovery. I genuinely thank all of my previous coworkers for your help and support over the last few years and wish you all the best.
Starting February 17th I’ll be an engineer at Trailhead, helping to build the APIs that power their newly launched mobile app. I started speaking to this team a few weeks ago and the problems they are trying to solve seem really interesting and I’m excited about getting to work.
After leaving a job, I try and think back on what I was able to do and the features I was able to ship. Here are a few highlights.
Author note: I don’t work for Heroku anymore so please avoid asking me questions about these features. Contacting Heroku Support would be your best option.
Postgres Connection Pooling (still beta, sadly)
Heroku Postgres has long had a hard cap of 500 connections which can often become a problem if your application is multi-threaded or runs a lot of dynos. We had offered a solution using a buldpack running pgbouncer but this didn’t work for cases where customers had a large volume of dynos.
To address this I worked on a feature that provided on-server connection pooling using pgbouncer running on the server itself. We had a few mishaps along the way but ultimately we were able to make it work with decent success. Unfortunately, decisions were made above my paygrade and this feature is stuck in perpetual beta.
Additionally, I was able to merge a few updates to the pgbouncer buildpack. One of the most difficult being removing stunnel in favor of native TLS in pgbouncer. This refactor took a lot more testing and research than I initially expected but shipping was able to remove a piece of clunky infrastructure from our buildpack that causes some customers endless headaches.
HIPAA Compliant Heroku Postgres
Let me first start by saying this was a full-team effort. I wasn’t a lead engineer and I certainly didn’t carry this project on my back. However, launching HIPAA compliant Heroku Postgres (now called Shield Postgres) was a huge task that opened up loads of possibilities for the organization.
While I don’t want to get in all the details, I can assure you that following the law when it comes to HIPAA is more difficult than you realize and most of them boil down to communication rather than engineering.
Seriously though, if you have personal health information in your app you should really consider Heroku Shield. You’ll thank yourself when the auditor comes knocking.
Heroku Postgres via PrivateLink
Since AWS launched PrivateLink awhile back, Heroku had always wanted to ship a feature that gives users the ability to build a secure link between their Heroku Postgres database and their own AWS account.
Previously, options for integrating your Private Heroku Postgres databases into your AWS accounts had previously been a tricky proposition. You’d have to open a support ticket to enable trusted IPs and it would require a stable IP in AWS for the connection. Alternatively, you would have to build a complex ETL pipeline, hack some kind of data replication, or build APIs solely to be consumed by services inside your own AWS account.
Thanks to Heroku Postgres via PrivateLink you can provide Heroku with your AWS account ID and we would make a service endpoint available to you. Pull it into your AWS account and it appears as just another network interface. Now you can integrate all your processes in your AWS account with Heroku Postgres.
Connecting to Heroku Postgres via Mutual TLS
This was one of the more interesting projects I was able to work on and bring to launch and it taught me a lot about the interworkings of TLS and AWS Private CA.
Mutual TLS is the process of opening a connection where both the client and the server offer certifications. The server verifies the public/private keypair and the client does the same. If there is a problem, the connection is denied.
This gives high-security (HIPAA/PCI/etc.) customers the ability to ensure their connections are safe and aren’t vulnerable to password guessing or brute force attacks. To open a Mutual TLS connection, the end-user needs their IP to be opened up server-side, the database’s username/password combo, and a set of verified client certs which are generated by Heroku.
This is the end of the dry technical talk. This next part is from the heart and also really weird because I’m about to share a lot of not-so-flattering photos because I hate seeing my own face.
Content warning: Mental Health stuff is mentioned towards the end.
Save me, San Francisco
“You’re braver than you believe and stronger and smarter than you think.”
— Winnie the Pooh
Even prior to the tech boom there was something that piqued my interest in San Francisco. I partly blame the 1968 film The Love Bug for this. If you haven’t seen it, The Love Bug is a film about a washed-up racecar driver named Jim Douglas who lives in a dilapidated firehouse with a friend (as an adult I realize they were probably squatting) who goes on to become a racing legend with the help of a magical car named Herbie.
It’s a silly movie filled with low-quality special effects, bad writing, worse jokes, racist stereotypes about the city’s Chinese population, and some casual sexism for good measure (wasn’t old Disney just magical?). The actor who plays Mr. Banks in Mary Poppins plays the main antagonist, so that’s a plus. By no means does it stand the test of time and should probably stay forgotten like most of Disney’s old non-animated films but it still has a special place in my heart.
However, something about the shots of the city, especially the various neighborhoods and the Golden Gate Bridge, make me take a look at where I lived and realized there was something different and interesting in other parts of the country. It felt like a city of misfits and I was certainly a misfit.
My first real visit to San Francisco was in March of 2016 for a job interview. I was turned down for that particular job (for complex reasons I won’t talk about but in-short it wasn’t because of my performance at the interview) but ended up being hired by Heroku a few weeks later.
The interview was at 10 am, over by 11:30 am, and my flight didn’t depart until 6 am the next day. The thought of taking public transit at that point filled me with anxiety (a fear I would later conquer) so I went on foot. I ended up walking 16 miles that day and kept going until my legs stopped working. My whole body hurt, I was dehydrated and hungry, and I could have just taken a nap right on the sidewalk.
Still, it was amazing. San Francisco was everything I thought it would be and more. Could this be home? Probably not.
Fast forward a few years to 2019. I’m going to skip over a lot of details here because they are personal. Lots happened behind the curtain but just know that my desire to leave Pittsburgh was strong. I finally decided that I needed to either put up or shut up. Moving was terrifying to me and I feared that I would be making a horrible mistake. Endless “what if”s flew through my brain but I realized that if I didn’t do this I would regret it forever.
Two things forced me to make a decision:
- My lease was up for renewal soon.
- A work offsite in California was approaching.
The offsite was set to be in Hollywood, CA and I had promised myself I’d commit to a decision by the end of the trip. Getting out of Pittsburgh always gave me perspective and the realized that the change of scenery in Southern California may help clear my head.
Finally, after a totally unrelated discussion with some coworkers, something clicked in me and I made my decision. I e-mailed my apartment manager to let her know I wouldn’t be renewing my lease. This was happening! I was thrilled, I was terrified, it felt like I was dreaming and I didn’t know what to do. So I did the least logical thing possible.
I went to Disneyland.
Dang, that was a good trip. I got the front seat in Pirates of the Caribbean twice! TWICE! Pretty sure I ate so much dole whip that it made me sick and I even may have run into a celebrity in line who I’m totally not going to name. Dang, some actors are really good at the incognito look but I guess I have a talent.
Hint: This person has been in a few Marvel films and no it isn’t RDJ.
While laying in my hotel room at the park, I booked a last-minute ticket to San Francisco and a cheap (by SF standards) hotel room near Heroku’s office at the end of Market Street. I ended up staying near Union Square in a stale smelling motel with no natural light and a shower with no noticeable hot water. Sure felt like I was making a terrible mistake but at least this time I knew how MUNI worked and had a Clipper card.
Amusingly enough, almost none of my coworkers knew I’d be there.
I was going to work most of the morning, then apartment hunt in the afternoon. The words “wtf are you doing here?” kept running through my head and there were numerous times where I considered heading back to the airport and trying to book a flight back to Pittsburgh.
By the end of the day, I had an apartment. It wasn’t amazing, I wasn’t a huge fan of the neighborhood, it was expensive, but it felt like it could be home. I signed the lease and the great westbound migration began. It was one of the most difficult things I had ever done. That night, as I sat in my shitty motel near Union Square, I shed tears of both fear and joy at the decision I had just made. This city felt so foreign to me and I hardly knew anyone. Having a beer with a friend would have calmed me down but I didn’t have any friends to call.
What have I done?
It was the best decision I ever made.
On May 13th at 9:50 am I arrived in SFO for the first time as a resident, having taken the 6 am “Nerd Express” from Pittsburgh. I was worried I’d sleep through my alarm and miss my flight, plus I’d rather say final goodbyes in the evening instead of at 4 am so I decided to stay at PIT’s airport hotel.
As luck would have it, work was doing an off-site in Berkley that week so I wouldn’t have to sit in an unfurnished apartment until the 21st when all my stuff would arrive. I hung out in a swanky hotel in Berkely talking about work stuff and trying to not have an emotional breakdown from all the new feelings. I’d say I did a 6/10 job at work that week, maybe 6.5 tops.
The off-site ended and I took the bus, then BART, then another bus back to my apartment. Thankfully I had already gotten the keys the day my flight landed so I didn’t have to go through that whole process. The place was totally empty aside from some toilet paper, a toothbrush, a towel, soap, and a few snacks I had bought the day I landed. Thankfully my stuff was scheduled to arrive in 2 days and I had the foresight to Amazon Prime a cheap air mattress and blanket to my building during the off-site.
If the reality of moving here didn’t set in when I was handed the keys to the apartment, it surely set in when I started unpacking. The smell of my old neighborhood in Pittsburgh still clung to a lot of my things, a smell I never noticed before until now. Lots of little dents and dings were visible from the shipping process and I even had to sacrifice a crappy end-table to the dumpster gods.
This was home. The adventure of getting here ended, now the adventure of living here began.
Author note: San Francisco still doesn’t feel like “home” quite yet. My social network is still evolving, I get lost all the time (thank you Google Maps), I mostly know restaurants by their presence on Doordash, and I cry a little bit every time I make a rent payment. This was the life I chose.
CW: The mental health stuff is inbound now so feel free to end here if you don’t want to read about some pretty personal stuff.
In the meantime here’s a picture of me with Spider-man.
You can’t stay in your corner of the Forest waiting for others to come to you. You have to go to them sometimes.
— Winnie the Pooh
(I have a thing for Winnie the Pooh. Wanna fight about it? Even the title of this blog post is a Winnie the Pooh quote. Maybe I have a problem?)
Life. Such a small word that is used so commonly that we forget its true meaning.
I had a life in Pittsburgh, but a combination of poor mental health, inability to better myself, stubbornness, and general neglect had taken something that should have been good and turned it into a pile of garbage.
It was my fault. All of it. I blamed others and forces outside of my control but in the end, I was the one who caused my life in Pittsburgh to fall apart. I’m finished trying to shift the blame or make excuses. Therapy sessions and medication changes certainly helped but I realized I needed to start taking my life into my own hands and stop living for others. Lots of us probably live this way and don’t even realize. I’m embarrassed at how long it took me.
I took a risk and success or failure was my responsibility (also the privilege of being able to move without changing jobs, a supportive manager, family who were willing to help me pack, my mom for listening to me panic on the phone about the moving process, friends who I cannot thank enough, being a single white male with disposable income, and loads and loads of luck). Seriously though, it is always a good thing to reflect on how your successes are shaped by your privilege in life.
The mental illness is still here and loneliness is a problem in a new city. There’s nothing romantic or magical about this kind of move, despite what my social media profiles may have you believe (my Instagram posts are primarily for my own benefit). San Francisco wasn’t a fix for my problems but rather it pulled away a lot of my coping mechanisms and forced me to deal with them head-on.
And for that, I say thank you. You did save me, San Francisco.
