Tackling Operations as Lucinity Grows Is A Welcome Challenge

I recently joined Lucinity as CTO. Considering my previous experience, it's only natural that I'm drawn to working here. I look forward to tackling the inevitable issues that arise within a quickly growing start-up.

Thursday Night Football  

Before joining Lucinity, I spent the last six years working for Amazon in beautiful Vancouver, BC, and the vibrant city of Austin, Texas. During my time there, I was fortunate enough to work on some exciting problems on an astronomical scale by leading the team responsible for video ads serving all Amazon-owned & operated publishing services (Prime Video, Twitch, IMDBtv, Alexa, etc.). That means, from a high level, during a live broadcast (i.e., on Prime Video or Twitch), ad breaks are defined beforehand with a specific duration.

When the ad break is about to hit, the service responsible for creating the playback streams and inserting the ads seamlessly into it (usually referred to as a stitcher) will call our service to retrieve all the ads. What happens behind the scenes is a whole bunch of ad selection logic in multiple downstream systems, which we've usually managed to cache beforehand.

What's left is a bin-packing problem, where we fill the requested break duration with as many relevant ads as possible to have a steady flow of ads rendered to the customer as if it were a typical tv broadcast. What is then returned in the service response is an XML document (yes, the ad industry still uses XML) containing content delivery network (CDN) links to all the ads, along with all kinds of information such as duration, frame rate, and tracker events (to track which ads have been watched).

One of the biggest challenges was supporting the tens of millions concurrent viewers of NFL's Thursday Night Football to receive video ads for each ad break. Due to the nature of a live event, all the requests to retrieve the ads come in around the same time, which results in a huge burst and high transactions per second (TPS) traffic for a short period. When dealing with TPS in the millions, every little detail starts to matter. So, keeping a high bar on operations, scalability, and performance becomes crucial to your success.

On raising the bar

I like the term operational excellence (OE). Its simplicity captures the mindset that teams should continue to strive for excellence in how they operate their products and services. Although, it does not happen automatically. It's a process that needs to be nurtured and iterated throughout the lifetime of your product.

A team with a high level of OE ensures their products have the proper monitoring in place, such as logs, metrics, and alarms. Dashboards are clear and concise, and runbooks for the on-call are kept up to date. It allows the team to learn and improve its operational mechanisms continuously. Automation is also crucial, where tests (unit, integration, and load tests) are run automatically during the build process or on a schedule. By setting the bar high for operations, the teams automatically need to be autonomous.

With autonomous teams, it is vital to demonstrate a strong sense of ownership, and with that ownership comes accountability. You have the necessary insights into your product's performance, which becomes key to its success and ability to scale.

To Lucinity and beyond!  

At Lucinity, what I'm most looking forward to is tackling precisely this. Going from a large software organization to a start-up means many opportunities to take my learnings and apply them in a fast-paced environment and be open to change.

During my brief time here, I've been amazed at how eager everyone is to improve and solve real problems for our customers continuously. And despite having no background in AML, I am thrilled about the fact that we're making the world a slightly better place - which is something I haven't really experienced professionally before.

Over the next few weeks, I will listen, learn, and understand where we can improve both on the technical side and how we methodically tackle work – by working backward from our customers' every step.

One of the main things I love about working for Lucinity is how customer-focused the company is. Everything is about the customer experience and how to solve our customers' problems effectively. The product is a sight to behold; Incredibly slick UI, fluid, and 100% focused on making the lives of our customers easier.

There are so many new things to learn and understand, and I can't wait to dive deep into the various aspects of our systems. It's still day 1.