Some Lessons Learned - Part 1

Pranav Sathyanarayanan

Tuesday September 7th 2021

Background

Over the past 9 months our team of engineers have transitioned from intrapreneurship within the corporate behemoth that is Google to entrepreneurship at a small pre-product startup tackling numerous large problems such as:

  1. Bootsrapping a market network of CRE professionals across the country
  2. Streamlining customer relationship management software in an industry with historically low tech-adoption.
  3. Automating significant portions of the investment sale and debt / equity fundraising process.

To name a few. Leaving behind the guide rails of an extremely mature and decades old engineering organization like Google's is no easy feat. It requires an aptitude to learn new technologies and tech stacks, be unafraid to pivot and make changes at lightning speed, and quickly learn from mistakes and remediate issues for our customers.

As we approach nearly 100K LOC, several large vendor integrations, some larger enterprise agreements as well as our 1000th pull-request, we figured this was a good time to share some of our learnings and tips to the broader community.

Lessons Learned

Use strict version specifications for your critical dependencies

Like most startups we leverage hundreds of open source packages to help build and ship our core SaaS platform. As useful as each individual package is they also increase your risk surface to potential outages due to bugs or odd and untested behaviors.

We recommend identifying your most crucial dependencies and strictly specify versions for these dependencies. When using package managers such as npm many will automatically upgrade dependencies with new minor or patch versions (i.e. use of ~ or ^ before version specification in package.json). You should ensure that this is not the case for that select list of core dependencies.

Although this is generally acceptable it can be detrimental for your most critical dependencies when they inadvertently introduce new unexpected behavior or more often the case... bugs. You can and should manually vet and upgrade these dependencies after some kind of QA process. .

To provide a concrete example the Revere engineering team specifically maintains the version of nextjs, react, prisma, nexus, and graphql as these provide the core open source libraries and frameworks we leverage to ship our platform.

Leverage a CDN for User Facing Platforms

Leveraging a CDN (e.g. CloudFlare) will definitely improve your page load times and result in a smoother user experience for your end users. It may even provide insulation from DDoS attacks and other security threats.

Setup Alerting / Monitoring Thresholds

You will inevitably experience some downtime or outage. The sooner you are aware of the problem the sooner you can react to it. Our recommendation would be:

  1. Leverage a tool like Sentry for error monitoring and integrate that with Slack for alerting.
  2. Create a simple Uptime Check with alerting to your Slack channel.
  3. For web services setup monitoring for important key metrics like 95th/99th percentile latency and response code frequency (2xx / 3xx / 4xx / 5xx).

Leverage Load Testing Before Marketing Events

Generally marketing events will lead to traffic spikes on your application. Whether they are ad campaigns, email campaigns, press releases or otherwise, being prepared to handle these large traffic spikes is key to ensuring success in your campaign and a smooth user experience for all your new potential customers.

Utilizing a free tool like k6 will allow you to load test your web application to simulate potentially thousands of concurrent users and ensure your system is up to the task! As an added benefit, depending on your scaling mechanism, this can also potentially warm up any necessary caches or turn up new instances to handle the load prior to real users encountering the site thereby ensuring low load times for your clients.

When using an RDBMS leverage SQL Based Migrations

There are many tools out there to assist in generating migrations for your database (e.g. Alembic, Prisma, Ruby on Rails)

Although we don't necessarily discourage their use (actually... we use Prisma's migration tool) we have found that nothing will beat the simplicity, portability, and functionality of plain old SQL. Eventually you will run into a use case where your migration will require some weird use-case not supported by your tool of choice such as:

  1. Installing or managing extensions in your PgSQL database.
  2. Running a complex transaction to migrate data into your new schema in a backwards compatible way.
  3. Implementing new functions and triggers in your database.
  4. Sometimes even just adjusting your referential actions on existing FKs.

By leveraging SQL based migrations you will never run into those dead ends and can ultimately position your team to seamlessly split a monolithic app and its database into microservices down the road in a framework/language agnostic way.

Pay attention to JS bundle size

A large JS bundle size (based on your dependencies and your build process) will adversely affect your SEO as well your page load times as well as your time-to-interactivity on your web page. Monitoring your build size frequently and auditing new dependencies with bundle size in mind will save the team headaches down the road by being proactive.

We leverage NextJS Script Optimization to ensure non-critical dependencies are not in the critical path of our page load.

Ensure SPF / DKIM Alignment with a valid DMARC Policy when leveraging Email Providers

Needless to say if you are sending out email marketing content from your email domain via tools like Constant Contact, HubSpot, SendInBlue, MailChimp or SendGrid... you don't want your emails to end up in spam.

When setting up these integrations it's important to ensure that you properly configure SPF and DKIM alignment between the provider's email servers and your domain so that the recipients of your email don't view your communication as fraudulent.

You can understand what these mean here but as general advice:

  1. Leverage mail-tester.com to quickly validate your setup (you need a 10/10 score).
  2. Leverage mxtoolbox.com to ensure you are not on a blacklist or SPAM list.

Prioritize setting up Product Analytics

We couldn't recommend Heap and HotJar enough. It's important to build product analytics into your engineering processes early on. Identify key customer journeys and funnels and ensure they are instrumented as part of the design and build-out of the features supporting those journeys. In our case this is a simple as identifying key elements on the page via id attributes and creating events and definitions in Heap around them.

Understanding your users' pain points is critical to improving your UX and ultimately your ability to sell and market your software. Screen recordings and concrete analytics will provide the basis for making informed decisions in your product to continuously improve and iterate in a data-driven fashion.

Utilize Sales Automation tools to consistently generate touchpoints

This isn't engineering focused but for any new startup this is a must. Using tools like MeetAlfred and Apollo.io to consistently reach out to potential clients via e-mail or LinkedIn will generate new hot leads for you to potentially convert to paying customers or even just brand ambassadors.

Very Specific Advice

This section is only relevant if you are building out very specific features similar to Revere's but you may still find it useful!

Utilize a paid Image Optimization / Resizing / Caching service

If your platform allows users to upload shareable image content (e.g. a social media network) you should leverage a paid Image Optimization/Resizing/Caching service such as CloudFlare.

You cannot (without some effort) control the size or quality of the images uploaded by users. If large or high quality assets are rendered as-is publicly they will impact page load times on your site. By proxying the loading of these images through a service like CloudFlare you can control the quality and size of the image and ensure it is cached for subsequent page loads to ensure a smooth experience for your users.

Expect Ad-Blockers and Email Protection Software messing with assumptions

It's hard to provide concrete recommendations here but your best laid plans can get interrupted by ad-blocking extensions on browsers along with some over-zealous email protection software.

An example of this occurring for us was when we encoded recipient information into a base64 string query parameter in our URLs for our homebrewed deal impression tracking system. The idea was that we would understand who viewed our deal pages since we could decode our base64 string which had the email of the user the marketing was sent to. However in practice we noticed that nearly 70% of the data in our CRM was gibberish since either an ad-blocker or email middleware was caeser shifting base64 encoded emails, first names, and last names!

Another example is that your product analytics may get interfered with by ad-blocking software preventing user interactions from getting logged to a system like Heap. This means that you may not have the "complete" picture of a user's interaction with your site.