Prefect Project Liftoff

Announcement
Feb 9

Click here to read the original post by Prefect CEO, Jeremiah Lowin: https://medium.com/the-prefect-blog/liftoff-fe5a117a94ff

Prefect’s growth in 2020 was extraordinary; it’s hard to believe that we introduced Prefect Cloud and Prefect Server less than a year ago. Our mission to eliminate negative engineering is resonating as loudly as ever, as demonstrated by the enormous expansion of the Prefect community.

Over the last year, Slack membership grew from 400 to 4,000 users, all collaborating with our team to create a new standard for dataflow automation. This momentum has allowed us to chart an intrepid course for 2021. Working with hundreds of partners in our open-source, commercial, and advisory networks, we have identified four key pillars that will enable us to continue supporting Prefect’s hyper-adoption within the data world: expansion, product, partnerships, and community.

Image for post

The scope of this effort is quite large, and it will result in a dramatic transformation and advancement of our product, business, and broad utility. It is the culmination of months of patient research and development, and as the capstone of such diligence we’ve given it a fitting name: Liftoff.

Today, we’d like to preview some of our major Liftoff announcements. In the coming weeks, we’ll provide further detail on each of these activities as we build toward the next stage of our plan. By the end of the year, Prefect will have delivered a new, modern solution for eliminating negative engineering.

Image for post
Liftoff: Expansion

Prefect has won accolades for simultaneously developing a thriving open-source community and also a successful commercial business. Our approach to open-source product strategy is fairly unique, both in terms of positive outcomes and capital efficiency. It is becoming increasingly clear that the 2015-era model of funneling investor dollars into open-source communities with ambiguous plans for future monetization only leads to deadweight loss across the ecosystem. The incentives of OSS commercialization based principally on hosting and services are just too misaligned. Therefore, we have been invited to partner with large companies and top-tier VCs to share our successful playbook and establish a new and more healthy pattern for open-source business models. Our expansion strategy is designed to ensure the success of that endeavor.

Announcing our $11.5M Series A

Prefect has always pursued a nontraditional funding strategy, which has delighted some of our customers who otherwise balk at VC-backed open-source fragility. Our approach of preferring strategic partners over capital partners continues with the announcement of Prefect’s $11.5 million Series A financing.

The round was led by our long-term partner Patrick O’Shaughnessy through a new investment firm, Positive Sum. In addition to benefitting from Positive Sum’s institutional resources, we have been delighted to work with the Positive Sum team — all of whom have been involved with Prefect for years — to establish new cooperative benchmarks for partnership and success.

The round was also joined by one of our earliest investors, Sam Teller of Valor Equity Partners (and formerly, Tesla / SpaceX), and also Gavin Baker of Atreides. We are also incredibly grateful to have the support of a new group of strategic investors, comprised exclusively of operators of some of the most successful data-driven businesses in the world: Nat Friedman, CEO of GitHub; Anthony Goldbloom, CEO of Kaggle; Michael and Daniel Broukhim, Co-Founders and Co-CEOs of FabFitFun; Daniel Gross, Founder of Pioneer; Zach Weinberg, Nat Turner, and Gil Shklarski, CEO, President and CTO of Flatiron Health; Matt Tucker, COO of Olo; Justin Gage, Growth at Retool; and others; and also all of our existing investors.

We have been collaborating with these partners for a long time, and the foundational advice they shared in 2020 will continue to compound in 2021 and beyond. We are delighted to work closely with all of them.

The Prefect Advisory Board

Prefect’s Advisory Board consists of select executives and influential leaders who share business insights and feedback from their various backgrounds and experiences. We firmly believe that diversity of thought is the catalyst to excellence and we work with our Advisory Board to shape our mission, strategy, products and services. Critically, we also rely on their collective expertise when we need to “level-up” on incredibly compressed timeframes, as exemplified by the strategic guidance we received in March 2020.

The size of our Advisory Board may be surprising to anyone unfamiliar with the advisory model pioneered by companies like Ariba and DocuSign, but it is one that we have seen work firsthand. Our Advisory Board is a key element of our core strategy — our “not-so-secret weapon.” As part of Liftoff, we will be announcing a major expansion of new advisors who bring unique perspective and enthusiasm to our company.

Growing the Prefect Team

Prefect is nothing if not for its people, both in our community and on our team. The primary use of our Series A proceeds will be to invest in the health of Prefect’s team, expanding wherever necessity and bandwidth determine. We are currently seeking enthusiastic new colleagues for a variety of engineering roles across open-source, backend, infrastructure, and solutions, and will be posting more roles soon. Our open positions are always posted on our website and we have referral incentives for anyone in our community that introduces us to someone extraordinary!

In 2020, we invested heavily in our internal policies and operations. The result is a company that operates with the confidence and productivity of a much larger organization, without compromising the rapid decision-making and execution of a startup. Our culture is candid but positive; we want to empower our colleagues to do the best work of their careers. We have always invested in our team, offering multiple healthcare plans, a matched 401(k), uncapped vacation, a $1,000 annual professional development budget, management and leadership training, home office stipends, subsidized high-speed internet access, and more.

Prefect Internship Program

In 2021, we will be launching Prefect’s first internship program across our engineering and sales organizations. We’re seeking talented individuals who are excited to build and distribute a new generation of data tools. Our interns will always be treated as employees, from the impact opportunities they are given to the compensation and benefits they receive. We look forward to sharing more details about this program soon.

Image for post
Liftoff: Product

For three years, we have worked directly with hundreds of partners to develop an innovative approach to dataflow automation. Prefect’s rapid adoption speaks to the success of our solution, and we are actually starting to see Prefect idioms and sometimes even vocabulary appear in other workflow management tools. But our goal is not only to promote a radically easier and more useful way of working with data pipelines; we have always said we want to build “a new kind of workflow system:” one that is fundamentally modern and native to the data science era. In 2021, we will build on this solid foundation to deliver a completely transformative way of orchestrating data. Our product vision will ensure that we continue deliver real value to our users.

Product Vision: A Modern Workflow API

The “modern data stack” has emerged as a key component in any data-driven company. However, the logistics of this stack haven’t changed in decades, and its layers are fundamentally ISTA: ingest / storage / transform / analyze. Legacy ISTA stacks were use-case specific, and stacks would have to be reconfigured or redeployed any time business requirements changed. What makes the modern stack “modern” is an API-first, universal design that allows companies to deploy a single stack with great confidence that it will scale seamlessly for future unknown use cases and requirements. This idea that great tools allow key business decisions to be deferred is critical to the modern data stack. Users don’t choose Snowflake or Looker because they know exactly how they want their tools to work; they choose them because they know they will adapt perfectly to any future use case.

Prefect has become a core pillar of the modern data stack, and as the stack modernizes, it is imperative for complementary tools like ours to stay one step ahead. For this reason, we are developing an evolution of our software that can be completely configured and controlled via API without sacrificing any of the expressiveness or user customization enabled by our Python tools. Like our hybrid model, executing this seemingly simple idea rests on a number of innovations that we have developed and patented. However, it’s also common sense: where a task should execute, where its data should be stored, how it should handle retries, what relationship it has with other tasks: these are all examples of decisions users may be better equipped to make (or change) after deploying their workflows, not while writing them. More directly, rarely is the same user responsible for both designing a data workflow and orchestrating it.

In 2021, Prefect will achieve our goal of cleanly separating orchestration from execution by building on our fundamentally modern approach to data API design. This change will be additive: we will not sacrifice any of the workflow expressiveness that our current users love. Our vision is that “any code can be Prefect,” whether fully instrumented with our Python SDK or written in another language entirely. Prefect has always met our users where they are, and our job is to ensure their modern data stacks operate as expected.

In addition, Prefect has always had a focus on privacy and security. Our unique Hybrid Model has allowed us to deliver solutions to companies in regulated industries like financial services and healthcare that have historically been unable to consume SaaS data products at all. To further enhance our ability to build secure and compliant products, we have begun the process of SOC 2 certification and expect to achieve SOC 2 Type 2 compliance this year.

Success-Based Pricing & Free Tier

We are excited to announce a new pricing model and free tier for Prefect Cloud. The result of months of conversations with Prefect partners, the new model successfully captures the spirit of our software in its simplicity and fairness. For most self-serve Prefect Cloud users, it will also be considerably cheaper than our current pricing.

Prefect Cloud is moving to a simple, usage-based pricing model that has no monthly fees or minimums. Users only pay for what they use, based on the number of task runs processed each month. But even that isn’t good enough, so we’ve gone a step further and created “success-based pricing:” only tasks that run successfully count toward usage. And because Prefect encourages small helper tasks for convenience, we don’t count tasks that run in less than one second. Lastly, we have automatic volume discounts that cut the per-run price as usage increases — no contract or negotiation necessary.

Most exciting, the first 10,000 successful runs each month on our Starter tier are completely free. To put that in context: 10,000 runs is enough for the median Airflow user to migrate their entire workload to Prefect Cloud, for free. With nearly 40% of Prefect users upgrading from legacy Airflow deployments, this will make it even more attractive to switch to a modern workflow platform.

We also have an option for users that prefer more predictable pricing. Our “committed usage” contracts offer significant discounts to customers that can commit to annual usage in advance. In addition, these contracts may include services like onboarding, feature upgrades, and other partnership benefits. For early access, please contact our sales team.

We are very excited about this new success-based pricing model. We worked hard to develop a plan that was as simple as it was fair, and it’s been warmly received by all early adopters in the last two months. It will be rolling out shortly, and current monthly customers will be grandfathered into the new plan to ensure that their usage does not result in any unexpected charges.

Prefect Runs on Prefect

One of our most popular slogans is “Prefect runs on Prefect.” From the beginning, we’ve believed that in order to ensure the quality of our product, we must be its largest consumer. Therefore, we run many of our internal processes on Prefect, whether our slackbot Marvin, our data warehouse, our CI, our unit tests, or even our billing.

In 2021, we want to ensure that our users run on Prefect, too. As we gain clear evidence for the long-term durability of the system, we will begin delivering entire features not as closed-source engineering efforts, but as open-source Prefect workflows that integrate nicely into our UI. Users will be able to modify the functionality of their Prefect deployments as easily as registering new versions of these flows. One of the first features we’ll deliver in this manner is something many Prefect users will be excited about: a flow that automatically monitors and registers new flows in a given location. We’re very excited to see what data applications the community builds.

Prefect 1.0

As we make progress against our product vision, we have begun mapping the road to Prefect 1.0. Prefect has been deployed in production settings since before it was open-sourced, so this milestone is not about acquiring user confidence. Nor is it about stability: with rare exceptions, our APIs still support almost two years of historical Prefect versions. Instead, reaching 1.0 is important to us because it represents a major achievement against the product vision we wrote three years ago. When we started, the idea of an API-first, infrastructure-agnostic, cloud-native, completely scalable approach to orchestrating and observing data seemed almost impossible. But thanks to the Prefect’s team dedication and steady innovation, it is within reach. We are excited to share this roadmap soon.

Image for post
Liftoff: Partnerships

For the first three years of Prefect’s life, we declined all formal partnership offers. Too often, partnerships can lead young companies to pursue suboptimal outcomes in the name of joint success. Instead, we committed to working with as many diverse interests as possible to mutually advance our product. In the course of developing all those positive relationships, a small number of especially strong connections emerged in a manner that was organic, sustainable, and aligned. These represented opportunities to create what we call “60–60 partnerships,” in which both parties get the best side of the deal. By working with such partners, we can lean on experts across the spectrum to maximize the delivery of value to our users. This allows our lean team to focus on our own edge: building a world-class dataflow automation platform. Therefore, we are excited to announce the first members of the new Prefect Partner Program, a key component of our scaling growth model.

Microsoft

One of the more interesting things about working at Prefect has been seeing our products pique interest from small startups and large corporations alike. We were very excited when Microsoft approached us about a multi-faceted strategic partnership. Working with them across a few dimensions including technology integrations, commercialization, and a unique approach to open-source strategy has been wonderful, and we are delighted to have the support and backing of one of the most powerful and influential technology companies in the world. This will manifest as a series of direct benefits to all Prefect users, whether open-source or commercial.

Slate

Since joining the original Prefect Cloud Lighthouse Partner program, Slate has been one of our most enthusiastic collaborators. Slate is an engineering company that delivers some of the best insight, guidance, and custom solutions we’ve ever seen. As we see growing demand for Prefect across enterprises with diverse technological requirements, we found a natural relationship with Slate to jointly deliver and maintain broad deployments of Prefect software. Thanks not only to their expertise, but also their warm and positive attitude, they have rapidly become our preferred way to ensure our customers’ success. Together, we have developed a best-in-class and in-depth onboarding program as well as infrastructure optimization services that will help our customers maximize the value of Prefect Cloud and their data programs more broadly.

Coiled

Prefect was designed to work with Dask since our very first prototype. As early users, contributors, and now sponsors of the Dask project, we have been especially excited to see its growth in popularity. Our journeys are intertwined: in 2020, Prefect was one of the most popular tools in the Dask community. With such natural overlap, we were not surprised to learn that a number of our customers were also working with Coiled. We have been working closely with Coiled to ensure our joint customers’ success, and as many of us already enjoy close personal and professional relationships with the Coiled team, we’re excited to cement a new commercial and technological partnership.

Image for post
Liftoff: Community

Community has always been a foundational pillar for Prefect: we are committed to ensuring a positive and engaging experience for every member of our community, whether they simply use our software or lead feature design discussions in our Slack. Our community support offering has been called one of the best in the industry, and we maintained our commitment to it even as our Slack community grew from 300 to nearly 4000 people over the last year. One of the benefits of maintaining such close relationships with our community is that we can directly perceive the emergence of important clusters in users and use cases. Wherever we see the opportunity to do so, we will continue to invest in creating a supportive ecosystem for the entire Prefect community.

Prefect for Academia

Prefect’s academic roots are strong, and we have built our internal culture around the same inquisitive exploration that characterizes the world’s greatest academic institutions. We are delighted to see that approach resonate with our users. Over the past year, we have seen many universities adopt Prefect, sometimes to govern internal data processes and other times as a teaching tool. However, we know very well that despite having needs that rival the largest companies, tools like Prefect Cloud are often unattainable for academic institutions due to reasons like cost or unique requirements around user management.

We have partnered with some of the world’s elite universities to create Prefect for Academia, a program that gives qualified institutions access to a tailored version of our enterprise-grade product at a significant discount (and often free). Please contact academia@prefect.io to join the Lighthouse Partner program.

Prefect for Startups

Due to Prefect’s extreme ease of use and minimal initial deployment, we have become the preferred workflow automation solution among startups. A number of VCs and incubators now recommend our product to their portfolio companies — a few have even become customers themselves! As a startup ourselves, we know that just because you’re small doesn’t mean your problems are small, too. Moreover, we love to grow with our customers; the best advances in our product come from the long-term relationships we’ve developed.

Therefore, we’ve created Prefect for Startups, a program that gives eligible companies access to Prefect Cloud at a significant discount. Please contact startups@prefect.io to join the Lighthouse Partner program; applicants should be less than three years old and have raised less than $10m in funding.

Prefect for Nonprofits

One of Prefect’s core values is empowerment — helping our employees do the best work of their careers — and this extends to our growing community. From day one, we promised that we would use our company not only to deliver value to our customers, but as a tool to constructively improve our world and help others when possible. There are many nonprofit organizations doing incredible work, and we are starting to see them adopt Prefect in significant numbers.

In order to better serve these organizations that often do not have the luxury of building large data engineering programs, we are excited to create Prefect for Nonprofits. This program will give qualified organizations access to Prefect Cloud at significant discount (and sometimes free). Please contact nonprofits@prefect.io to join the Lighthouse Partner program.

Prefect Referral Program

More than a third of Prefect users hear about us through a friend, colleague, or someone they trust. This is a real testament to the positive impact of our focus on solving our users’ frustrations. We so appreciate this particular form of engagement that we’re creating a Prefect Referral Program to reward our community ambassadors for doing what they already do — share their enthusiasm for Prefect! We look forward to sharing more details soon, and thank you for being part of our community.

Image for post

With the Liftoff initiative, we are embracing a bold vision to enhance our product, accompanied by a new group of users, advisors, and communities. In the coming weeks, we will be covering each of the Liftoff items in greater detail. We are excited and grateful for the opportunity to work with such a vibrant community, and look forward to delivering further value in turn.