We love the Serverless Framework at Formidable. We get to focus on application development while it takes care of details like packaging and deployment—all with a rich plugin ecosystem and a great technical community!
However, an oft-recurring pain point for Serverless projects is that packaging and deploying can be really, really slow.
We're thrilled to introduce the serverless-jetpack plugin, a drop-in replacement for built-in serverless
CLI packaging. For many large, real-world Serverless JavaScript applications, faster packaging may be only a few lines of configuration away!
What slows down Serverless packaging?
A few months ago, a co-worker noticed that Serverless Framework deploys for a client project were taking more than 10 minutes in just the packaging stage. This made the developers miserable, and even worse, significantly hindered our production deployment speed.
We dug in to how the serverless
CLI packages files and discovered that a typical / default packaging arrangement leads serverless
to:
- Read nearly all of
node_modules
(and other sources) from disk into a preliminary file list. - Then exclude files from the list detected as
devDependencies
. (And, perform additional disk I/O innode_modules
to infer what aredevDependencies
to exclude.)
This approach hit us hard, as our client project had a git monorepo with many individually packaged Lambda functions, meaning lots of large node_modules
directories driven primarily by devDependencies
.
A simple, faster idea
So boiling the problem down: the vast bulk of files in node_modules
read during packaging end up being excluded later because they're devDependencies
. Then we thought, what if we didn't have to read all of those devDependencies
? What if we somehow had the serverless
CLI read just the production ones?
Well, we would need a tool to quickly get us production dependencies... But, as it turns out we have not one, but two! Modern npm
and yarn
, the ubiquitous package managers your project already uses, install production dependencies very quickly when used with a lockfile.
We postulated a theory: the cost of an additional, temporary yarn|npm
production install would be far less than the current time the serverless
CLI uses in excluding extraneous node_modules
files post-read.
How it works
With this hunch in hand, we tried a simple packaging experiment:
- Do a fresh production
yarn|npm
install into a temporary directory. - Do normal exclude/includes on project sources except
node_modules
, instead matching against the production one in our temporary directory.
The results were amazing. Across various packaging [scenarios][jp_repo_scenarios], we were able to see consistent, impressive packaging speedups. And the speedups increased as development dependencies increased.
Some timings for relative comparison from our project benchmark:
Scenario | Mode | Type | Time | vs Base |
---|---|---|---|---|
simple | yarn | jetpack | 5687 | -42.16 % |
simple | yarn | baseline | 9832 | |
simple | npm | jetpack | 6163 | -31.37 % |
simple | npm | baseline | 8980 | |
individually | yarn | jetpack | 8259 | -50.44 % |
individually | yarn | baseline | 16665 | |
individually | npm | jetpack | 9787 | -50.10 % |
individually | npm | baseline | 19613 | |
huge | yarn | jetpack | 13473 | -71.85 % |
huge | yarn | baseline | 47864 | |
huge | npm | jetpack | 9451 | -71.45 % |
huge | npm | baseline | 33105 |
(The Mode column has clickable links to scenario projects on GitHub. For Type, baseline
is built-in Serverless Framework packaging, jetpack
is our new plugin. Time is in milliseconds.)
Try it out!
... and thus the serverless-jetpack plugin was born!
Installation should be a breeze for most projects. Jetpack works seamlessly with built-in Serverless packaging configuration, including service and individual function-level configurations with exclude
, include
, etc.
yarn
and npm
users should add this to serverless.yml
:
plugins: - serverless-jetpack
and npm
users will also need:
custom: serverless-jetpack: mode: npm # (default `yarn`)
And that's mostly it!
We take correctness seriously at Formidable. In addition to the usual tests and style checks, we validate that Jetpack's output in each scenario identically matches that of built-in Serverless packaging.
As a few parting thoughts, you should be using yarn
with a yarn.lock
file or npm@5.7.0+
with a package-lock.json
to get a full speedup. And, there are a few esoteric complexities and additional configuration options that you may want to review before integration.
All in all, if Serverless Framework packaging/deployment speed is cramping your style, hook up a jetpack and see if your deploys take off! 🚀