KENT: We're live with JavaScript Air, for the second to last time. That's right, this is the second to last show for indefinitely. And today, we're going to be talking about Yarn. And this is episode 47, meaning that this is our 48th episode, because of course we started at zero. And yeah, we're gonna be talking about the Yarn package manager with some subject matter experts. So I'm really excited about this show. Yarn is a pretty cool tech.

And yeah, so before we get into things, I need to give a shout-out to our sponsors who are super awesome and have supported this show. So first Egghead.io, the show's premier sponsor, has a huge library of bite size web development training videos. Check them out for content on JavaScript, Angular, ReactNote and more. Egghead.io is also the host of two free courses from Dan Abramov. Find them at egghead.io/Redux.

And Frontend Masters is a recorded, expert-led workshop with courses on advanced JavaScript, asynchronous and functional JS as well as lots of other great courses on front end topics. Find them on frontendmasters.com. Special shout-out to them, they did just add a cool thing for their subscribers. They used to have, like, the workshops be separate from the subscription. Now, the workshops are totally free for all subscribers. So there's no fee for the tickets, and you, as a subscriber you also get early access to the raw footage from the workshops as well. So yeah, that's pretty sweet. I actually have two courses that you could go watch right now if you're a subscriber that aren't technically released yet, so that's fun. So check that out.

And TrackJS reports bugs in your JavaScript before customers notice them. And with their telemetry timeline, you'll have the context to actually fix them. Check them out and start tracking JavaScript errors today at TrackJS.com. And I noticed last week Tyler McGinnis had a little bit of a hard time saying telemetry timeline which was hilarious to me because I had a hard time learning how to say that word too. (laughs)

Cool. So we do have the fact that this is a live show and so if you're on Twitter, go to Twitter and if you have any questions, with the #jsAirQuestion, you can ask them and we'll go through those at the end of the show. And then we do have a show that's going to be, like, a half hour after this one, which will be the grand finale with Brendan and I, it's JavaScript in the web platform. So that will be super fun. And as always, follow us on Twitter. I'm going to ignore Facebook and Google+ right now. I don't really look at those a whole lot for this show, but yeah, Twitter will still be a good place to follow JavaScript Air even though the show is technically ending because if I start it up again, that's where I will start tweeting out stuff. So, yeah. Don't unfollow JavaScript Air.

So, yeah, sweet. Let's get into things, introduce people. First, we have Yehuda Katz.

YEHUDA: Hey.

KENT: And then oh, shoot, I didn't even practice your name. Konstatin Raev?

KONSTANTIN: Konstantin Raev.

KENT: All right, awesome, I wasn't way too off. (laughs) Thank you for coming. And then, Sebastian McKenzie.

SEBASTIAN: Hi.

KENT: And Christoph Pojer. I don't know how to say your last name, Christoph. (laughs) Christoph, how do you say your last name?

CHRISTOPH: Oh, it's "Poyer."

KENT: "Poyer," okay. I wasn't, like, way too--

CHRISTOPH: You got it. (laughter)

SEBASTIAN: I pronounce it "Persia." (laughter)

KENT: Cool. Let's get to know you all really quick before we start talking about Yarn. Yehuda, do you want to give us an intro to yourself?

YEHUDA: Sure. I'm Yehuda. I work on a lot of things, including, most recently, Ember and work on the Rust core team. And I got into Yarn after doing both Bundler and Cargo and feeling like there were lessons that we could learn from those ecosystems to make the npm situation better.

KENT: Fantastic. Thank you. I love when people with a lot of really great different perspectives and experience come on to the show. You bring a lot to the show, so thank you. All right, Kostantin?

KONSTANTIN: I work at Facebook on React Native build tools, and dealing with Node modules was a big issue for quite some time. That's the reason why I joined Yarn project. Here I am.

KENT: Awesome, thank you. Sebastian?

SEBASTIAN: I'm a software engineer at Facebook working on the JavaScript team. I previously started a project called Babel. And I started Yarn at Facebook after we ran into a bunch of ironies dealing with npm.

KENT: Sweet, yeah, solutions out of actual problems are the best kinds of solutions so that's awesome. Although you mentioned Babel, and I'm pretty sure that was a solution out of an interest to learn JavaScript, if I'm not mistaken, right?

SEBASTIAN: Yeah, so, this was taking a bit of a different starting point.

KENT: Yeah.

SEBASTIAN: Kind of a different project from the start, I guess.

KENT: Yeah, totally, cool.

KENT: Christoph?

CHRISTOPH: Hey, so, I most recently worked on Jest. I've also been at Facebook for a while. And now, I'm actually moving to London and we're starting a new team here. We're gonna take care of Jest and Yarn and React Native Packager, so that's really exciting. Yarn is just an awesome project, and so far I've only helped getting it out the door, kind of, like, for the first release. But I'm excited to be more involved with the project.

KENT: Awesome, so that's actually a great place to kind of start off, so Yarn is going to get a dedicated team and that team will also be working on Jest, and what else will that team be working on?

CHRISTOPH: Oh, React Native Packager.

KENT: Mmm, cool. Can we talk about that for a sec?

CHRISTOPH: So, basically, we're not just starting a Yarn team or anything. So Yarn is not really a Facebook project, but we're still adopting it internally and we're trying to figure out how to make it work really well for Facebook. But we're also trying to build up this community of external contributors, and then we're trying to figure out how we can work well with the entire JavaScript community to build, like, the best JavaScript package manager.

KENT: Fantastic. Awesome, so that was, like, a little bit of a detour around probably where we should have started, so sorry about that. But let's get an idea of what this Yarn thing even is. So, if people haven't seen the blog post, I'll add a link to that in the show notes. But does anybody want to give us a quick idea of why Yarn was created, and then what it is?

YEHUDA: I wouldn't mind jumping in but I think Sebastian will probably have things to add after that. I can't speak to why it was created, 'cause I was not there. So, I think there are two ways to think about why Yarn exists. One of them is, basically everyone will have a story that-- everyone that was working on it will have a story that approximates-- I was working on actual Node project, and I noticed that npm was taking or my team or myself, some number of hours per day or week, and I figured, "Oh, we could do better than that and maybe there was a solution for that." So I think in terms of why Yarn exists, it's basically just to bring some best practices about how to reduce the cost of package management into the Node community.

In terms of what it does technically, it sort of does the boring stuff that a lot of others' ecosystems have done. Sebastian actually has done some stuff in Yarn that are better, like, I think focusing on security and check sums earlier is not actually so ubiquitous outside of Yarn, but just making sure that when you say, "yarn install" and you hit Enter, you get a predictable output. I guess everyone watching this is from the JavaScript community, but if you're not from the JavaScript community, it can be easy to miss the importance of that, 'cause you probably have it already. But basically saying, "yarn install", hit Enter, there's some manifest that represents the last thing that happened and you get the exact same results every single time you hit Enter, no matter what. And, of course, if you're getting the exact same results, the process of actually figuring out what you need to put on your computer is a lot shorter, right. So I think those are the big-picture things that you get, very, very strong predictability across machines, environments, et cetera. And you get much better performance 'cause you're not spending time figuring out what to install every single time.

KENT: Cool. So Yarn is basically a replacement for the npm client, but not the npm registry, right?

YEHUDA: Yeah, I think that's super important, is that the Yarn client it assumes that everything about the, everything about people who use packages, that includes things like Git packages, npm link, things that, things in the registry, scopes, some, you know, private registries in your company. All those things are how people use packages in Node, and those things should keep working, so Yarn doesn't-- I mean, my company is very small, it's only, you know, six people. We all use Ember and we were able to adopt Yarn more or less the first day it came out because Yarn doesn't change anything about how Node works or how the registries work, it just changes how the things get installed and put on your computer.

KENT: Very cool. Actually, Yehuda, maybe you could have something in addition to speak to on this, but if I understand correctly, many other package managers don't work the way that, or like dependency management doesn't work the way that Node dependency management works. With Node, everything's local, and I could be totally off because I haven't really been in other ecosystems but if I understand correctly, it's more of a global thing for lots of other systems. Can you talk about that a little bit?

YEHUDA: Uh, yeah, I don't agree with that but I understand why people think that. So, sure. (chuckles) So I just gave a talk actually at RubyConf in Portugal about this topic and I separated package managers into local and global package managers, as a firm. I, basically any package manager that works like yarn install, bundle Install, Cargo Build, anything like that, I call a local package manager. They some, a lot of them don't put the actual packages physically into your directory but the decision about which packages to use are always based on a manifest and a lock file that is physically in your directory. The main benefit that other package managers get out of that is that you don't have to copy so many files, which I think Sebastian can talk more about, but is actually a dominant cost in the npm install process. So I think that's not as different. I think that is pretty different across ecosystems is duplication. So, for example, Bundler, which was the first package manager I built, doesn't support duplication at all, which has some benefits, actually. Cargo, which is the second package manager I built, allows duplication just like npm. So I had to sort of figure out how to make the Bundler model make sense in Cargo. And so when I pinged Sebastian originally to get involved in the project, I sent him, he'll probably confirm, a big list of questions that was basically like the torture memo, right? It was basically like, "Here's the list of-- I need you to go through and answer a billion questions about how things work." And he already had gotten on the right page about all this stuff. I was pretty shocked. Basically, Sebastian's good at programming, it turns out. (Kent laughs) But I think, basically what I'm saying is, duplication or non-duplication is a real difference between ecosystems, but Node is not unique in duplicating.

KENT: Cool. Sebastian, did you wanna talk about any of those things Yehuda was mentioning?

SEBASTIAN: Not really, I think he pretty much nailed it. As to why it was created, so I guess... so internally, we've kind of had a lot of, Konstantin can also talk more on our particular set of problems that we had. But particularly, inside of Facebook, we have a lot of engineers who are working on JavaScript projects, and so reproducibility is kind of like is one of several big things we really care about. So actually being able to have consistent builds across development machines like, really matter. If you have a team of, like, 20 people working on a mobile app that's in JavaScript, there might be difference in dependencies can like, introduce, like, really bad bugs or just, like, completely break the environment where they can't even work. And especially when the app is actually pushed to production it's kind of really hard to... You obviously wanna, like, go, the development that you're developing in to be as close to the production mode as possible, and the existing npm client kind of has some weak quirks around how that works. There's also a lot of things like us wanting to check in our Node modules folder and then like, and Node modules being too big for version control, so needing to split that into the toggles that are extracted, really helps version control size.

And I guess it's, yeah, so around performance as well, so we've got, like, a lot of dependencies. As our dependencies grow, we always sit there and want the performance of our package manager to grow exponentially. And just having more, like, fine-grain control over the restrictions that we place our dependencies and where they are and yeah. So yeah, and the npm, the npm client didn't really allow this, so, you know, kind of fix these problems we kind of needed to try new things, especially since our work flow with using Yarn is quite different from using npm, so it's based around this lockfile, which means that you can't sync it like the same thing. This is kind of like a really big deviation from the existing npm work flow, and so just the way that they currently work isn't really compatible. Npm has a Shrinkwrap, but it's not really like a first-class citizen, there's a whole bunch of issues around it. And it's kind of (mumbles) to the second-class citizen platform where it's kind of not so easy to use aside the priority, so yeah. I guess that kind of has a bit of context. Maybe Konstantin can speak to some other particular issues that we faced.

KONSTANTIN: Oh, yeah. For example, as for the number of builds we're making, at Facebook, we have a large monorepo where we have thousands of commits a day and it means that we have tests running ten, hundred thousands of times a day. And having a reliable network is not an option because sometimes things are down, sometimes things are deleted from network. So we definitely needed offline support. And checking in Node modules was an option in the beginning but as the repositories grow, the size of Node modules folder becomes unmanageable. It could be 100,000 files, it could be 30,000, depending on what npm client we are using. And in the end, we tried doing that but it was unmanageable because of the performance implications. When you check in a large number of files, switching branches becomes really, really slow. So we came to a conclusion that we actually should check in the vendor packages, the ones that are downloaded from npm, there is a lot less of them. And we actually tried to use npm, original npm clients with that, but this feature hasn't been supported for a very long time. So we just needed a way to install from the vendor clients like if it was coming from the network. There are some solutions with original npm and you can build a small proxy server in your company. But the problem is that, the problem with proxy server is actually quite an effort. Not many people wanted to own it. And having some (mumbles) storage where we could just check and review them and not worrying about ports being available or daemon running, it's a huge time saver for us.

YEHUDA: Yeah, one thing that's pretty cool about how all these package managers work is that you almost get-- so, what Facebook is doing is interesting around vendoring. And we've been doing some similar stuff in Cargo, to help big companies like Mozilla (mumbles) with Echo, which is basically a monorepo. But one thing that's pretty cool for most people is that if you run "yarn install" and hit Enter, and then go offline and run "yarn install" and hit enter again, that actually conceptually works inherently because of the fact that you-- the lockfile, you can check the Lockfile to see whether you have the local files present. And if you have the local files present, you're done. So, obviously, that doesn't solve everybody's offline problems 100% of the time but it solves a huge amount of flaky connection, offline, et cetera issues, just by doing things in a more consistent way.

CHRISTOPH: I just wanna round this all up. So Sebastian started working on Yarn for the reasons that he mentioned. And then at the same time, we were running (audio cuts out) was bringing up on React Native in our big monorepo. And we were building all these mandates around those issues we were facing, like Shrinkwrap, shrink-- blah, I cannot say that, shrinkwrap, like, it's just using JSON.stringify, which doesn't preserve the order. So it was like a 10,000-line long file and every time we install a single module, it would change 5,000 lines and nobody could review that. So we built, like, a small script that would sort the output in the Shrinkwrap file. We built the script to upload the entire Node modules folder to our internal storage, so that the CI systems can pull them down. We built all these, like, tiny little scripts and we just realized that that doesn't really scale if we just try to keep fixing it, those issues as they come up. And then, like, it all happened here in London, it's kind of nice that Konstantin and Sebastian were both here. And then there were some people involved also in (mumbles) and when Sebastian started talking to a bunch of engineers at (audio cuts out) to Yehuda, and we were talking to the exponent team a lot. And everybody was kind of running into the same issue, and then we kind of just brought everyone together and then started solving the solution. And that is basically what happened. This is, like, how Yarn was created. And it's been working really well for us at Facebook so far.

YEHUDA: Yeah, and Ember Project also had sort of the same equivalent thing, like, basically Ember has a CLI that everybody uses, so when people have problems with npm, they report them as bugs (mumbles) Ember, so we've had to do equivalent kinds of crazy hacks, like we vendor npm as part of Ember to try to keep things consistent. We have, we built our own tool to add checking to make sure that the stuff that was in your Node modules is actually what's in your package at JSON, right, and that we run, every time you run Ember S. So there's all the, I think every, a lot of people who have taken this seriously have done a huge amount of work to work around limitations that were taking a lot of time. And indeed, that's why I got involved in Yarn because when I, I always knew that these problems existed. I was, in fact, interested in doing it on my own at some point but when I pinged Sebastian to ask him about what I thought were the key questions, he had the right answers, so collaboration's great.

KENT: Yeah, that Sebastian. He's got all the right answers. (laughs) That's awesome. And as a user of Yarn for the last little while, those answers were awesome. (laughs)

SEBASTIAN: Use them?

KENT: Yeah. I, actually I'm kind of curious. Christoph, you mentioned that you were writing a bunch of little scripts to kind of work around some of the issues with Shrinkwrap. Is there any reason why Yarn, and this kind of applies more generally, like, a lot of people on, like, in the world, are wondering why Yarn wasn't just, like, a more collaborative thing to iteratively make npm's client better. Is there any reason why these, like, little fixes and workarounds that you made, weren't just contributed back into npm?

SEBASTIAN: So I guess I can answer this. So to start off with, there was a very iterative kind of thing so it actually investigated improving the npm client. The npm client is very, like, it's really unwieldy in terms of its current base. It's kind of get these things that just be like, hacks on hacks. A lot of the wrappers that we would make, like scripts around improving npm, would be, like, very specific to Facebook infrastructure. So like uploading the Node modules folder we would have there instead of infrastructure that kind of like handles file storage. So you know what it's like, making more general and generic fix, like we couldn't have used the existing solutions that we had. And especially incorporating them into an existing client that already has a lot of preconceived, kind of notions about how it's supposed to work, is kind of really tricky. Since npm cared a lot about backwards compatibility, it's kind of, and like the changes that were made were very substantial. And in order to actually, like, convince a large project, you know, to like, adopt all these, like, I think you could make a massive list of things that have actually changed that are different between npm and Yarn. We've actually got a list on the actual Yarn website and how to actually migrate. A lot of these decisions have been deliberately made, due to our experience and other people's experience with the npm client. And the, yeah, so most of the reasons kind of come down to backwards compatibility. The (mumbles) work fundamentally different in basing it around this Lockfile. Since the npm client, like, at its own nondetermininsm (audio cuts out) bug. I can actually go on to the npm website, to the docs, there's actually a docs page that kind of describes, you know, determinism. Don't actually (mumbles) install order. And it kind of explains why this behavior isn't a problem in practice, but I believe that a lot of people will actually run in to this where the non-determinism is an issue. Reproducibility is, like, really important and kind of, not having the exact same file structure especially when it comes to dependencies across, like, office machines, you know, production machines, testing machines is not really ideal.

CHRISTOPH: And I think that the pain that people feel, like, that people can sympathize with, (static) morning, and you go to work and, like, the first thing you do is you delete your Node modules folder and you do a clean install and then you, you know, like, you hope everything is still working. And that's something that's really common, right? Like something's going wrong with your dependencies, you just wipe away the whole thing and then you reinstall and that often fixes problems, you know?

KENT: Yeah, I have an alias that actually does that for me. It's called flesh-npm. (laughs)

YEHUDA: Yeah, the whole Ember community has one called nom and it's this, npm cache clear (mumbles) ember f node modules and npm install. We also have nombom, which is the same thing for Bower. But I think what Sebastian just said is really the key set of things. I think the npm team is not massive, they're all really busy. They do things like support Node 0.8 which are all, I think, quite important things to do, I think the main npm client is way more concerned with backwards compatibility. And that's not a bad thing. I think it's really, like, there are people out in the world who have very severe compatibility concerns. And in fact the fact that not everyone in the world switched to Yarn on day one illustrates that, right. And so, basically, the npm client exists to be very compatible, as a person who works on Ember, and we have extremely strict compatibility requirements, I understand the tradeoff. Or, like, I work on JavaScript also on TC39. There are, the tradeoffs of compatibility are very strict. And in the case of npm there were some foundational things that I think, in theory, could have been changed compatibly, but not without pretty serious compatibility voodoo. And asking a team to do some pretty serious compatibility voodoo before you've proved that the thing you're doing even matters, is a pretty heavy ask.

So, like, I think several of us have actually talked to many people on the npm team, brought up these issues. I talked to, for example, Forrest, about a year ago, about exactly all these things, and more or less what they say, and I think this is just reasonable, is, "Look, we're very busy right now. I don't think we can prioritize what they fix in Shrinkwrap," which is what they would say. I said, "That seems fine. You guys are indeed very busy, and you have a bunch of, like, whatever, like, the things that your customers need don't necessarily align with what we need." That doesn't necessarily mean that, like, that doesn't really say a lot about what the long-term story here is, right? So I think the goal of Yarn is basically to illustrate to prove, that the set of things that many people thought were important that the npm team couldn't prioritize or didn't want to spend the-- like, to figure out the compatibility voodoo for, prove that those things actually matter and work. And who knows, maybe they, at some time in the future, the npm team will have been persuaded by the proof and we will all work together. It's hard to say. That happened with me in Merb and Rails, right, back in 2007, 2008 I worked on a thing that was quite similar to Rails and the entire purpose of it was to prove that modularity in Rails was worth it. And eventually, what happened was DHH said, "Oh, you've proved enough of that, you have enough users and you've shown that it doesn't actually have too much conflict with the goals we all share. Let's just work together." That can happen but it's just a lot easier to prove your point without having to also deal with compatibility at the same time.

KENT: That makes a lot of sense, yeah. I think that's great.

CHRISTOPH: I wanna just point out again that we didn't build this in isolation. Sebastian was really active in talking to people and like, as Yehuda said, that they were in touch the whole time and we were in touch with Sam Saccone from Google, Paul Irish as well, and then the entire Exponent team. So we were collaborating with them and figuring out what kind of issues they are also running into and then how we can solve all those. And also, I just want to say again that Yarn is not just, like, a Facebook project. We built this in collaboration and we wanna really focus on building up a community and figure out how we can support a core (mumbles) from outside of those companies that have invested in Yarn so far. It's really exciting. So we have more than 100 contributors right now. And that's pretty crazy. It's only been three weeks or so since we open sourced this.

YEHUDA: Yeah. I just wanted to say on that topic that, I think the Ember community, like, one of the reasons I got involved was I didn't want Yarn to be another thing that divides the community. I think the Ember community and the React community basically have exactly the same requirements here. And we're, like, the Ember team is pretty much racing right now to get Yarn built to be the default thing that Ember users use. And I think the Ember community is a pretty enthusiastic and active community and I suspect that there will be a lot of contributions coming from the Ember community, Ember CLI team, in the near future. I think, one of the things that I really I give Sebastian a ton of credit for was, when I first talked to him, obviously there was some technical considerations, but there was a bunch of just project structure considerations, like if Ember wants to contribute to Yarn, we need to make sure that there's an actual way for us to contribute that does not involve us having lunch at Facebook, right? And that, I think, Sebastian was extremely, extremely receptive to all those things to the point where, like, Chris and Sebastian and I are now, after Yarn came out, it's easy for us to say that it's a community project with a community process. And I think people who worked inside of Facebook early on were, like, I guess a lot of the artifacts are still in the repo comments that I made about it, but I think, I give people who started working on it internally a lot of credit to do a lot of, perhaps somewhat uncomfortable or difficult things, during a period of time where the release was the thing that was on everyone's minds, to get to the point where when we shipped, it was easy to see that it was a community project that anybody could contribute to.

SEBASTIAN: Yeah. It's really hard to, like, from the time of the inception once something is released (mumbles) producing private it's kind of really hard to in fact, get around the fact that it's kind of like a community collaboration. And I guess in a way, that was kind of a necessary since this is kind of like, disruptive. And getting something, like, unreleased, like, unfinished out, where the "vision," I don't really like using that word, was in the clear. So you know, for example, we've talked to basically, I wanna emphasize that, like, as Yehuda, just Yehuda being here is kind of like, (mumbles) and his collaboration is kind of proof that we didn't really like, develop this independently or like, at least develop for our own needs. Like, I think basically, all the major frameworks kind of represented in a way. For example, I went to Angular Connect and talked to Igor Minar kind of like on what Angular, their concerns are. I went to Polymer Summit in London and talked to a lot of the Polymer team about particular issues that they have, and they'll also be adopting it. We've added specific features around, kind of like, all of their requirements around HTML imports and needing a fat module structure. And these are concerns that the existing npm client is like, have been raised there for, some of them for years, and haven't really been prioritized, and actually getting these into the npm client is really, it's really difficult. Especially around just, just around the project kind of management and kind of the governance of it. (mumbles) super clear. And so one of our priorities with Yarn is that everything is super transparent and the kinds of decisions that we make, and it's very clear for external people to contribute and kind of like have their voice be heard, especially if, in the future, once we start rolling out, kind of like more broad, like, broad features.

CHRISTOPH: And then, I just also wanna say, there's always this notion of this turf war going on between all those projects, you know, like Yarn versus npm, React versus Angular, Jest versus Mocha, and all this. And that's something that always sounds fun, and you can take sides, and like, that's something where people tweet about all the time but that's not really what drives us or motivates to build those things, right? Like, for us, we work at Facebook and we wanna build JavaScript tools that work well for engineers at Facebook. And when we get to, when we have opportunities to collaborate with other people, then that's something that's really awesome. But we're here to solve the problems that engineers at Facebook are facing, and then we're hoping always to also solve problems that the JavaScript community faces. So this is really what drives us and motivates us, at least me (mumbles) and, like, with all the engineers that I work with are usually driven by. So it's not really about, like, this turf war kind of thing, at all. We just wanna build really good tools.

SEBASTIAN: Yeah, I think Facebook's issues do not exist in isolation either. I mean, like, when we say like, Facebook developing stuff that solves Facebook concerns, and then developing stuff that addresses the JavaScript community's concerns, I guess Facebook engineers are a part of the JavaScript community. It's not like everybody here is kind of like from other companies and has, had like, other prior experience in these, involved, like, in the community. So I think that when we have solutions to our own problems, basically it's almost certain that other people face them, so I think it's important for this kind of collaboration to happen.

YEHUDA: Yeah. So I work a lot with (mumbles) their Ember app, part of my day job, actually and a lot of the concerns that other big companies have are similar. I think Facebook is pretty unique, in that Facebook actually does the heavy lifting sometimes to build new tools that need to exist, and I think that's awesome. Most big company, the npm situation was so acute, I think, for both companies like LinkedIn and just big Ember projects that, a lot of companies that really don't usually spend so much time building these kinds of tools, like, nobody wants to have their own npm. That's pretty ridiculous, right? But there was a lot of companies that had a lot of ad hoc tooling around npm just to deal with basic, basic problems, basic work flows that a lot of people have. So I agree with Sebastian that, I think the way that Facebook uses Node and npm and React is not that different from how, like, LinkedIn uses Ember, especially as your project size gets bigger. And I think, I think something we didn't say about Yarn yet, but I think is pretty great, so Yarn is a good npm replacement for a lot of different kinds of projects, but really, I think Yarn shines when you're dealing with large projects. And I think that really npm's attitude about large projects historically has been like, "Why are you writing a large project?" And I'm not saying there's anything-- there's a lot of reasons why people have had different attitudes, I think when you're stuck with really hard technical problems sometimes you rationalize. I've done it plenty in my own career. But the bottom line is that actually trying, like, having a place where we can all work together, or working on projects that are not a small package and actually deal with it, I think is a huge improvement in the ecosystem.

KENT: Wow, that was a lot of really great stuff. And I have, like, a million questions about all the things you all just said. (laughs) One thing that I wanted to touch on, you were talking about the difficulties of coming out from closed source to open source and still making it feel like it's a community-driven thing. I was having a conversation with Lee Byron a couple weeks ago at Strange Loop, and he mentioned that and he gave a talk, actually, about some of his learnings with the open sourcing GraphQL. And he kind of talked about this idea and at first I was like, "Why don't we just always develop everything in the open, like, all the time?" And one of the things that he said that really kind of convinced me is, when you're building a giant, or like a project that has huge implications, like GraphQL or Yarn, you can get really overwhelmed with support requests or with contributions before you even really know what shape this thing that you're building is going to have. And it can, yeah, it can overwhelm you and make it really difficult. So because of that, when Yarn came out, because of that conversation I had with Lee, when Yarn came out I was like, oh yeah, I had no problem with the fact that it was developed in a, not open source, and then, like, open sourced when it was fully formed, well, maybe not fully formed, I'm sure it's still evolving, but, like, when it was actually a practical thing for people to use. And so I think people should hopefully be able to understand that a little bit. It's like, a really, really, pivotal technology. In fact, just the other day, yesterday, I think, the stats came out that three percent of downloads on the npm registry are coming from Yarn users, which is really incredible.

YEHUDA: That's actually, that has to be an understatement because Yarn significantly reduces the amount of downloads you need to do.

KENT: Absolutely, yeah. Actually Ryan Florence responded with that same kind of thing, which is, it's kind of funny, like, 'cause he's, I think he graduated in finance and so statistics is something that's important to him. But yeah, like, it totally, you don't hit the npm registry as much with Yarn, which I think is pretty cool.

CHRISTOPH: To go back really quickly, sorry, to what you just said about your conversation with Lee and about it being overwhelming when you release too early. The case in point here is actually the Yarn release itself. After the open source release, after 24 hours, we had 300 open issues, we had about 30 pull requests, and then a bunch of people (mumbles). That was really awesome. And then we were just completely overwhelmed. And at this point, I just wanna give a shout out, like, the weeks leading up to launch, James Kyle has been a huge help. He built the website, wrote most of the documentation. We had Andrew Suarez, who works at Facebook, he helped us improve the startup time, (mumbles). He just got everything into good shape. We had Danielo Nigro, also from Facebook, working on the Ads team. He built all the installers for Windows and Ubuntu. And then also the entire Facebook open source team that helps with documentation and getting projects ready. All those people, they supported us so much that we could even do it. And then we open sourced it and it was still too much, you know? (Kent laughs) But yeah, I would give a shout out to all those people that helped so much leading up to the release.

YEHUDA: I totally agree, I was pretty overwhelmed inside at how many, like, all the things you just said, like getting installers is such a grunt work task and I was pretty psyched that we had all that stuff on release. One thing that I would, that I personally worked on a lot, if you look at my issues I'm, I have like a combination of really detailed technical issues and governance issues. I think one thing that went really well for Yarn is that we spent some time before the open source release just to get the governance story-- I wouldn't say that we nailed it down perfectly yet, and governance is hard to nail down, period. But actually just having some open issues, having roc repository, this is stuff that I did. But basically having, making it clear what the plan is because obviously, when you go from closed source to open source, the people working on the closed source project are 100% of the core contributors at that point. We got lucky in that we already had people who weren't at Facebook working on the project, but I think it was, it's really important when you do that transition, and I've done it, I've tons of times worked on closed source projects that became open source, it's really important to say right away, like, "We plan for this to be a community project," and not just lip service, but "Here's the actual plan, "here's how people will contribute, here's how people add, present ideas, here's how the core contributors will interact with the rest of the project, and hopefully the answer to that is, core contributors and the rest of the project are on pretty even footing." And all, I think all that is also, in the same way that it's easy to forget to do installers, it's easy to forget to do governance. And I think the way that we did that really proved the point that we were trying to make when we shipped.

KENT: Yeah, that's fantastic. I actually really love the governance, like the fact that that was a big part of the release, as well.

CHRISTOPH: Did you wanna say something?

SEBASTIAN: Oh, yeah. I think one of the things that kind of made it, like, easy to me, not to be convinced, I actually didn't really initially think of governance, Yehuda was definitely the main advocate behind it. I think I was convinced really easily mostly because of my experience maintaining Babel, just around, like, getting something sustainable that has a dedicated kind of like, contributorship. That was one of the things that I really regret about Babel, not getting more people involved other than myself, really early on. I've been kind of like, ruling with an iron fist. I mean, it feels good at first, but I think it gets (mumbles) and a lot of the satisfaction that you get early on kind of begins to fade and the extra responsibility isn't really something that's manageable long term. And especially, it's just not good for the project, period, since it's kind of hard to kind of get empathy for a lot different use cases, so you kind of have to have more people involved. So, yeah. I mean, being inclusive is definitely a really big thing. Yeah, so I was really easily convinced that, kind of like, is governance more or less the correct one or at least somewhat more open would make the project more successful.

YEHUDA: Yeah, we already have an arc, so the Ember community really needs Yarn to commit to being on the node ops schedule. And there's already an roc from someone who is not me, Nathan from LinkedIn, that's just like, explaining what that would mean, what the details are, et cetera, and I've been talking to Sev about it a little bit. But I think that's also pretty good, like, you could imagine that if there was no roc process, what you have to do? You would have to, like, find somebody, you would, like, go find somebody at a conference, maybe you would open an issue and in a normal project, the BDFL would be like, "Oh, we're not really interested in that right now," then, then what? But this is a real blocker for us, like, we actually need to be aligned with the rest of the ecosystem in Ember. Of course, there are gonna, every one of these issues has conversations, has already, like, tons of comments on that issue and that's great, on that ROC, but I think that just letting people know, like if you have an idea, there's a way for you to bring it to the attention of the core contributors without finding them in person. And there's a process for getting through that that doesn't involve harassing or creating campaigns or whatever other things happen in other ecosystems. I think that's just a huge improvement.

KENT: Awesome. Cool, we've talked about a lot of really cool things and I wish that we had more time because I have more things I wanna ask about. So one thing that I think would be kind of interesting is to get an idea of how Yarn technically works. So, if you go to, like, the Yarn home page, there are, like, three big things that it talks about, it's ultra-fast, it's mega secure and super reliable. So could we talk about how Yarn works, to make it those things?

CHRISTOPH: Also, you have to call out, sorry to interrupt, you have to call out the awesome artwork on the website.

KENT: Yeah!

CHRISTOPH: Al worked with Shitty Watercolour to make that happen. It's a very popular person on the internet. So those are--

YEHUDA: I believe you have just violated the code of conduct. (laughter)

KENT: Yeah, (laughs) we'll let that one slide. (laughs)

SEBASTIAN: Yeah, so I guess you can talk about the three different things, I guess, like, during our messaging there's kind been, like, the three things which is reliability, performance, and then security. So from reliability you have determinism. So I guess I'll start with, I kind of early in the show, kind of like went over why determinism is important, especially in the context having multiple people working on the same projects in a lot of different environments. Security is also a really big deal and with (mumbles) the initial thing that kind of seeded Yarn, in a way. So I was actually talking to Sam Saccone of Google, who was kind of going over how-- and they eventually released a cve of this, basically this is just like a fundamental way of how npm works where you have these live cycle scripts where you can basically execute arbitrary commands. And then one thing that you could do is publish other packages inside of this. So you could do something, so when you install a package, it could go through and check, are you currently logged into npm? Yes you are. I'm going to publish every single package that you have permission to and add myself to it. So you would basically have this thing where authors who then have all their published packages published with the new version or something, and it could just include an arbitrary payload that could do, like, whatever, install a virus, you know, information. And this is obviously a really big deal or issue for companies, especially if you value privacy and security.

And so one of the ways we kind of, we resolved this in a few different ways. One is that you actually have to log in each time, so like publishing package and don't actually store the npm login token. That's kind of a hassle if you're in a CI environment, but we actually offer a way that you can then log into npm and then save that token somewhere and then reload it. So, like to automating that is kind of like, the same. And mostly around-- so, determinism is also kind of like a big thing that goes into security, since, if you're constantly pulling in new versions of packages, they could introduce whatever, you don't really have a chance to audit those. So having this lock file kind of prevents this automatic updating. So even if, in the existing ecosystem, if this hypothetical npm worm were actually a thing that would go through everything, if you had Yarn lock file, you wouldn't use, like, infected versions of something. And this kind of gets in the way of, kind of making it as secure and reproduceable as possible. And to go back to the lock file, we actually hash of all the network toggles that you might use if you use a Git repo references of package, we will store the commit hash to make sure we only ever restore back to that, to add a bit more network security. But it's kind of like, small things that a lot of, like (mumbles) package managers do, but as Yehuda alluded to earlier in the show, like, this isn't kind of like, this isn't something that everything does, especially not npm. So npm doesn't store the tarball URLs inside of your Shrinkwrap, and so it has nothing to authenticate that. What the network actually gives you is what you, (mumbles).

And going back to performance, I guess we're focused a lot on just parallelizing everything better. I think npm has an issue where it has a lot of waterfalls, so what that means is that, say you have to install five different packages, it will install them one after another, and it will be blocking. So if it takes one second, and the next takes one second, it kind of stacks on and everything takes two seconds, but Yarn will, like, parallelize that and do them both at the same time. So if five packages take one second each, they will all complete in one second. And this kind of allows you to exponentially grow, like, your (mumbles) graph, and you obviously always have to pay a cost. But if the cost is, like, really small, then you can basically stack onto that to, like, insane limits. So I guess that kind of goes over the three things and kind of what we've done to address them and how npm doesn't really solve those in a way that we think is acceptable.

YEHUDA: Yeah, just wanted to add one thing to that which is, I think one thing that the npm community really jumped on, and I think this is great, actually, the Cargo community, the Rust community has done a similar thing, is just argue that people should build small packages, if the small package can be reused in some way. So I think that the Rust community has gotten a little less extreme about it. Although I think there's a left ad package in the (mumbles) file. But the basic idea that if there's more than one user for a thing, give it a public API, ship it, version it, whatever. But, as Sebastian was pointing out, npm itself has problems with really big dependency graphs and npm projects tend to have really big dependency graphs. Again, the story here is, a lot of times people will, you know, I feel trolled sometimes at the npm ecosystem saying like, "I opened up my Node module directory and I have so many packages," it's like, well, what did you think? That was the thing everyone told everyone to do is to make a lot of packages, and everyone did that, so that's why we did that, right? And so I think we just, we have a community that actually is pretty good at code sharing and code reuse, and I think that's great. There's still some things that we could do better. But I think making the package manager good at doing things like reducing waterfalls doing parallel builds compilations, sometimes when I say "build" I mean, sometimes there's like C code that has to be compiled in your npm install. Like, sometimes your pre-published hooks are actually downloading things from the internet. So basically, being able to do a lot more things in parallel, means that you could do the thing that the npm community and the Rust community both believe in, which is, make a lot of little packages without it becoming exponentially expensive.

KENT: Excellent, yeah, that's great. So, we just have a couple minutes left and I want to ask about the future of Yarn. So, one thing that, I actually remember, Sebastian, awhile back we were going back and forth on Twitter about, like, I made the assertion that, why would anybody ever use Bower. You can use npm for everything. And one thing that you brought up that that makes a lot of sense is, Bower has dependency resolution, where if there are two dependencies that conflict, it will ask you which version you want to use. Is that a feature that we'll see in Yarn at any point?

SEBASTIAN: That's a feature that's already in Yarn, actually.

KENT: Oh, snap. Let's talk about that. (laughs)

SEBASTIAN: Yeah, so, I think that the night we had the discussion on Twitter, I believe I was in the middle of actually implementing that. So, that feature was developed in basically close collaboration with the Polymer team. So they had specific requirements where, with HTML inputs, you actually have to always know the location of a module or a file on disk so it can go and fetch it, since, when you're in the browser, you don't really have this luxury of being able to do, like, a lot of network or file requests, so you need to be able to resolve it in a, like, in an absolute way. So that's kind of why that feature exists. And so Bower, when you have multiple versions of a package that share the same name, a lot of them, say, just like pop open a window that just asks, "You have these several, could be several dependencies that all collide, they have the same name, they're different versions. Here are the ranges that they're looking for," and say that a package might depend on jQuery 2.1.0, and another might depend on 2.2.0. Now, it's more than likely that just depending on 2.2.0 will work, just like across all over. And this kind of reduces duplication and especially is useful in a browser context where you actually have like local name space, where you're just, like, shoving everything and you don't have anything isolated like, in the context of how Node has modules. I think it's a really useful feature, especially (mumbles). It's even kind of useful for other projects, the (mumbles) by Polymer HTML import semantics. So, yeah.

YEHUDA: Yeah, one thing I'm pretty interested in, there's a really long open issue that I have about this. I'm really interested in making the flat, so the thing that Sebastian was talking about is, like, Flat mode, which basically insists that there's only one copy. I usually call that thing the highlander rule, where there can be only one. And I'm really, I've been doing work in Cargo for a long time now, and I'm interested in Yarn, in combining, or unifying the Flat mode and the non-Flat mode, so that, in other words, I think doing a dependency resolution-- it's just a hard problem, dependency resolution, that is more like the Flat mode or the Bundle mode or the Polymer ones, but still allowing duplicates as an escape valve. I think is a, it's what we do in Cargo, basically, I think it's a nice feature. The complexity of the problem is that if you reach a point where you're not sure if you're-- so in Bundler, if you're doing a dependency resolution and you find a conflict, it's actually very easy what to do, you backtrack and you try another thing until you find the solution. When you have duplication allowed, when you hit a conflict, when you say, "This guy needs 1.1, and this guy needs 1.2," you basically have two options. You can either backtrack or you can just duplicate. And knowing whether you should backtrack or duplicate is just a new problem that, to my knowledge, no one has addressed directly in a package manager before. So, I have an open issue about this, people can read it, it's detailed. But, I think there's a way to do what-- one way to, like, a very easy way to think about this is I think you could imagine having like, a sub graph that's Flat mode, right? So like, the whole Repo is normally npm, but for these, all dependencies below this point, it has to be exactly Flat mode, would be a thing that you might want to be able to say, but would be hard to say unless you unified the modes in some way. Sorry, that was a bit rambly. (laughs)

KENT: No, no, it's an interesting problem. Cool. So, actually, another Bower-related question. So, can we now just use Yarn? I know that the PureScript community has been using Bower exclusively, for some of their requirements, and I think the Polymer community uses Bower. Can Yarn be a replacement for Bower in those cases?

SEBASTIAN: Yeah, so we actually, initially, had integrational compatibility with Bower. This is actually really easy in the Yarn client itself, since it kind of like abstract things such as the manifest names, so it could internally it can support, like, multiple things. So this meant supporting Bower was quite-- to start off it was quite easy since Bower JSON basically follows more or less the same kind of "standard" which isn't really a standard, it's just kind of the other standard of package JSON, so it's kind of like, npm, whatever npm supports, like the dependencies object, which loosely kind of-- the whole package JSON file that npm uses kind of loosely follows the CommonJS module, package JSON format. So this allowed us to add, initially, support for Bower. And for open sourcing, we actually had Bower support. This was initially for compatibility with, like, existing Bower projects. But after open sourcing it, just, there was a lot of issues just related to how it actually worked. There's a lot of different ways in which Bower kind of handles things, different config options that npm might have a parallel for but we just don't have the support to like, kind of convert them. And so we actually removed Bower support, mostly around just, like, unifying how, what the project is actually about, which is kind of around this same work flow. And I think it's quite difficult to support, like, these two different registries, especially since Yarn is a much better replacement for npm than it is for Bower, at least currently. And so, you run Yarn and so on and you can discontinue using Bower. And I think a much more realistic or at least, more efficient way to kind of, like, replace this, to at least replace Bower in your projects, is add the features that people use in Bower to Yarn. So there's the stuff like the Flat mode, it couldn't be in the future stuff like supporting Svn repos, which is a feature of Bower, Mercurial, and things like this. I think it's a much-- (silence)