KENT: And we're live. We're the JavaScript Air. Hello everyone. I just almost had a heart attack. It said "Failed to go live" and I was like "That's never happened before!" Okay, we are live. And this is a good thing, so. This is JavaScript Air, your favorite broadcast-podcast. All about JavaScript and the web platform. And today is episode 041 and it's called "Test All The Things With Cypress." So we got a couple of subject matter experts on the subject here with us today. We're excited to chat with them. Before we get into our conversation though, I want to give shout-outs to our favorite sponsors, which are all of our sponsors 'cause they're all great.

So Egghead.io is the show's premier sponsor and has a huge library of bite-sized web development training videos. Check them out for content on JavaScript, Angular, React, Node, and more. Egghead.io is also the host of two free redux courses from Dan Abramov. Find them at egghead.io/redux.

And then Frontend Masters is a recorded expert-lead workshop with courses on advanced JavaScript, Asynchronous and Functional JS as well as lots of other great courses on front-end topics. Find them at frontendmasters.com.

And TrackJS reports bugs in your JavaScript before customers notice them. And with their telemetry timeline, you'll have the context to actually fix them. Check them out and start tracking JavaScript errors today at trackjs.com.

And WebStorm is a powerful Javascript IDE. Working with Angular, React, or Node.js? Then you don't wanna miss its super intelligent coding assistance. Use the discount code "JAVASCRIPTAIR" at checkout at jetbrains.com/webstorm to get 20% off of your WebStorm personal subscription.

And finally, Trading Technologies is looking for passionate and inventive full-stack JavaScript developers who want to work on cutting-edge solutions in a collaborative and challenging environment. Go help them build the top choice for derivative traders.

Alright, sweet! So today we don't have any panelist. Sad day, but we do have two guests. So let me introduce them to you. They are Brian Mann.

BRIAN: Hello.

KENT: And Gleb Bahmutov.

GLEB: Hello, everyone.

KENT: And Gleb was actually on my show at least once, maybe twice, on Angular Air, and I think he's been on since I left too. So Gleb is pretty used to the camera broadcast-podcast format here. But let's get an intro to our guest here. We'll start out with Brian. Can you give us a quick intro to yourself?

BRIAN: Sure. I'm the creator and founder of Cypress.io. I've been programming with JavaScript now for probably like eight or nine years. I've built a lot of different products so really passionate about testing obviously. And just here to basically help the community and try to build a great product.

KENT: Thank you for your work. That's awesome. Gleb?

GLEB: I'm Gleb Bahmutov. I've been doing JavaScript at Kensho, which is financial analysis start-up in Boston and New York. And personally and as at Kensho, we've been very happy Cypress user and paying user for the last, maybe, three or four months. And I think this completely changed the way we test our software and we've improved the quality tremendously.

KENT: Awesome. I wanna hear all about how that has improved your software tremendously. Cool. So let's, just to kinda kick things off for us, it's always good to get kinda a foundational understanding of what the topic that we're talking about even is. Many people probably have never heard of Cypress before at all, and so let's go ahead and get that. I'm sure, Brian, you, over the last couple of years, you've probably honed your 30 second elevator pitch on Cypress. You can give us a little bit more than 30 seconds if you want, but what is Cypress?

BRIAN: Sure. Cypress is a testing tool that is aimed, I would say, at developers or QA engineers, and really just helps them write automated tests for anything that they are putting on the web, whether it's a web application or even just a website. And I would just say that really the biggest difference is that the architecture of the way that Cypress is designed, is really fundamentally different than the way that Selenium or WebDriver works. It doesn't actually use Selenium or WebDriver under the hood. And it's built with modern developers building modern applications in mind. So the best use-case of it is you're using Angular, using React, using Backbone, you're building something that has a rich amount of interactions. That's sorta its position and that's who it's marketed to. It's built on top of sorta all the best in class except in JavaScript Tooling, which is Chai, Mocha, and Sinon. And it's basically a comprehensive tool that helps with the setup of testing, the actual test writing, and then, finally, when you go to run your test in CI, it helps diagnose and pinpoint failures at that point.

KENT: Cool. Awesome. So I've been, sort of, casually reading this book called "Start with Why" and you might be familiar with that. But I've actually been trying and failing sometimes that, starting kinda with the "why", and I think that we can get even more deep on convincing people of why Cypress is so great. If we could talk a little bit about why Cypress even exists, like Selenium seems okay, right? Or do I even need end-to-end tests? So could you talk about the "why" behind Cypress, like why did you even build it?

BRIAN: Sure. I mean going all the way back, why I built it. It sorta originated out of my own frustrations of using tooling, of running a developer team, bringing new developers sorta into the testing ecosystem and watching them struggle. I mean I would say that unit testing is pretty much figured out. The approach to doing it is really the same in all languages. There are great tools across all different languages that help you do that. I mean, obviously, there's still lots of innovation going on even in the JavaScript space related to this. But it's really what, what we're focusing on right now is like the end-to-end integration stage. And Selenium has been around since 2004. And it has had tremendous amounts of development and tremendous amounts of innovation really. You know, WebDriver was sorta the biggest thing that's ever happened. But ultimately, where it falls short is that WebDriver is a stateless HTTP API that is attempting to test a staple system, which is a web application, right? And architecturally, it is just fundamentally impossible for it to really understand all the things that are happening inside of an application, and therefore it can only make really the best guesses.

It's kind of like, imagine you were basically just firing remote commands, like you're outside the box and what's inside the box is black and you don't know what's happening, and all you're able to do is fire sorta commands over the wall into the box and query just this one instant, you know, snapshot of the state. That's not necessarily representative of how the web application is actually performing or sorta where it's at. You could catch something like one millisecond before it happens or one millisecond too late. And Cypress is basically designed with the exact opposite architecture, which is that Cypress is actually inside the app, you're testing your app through Cypress. And therefore it is able to understand everything that is happening. It is in the same run loop. It can react immediately to events and modify its behavior dynamically based on a set of rules to understand exactly what you're wanting to do so we can immediately pause commands, you know, we detect that page is loading. You have native access to every single object so you can tap into the window, you can tap into your own application code and control it, to stub it, to basically you change the way the application performs in real time. And because of that, you're able to write much more resilient tests that aren't flaky or brittle and that ultimately leads to much better experience.

KENT: Yeah, that makes a lot of sense. I'm convinced. (chuckles) So, like, can we get into some of the nitty gritty details of like how I would use Cypress? Like does it work with other testing frameworks that I'm familiar with? What's the syntax like? Or what's the API like in actually using Cypress?

BRIAN: Sure. So I think I'll, you know it's, Cypress is kinda a difficult tool to sorta describe so I'll just kinda like walk you through a day in the life of like how you did it and what that experience is. And also kinda highlight some of the design decisions around what we wanted. So obviously one of the biggest differences between WebDriver and Cypress is that you only write your tests in JavaScript. That's the only thing that we support, it's the only thing that we will ever support. And, you know, that's something that's only happened in the last few years where it would be like acceptable to a team that they only write in JavaScript. So let's kinda get that out of the way. So we know like our target market is JavaScript developers.

But then the next thing was is that we didn't wanna necessarily pin ourselves directly into the npm ecosystem, like you can use, obviously we support it, you can npm install Cypress. But we wanted it to be like a zero dependency environment. So actually the way that we distribute Cypress is as a desktop application where we pack up every single dependency that it has, right? And we actually use Electron to do that. So the first part of Cypress is just downloading the app and it runs as a legitimate first-class desktop application that you go out and that you install. Because of that, there's basically no chance that anything can go wrong, we build it for every different operating system, and that's where you start.

And then really the next step is that there's an entire GUI that sort of walks you through getting your project initially set up. So you go in, you drop in your project directory, we automatically scalpel out, sorta suggest structure for it. We scalpel out like a specific spec file for you. And then you really just use a GUI to launch the browser and just start testing. But then what that does is that brings up a real browser and that's where all your work is performed. I mean we didn't wanna create like some desktop application where you're like trying to test the web but you're really not in the web, no. You work in the web the entire time and so you have native access to all of your DevTools and everything else that you're comfortable with that.

From there, we have a DSL that's about 80 different commands that basically simulate all the things that you'd wanna do in a browser. So you can get elements, you can interact with them and click them. You can do things like stubbing actual XHRs to force their return values so you can simulate network delays. You can actually test without having a backend at all. That's actually what we do at Cypress, we dogfoot in on our own projects. Before we even have an API server, we basically bill out the entire frontend and stub out every single route, every single XHR. So we force it to have the data that we want and we can essentially build out to all the edge cases and situations.

And while you're working inside of this, you're basically are inside of, you're inside of Cypress and you have a GUI that basically breaks down and shows you every single command that you run. So typically in a test it might be 20 or 30 or 40 different commands in order to test a situation. "Go get this element and interact with it and fill out this form and click this." And so we indicate to you the progress of every single command as it's happening in real-time so that, when there is a failure, you know that it failed on command 11 and it wasn't a different problem. And because you're in the browser the whole time, it's like lightning fast reloads. You make a code change and, you know, it's sub-second refreshing it and it's triggering the browser really as fast as your application can go. So does that kinda makes sense of sort of what the experience is, what it looks like to work inside Cypress?

KENT: Yeah, yeah. So I just had a question that came up as you were talking. What about code coverage? Are you able to record and report code coverage?

BRIAN: Yeah that is definitely challenging because you don't really write unit tests. You can write unit tests in Cypress. And that's something that we're gonna be doing better down in the road map, but it's mostly for integration and end-to-end testing and then do coverage through that. Cypress doesn't know anything about your application. You know? Like you can just use Cypress today to go test google.com, right? And to do code coverage when it comes down to integration and end-to-end testing, you have to instrument the JavaScript files sorta on the applications' end. And that's just way, it's fundamentally harder than doing it with unit where you're not requiring files in-line, right? Like the JavaScript files are being served by a web server way later. And Cypress isn't really conscientious of those. It doesn't know anything. Your application really just does the same thing it does when you fire up the browser, right? So that is not something that we have solved. And it might not even really be solvable in like a generic way. You might have to like, you know, specify exactly what you want or something like that. So, yeah.

KENT: I can see that being solvable. Gleb, you got an idea.

GLEB: I got an idea. So Cypress allows you to specify the base URL kind of like what your router log. And you can easily change it for a config file or environment variable or even command-line as you start Cypress. And if you really want code coverage for end-to-end tests. I have a project called WorkFaster, which is a code coverage proxy that can run between google.com and your machine. And so you can actually run Google tests by pointing Cypress, not at google.com, or that was past the proxy, which will cover all the code from google.com that is received. And so you'll get a code coverage. But it's not Cypress's job, it will be like an external thing.

BRIAN: Sure. I mean it would be possible. So Cypress does crazy amounts of proxying, where it forces every request to go through the proxy. And therefore it can modify and change the world however it wants. It would be possible that, as JavaScript files are being served, that we would, you know, basically go through and use a (mumbles) or whatever else in order to break it all out. But the problem is is that what is being served is not necessarily, it's not your source. And so you would get code coverage like in the final build, right? Like after you've gone through Webpack or Browserify and it wouldn't necessarily point. Might be maybe like if you gave a source maps and we instrumented those, we could point it back. But I don't know, it's an interesting problem to solve. And I'm just, you know, it's like one more thing on our radar and it's just not a priority at the moment. But I would love to have some ideas and talk through maybe how it's possible to do.

KENT: Yeah. Honestly, with the way that I see Cypress positioning itself, it's not as much of a unit testing tool where code coverage actually matters. It's more of a like end-to-end or integration. And at that point, code coverage starts to matter less and what you're more concerned about is use-case coverage where, you know, and you're testing certain flows and you want to make sure that you have those things covered pretty well. But I wanna ask you, Gleb, about your experience adopting it. I imagine that, like in the process, you were probably replacing something else or maybe you had to convince people to look into Cypress and stuff. Can you walk us through like the process of adopting it and how that experience has gone for you?

GLEB: Sure. So we've been around for three years. And we've been doing end-to-end testing pretty much from the beginning, in addition of unit test. So we started with PhantomJS. You know, we wrote into it a test. Okay, fine. And then Casper appears. So we actually switched from Phantom to using Casper for better API. Then we decided, "Okay, we need multi-browser testing or cross-browser testing." So we actually have protractor tests. So we've been progressing with technology, every year it's something new. And yet, when I saw Cypress video maybe five months ago, I was just hooked. I can see my test running, I can write test, we will be using very same API compared to, you know, Casper or anything else.

And the best thing is that the value of end-to-end tests is not what it actually passes, right? The most value I get as a developer is when it fails. So when a Cypress test fails, I can see the screenshot, I can see the steps that it took and where it failed. And if I'm running in the browser, I can actually see the element, I can inspect everything when it failed in DevTools. I can point at each step in my test and see the snapshot of the DOM when that step happened. So, as a developer, if a crash happens in Cypress, I can debug it very, very quickly and easily. Everything else before like Phantom, Casper, when a crash happens, you pretty much have no idea of what really happened. The "why" of a test failure would be not, "Oh I have a stack", it would be what happened before that, how did my page look. And I had no idea.

So, as technology progressed and we reached Cypress, we're like "Okay, "now end-to-end tests are actually useful." And they give us so much information as a developer. So I tried them on a couple of work-from-home projects and Cypress was in a private beta, you just signed up. And when I started showing it at work I said, "Okay, we should not be writing Protractor tests. We should just use Cypress." And there was a lot of convincing to do because, well, for different business reasons. But the developers were sold after one hour. Pretty much as soon as I show them, "here's how we task it, here's how you would write a task, here's how you watch what test happens, and you can look at it step-by-step." The developers are sold immediately. But business people would probably required some convincing to do.

KENT: What do you say to business people? I mean like, we kind of put the business people over here in like, as if we're fighting with them to get the things that we want. But I think any reasonable developer also cares about the business. And so like I was at a company and I wanted to rewrite from Angular 1. But it didn't make business sense so I was like, "I'm not going to recommend that we rewrite, even though I want to. I'm not gonna recommend that because I know that would really hinder our progress as a business." So I don't think that, like it's great to frame things as like how you, you know, coerce the business in doing something. But how did you convince yourself that this was worth it from a business perspective and then how did you shared it, that convincing with the people actually making the decision?

GLEB: So I will start with kinda one concern that we answered and I will kinda punt and throw it over to Brian after that. So the biggest concern for us was sunken cost. We already know how to write Protractor tests, right? Well we have a bunch of them, so why should we care about a new tool, right? Care about writing the test in a new manner and then support both types of tests? So that was a big primary objection. And the second objection we had to overcome was controlling the source code and controlling the CI itself and, you know, the test process. So by default, if you sign up for Cypress, you can use their own cloud to run the tests. And it's really easy to do. But in our case, because we are trying to really keep the source code private, we run our own CIs, our own boxes. The business side and the management side felt very strongly that kinda surrendering and running our end-to-end tests on Cypress's CI cloud was too much, right? And so we had to look for ways to actually run the Cypress tests ourselves. So we build docker images with Cypress backend and all the dependencies so we could run it easily ourselves and control everything. But that's something we had to do. A lot of people probably don't have that concern, especially for public-facing sites. But maybe Brian can tell us more about the CI cloud that Cypress has.

BRIAN: Yeah, so, let me, and I guess let me kinda clarify on a few points. So, you know, when we talk about Cypress, we're mostly talking about the desktop application, right? And the desktop application's what developers use. It's what you gotta install. And that solves the first two legs, right? Makes getting tests set up really simple and then the actual developer experience day-to-day is really straightforward, right? And, of course, all of that you install. None of that is connected to us in any single way. And all of that is absolutely going to be open source. But then the third part, the CI piece, is really where, to do that well, you sort of have to add, you have to add value to that part and you have to pull out those assets and you have to you make those accessible to all the users. And that's sort of like where our platform comes in.

And just to kind of explain, so this is a project I've been working on for over two and a half years full-time now. I worked the first nearly 18 months just completely proving this out completely on my own. And then when I got to a certain point, I realized this is way too complicated of a product. I'm gonna actually have to have other full-time developers work on it. And, you know, we went through a lot of discussions. We ended up raising capital with the 3DC company. So we're actually founded in Atlanta. We're a team of five. We're obviously hiring. We're gonna be raising a (mumbles) in the next few months, that sort of thing and growing from there. And so, obviously, like a part of Cypress is the business component. And that has evolved a lot, like we have gone through many different revisions, many different ideas of how to really do this. And so we talk a lot about like spinning up our own servers in the cloud and, in fact, we spent months building that out. We got that working. And I guess I'll just sort of explain that. We're probably actually not gonna do that anymore. And the story for CI and the story for Cypress is actually gonna be a lot simpler. And this has obviously evolved a lot from working with early adopters to hearing their objections to trying to sort of like make all these things work. And so I can sort of explain that what we're doing now is definitely not we're gonna do in the future.

And really, there's like two major steps to this product. The first is that, you know, we need to open-source the entire desktop application, right? And that's actually really close. We've actually already open-sourced like seven or eight repos. There's close to 20 repos. We're actually close to converting that back to like a mono-repo. We like split everything out into like a million npm packages and it's actually super difficult to manage. So we're gonna push that all back into one repo. Sort of how Babble does it, right towards moving packages and sort of everything lifted out. And all that's gonna be open-source. So what do we mean by that is that you'll be able to use Cypress in your own weird way, however you want. You can fork it, you can do whatever, and then you can have an amazing local experience when you go to run your test, right? And you'll be able to run that product in CI as well as long as you've basically manage that process yourself. And we'll still like deliver the final assets, like after a build, to help you with that.

But our cloud platform, like where we're gonna come in and where I see us working for the vast majority of people is that when test go to run in CI, there's really three things that you care about. There's screenshots of the actual failures, there is a video of the entire run, and then there is like a massive amount of logs. And based on the way we design Cypress and the position that it's in, it's capable of understanding everything that happened inside the browser. And so we're capturing all three of those things and were going to, as part of the platform, automatically upload those to ourselves, and then make those results immediately available back into the desktop application. So within the desktop application, you already have the site's GUI, you have the ability to launch browsers, you have the ability to set up projects and see where everything is at. And then you're gonna have the last leg of it really at your fingertips, where you will see what happened in the CI run.

Past that, there's all kinds of cool things that we can do with the platform like giving you real-time access to your CI build, like basically shipping and embedding VNC and then automatically handing the tunneling to enable you to remotely connect to it, to watch the build as it's happening, or potentially even put Cypress in a mode where you don't just watch what's happening, but you basically prevent Cypress from exiting CIs. So it's in the real environment, it's running your tests. But then you want to come back and modify a source code locally, we'll automatically like synchronized that with the CI server so you can essentially iterate on the failure directly.

Those are sort of some of things that we're gonna do in the platform. We'll likely add in screenshot dipping later and then give you like an interface and understanding that stuff. But I don't know, that's sort of where we're going and we're actually. We're actually at this point, likely not gonna be spinning up instances anymore. The plan of what we, what we originally were thinking for our platform is that, that basically we have to control the environment that the browser runs in. There's all kinds of situations, like browsers are amazingly complex things. And all the way down to GPU drivers can affect it and can cause crashes. So we were like "We'll we really need this like pristine cloud environment where we control the resources given to the browser." So that means that when it goes to run in CI, we have to automatically handle the tunneling and, of course, that's like where we can record the video and all that stuff.

And at the end of the day, I think we don't need to do that and we can better align ourselves to sort of what the community expects in an open source. We're really worried like, "Well if we do this, are we also gonna be designing features inside of Cypress that potentially push out our own platform or like reduce its value."

For our team, I don't want to be in conflict. I don't want to have these questions where we're like refusing to build out features that people need in the open-source product just because we want our platform use to be more useful. You know, I want about these things to make sense, to be used everywhere. And you know, like you built out all these docker images, you handle automatic parallelization at the operating system level. Like all we need to do is basically just expose you and API on our backend that we understand that you parallelize the builds so that when we show you the results, we can associate all of these things together and give you like an interface to inspect the results. So, I don't know. That's some of our ideas. Running in CI is a fairly complex thing. And we can talk a little bit about the way that it works now. And we can also talk about like in the future, like how we're gonna support multiple browsers and things like that.

KENT: You said a lot of stuff there that I'm just like trying to make sure that I don't miss. So one thing that you said there that I have a question about was the like automatically uploading assets to your server and then having that available in the desktop client. So I'm trying to picture my workflow here because this happened like, here's my current workflow. I make a pull request, the build starts after, you know, 'cause we're using Jenkins and everybody's favorite tool. It starts after a couple of minutes, takes forever to do stuff, and then eventually it breaks. I have no idea what's wrong. We have this plug-in where I just type in "Build me" as a comment and then it'll restart the build and something magically happens and works. If I wanted to look at the build, then I go to the enormous amount of blogs to try and figure out what part of it broke and most the time I can't, so I just say "Build me" again and somehow it magically works and I just don't understand it. It's a real pain. so what would my workflow look like in like a pull request scenario with Cypress?

BRIAN: Sure, sure. Yeah and I love to talk about this 'cause I think this is the thing that we're nailing the best, right? And kinda let me take it to the back, right? So it's sorta like testing is hard as-is, just like the concept of testing. And then actually writing tests, it's extremely difficult as well. But we don't ever think about what you just described, which is just that after you sort of made the investment and you built up this suite of test that's only ever growing, when failures happen, that is often times the hardest part, right? When we can't figure out why something fails, we can't easily see inside into it. And then we, you know, we lose faith and we lose, we feel like we don't have control over what's going on and then we basically stop testing altogether, right? Okay, so now--

KENT: Yeah, that's completely what I looked like a lot earlier.

BRIAN: Yeah. Okay, so how is this gonna work, right? I mean it's so, it's so simple. But basically, where you run in CI doesn't matter to us. And in fact we call it CI, but you can do all this from your own computer 'cause it's really no difference. But the idea here is just that you have your CI scripts and part of your CI scripts you have a couple of little commands that basically calls Cypress to run. Right now, you have to do npm install Cypress CLI, and then you have to do Cypress install, and then you have to say Cypress CI, right? So three commands. We're actually gonna be reducing that down to one command. But, nevertheless, that's really it. You have to write three lines of code in order to have Cypress to run. And Cypress is installed on the code base and we don't ever, like when Gleb would change source, we have nothing to do with your source. Source will never be transferred, it will always stay on your system. Even when we were doing the platform, all that was was that was a browser in the cloud connecting. And a browser obviously doesn't have access to source control, it only has access to, you know, the stuff that, you know, the files that it give us, download to over HTTP. But at any rate, so your CI run has spun up, it has called out, it has installed Cypress. And of course your source control has the Cypress test and now the app is installed, Cypress begins to run right? So Cypress is running and, as part of the desktop app, it's basically handling the three things that I mentioned. It's basically building up a huge amount of logs, it's building up screenshots and failures, and it's building up the video.

Well after the end of the run, right, like before we exit with the number of failures, all it will do at that point is basically take those three things and it will upload it into us. And when you upload it into us, we can then basically parse through all the logs, parse through all the screenshots, parse through the entire video, and basically hide everything that was passing and only focus on the failures. So that's like the raw data is now on our end. So what we can do is that yes, your flow stays exactly the same. You would see like, "Oh, exit code was three". Three failures, right? But instead of scrolling through text and standard out, which doesn't help with, it's just like a hill of beans. Like what happened in a GUI browser? You cannot compress that down into text, it's impossible, right? The thing is is you need access into sorta the raw material that is gonna indicate to you what failed. And, I mean yes, if you build in like, "Hey, okay. As far as my CI stuff, I don't want to send these assets to Cypress and I just want to house them myself." You can do that, you can just manage those in whatever way. But trust me, the logs are like megabytes of traffic. So you're gonna send that out to us, and then we're going to give you basically just a GUI that where the results are immediately available inside of the desktop app or you would just go to like admin.cypress.io. You go to the same thing that you have with Travis or with Circle, right? A web app that gives you all the results. And then you have like a UI that only shows the failures. And then you can associate, not just the screenshot in the video, but also all the logs that happened as the test was running. So you can see like, I mean you can see visually what happened right? But also, you didn't have DevTools open, like there's still so much more knowledge there that you might be missing. And so by using this, you be able to see like "Okay, this command started. And oh! I saw network traffic." And then, you know, I saw something that I caught the log that we serialized. And then I saw like this command failed here. Well I can see everything associated before that ran, and I have a pretty clear idea of what failed.

So that's sort of, the extension beyond just your normal flow. All these are like additions. And really liked part two of that is that, to really solve this problem, you know you're like, "Okay I see that there's a failure of 90 something didn't appear". And I look at the logs and I have a pretty good idea, but you still need to then iterate on that problem directly because maybe you fire up Cypress in your local environment and it works straightaway, right? Because, you know, CI environments are different than local environment. You could have written your test in like a flaky way, which we can talk about. You can still definitely do that in Cypress. So the last leg here is that you need to iterate on the problem. So then what I would do is I would then run the build again, except this time pass like an environment variable that just tell Cypress like, "Hey, don't like start running until you receive a remote connection." So basically everything starts up, Cypress is about to run but it's like "Okay, I'm halting here until I receive a remote connection." Then from the desktop client, we know that our run is happening. We give you like a button that allows you to remotely connect. We'll automatically handle the tunneling and, so there's no firewall or anything like, you would get into your build and then you would just watch it and iterate on that failure directly where it happened. So this is like the comprehensive solution.

For people that don't want us to upload assets, like you don't want us to have like the raw logs and stuff like that. You don't have to use that part of the platform. Likely what we'll do in the future is we'll build something like an on-premise version of Cypress where you can like install it in your AWS cloud, and it's really the same thing. That's likely what we'll do in the future. But really it's just, when I feel like we're better aligned with the community, the story behind like using Cypress is gonna be very explainable. Like if you don't want pay for the stuff, you won't get this value but you can just opt out of it and then you can just manage them in your own way. And you won't get like these extra things, but it make sense, right? There's no chance that, like you're investing in Cypress's tests. There's no chance that that's ever gonna stop working. There's not gonna be a paywall between you and your source code. There's really just gonna be a paywall between you and the additional valuable things that we think you will need to really solve Cypress. I mean we see people struggling with this all the time. But it's gonna be a choice that you have.

KENT: Cool. Awesome. That sounds like an amazing workflow. Gleb, can you talk about what your workflow has been with Cypress since you started like four months ago?

GLEB: Absolutely. So one thing that I like about Cypress right away was when you add a new project, (mumbles) immediately has all the configuration and example spec file for you. So you can start from nothing, it gives you things to run right away. And then you can look at the example spec and it includes pretty much the kitchen sink of everything Cypress can do for you. So we started by just taking the file and removing pretty much everything but what made sense for our website, and that was a start. And you can literally do this in maybe five, 10 minutes if you start from nothing. So we just started by logging our page, kinda clicking around, verifying method, it breaks. I mean, it doesn't break. Once we solve a CI problem, we decided, "We like what we see but we have so many web apps, it makes sense for us to actually run a bunch of test in parallel for different apps."

But interesting design decision, and I agree with it, in Cypress was that Cypress doesn't actually bundle your spec files, like your spec source code. Because there are so many choices. People like WebPack or they like Browserify or anything else. So by default, Cypress just logs your spec file and it assumes it has everything. So it doesn't assume that you have require or MD or we can log anything else. You have to pack it yourself. So we had to write a still piece that we called Multi Cypress that can take multiple spec files and generate multiple bundles. So you have complete bundles that you can run separately and that tests different web apps for us. So it's open-sourced. We released it under Kensho GitHub account. So now we can use ES6, we'll run a roll-up. We reuse different dusting utilities. For example, you could say, you know, we'll run an analysis. And it is now just a single function call and it does things and we can use it for multiple projects.

We also wrote a lot of utilities but our outcome, Cypress commands. So Cypress allows you to define a command and it becomes almost a part of its API, just like any command in its DSL. So for example, we could say "cypress.login". And it'll do all the custom steps for our website and login. We can do, you know, click some on this weird widget that's not really supported natively by Cypress and its ports. So we got a bunch of projects, they all have multiple specs. Because specs are separate, we can run them in parallel. We use GitLab CI to run everything in parallel. The biggest challenge for me right now is not writing the test, but making sure that everyone in the company that's touching something can just go on (mumbles) and run test, right? So they know if something breaks. So if something changes, everyone can update the test. So far, it has been very successful. People who are not front engineers, they go and update end to end specs and see no problems running them again or running them locally or making sure the CI passes. So nothing but positive things to say. We're experienced as a developer and, you know, as a person who cares a lot of quality, it has nothing but positive.

KENT: That's great. So where do you draw the line for the types of test you write with Cypress? And then, kinda maybe that sorta goes along with it, where do you put your test files, in your file?

GLEB: So we keep unit tests and for unit tests we use Mocha and AVA right next to a source code. But with the just unit tests, they're on another environment. Cypress end-to-end tests live in separate repos. Just because we have one giant repo really and we don't wanna pollute it with end-to-end tests. And we want to keep it separately from our source code. I think it makes sense. So it's all separate repos. We have separate repos for end-to-end utilities for Cypress, which can be reused from other things. And use internal npm registry to import utilities as well. But we found that to be optimal choice.

KENT: Interesting. So when you, like are running a build on your project file, then do you install the Cypress test to have them run on CI? Is that kinda how it work?

GLEB: So end-to-end tests happen after deployment, right? So when our Jenkins build everything, it runs unit test. It runs Mocha, AVA, and also we have a bunch of Angular. So we just test it using Karma. Once everything passes, it deploys things to a sever. And then it triggers GitLab CI job and says "Okay, end-to-end test. Run on this." And so a bunch of test jobs will be ran in parallel, testing the new deployment. And I think that's a good strategy, right? You want to deploy something and then run end-to-end tests and kinda separate the two steps. And we're kinda taking it now further. Like I'm working on a tool that can test any deployment that's done using, if you have heard of a tool from Zite called Now, right? An instantaneous, immutable deployment. So we have now pipeline that can install using Zite Now. You get the new URL when we run Cypress end-to-end test. And if everything passes, we switch for DNS alias. So Cypress became kinda our go-to thing that we run separately after a deploy.

KENT: Okay. That's interesting 'cause the way that I envisioned it was you would run like your unit test and then you would fire it, like build your app and then fire up a local server of that local app, test it was Cypress, and then do the deploy. Do you deploy to production before you run your Cypress test, or do you have different environments that you deploy to?

GLEB: So I completely agree with you for smaller projects with a kinda standalone and self-contained. As part of a build, we do fire up local server, we do run Cypress test. But for a large project that requires a huge stack of services, we actually deploy somewhere and we deploy to different stacks before we deploy to production. So we actually go through several stages and we're testing each one using Cypress before we actually deploy it to production or deploy production at a good stack. So it's all about immutable infrastructure. You know, deploying fresh stack, running Cypress end-to-end test, and then pointing production DNS.

BRIAN: Very cool. The one thing I wanna to say is so most people when they think about sort of adopting Cypress, they think about it as like, "Oh this triple equals Selenium. This is like just for end-to-end tests." So if you think of it like the testing pyramid, testing triangle, you got most of the units at the bottom followed by integration, followed by end-to-end. And with Selenium, you will only ever be able to do the tippy top tier. And when Selenium runs, you have no control over the environment, you have no control the browser. The browser's just gonna do exactly what it's programmed to do. And when it makes a request to that server, that server better be there, right? But with Cypress, it's actually much, much, much lower.

And, in fact, while you can do just pure end-to-end testing, it's sort of like you're kinda giving up on all the most powerful features. I really strongly believe that only a fraction, a tiny fraction your test should actually be end-to-end smoke test. Like maybe around logging in and maybe like that your website is like up and running. And that should be it. Mostly everything else should just be like pure integration tests, where you were basically controlling the exact data that your application needs. You're never talking to a real database, you're just stubbing out every single request, you're forcing it in that position. And it just solves so many issues if you don't deal with the backend, if you don't deal with like having to seed the database with the (mumbles). Just think about like Pagination, you know, like if you like wanted to truly test Pagination. Or like search results or anything like that. It makes no sense. It's not scalable. And so what we'd like to see happen is really just show people how to do that to where they're sorta pushing down the stack. They're writing less, they're running tests, they don't depend on so many other things.

And we really would like to solve the unit testing scene as well. And when I say solve, it won't really ever solve. Like there's obviously advantages to running unit tests in Node that the browser can never replicate. Like parallelization is basically impossible in a browser because there's really only one DOM, there's only one URL. There's things that you're bound to one of. And even spinning up multiple browsers on the same machine, it's not the same thing because operating systems have, you know, they understand what application is in focus, and that changes the behavior the browsers. So really parallelization can only ever happen at the operating system level, which is what what Gleb does with his docker containers. Anyway, but I digress.

But we really would like to also solve the unit tests. It wouldn't be for everything, it would be for everyone. And it would be slightly less flexible but that's something that we also like do as well. And we sort of built Cypress with this concept that, you know. Right now, it enforces a certain amount of lifecycle around your test, like it insures like between test that things are cleaned up, like that the application is new and like the cookies are new, the local store is new. Because really that's like how you should be writing the tests, not dependent on one another. But obviously when you're writing unit tests, well there's no concept of this and in fact you might not even render anything to the DOM at all. So, and it's weird. Like we've even seen like some projects sorta do like a hybrid approach. Like some people, they builder spec files with their react components render directly in the spec file. And what they do is they end up like rendering just that component to the application area inside of Cypress. So it's like this weird hybrid-like, it's not quite a unit tests. But it's also not just like a pure integration test. You're just taking a tiny slice of your application.

So all I'm saying is what I'd like to see is like that there's so much more power and flexibility that Cypress offers you, that there's so many different ways to take advantage of it. Where, yeah, it might not be like as raw and raw speed terms as fast as node and it might not, you know, might not ever work for projects that need that. But the same time, you know, you don't have to use it purely as Selenium. You can like use it to control more of what's actually happening on the page and write more unit-y type tests. That's all.

KENT: Cool, yeah. Sounds like a pretty powerful tool and something that I'll probably look into. I often, when I find cool tools like this, I try to think of one of the libraries that I built that I can test it out on. And is thinking, "I can do it on the JavaScript Air website". But that's like pretty limited. In fact, there's like no JavaScript running on JavaScript Air. Now that I think about, it seems wrong. (chuckles) But yeah, cool. Maybe I'll give it a shot anyway. I'll just like try clicking on a link and making sure it takes me to the right page or something. Cause yeah, I want to play with this tool.

So were coming down on our time. Is there anything else that we should make sure we bring up before we? Oh, and I actually, I forgot to mention. Anybody watching live, if you have questions about Cypress, feel free to ask on twitter with the hashtag #jsAirQuestion and we'll get to those. We don't have any right now, so if you ask, you're very likely to get an answer. But yeah, while we're waiting for those to roll in, is there anything else that we should talk about Cypress before we wrap up? I'm sure there's like plenty of stuff we could really get into like really deep. But yeah. It sounds like Cypress is pretty awesome. And people should go check it out. Actually, what is the best way for people to learn Cypress, who're getting started in Cypress?

BRIAN: Yeah so, I mean it can, not great because we're sorta doing these podcasts. But were in like a closed beta right now so you can't really even get access to it. And that's actually a lot to talk about as well. I think most people, you know, they assume you should just like ship early and often, you should just have this out there. And we do have, we have like hundreds of active users where we sorta like handpicked these people that are good fits. We have a flood of feedback that comes in, but we also know that there are a lot of things that it's just not ready for and it's not solving super well. And we want to release it when it is like ready to be consumed. Not perfect, there's gonna be plenty of bugs and whatnot. But mostly where like there's enough documentation where there's a clear path to success and, you know, that will relieve a lot of the support needs from us.

So it's been difficult balancing that. Like on one part, I'm like I just want release and get it out there and start getting community adoption. But we have to show a path to, well, how would you contribute to this. But there's a lot of like infrastructure that would have to have in order to like support a lot of users using this so it's been a challenge. But to answer your real question, you just go to Cypress.io. You can sign up, you can get on our newsletter which we send update to. We also have, if you sign up for the newsletter, you're gonna get a link to fill out an early adopter form. We look at every single one of those and we invite people off that.

Also, protip. If you just come into our Gitter channel, we will just invite you basically immediately. Anyone that comes in that sort of seeks us out is obviously like, you know, sorta jumping through hoops to get it. And we unlock anyone that comes into the Gitter channel. And that's actually what we prefer. We want people to come in. We don't want you to just use this and either have a good experience and bad experience and go away. We want to talk to you. We will do screen shares when we diagnose bugs and things like that. We're looking for sorta early adopter champions that we can have honest conversations with as we get closer and closer to open source.

KENT: Okay. So that's just like the paid product version, right? Like I was under the impression that I could just like npm install Cypress and start using it today. Is that not right?

BRIAN: You can, but when you go to login, it will not enable you to work. But you can look at all the source code. It's all, you know, compiled into an application but, I mean that's an artificial limitation and it's just like, it's just the way that it works today. Like, we actually had to do work to like restrict you from being able to log into Cypress. It's kinda ridiculous, you know. But that's just what we do to basically control the floodgates. 'Cause we've had a lot of submissions. Like thousands and thousands of submissions. And so we're pretty sure like we finally like get this going, we're going to be completely inundated and we're not gonna be able to respond to all of it. And then, you know, people are gonna get frustrated. It's weird. Being in this community and seeing a lot of projects sort of come up.

I took a lot of lessons from a lot of different communities that I've been a part of and I will say kinda openly that I believe that one the reasons that Ember never caught on as well as the other projects. I mean Ember's a phenomenal tool. So comprehensive. It's because they released too prematurely, to soon without documentation and the constantly pivoted. They constantly make changes. And what happened was is that when you tried Ember in the early days, there was not a clear path to success. It just ended in frustration or you would build your app in one way and the rug would be pulled underneath you. And I cannot express how immensely complicated it is to write Cypress. Our code base is massive, it's already over a hundred thousand lines of code. And every single time we make a change, it causes us to do so much work to basically have like a migration path. Like there's so many things that we just do like even when we rename a config variable, like we write a migration of where existing projects will automatically like move up into the new way of doing it. Like when we deprecate a feature, we write explicitly like what you used to write this signature and how you'll now write in the new one. Like so much of our work goes into just making this like a good experience. And we can not, we just cannot possibly manage. And the project is just not ready to be open-source yet. It's just really that simple. But it will be. We're a lot closer to it now than we ever have been.

KENT: Okay, cool. Good to know. I wasn't aware of that and that it wasn't just like usable right now. I'm actually a little bit curious though. So even if I am not gonna be paying for the product, I'm just gonna use the open-source version, I still have to log into the desktop application? Can you explain why it's not so necessary?

BRIAN: It's a completely artificial restriction, right? Like obviously, that, when we open-source, that's one of the first things that's gonna be immediately ripped out, right? Which is like obviously, we're not gonna require anyone to have. It's gonna be complete. It's gonna make sense, right? And then we will require you to login only when you're like ready to basically set up a CI part and you want us to record results. Like we're gonna push the login back to only requiring around the thing that actually looked legitimately like requires logging in, right? When you're finally ready to run on Circle CI, you gotta like sign up for an account, right? That you write your test all ahead of time. That's all I mean. There's still lots of, there's still lots of things to shift and change about the product to best support the story that we're trying to tell and like the path to success.

We already have some like 120, 160 pages of documentation, but we don't have screencast that show you how to debug in Cypress. There's so many things that we want to have ready to where like people start using this, they can be can be successful on it. They can like see it working and they can understand even just how to contribute, how to like build Cypress, what the development flow is like, so. We're waiting for all those things at the same time. You know, we're out there. The users that use us give us a massive amount of feedback. And so we feel pretty good. Like we're not building this in a blind box or whatever. We're constantly reevaluating what we're doing based on the feedback that we're getting. So it's tough, it's tough balancing these two things.

KENT: Cool. Okay, the make sense. So we are down on our time. We do have one question from Marcel Barrus, or Ba-rose. The question is "will Cypress.io support promises and async awake?" I'm assuming that your answer's going to be yes, transpile first. Or the browser needs to support promises

BRIAN: Yes. I mean we support everything that JavaScript supports, we support everything that the browser support 'cause it's all executed in the context of the browser. So you can use whatever you want as long as the browser can execute it. Our DSL that drives the browser, right? Everything is a promise, everything is a sync. That is a design decision so that everything can be retried and it can be leisurely figured out. Surprisingly, when you're using our DSL, there's not a lot of programming that goes into Cypress. We sort of simplified the process to where you don't, your code shouldn't look like gnarly. It should be fairly readable, straightforward, and understandable, even if you've never seen it before. So, yeah.

KENT: Cool. Awesome. Maybe uh. No. That was a terrible idea. I'm not even gonna say it out loud. Okay, cool. (chuckles) Great. That's it. So let's go ahead and we'll jump into our tips and picks. So I'll go ahead and go first, and then Gleb and Brian, you can give your tips and picks. So first my tip is if you're ever going to be using Hangouts on Air, don't forget to set your audio input setting. I totally botch this at Strange Loop and last week show, if you're listening to the audio podcast, you're gonna get a rerun. If you watch the video, you're gonna see why. I totally forgot to set my audio input to be the microphone or like the microphone interface that I was using and it was using my MacBook built-in microphone, which it was like really bad because we were sitting far away and stuff. So I'm really, really, really, really sorry about that. I feel so bad but that's, that's my tip. Just don't forget.

And for my picks, I have two. The first is one of my favorite talks at the Strange Loop from last week. It's called "Idealized Commit Logs: Code Simplification via Program Slicing" by Alan Shreve. And this was an amazing talk. Alan Shreve actually is the guy who built ngrok, which if you have it used, you should. It's fantastic. You should look at it. But yeah, program slicing, totally new concept to me. And now I'm building a tool in JavaScript to add program slicing so that you can understand your JavaScript programs better. And it's, yeah. I'm super, super, super excited about it. It's a lot of fun. And yeah, I could talk about forever, but this is end-to-show about that.

So then my next pick is a tool that I built on the plane on my way to strangely called Split Guide. So at Strange Loop, I gave an ES6 workshop. And one of the cool things about like topical workshops is you can have like a folder and a file for each topic. Like "Okay, here are the test for this topic. "Make these test pass and if you do that, then you learned the topic." Right? But you want to have a final version for people to reference. And so you have like exercises folder and an exercises final folder. Well the problem there is like maybe you're working in exercise final and then you go and like copy and paste it over to exercises folder, then remove all the solutions. And that's really tedious to do if you're making updates. And so Split Guide basically does that for you. You have a templates folder, you have some special comments in there. And then Split Guide will generate the exercises and exercises final for you. But you can keep things in sync lot easier. So it's actually really, really cool, and if you've ever given a workshop before, like a topical workshop, then this is a really fantastic tool to help simplify that. So those are my tips and picks. And Gleb, can you give us what you're thinking of?

GLEB: So I have one tip. It's that if you use Graphs Reporting on monitoring service for your frontend code like TrackJS, or Sentry or Raven, it becomes really, really useful during end-to-end tests. Like when Cypress runs, it actually runs as a user, or as if it were a user. So anything that goes wrong that's not visible in the UI, that the user probably has not noticed, will probably will become visible in your error exception logs. So it became really, really useful, you know, to have crash reporting during end-to-end testing.

And my pick is only one. It's kinda similar to your Split Guide. It's a website called katacoda.com. And you go there and it's a free training where you can do Docker and Kubernetes training, online, step-by-step for free. And it's amazing because you're actually execute commands. You'll run in a full environment. And there are many, many tutorials and many, many exercises. So it's great for learning Docker. Right.

KENT: That's fantastic. I need to learn that. (chuckles) I still have no idea what this Docker thing is all about. Cool. Brian.

BRIAN: So my one tip I'm actually super stoked about the latest GitHub changes that's come around, the new project tab. Now we have a kanban board, however you pronounce it. We're like gonna start using the heck out of this and actually show people what's coming up on our roadmap. And they can actually see the progress of all these different issues. I mean, obviously, we used external tools like Simple Tracker or Jeer or whatever. But like the integration into GitHub, because that's where everyone goes to. I mean it's just really great. I'm super excited. And everyone should probably start using it immediately. The only thing I'm a little concerned with it's like now we have like milestones. Can we have projects it should just like ditch milestones altogether and just use projects? But I don't know. We'll see.

What else? I would just say Electron, if you haven't checked out Electron, Electron can absolutely change your JavaScript life. It's sorta like the other leg of it. We've had the frontend obviously, we have the server now. And now you can just do completely native desktop applications. That can be a total game changer based on what you're trying to build. I cannot express how easy it's been. And without this, none of Cypress would really be possible. So check out Electron.

KENT: Sweet. Alright, I've been meaning to do that too. There's just so much stuff that is awesome in this world! Okay great. So let's, I'm gonna wrap up with a couple closing announcements. I wanna give a special shout out to our silver sponsors. First, ReactJS program helps you master React and React Native. Find them at reactjsprogram.com. And Sentry gives you cross-platform crash reporting so find them at century.io. And then a couple of links for you. If you have suggestions for topics and guests, either or, go to jsair.io/suggest. And then if you have feedback for the show or specific episodes, go to jsair.io/feedback. And if you want to get in on our newsletter that we send out weekly about each show, go to jsair.io/email.

And I totally forgot to announce at the beginning, but we do ask about web components, so it's gonna mean a good thing. I'll be honest, I'm in the skeptical crowd. And so I'm looking forward to chatting with some people who actually know a thing or two about the web components specification or specifications, I guess. We can actually talk about the benefits and use cases of web components. So yeah, that's it. And that'll be same time, same place next week. Yeah, so that's our show. Thanks so much Brian and Gleb for coming on, and we'll see you all around the Internet.

BRIAN: Great. Thanks for having us.

GLEB: Bye guys.

KENT: Bye bye.