HashiTalk Recap: Writing Your First Waypoint Deploy Plugin

Seaplane team members Fokke Dekker and Bram Vogelaar gave a talk for HashiTalks 2023 about their work on a Waypoint integration for Seaplane. You can read our transcript below, or watch the full recording on the HashiCorp YouTube channel.‍

Transcript

Fokke Dekker: So my name is Fokke Dekker and I'm part of the customer team at Seaplane. I'm also our resident Golang noob and before doing this I'd never used Waypoint before. And you might be wondering why I'm telling you that, but I think it's important to know that even if you feel like you don't have the required background knowledge, writing a plugin for HashiCorp Waypoint really isn't that complicated. There's great documentation, there's great people that might be able to help you, so if you feel like you want to do it just jump in and build one yourself.

I'm originally from Amsterdam. I moved to California in 2018. So it's 6:30 local time for me, which is pretty early [laughs]. If I look a little bit tired if I talk a little bit slow, that's probably why. I haven't had my first coffee yet.

Bram Vogelaar: That brings us to me! Hello I’m Bram, for the English speakers. So I actually went to university be a molecular biologist and over the years I have slowly descended to the dark side to be in operations. Since January 1st I joined Seaplane to do cloud engineering, started off as a community member ambassador, and now I’ve joined the club.

This is my fifth year speaking at HashiTalk and I'm very, very excited to do a talk today!

Bram Vogelaar:  Let's start by introducing Waypoint. Waypoint is one of the newish tools that the lovely people at HashiCorp came up with. It’s in the developer experience space and it allows you to build workflows in a language you should be already familiar with — HCL, the HashiCorp Configuration Language. We've used it in Packer, we've used it in TerraForm just today in the previous talks, and you can extended with your own plugins. So if the tool doesn't come with the thing you want to deploy, then you can write your own. That's how we ended up with Waypoint.

We probably should introduce the vendor company at the moment. I promise it’s not a vendor pitch, but we need to set up the story for today.

Bram Vogelaar: Seaplane is a multi-region, multi-cloud platform that helps development teams basically abstract away all the difficult stuff when you want to go global. When people in America wake up, that's where you want to have your workload, while Asia goes to bed. There's no reason to have it everywhere. And that's what our heart is, that's the platform we've built.

Bram Vogelaar: So if you go to onboard on Seaplane, you give us your Docker image and we make sure that it lands fairly close to where your users are. We're in quite a few locations in the world, as you can see, and we're on pretty much any cloud provider, including some of the edge providers. Why do we do this? As I said, we want to abstract away all the difficulties of going global. So we came out with CLI and API, even an SDK, so whatever you want to use, we'll provide it to you to stick with your standard developer experience.

But, we want to go one step further and provide you with the full CI/CD developer experience integration. So, how do we get to Waypoint?

Bram Vogelaar: [We’re working on a Mac so] basically it's as simple as a brew install. Then when you run it locally, there's a one liner on the screen that you can use to just get it going if you have a Docker running and then we can get to the UI. If you want to have access to the other ways you can install Waypoint, there's a link on the screen right now.

Bram Vogelaar: What are we going to deploy today? We are going to have a simple little Python application that has three routes: it gets a 200, it shows you a seaplane — we're gonna stick with the aeronautics theme today — a 404 will [show you nothing] and the 500 shows a crashed plane. Don’t worry, the people made it out safe. If you want to have a look at what the application is all about, there's a link on the screen right now.

Bram Vogelaar: Then the second part. We have Waypoint, we have it installed, we can load something on it. This is actually what we need to model today into our Waypoint deployment. Seaplane has the notions of “flights” and “formations.”

A “flight” is basically a one to one map to your image. You can build locally- like you can do with any Docker image- then you push it to our registry and then we deploy it using a formation. And for today, [because] naming things is hard, our flight is going to be hashitalks-flight and our formation is going to be a hashitalks-formation.

When we deploy the hashitalks-flight you must believe us when we say we ran it locally and it was successful. There’s a URL where we can go to check if it's works. So you're gonna see your local pub somewhere in the world. For me, it's going to be Amsterdam, for Fokke is going to be probably San Francisco or Los Angeles, and for you it will change. If you deploy, it will take a while for the platform to converge but this is the simple notion of Seaplane.

Bram Vogelaar: How do we map this process into Waypoint? Waypoint has a basic three step [process] to get your deploy in front of customers. There's a build step, where we're going to build our container and push it to a registry. Deploy, in this case it goes out to us. Then the release of the application. So you can potentially do things like blue-green or canary releases, or you flip a load balancer onto the new deploy.

Bram Vogelaar: Today, we're mostly gonna show you the deploy section of writing this Waypoint plugin which includes deploying in your application, getting update status in the Waypoint UI and/or in your CLI. The Docker build is actually fairly simple. It’s the standard build step you're used to, just wrapped in a little bit of Waypoint.

Bram Vogelaar: The lovely people at HashiCorp have already have an example repo set up for us, which you can just clone CD into the right subdirectory. There's a clone script. Then you take that code and you take that into your own Git repo and you start developing from there using your local Waypoint setup. Or, if you have more elaborate setup, you can run it inside your Nomad cluster. But today, we're going to keep it simple and run it on our local machines.

The Waypoint Integration Framework has bare bones code set up already for the three steps that’ll be discussed. If you drill into deploy of the platform [we see it’s split up into three things]: deploy, destroy, status. The things we're interested in.

Bram Vogelaar: For this deploy step, we need to have a Waypoint file that we can call using the Waypoint CLI. We're gonna give it some input, because we are going to share some secrets — that's how we communicate who's who and to deploy our code. So we're going to have a secret, which we're not going to commit into our code- we just feed it as a variable. We're going to push our image. We're going to deploy it to Seaplane. We’re going to get that status, whether or not it was successful or if it's still converging. Then we're going to implement destroy because when we live in the cloud, the way you spend the most money is by not turning things off. We made really sure that we implemented this destroy function so we can clean up after ourselves.

Bram Vogelaar: When we looked at building this plugin, we sort of identified three maturity levels. One is recall. We basically drop out away from the local exec to our Seaplane CLI and then we call the Seaplane CLI. The next one we have we'll wrap it around in our plugin, but still call the local CLI. And the third one will directly call the Seaplane API from the framework so we don't need to have the Seaplane CLI somewhere so we don't have to build a custom image as a Waypoint runner.

Bram Vogelaar: When you start with this maturity level one, local exec, it actually wasn't too hard. You have to deploy the resource in your Waypoint file with this local exec you know, we make sure that it's all commented out properly in our ACL file, and then we can actually run it.

Bram Vogelaar: [Well] we could sort of run it because as we discussed before, Seaplane deploy actually just takes two commands to get stuff running. We need to declare a flight, then we need to start the flight in our formation.

There is no easy way to do two local execs in a deploy section. So that's why we got a little bit stuck. [Because] it's a local exec we can't really pass around variables, which is a bit annoying. We don't actually get [a very good] update status on these deployments. As I mentioned, this CLI still needs to be somewhere present, needs to be up to date of course. LCM is always an issue, and the validation of these these parameters is not always what we want.

As a plugin (it’s not really a plugin yet) we’ve shown to ourselves [that] we could deploy to Seaplane with Waypoint. But, we want to do better and this is where I’m going to hand it off to Fokke for level two and level three.

Fokke Dekker: Thank you! As Bram said, we can still do better.

Fokke Dekker: The [maturity level one example] wasn't really a plugin. It also had some issues around status reporting, it had some issues around actually deploying as it was taking two arguments. What we're going to do now is we're going to use the same tactics, so deploying through the CLI, but integrated into the framework, which we thought would be probably the easiest way to get things done without writing too much code. I think we're all probably a little bit lazy and so we just wanted to get it running as quickly as possible [laughs].

We're still using the standard Docker builder — we're going to do that throughout all of these steps — that pushes the app to the Seaplane registry. But, we are going to add a couple of required variables to our HTML file- our Waypoint file- that we’ll use as input arguments for the CLI. If you remember the steps that Bram showed before, when we were looking at the flight creation and the formation creation through the CLI, what we need there is a formation name and a flight name and an image URL. If you look here at the deploy, block in the middle, you can see that those variables are available there.

The last thing we're also going to ask is the Seaplane tenant ID, which we’ll use in the image name to identify which image belongs to which tenant. So with that, we can then start implementing the v2 plugin.

Fokke Dekker: There’s a couple steps that you do in this framework. The first thing we want to do is verify the input parameters. For the Seaplane CLI, there's a couple things that we need to take into account. For example: formation name cannot be longer than 27 characters, we can only use alphanumeric characters, and we cannot use double dashes in the formation name.

The reason for all of that [is because] the formation name is part of the resource URL. So you'll have something like “your formation name.on.cplane.cloud,” and that is the URL to your app. So that's what we do in the configurable notify function in our deploy step.

Fokke Dekker: This looks very similar to the local exec so inside the deploy function we're going to execute a couple of commands. You can see that the first step is the same as before, so we're recreating our flight. Then the second step, which is now possible (which wasn't in the maturity level one plugin) is creating the formation. Once that's done, the app deployment is live.

The next step would be to implement the status function using the CLI, and you do this in the status function inside of the deploy component. This was a little hard though. The way our status update works in the CLI is that it auto updates the terminal. It’s similar to [Waypoint] where you see the updates come in, and then the other one disappears. That is cool, and it looks great in a terminal, but it's a big pain to process it. Especially in Go and especially for somebody that has never used Go before [laughs].

So that was a bit of an issue, and sort of where this development stopped. We realized that this wasn't going to be the end-all be-all for the for the plugin either, so we can still do better!

Fokke Dekker: Yes, we can still do better. It still requires the Seaplane CLI, which isn't a massive problem but it does mean that you have to install it wherever you run this plugin so it's a little bit less flexible. We're unable to report the status of the deployment in the current configuration, which means that if you would go to the Waypoint UI it would tell you that it's down all the time which isn't great.

There's also not really any flexibility to add more deployment settings. Within Seaplane, one of the things that we do is we deploy your application globally but then we give you a set of configurable settings that allow you to restrict workloads to certain regions, for instance, to comply with things like GDPR, so you can take your workload and say “this can only run in Europe,” or “this can only run on AWS or GCP.”

That is what a lot of people like about Seaplane, but to be able to integrate that in this version of the plugin, we would essentially have to have all combinations and permutations of those settings — or work with some dict where we store them and then process them — just way too much work because we knew we were going to build a third version of this which would use the API, so we didn't actually do that in the maturity level two [version].

Lastly, I think pretty importantly, we're unable to use multiple tenants or API keys because you'd have to log in to the Seaplane CLI. So we would have to create a whole other step where you would log in, provide a new API key to deploy to a new tenant, and that was a bit of a pain so we decided not to do that as well and move on to maturity level three, which is the full integration with the Seaplane API.

Fokke Dekker: So for one last time, we have to update our Waypoint file. This time, we still need the formation name, we still need the flight name, we've added the API key, and we've taken away the image URL because what we're going to is pass the image URL from the build step to the deploy step within the HashiCorp framework.

If you start to develop the your own plugin, you'll notice that between all these steps there are components that you can pass along. So if there's an image URL that you set in the in the build step, that is available in the deploy step. You don't have to ask your user for that information, which makes it a lot more convenient to use.

Fokke Dekker: Then we're going to adjust our deploy and destroy files. All the operations are going to run through the Seaplane API. We’re going to reuse the same ConfigurableNotify, because we're still using formation name and flight name and the same requirements still are true so the name can't be longer than 27 characters. So we don't have to make any changes there. But we do need to make some changes to our deploy function.

What we do here is we create a simple post request in Go and fire that off to Seaplane. So you can see that we construct first the flight, and then add that to the post request and fire that off to Seaplane. Above this code [shown in the slide] there's some code to authenticate with the Seaplane API, but I didn't think that was interesting and this is already hard to read so I decided to leave that out of here. But if you wanted to take a look at the end of the presentation, we'll share a link where you can see the full code base and [you can] contribute if you want!

Alright, then we have to implement the status function and now that we're using the REST API, rather than the CLI, we can actually do this. So inside the deploy framework there's a status function in which you can call and then report on the status of your deployment. That’s what we've done here as well. I'll talk a little bit more about that in a second.

Then finally, we’ve got to implement the destroy function, which is just another simple delete request with the formation name to take down the deployment. It’ll look very similar to this. I didn't think it was interesting to include all the REST calls, because they're basically all the same. It's just that one is a get request, one is a post request, and the other is a delete request.

Fokke Dekker: Then there's updating the status. We're using the built in status SDK provided by Waypoint. So if we get back a 200 OK from our API, we update the user in the terminal and then we set the status. So SDK, that status report ALIVE will store the status of our deployment. This means that [it's available in the terminal] and then it's also available in the waypoint UI so it will show you that the deploy was successful.

Now in the best case scenario you would update this every couple minutes. We haven't done that yet. That's on the to do list for us. Because then Waypoint would keep pulling the status of your formation and making sure that it is live and if not you'd be able to see that in the Waypoint UI moving forward. That's something that we still need to look into. We had some issues there.

Fokke Dekker: So here, we're in a simple project. We’re just going to update our HCL file, the Waypoint.hcl file. We'll give it a formation name and again, naming is hard, so we went with hashitalk-formation, and then the flight name hashitalk-flight. And once we save that and update the API key then we run Waypoint in it to initialize the file and Waypoint up with our tenant ID. This will trigger all the build and deploy steps. So the first thing you're going to see here is a simple Docker deploy as a standard into Waypoint. We've done nothing here, this is just built in. It's also sped up a little bit because it can take a little bit of time here.

Once that's deployed and pushed, it will start to deploy on Seaplane and will provide you with a formation URL that, given enough time, will show you that it's a running application. Now, this is all sped up a little bit and I've used the power of the internet to make it look better. I'll tell you a little bit more about that in a second, but here's a destroy step as well. It will ask you if you actually want to do that, and then it's going to fire off the destroy and remove the application.

I said I made it look a little bit nicer there, and we obviously sped it up a little bit because this is going to take a little bit more time than what you just saw, but one of the other things is that the Waypoint SDK- the plugin- has a built-in feature for the resource URL. We couldn't get that to work and I'm not entirely sure why. I've been talking to a couple people over at HashiCorp, there were some known bugs with that URL showing up into the UI.

So what we've done instead is we've just literally printed it as an output of the application. I think that's one of the things we still want to improve because once we do that you can just have it there. It's easier to then move on to launch the application and to run blue-green deployments and whatnot.

Fokke Dekker: That is the integration we've built! It's all up and running, you can you can look at the plugin itself with the QR code at the bottom left. The example project is going to be at the bottom right and the example project also contains a configured Waypoint HCL file. If you wanted to test it yourself with your own Waypoint installation, you can do that.

The basic integration is done, but complex deployments aren't possible yet. I talked a little bit about the features of Seaplane where you can limit workloads to run in certain regions, or don't allow certain cloud providers, [none of that is] implemented yet. But it's fairly easy to do, because it's just an extension of the JSON that we pass with the post request.

We are looking for open source contributors. Originally the plan when we were building this was that Bram, as an open source contributor, was going to build this together with me where I was the Seaplane employee, and Bram was the was the Community Contributor. But then Bram and I started to talk more and it turns out we really had a need for his skills and he ended up joining the team!

So if anybody's interested in contributing, we still think that this should be an open source project. We'd love to have your help, to get you involved. I meticulously commented all the code because I was trying to make sense of the framework myself, so even if you're a pure beginner, I'm pretty sure you can just dive in. I can give you a little bit of a walkthrough as well if needed I’m more than happy.

Email me at my email address shown there and you can get involved.

Fokke Dekker: Any questions before takeoff? If you have any questions feel free to email me or reach out over Twitter, we'd be more than happy to chat. Also, if you have questions about building your own Waypoint plugin there's great documentation as we shown before, but it still takes a little bit of time to kind of understand what's happening and the framework isn't always logical [laughs]. I've had a couple of steps where it was a little bit interesting, at least. So if you're stuck, or if you have any questions, just reach out, I'd be more than happy to help!

‍

Thank you again to HashiCorp for hosting and Fokke and Bram for their time and expertise! You can follow Fokke on LinkedIn and Twitter and Bram on LinkedIn and Twitter.

Seaplane regularly hosts and participates in talks (like this one!) so be sure to follow us on LinkedIn and Twitter for updates.

‍

Some quotes in this transcript have been edited for clarity.

‍