Logo Fady Nagh's Blog
Securely Accessing Private AWS Resources with TailScale

Securely Accessing Private AWS Resources with TailScale

November 10, 2023
8 min read

The Problem

One of the common problems of CI/CD automation is how to handle private resources without exposing these resources directly to the public or relying on complex VPN setups may not be the ideal solution.

Typically, you go with something like OpenVPN, but OpenVPN can be complicated, with something like OpenVPN,you have to set up your own server, and configure, maintain, etc. On the other hand, it takes less than 2 minutes to set up TailScale, Another thing, if you live in a country where OpenVPN is blocked, you don’t have many options.

diagram

Here is the diagram that we are going to be implanted in this blog, the blog post will guide you with step-by-step through its implementation. The initial steps involve setting up a GitHub Actions workflow that installs the TailScale client and authorizes it into a TailScale network. Subsequently, the blog post will cover the creation and configuration of the TailScale router, ensuring proper traffic routing and security group settings on AWS.

Connecting GitHub Actions to the Tailscale Network

Before starting this guide, you’ll need to have a Tailscale network set up and configured with at least one existing device, If you do not know how, follow this guide and of course you’re going to need an AWS account as well.

Now the next thing that I’m going to do is go out to GitHub and create a new repository,You can find the code for the end result in this repository.

You will then need to create an OAuth client to use with the Github action via this link, copy the Client ID and Client Secret to a safe place for now. image2

Go to the GitHub repo you created earlier and go to Settings > in the Security section of the sidebar, select Secrets and Variables, then click Actions, Create a GitHub secret with the name TS_OAUTH_CLIENT_ID and assign your OAuth client ID as the secret value and name TS_OAUTH_SECRET and assign your OAuth client secret as the secret value.

image3

You will then need to create a server role account using ACL tags in Tailscale, basically ACL tags allow you to assign an identity to a device that is separate from human users, and use that identity as part of an ACL to restrict access. This should be used when adding servers to your Tailscale network, to learn more about ACL tags, click here

Open your favorite editor and create a .github directory and underneath .github we’ll create a Workflows directory, this is the special directory that GitHub Actions is going to look for workflow files in, create a new workflow, I’m going to call it tailscale.yaml.

image1

And inside of this file, we’re going to go ahead and set up our basic workflow structure.

tailscaleAction.yaml
on:
  workflow_dispatch:
  push:
 
jobs:
  deploy-via-tailscale:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Tailscale
      uses: tailscale/github-action@v2
      with:
        oauth-client-id: ${{ secrets.TS_OAUTH_CLIENT_ID }}
        oauth-secret: ${{ secrets.TS_OAUTH_SECRET }}
        tags: tag:ci
 

Let me break it down for you.

  • the workflow dispatch event. Allows us to trigger workflows manually, which is great for debugging.
  • The push event means that whenever we push a change, it goes ahead and triggers a build.
  • Define the job and name it deploy-via-tailscale. It will run on the latest release of Ubuntu.
  • connect to your tailnet by using the Tailscale GitHub Action

Go ahead and push the code to Github, you should see an ephemeral node. The node can access nodes in your Tailnet, subject to the access rules that apply to the specified ACL tag(s).

image4

Set Up Amazon VPC and Tailscale Router

What we’re going to do is we’re going to create a VPC, and then we’re going to have this public subnet, and we’re going to put an EC2 instance on that public subnet, and we’re going to have that EC2 instance connect out to our Tailscale network as well, and then we’re going to set that EC2 instance up as a router.

image5

Let’s go to the AWS Management Console. First, we’ll create an elastic IP and EC2 instance on the public subnet, which will act as a subnet router.

image6

image7

After the instance has passed the checks we need to download and install Tailscale onto your subnet router machine. image8

Step 1: Install the Tailscale client

ssh into the instance and follow the instructions

Install Tailscale
curl -fsSL https://tailscale.com/install.sh | sh

image9

Step 2: Enable IP forwarding and Advertise subnet routes

Enable IP forwarding
echo 'net.ipv4.ip_forward = 1' | sudo tee -a /etc/sysctl.d/99-tailscale.conf
echo 'net.ipv6.conf.all.forwarding = 1' | sudo tee -a /etc/sysctl.d/99-tailscale.conf
sudo sysctl -p /etc/sysctl.d/99-tailscale.conf

image10

Advertise subnet routes
sudo tailscale up --advertise-routes==10.0.16.0/20 --accept-dns=false

Replace the subnets in the example above with the correct ones for your network, it will give you a link that you will need to go to in order to authenticate. images11

The EC2 instance should appear in the Tailscale console.

images12

Step 3: Enable subnet routes from the tailscale console

Locate the device that advertised subnet routes, from the ellipsis menu at the bottom of the table, select Edit Route Settings. This opens the Edit Route Settings panel.

images13

Everything should be working to test connectivity, we will add test connectivity to the Github action, commit and push the code.

tailscaleAction.yaml
on:
  workflow_dispatch:
  push:
 
jobs:
  deploy-via-tailscale:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Tailscale
      uses: tailscale/github-action@v2
      with:
        oauth-client-id: ${{ secrets.TS_OAUTH_CLIENT_ID }}
        oauth-secret: ${{ secrets.TS_OAUTH_SECRET }}
        tags: tag:ci
    - name: test-connectivity
      run: |
        ping -c 10 100.73.32.96

Replace the IP address with the IP address of the subnet router.

images14

Create Private Amazon RDS Instance and Test Connectivity

After reviewing and verifying, we will provision an Amazon RDS instance in our VPC and test connectivity from our GitHub Actions workflow.

First, we will create a security group that will allow connection from the subnet router, I’m going to call it the database security group.

images15

Note

In a production environment, it is bad practice to allow all connections from a security group, Only allow access from trusted sources for specific ports and protocols, following the principle of least privilege.

I will not cover how to create an RDS database here. I assume you have already created an RDS database, please refer to the AWS documentation if you don’t know how to create one.

images16

If you want to test connectivity between your machine and the private Amazon RDS instance type the following command. This action allows you to connect to the MySQL DB instance using the MySQL client, Replace endpoint with the DB instance endpoint (DNS name) and replace admin with the master username you used. Enter the master password that you used when prompted for a password.

Test Connectivity
mysql -h endpoint -P 3306 -u admin -p

images17

If everything is configured correctly, you should be able to see this massage. images18

We will return to the Github action workflow to test connectivity, we will create three secrets for the database endpoint, password, and username.

images19

Update the workflow with this to test connectivity. We can now connect to RDS from any node in the Tailnet, using the same DNS name used inside AWS.

tailscaleAction.yaml
on:
  workflow_dispatch:
  push:
 
jobs:
  deploy-via-tailscale:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Tailscale
      uses: tailscale/github-action@v2
      with:
        oauth-client-id: ${{ secrets.TS_OAUTH_CLIENT_ID }}
        oauth-secret: ${{ secrets.TS_OAUTH_SECRET }}
        tags: tag:ci
    - name: Test-Connectivity-to-AWS-RDS
      run: |
        mysql -h ${{ secrets.ENDPOINT }} -u ${{ secrets.USER }} -p"${{ secrets.PASSWORD }}" -e "show databases;"

images20

Conclusion

In this blog, we’ve demonstrated how to create a secure and seamless connection between GitHub Actions workflows and private AWS resources. This approach not only improves the overall security of your CI/CD pipeline, but also ensures that sensitive data and interactions remain within your network, TailScale is easy to use, and has robust encryption mechanisms.

Note

This is not sponsored or endorsed by TailScale, I use the TailScale at work and in my home lab, I am just a happy customer. One more thing, you can host Tailscale Control Server yourself if you want, which is a plus.