*record scratch* *freeze frame* Yup, that’s me. Sitting in front of my monitor, crying, because I have to do everything again. You’re probably wondering how I ended up in this situation. Let’s go back to the start. I wanted to test our infrastructure setup and to do so, I need to boot up a cloudformation stack and access its resources to call an API Gateway.

 

I did all that. Some things had to be adjusted. I did it again. Some time passed. Some more things changed. I need to do it again. How do I have to call everything again? How does that set up work?

 

When performing potentially recurring tasks, it’s a good idea to automate them. Deployment scripts, test scripts,… Luckily the AWS CLI allows for relatively easy automation of those tasks. This blog post covers some easy ways of obtaining the required information to work with the CLI in scripts.

 

Prerequisites

• A Unix-Shell
• Installed and set up AWS CLI
• Ensure that the AWS CLI is fully configured
• AccountID and Default region need to be set up

Automating Region and Account ID

The region and account ID are often used in identifiers, resource names (ARN) and URLs when working with AWS. For example, the Elastic Container Registry has the following url:

${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com

To simplify working with those, we can automatically obtain the two variables based on the cli configuration:

# get current AWS Account ID
ACCOUNT_ID=$(aws sts get-caller-identity --query 'Account' --output text)
# get current AWS REGION to use in URLs
REGION=$(aws configure get region)

The call to the Security Token Service (STS) is a workaround to obtain the account ID, and it already showcases the query parameter, but more on that below. The region is simply read from the local aws cli configuration.

 

These things are required for some parameters, but can also be used to construct URLs that are built by convention. For example repositories in the Elastic Container Registry (ECR) are constructed this way

${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com/${REPOSITORY}:${TAG}

 

Your friend and helper the query

There is no way of obtaining the URL of an API gateway directly via the CLI. However, it can be constructed easily, just like the ECR registries.

https://${REST_API_ID}.execute-api.${REGION}.amazonaws.com/${STAGE}

We already know how to get the region, and we have to know the stage anyway, but how do we get the REST API ID? There is the `aws apigateway get-rest-apis` command which lists all the APIs as well as their IDs, but how do we get the ID we want?

 

Query parameter to the rescue

Every AWS command can be called with a `–query` parameter. This allows us to query the json output, getting only the specific information we want. Furthermore, it can be combined with the `text` output-format, which results in only the queried result being returned, and nothing else. Perfect for further processing.

 

Here is an example output when running `get-rest-apis` without additional parameters:

{
    "items": [
        {
            "apiKeySource": "HEADER",
            "name": "foo",
            "endpointConfiguration": {
                "types": [
                    "EDGE"
                ]
            },
            "version": "2016-07-26T07:34:38Z",
            "createdDate": 1549960338,
            "id": "04hui93ben"
        },
        {
            "apiKeySource": "HEADER",
            "name": "bar",
            "endpointConfiguration": {
                "types": [
                    "REGIONAL"
                ]
            },
            "version": "0.0.1",
            "createdDate": 1544864463,
            "id": "si[4htmoeu"
        },
        {
            "apiKeySource": "HEADER",
            "name": "baz",
            "endpointConfiguration": {
                "types": [
                    "REGIONAL"
                ]
            },
            "version": "0.0.1",
            "createdDate": 1544866973,
            "id": "abcd123456"
        }
    ]
}

Now let’s say we want to construct the URL of the `foo` API:

aws apigateway get-rest-apis --query 'items[?name==`foo`]'

Note that the name of the API is wrapped in “. What we’re doing here is that we access all elements in the “items” structure (see JSON above, ‘items’ is the label for the top level object) which have the name ‘foo’. We could also use wildcards, a * would give us all elements, but that’s not very useful on its own since it’s the same output as no query at all. The interesting part is combining the query with object traversal. For example, this snippet here returns all of the items’ names.

> aws apigateway get-rest-apis --query 'items[*].name'
[
    "foo",
    "bar",
    "baz"
]

Finally, let’s construct a query to get our API ID:

aws apigateway get-rest-apis --query 'items[?name==`foo`].id' --output text

Putting all of the above together, we can query the API to obtain its ID. Furthermore, we change the output format from JSON to text. This way we do not get a wrapped object, but only the ID itself. Here is the complete snippet for constructing our API url:

STAGE=test
REGION=$(aws configure get region)
REST_API_ID=$(aws apigateway get-rest-apis --query 'items[?name==`foo`].id' --output text)
API_URL=https://${REST_API_ID}.execute-api.${REGION}.amazonaws.com/${STAGE}
curl ${API_URL}/helloworld

For more info on the query parameter, check out its AWS documentation.

 

Filtering Results

In the first section, we built ourselves the ECR repository URL, so we have an automated way of obtaining all the info. That’s nice, but what if we could just ask Amazon directly? The REST API Gateway above is actually the exception, most resources can be obtained directly through the AWS CLI. Some AWS CLI commands allow filtering the response in addition to the `–query` parameter, which is a global parameter and works for all commands. They either support a `–filter` parameter or provide other parameters that allow limiting the response.

 

Let’s take a look at the ECR URL again. Instead of constructing the URL ourselves we can obtain it by querying the repository information itself:

aws ecr describe-repositories --repository-names ${REPOSITORY} --query 'repositories[*].repositoryUri' --output text

`ecr describe-repositories` returns a listing of all repositories and their information. The command does not support `–filter`, but provides an alternative with `–repository-names`, allowing us to only obtain the information of specific repositories. We limit the response to a single repository, and then query the URI field of the response. Without limiting to a single repository, we would obtain a delimiter separated list of all repository URLs.

 

Summary

To recap everything in a few condensed lines:

• Automate where possible, it makes life easier
• Use the CLI commands to obtain as much information automatically, removing the need to hardcode IDs and URLs
• Use –query to obtain only the info wanted from the command output
• Use –filter or whatever parameter the command provides for limiting the output

Bernhard Bonigl, Software Engineer at viesure