API Gateway

This package features a REST-API that serves as a gateway between the user realm and the backend of the climate action architecture. It provides endpoints to list available plugins, trigger computation tasks and retrieve computation results. Each computation generates a correlation id that uniquely identifies a computation request. Result retrieval is bound to these ids in a two-step procedure:

1. All results generated through a given id can be listed. The list remains empty as long as the computation is not finished or in case it failed.
2. The listed results (artifacts) provide a filename which is a unique identifier for that element. The element can then be downloaded in a second API call.

For more information see the API swagger documentation.

One example for a user realm application is the climate action frontend/web-app.

Run

The API is embedded in a full event-driven architecture. All interaction with that architecture is provided by the climatoology package. It requires multiple services such as minIO and RabbitMQ to be available and the respective environment variables to be set. The simplest way to do so, is using docker. You can use the infrastructure repository to set up the architecture. Afterward copy .env.base_template to .env.base and fill in the necessary environment variables.

Local run

Follow the database migration instructions, then start the api with

 poetry run python api_gateway/app/api.py

and head to localhost:8000 to check out the results.

Of course, you won't see much until you also launch a plugin that can answer your calls. You can try the plugin-blueprint or any other plugin listed in the infrastructure repository.

Database migrations

The backend database must have the correct schema, as required by climatoology. When running the gateway from the infrastructure repository, the database upgrades will automatically be run. Otherwise, use one of the following methods to manually run the migrations.

For more advanced alembic usage and commands, such as how to downgrade your database, see the climatoology documentation.

Docker

If you have the docker container available, run docker run --env-file .env.base --entrypoint ./run-alembic.sh repo.heigit.org/climate-action/api-gateway:devel upgrade head

Local

To run the database migrations locally, first create a .env.migration file containing the database connection specification (as per .env.base_template), then run ./run-alembic.sh upgrade head.

Managing expired tasks (DLQ)

We use celery as a task management system. The gateway sends computation requests to a specific queue per plugin, while each plugin then consumes tasks from that queue.

Computation requests may be sent with a q_time limit, which determines how long a task can wait in queue without being started. If a computation request is not started within the q_time, RabbitMQ will automatically move that task to the dead letter queue (DLQ), without alerting celery. Therefore, we need a 'DLQ handler' to ensure that computations that end up in the DLQ are marked in the backend as REVOKED. This handler is provided in dlq_handler.py.

When running the gateway from the infrastructure repository, the DLQ handler will automatically be started. Otherwise, if you require the DLQ handler, use the following command to start it:

poetry run python ./api_gateway/dlq_handler.py

Testing

We use pytest as a testing engine. Ensure all tests are passing by running poetry run pytest.

Coverage

To get a coverage report of how much of your code is run during testing, execute poetry run pytest --cov.

To get a more detailed report including which lines in each file are not tested, run poetry run pytest --cov --cov-report term-missing

Docker

The tool is also Dockerised. Images are automatically built and deployed in the CI-pipeline.

In case you want to manually build and run locally (e.g. to test a new feature in development), execute

docker build . --tag repo.heigit.org/climate-action/api-gateway:devel
docker run --env-file .env.base --network=host repo.heigit.org/climate-action/api-gateway:devel

and head to the link above.

To run behind a proxy, you can configure the root path using the environment variable ROOT_PATH.

Deploy

Build the image as described above. To push a new version to the HeiGIT docker registry run

docker image push repo.heigit.org/climate-action/api-gateway:devel

Canary

To build a canary version update your climatoology dependency declaration to point to the main branch and update your lock file (poetry update climatoology). Then run

docker build . \
   --tag repo.heigit.org/climate-action/api-gateway:canary \
   --push

Further Optional Parameters

env var	description
MINIO_SECURE	set to True to enable SSL in Minio connections
FILE_CACHE	location where files are temporarily stored
API_GATEWAY_APP_CONFIG_DIR	The directory holding configuration files
API_GATEWAY_API_PORT	The port, the api should start under
LOG_LEVEL	The api logging level