Skip to content

HOWTO-DEV.md

Latest commit

 

History

History
215 lines (147 loc) · 7.35 KB

HOWTO-DEV.md

File metadata and controls

215 lines (147 loc) · 7.35 KB

camptocamp.org API developer manual

There are two ways to set up your development environment:

  1. Dev Container — the recommended approach; everything runs inside a pre-configured container, no local Python install required.
  2. Manual setup — set up Python, services, and configuration by hand on your host machine.

Note: For details on the "Mobilité douce" (public transport) features, see SMART-ORIGIN-README.md.

Option 1: Dev Container (recommended)

A Dev Container configuration is provided in the .devcontainer/ directory. It gives you a fully configured development environment with all companion services (PostgreSQL/PostGIS, Elasticsearch, Redis) running as side containers.

Prerequisites

Getting started

  1. Clone the repository and open it in VS Code:

    git clone https://github.com/c2corg/v6_api
cd v6_api
code .

  2. When prompted, click "Reopen in Container" — or open the command palette (Ctrl+Shift+P / Cmd+Shift+P) and select:

    Dev Containers: Reopen in Container

  3. Wait for the container to build and the postCreateCommand setup script to finish. On the first run this takes a few minutes — it will:

    • Install all Python dependencies
    • Generate the configuration files
    • Create and initialize both the dev and test databases
    • Populate the Elasticsearch indexes

  4. You're ready to go! See Running the API, Running tests, and Linting below.

Services available in the dev container

Service Hostname Port
PostgreSQL postgresql 5432
Elasticsearch elasticsearch 9200
Redis redis 6379
API (when run) localhost 6543

All ports are forwarded to your host machine automatically.

Rebuilding the dev container

If dependencies or the Dockerfile change, rebuild via the command palette:

Dev Containers: Rebuild Container

Or to re-run the setup script without rebuilding:

bash .devcontainer/setup.sh

For more details, see .devcontainer/README.md.

Option 2: Manual setup

Requirements

  • A computer running Linux or Windows (on Windows, take a look at WSL, it will make your developer life easier). Maybe MacOs can work, but it was not tested.
  • Python 3.9.x installed with pip and the virtualenv package (currently, it will not work with python 3.10+)
  • A docker environment with docker compose plugin (either docker desktop or a manual installation)
  • A development IDE is not mandatory, but highly recommended (pycharm, vscode...)
  • If you have GNU make installed, you can use it to run all the commands below to initialize and run your development environment (run make help to see what can be done) instead of running them "by hand"

Installing the development environment

  • Clone the git repository
  • Cd inside your cloned repo
  • Create a virtual environment in it:
python3.9 -m venv venv
  • Activate the virtual environment (⚠️ Do not forget to activate it in all terminal instances you intend to use)
# Linux (or WSL)
source venv/bin/activate
# Or Windows Powershell
.\venv\Scripts\Activate.ps1
  • Install the API and its python dependencies in the virtual environment with the following command:
pip install -e ".[dev]"
  • Start the necessary tools (postgres database, redis, and elasticsearch) in docker containers:
docker compose up -d
  • Go to the config directory and copy the env.local.sample to env.local
  • You should not need to modify it as it is already filled with the necessary values to be compatible with the docker compose defaults but if necessary you can
  • Update the *.ini file:
# Linux or WSL
./scripts/env_replace config/env.default config/env.dev config/env.local < alembic.ini.in > alembic.ini
./scripts/env_replace config/env.default config/env.dev config/env.local < common.ini.in > common.ini
./scripts/env_replace config/env.default config/env.dev config/env.local < development.ini.in > development.ini
# Or Windows Powershell (Yes you should really have a look at WSL)
Get-Content alembic.ini.in | cmd /c "python .\scripts\env_replace config/env.default config/env.dev config/env.local > alembic.ini"
Get-Content common.ini.in | cmd /c "python .\scripts\env_replace config/env.default config/env.dev config/env.local > common.ini"
Get-Content development.ini.in | cmd /c "python .\scripts\env_replace config/env.default config/env.dev config/env.local > development.ini"
  • Initialize database:
docker compose exec -u postgres -T postgresql /v6_api/scripts/database/create_schema.sh
initialize_c2corg_api_db development.ini
  • Initialize test database:
docker compose exec -u postgres -T postgresql /v6_api/scripts/database/create_test_schema.sh
  • Initialize elastic search indexes
fill_es_index development.ini

Running the API

  • Make sure the postgres redis and elasticsearch docker containers are running
docker ps
  • If you do not see them running, restart them (without the initialization steps that should have been done just once when setting up the environment):
docker compose up -d
  • Run the API (this command will not exit so you should run it in a separate terminal):
pserve development.ini --reload
  • Run background jobs (run each of these commands in a separate terminal, as they will not exit):
python c2corg_api/scripts/es/syncer.py development.ini
# Does not work in Windows, as it uses signal.pause() which is only available in unix based OSes (Did I already tell you that you should really look at WSL ?)
python c2corg_api/scripts/jobs/scheduler.py development.ini

Running tests

  • Update the test.ini file:
# Linux or WSL
./scripts/env_replace config/env.default config/env.dev config/env.local < test.ini.in > test.ini
# Or Windows Powershell
Get-Content test.ini.in | cmd /c "python .\scripts\env_replace config/env.default config/env.dev config/env.local > test.ini"
  • To run tests , you can either run them directly with pytest (Again, it looks like some tests will not pass in Windows):
pytest
  • Or directly in your favorite development IDE (pycharm, vscode...)

Linting

  • Run the linter:
flake8 c2corg_api es_migration

Other

  • Flush redis cache:
python c2corg_api/scripts/redis-flushdb.py development.ini