Skip to content

Fixes related to handling of tunnel type connections #100

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
mgordel wants to merge 47 commits into
base: feat/tunnel-endpoint
Choose a base branch
from
Draft

Fixes related to handling of tunnel type connections #100

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
24ba6fa
fix python 3.9 tests
shadeofblue Aug 22, 2024
0bb88b1
feat: admin panel optional params
Aug 23, 2024
eaf616a
Merge pull request #88 from Web3-Pi/blue/proper-py-3.9-tests
shadeofblue Aug 26, 2024
ac8ac2b
feat: nodes sync test
Aug 28, 2024
ec84f71
feat: admin panel colors
Aug 30, 2024
8b0e427
fix typing issues on python 3.9
shadeofblue Sep 2, 2024
5700e9b
Merge pull request #91 from Web3-Pi/blue/py3.9
shadeofblue Sep 2, 2024
557987a
Merge branch 'main' into feat/admin-optional-params
Sep 3, 2024
9b78400
Merge pull request #89 from Web3-Pi/feat/admin-optional-params
lukasz-glen Sep 3, 2024
3670fb0
Merge branch 'main' into feat/sync-test
Sep 3, 2024
03b0552
v0.4
Sep 4, 2024
06bb87c
feat: endpoints management in admin panel
Sep 11, 2024
b2a68c8
fix: admin panel refresh after endpoint change
Sep 12, 2024
35be50f
wip - enable web3pi proxy on oses other than linux
shadeofblue Sep 25, 2024
b743a72
... fixes
shadeofblue Sep 25, 2024
9cf6c26
Wallet integration docs
lukasz-glen Oct 7, 2024
45c2ab2
Add files via upload
lukasz-glen Oct 7, 2024
4640df0
Add files via upload
lukasz-glen Oct 7, 2024
8a8e800
Update README.md
lukasz-glen Oct 7, 2024
49f109f
Update README.md
lukasz-glen Oct 7, 2024
189a671
Update README.md
lukasz-glen Oct 7, 2024
a7a36ab
Update README.md
lukasz-glen Oct 7, 2024
14314d0
Update README.md
lukasz-glen Oct 7, 2024
a49af11
Update README.md
lukasz-glen Oct 7, 2024
5aa432b
Update README.md
lukasz-glen Oct 7, 2024
d573619
Update README.md
lukasz-glen Oct 7, 2024
15ac8b4
docs(readme.md): typo in remove_endpoint usage
Oct 9, 2024
0f30c24
Merge pull request #94 from Web3-Pi/blue/poll2
cmd0s Nov 20, 2024
c81b7a8
Merge pull request #96 from Web3-Pi/fix/docs-typo
cmd0s Nov 20, 2024
df2a28e
Update README.md
cmd0s Nov 20, 2024
a4da883
Added url validation when adding and editing endpoint
mgordel Jan 14, 2025
d623e44
Merge pull request #97 from Web3-Pi/mgordel/endpoint-validation
mgordel Jan 14, 2025
a1bd910
refactor: renamed internal function as suggested by review
mgordel Jan 15, 2025
68f88d0
Merge remote-tracking branch 'origin/main' into feat/tunnel-endpoint
mgordel Jan 15, 2025
3231f9b
refactor: Small corrections as suggested by review
mgordel Jan 15, 2025
141a85f
Merge remote-tracking branch 'origin/main' into v0.4
mgordel Jan 15, 2025
03ddd38
Merge pull request #92 from Web3-Pi/v0.4
cmd0s Jan 15, 2025
83f0e08
Merge pull request #90 from Web3-Pi/feat/sync-test
cmd0s Jan 15, 2025
bb2bfe7
Merge pull request #95 from Web3-Pi/lglen/wallet
cmd0s Jan 15, 2025
ff69a84
Merge pull request #93 from Web3-Pi/feat/enpoints-management
mgordel Jan 16, 2025
b33e5cd
docs: Updated readme. Added instructions for configuring and installi…
mgordel Jan 21, 2025
fc5a14d
refactor: list replaced by table
mgordel Jan 21, 2025
7fdf9cf
refactor: list replaced by table
mgordel Jan 21, 2025
dd727ab
refactor: list replaced by table
mgordel Jan 21, 2025
576c1b5
Merge pull request #98 from Web3-Pi/mgordel/readme-update
cmd0s Jan 22, 2025
60d6ae7
Merge remote-tracking branch 'origin/main' into feat/tunnel-endpoint
mgordel Jan 24, 2025
4cfa1ca
fixes related to handling of tunnel type connections
mgordel Jan 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 7 additions & 4 deletions .github/workflows/tests_unit.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,11 +23,14 @@ jobs:
with:
python-version: ${{ matrix.python-version }}

- name: Install poetry
uses: Gr1N/setup-poetry@v9
- name: create the virtualenv
run: python -m venv ./.venv

- name: install poetry
run: .venv/bin/pip install poetry

- name: Install dependencies
run: poetry install --no-ansi
run: .venv/bin/poetry install --no-ansi

- name: Run tests
run: poetry run poe tests_unit
run: .venv/bin/poe tests_unit
81 changes: 70 additions & 11 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Web3Pi Reverse Proxy
# RPC Reverse Proxy

A reverse proxy for Geth intended for use within Web3Pi ecosystem.

Web3Pi Reverse Proxy comes out-of-the-box with several features:
RPC Reverse Proxy comes out-of-the-box with several features:

- Multiple geth nodes - you can hide multiple Geth nodes under single instance of reverse proxy
- JSON-RPC parser - our custom parser validates JSON-RPC requests before they reach the nodes
Expand All @@ -14,27 +14,67 @@ Web3Pi Reverse Proxy comes out-of-the-box with several features:

## Setup

You can install `web3pi-proxy` in one of two ways:

### Install via PyPI

Simply install `web3pi-proxy` package using your Python package manager, using **pip** for example:

```bash
pip install web3pi-proxy
```

Web3Pi Reverse Proxy expects you to provide **ETH_ENDPOINTS** environment variable to your system.

It should be a list of endpoint descriptors for JSON-RPC over HTTP communication with Geth.
### Install from source

Refer to the following example:
To install the package from source, follow these steps:

```bash
export ETH_ENDPOINTS='[{"name": "rpi geth 1", "url": "http://eop-1.local:8545/"}, {"name": "infura", "url": "https://mainnet.infura.io/v3/<YOUR_INFURA_API_KEY>"}]'
git clone https://github.com/Web3-Pi/web3-reverse-proxy.git
cd web3-reverse-proxy
python3 -m venv venv
source venv/bin/activate
pip install poetry
poetry install
```

You can define as many endpoints as you wish and chose their names however suits you.
## Configuration

You can define the following environment variables, and you can place them in the .env file (all are optional):

| Variable | Default | Description |
|---------------------------------|----------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `LOG_LEVEL` | `INFO` | Specifies the logging level. |
| `ADMIN_AUTH_TOKEN` | Randomly generated | Admin authentication token. |
| `ETH_ENDPOINTS` | `None` | A JSON list of endpoint descriptors for Ethereum nodes. Example: `[{"name": "rpi4", "url": "http://localhost:8545/"}]`. If defined, this list becomes static and cannot be managed via the admin panel. Leaving it undefined enables endpoint management via the admin panel and local database. |
| `DEFAULT_RECV_BUF_SIZE` | `8192` | Buffer size for socket receiving. |
| `PUBLIC_SERVICE` | `False` | Whether the service is public. |
| `USE_UPNP` | `True` | Enables UPnP if `PUBLIC_SERVICE` is `True`. |
| `UPNP_DISCOVERY_TIMEOUT` | `2.5` | Timeout for UPnP discovery in seconds. |
| `UPNP_LEASE_TIME` | `18000` (5 hours) | Lease time for UPnP in seconds. |
| `PROXY_LISTEN_ADDRESS` | `0.0.0.0` | Address for the proxy to listen on. |
| `PROXY_CONNECTION_ADDRESS` | `None` | Address clients use to connect to the proxy. Default is `None` (auto-resolved). |
| `PROXY_LISTEN_PORT` | `6512` | Port for the proxy to listen on. |
| `NUM_PROXY_WORKERS` | `150` | Number of workers handling proxy connections. |
| `MAX_PENDING_CLIENT_SOCKETS` | `10000` | Maximum number of pending client sockets. |
| `MAX_CONCURRENT_CONNECTIONS` | `21` | Maximum number of concurrent connections. |
| `IDLE_CONNECTION_TIMEOUT` | `300` | Timeout for idle connections in seconds. |
| `SSL_ENABLED` | `False` | Whether SSL is enabled. |
| `SSL_CERT_FILE` | `cert.pem` | Path to SSL certificate file. |
| `SSL_KEY_FILE` | `key.pem` | Path to SSL key file. |
| `CACHE_ENABLED` | `False` | Whether caching is enabled. |
| `CACHE_EXPIRY_MS` | `300000` (5 minutes) | Cache expiry time in milliseconds. |
| `JSON_RPC_REQUEST_PARSER_ENABLED` | `True` | Enables JSON-RPC request parsing. |
| `STATS_UPDATE_DELTA` | `12` | Update interval for stats in seconds. |
| `ADMIN_LISTEN_ADDRESS` | `0.0.0.0` | Address for the admin panel to listen on. |
| `ADMIN_CONNECTION_ADDRESS` | `None` | Address clients use to connect to the admin panel. Default is `None` (auto-resolved). |
| `ADMIN_LISTEN_PORT` | `6561` | Port for the admin panel to listen on. |
| `DB_FILE` | `web3pi_proxy.sqlite3` | Path to the database file. |
| `MODE` | `PROD` | Proxy mode (`DEV`, `SIM`, `PROD`). |
| `LOADBALANCER` | `LeastBusyLoadBalancer` | Load balancer strategy (`RandomLoadBalancer`, `LeastBusyLoadBalancer`, `ConstantLoadBalancer`). |

## Run

After configuring endpoints, you can run your reverse proxy with command
You can run your reverse proxy with command:

```bash
web3pi-proxy
Expand All @@ -58,7 +98,7 @@ http://0.0.0.0:6561/?token=<ADMIN_AUTH_TOKEN>

The **admin auth token** will be output to your terminal, during the launch.

Token is not stored and will be randomly generated on each launch.
Token is not stored and will be randomly generated on each launch, unless it has been defined as an environment variable or in the .env file.

Outside of admin portal, the admin service allows several operations, performed by submitting JSON-RPC requests.
Use **admin auth token** in **Authorization** header of your HTTP POST request for authentication.
Expand Down Expand Up @@ -88,7 +128,26 @@ Change existing endpoint's configuration at runtime by providing its **name** an
Remove endpoint at runtime by providing its **name**. For example, in order to remove endpoint ***local*** :

```
{"jsonrpc": "2.0", "method": "update_endpoint", "params": ["local"], "id": 0}
{"jsonrpc": "2.0", "method": "remove_endpoint", "params": ["local"], "id": 0}
```

**IMPORTANT:** Resulting changes are saved in local `.env` file for reuse.

## Wallet integration

Any client can connet RPC Reverse Proxy.
A web client, scripting tools, backend servers etc.
It is the same as with other Ethereum RPC providers: you need to use a user's access URL containing API key.
The proxy supports CORS, which enables usage within web browsers.

Users may wish to integrate the proxy with wallet applications, e.g. Metamask.
Below is the example of a proper configuration.
It is convenient to create a duplicate of Mainnet network configuration.
The only specific piece of setup is the RPC URL, which needs to be set to the access URL (that contains proxy's RPC endpoint and user's API key).
See the example below.

![Metamask](./admin/docs/metamask-1.png)

In case of any problems with the verification of the newly added network, the best place to start the investigation is to check the network traffic,
for instance with the web browser's dev tools/developers tools (Ctrl+Shift+C).

Loading