Fix adaptor docs fetching

[hanna-paasivirta]

Fixes #377

AI Usage

Please disclose how you've used AI in this work (it's cool, we just want to know!):

• Code generation (copilot but not intellisense)
• Learning or fact checking
• Strategy / design
• Optimisation / refactoring
• Translation / spellchecking / doc gen
• Other
• I have not used AI

You can read more details in our Responsible AI Policy

[hanna-paasivirta]

This is the Claude analysis about the Bun issue @josephjclark

JSDoc/Bun Compatibility Error Analysis

The Error

Python Side (job_chat service)

ERROR:job_chat.prompt:Error retrieving knowledge: Expecting value: line 1 column 1 (char 0)
INFO:search_adaptor_docs:Checking/loading adaptor docs for @openfn/language-mogli@0.6.10
INFO:load_adaptor_docs:Checking if docs already exist in database
INFO:load_adaptor_docs:No existing docs found for @openfn/language-mogli@0.6.10
WARNING:search_adaptor_docs:Failed to load adaptor docs after 2.813s: (500, 'Failed to fetch docs for @openfn/language-mogli@0.6.10')
WARNING:job_chat.prompt:No adaptor signatures returned from search_adaptor_docs for @openfn/language-mogli@0.6.10

Server Side (TypeScript)

✅ adaptor ready
 Preparing common adaptor docs
Using cached files for @openfn/language-common 3.2.2
❌ error loading common
Error installing adaptor docs
Error fetching adaptor @openfn/language-mogli@0.6.10: {
  message: "7 |  */\n 8 | const _ = require('lodash');\n 9 | const fs = require('fs');\n10 | const Module = require('module');\n11 | \n12 | const originalWrapper = Module.wrapper.slice(0);\n                                    ^\nTypeError: undefined is not an object (evaluating 'Module.wrapper.slice')\n      at /Users/hanna/openfn/ai_experiments/apollo/node_modules/requizzle/lib/loader.js:12:32\n      at /Users/hanna/openfn/ai_experiments/apollo/node_modules/requizzle/lib/requizzle.js:10:7\n      at /Users/hanna/openfn/ai_experiments/apollo/node_modules/requizzle/index.js:8:7\n      at /Users/hanna/openfn/ai_experiments/apollo/node_modules/jsdoc/jsdoc.js:17:5\n      at /Users/hanna/openfn/ai_experiments/apollo/node_modules/jsdoc/jsdoc.js:34:36\n\nBun v1.1.34 (macOS x64)",
}

Root Cause Analysis

The Error Chain

1. Python service calls TypeScript service

   • load_adaptor_docs.py → apollo("adaptor_apis", {...})
   • Location: services/load_adaptor_docs/load_adaptor_docs.py:302

2. TypeScript service invokes JSDoc

   • adaptor_apis.ts → installAndGen(adaptor) from @openfn/adaptor-apis
   • Location: services/adaptor_apis/adaptor_apis.ts:6

3. JSDoc crashes in Bun

   • JSDoc depends on requizzle package
   • requizzle tries to access Module.wrapper.slice(0)
   • Bun doesn't implement Module.wrapper property
   • Error: TypeError: undefined is not an object (evaluating 'Module.wrapper.slice')

4. TypeScript service exception not caught

   • The exception bubbles up through the handler
   • Returns an error response (likely HTML or plain text, not JSON)

5. Python fails to parse response

   • apollo() function at services/util.py:96-97
   • Calls r.json() without checking response status or content type
   • Results in: JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Core Issue

Bun Node.js Compatibility Gap: Bun's Node.js compatibility layer doesn't implement the Module.wrapper property that JSDoc's requizzle dependency relies on. This is a known limitation when running JSDoc under Bun.

Suggested Fixes

Fix 1: Improve Python Error Handling (Immediate Fix)

Problem: The apollo() function blindly calls .json() without checking if the response is valid.

Location: services/util.py:87-97

Current Code:

def apollo(name, payload):
    global apollo_port
    url = f"http://127.0.0.1:{apollo_port}/services/{name}"
    r = requests.post(url, json = payload)
    return r.json()

Proposed Fix:

def apollo(name, payload):
    """
    Call out to an Apollo service through HTTP.
    :param name: Name of the service.
    :param payload: Payload to send in the POST request.
    :return: JSON response.
    """
    global apollo_port
    url = f"http://127.0.0.1:{apollo_port}/services/{name}"

    try:
        r = requests.post(url, json=payload)
        r.raise_for_status()  # Raise HTTPError for bad status codes
        return r.json()
    except requests.exceptions.JSONDecodeError as e:
        # Response was not valid JSON
        raise ApolloError(
            500,
            f"Service {name} returned invalid JSON response. Response text: {r.text[:200]}",
            type="SERVICE_ERROR"
        )
    except requests.exceptions.HTTPError as e:
        # HTTP error status code
        try:
            error_data = r.json()
            raise ApolloError(
                r.status_code,
                error_data.get("message", str(e)),
                type=error_data.get("type", "HTTP_ERROR")
            )
        except requests.exceptions.JSONDecodeError:
            raise ApolloError(
                r.status_code,
                f"Service {name} failed: {r.text[:200]}",
                type="HTTP_ERROR"
            )
    except requests.exceptions.RequestException as e:
        # Connection error, timeout, etc.
        raise ApolloError(
            500,
            f"Failed to connect to service {name}: {str(e)}",
            type="CONNECTION_ERROR"
        )

Benefits:

• Provides clear error messages when services fail
• Prevents cryptic JSON parse errors
• Helps debugging by showing actual error response

---

Fix 2: Add Try-Catch to TypeScript Service Handler (Defense in Depth)

Problem: Exceptions in TypeScript services aren't caught, leading to unclear error responses.

Location: platform/src/middleware/services.ts:35-49

Current Code:

app.post(name, async (ctx) => {
  console.log(`POST to /services/${name}`);
  const payload = ctx.body;
  const result = await callService(m, port, payload as any);

  if (isApolloError(result)) {
    return new Response(JSON.stringify(result), {
      status: result.code,
      headers: {
        "Content-Type": "application/json",
      },
    });
  }

  return result;
});

Proposed Fix:

app.post(name, async (ctx) => {
  console.log(`POST to /services/${name}`);
  const payload = ctx.body;

  try {
    const result = await callService(m, port, payload as any);

    if (isApolloError(result)) {
      return new Response(JSON.stringify(result), {
        status: result.code,
        headers: {
          "Content-Type": "application/json",
        },
      });
    }

    return result;
  } catch (error) {
    console.error(`Error in service ${name}:`, error);
    const errorResponse = {
      code: 500,
      type: "SERVICE_ERROR",
      message: error instanceof Error ? error.message : String(error),
    };
    return new Response(JSON.stringify(errorResponse), {
      status: 500,
      headers: {
        "Content-Type": "application/json",
      },
    });
  }
});

Benefits:

• Ensures all service errors return valid JSON
• Consistent error format for Python clients
• Better error logging

---

Fix 3: Run JSDoc Under Node.js (Root Cause Fix)

Problem: JSDoc doesn't work under Bun due to Module.wrapper incompatibility.

Option A: Use Node.js to run adaptor_apis service

Create a wrapper script that invokes Node.js for this specific service:

New file: services/adaptor_apis/run_with_node.sh

#!/bin/bash
# Run the adaptor_apis TypeScript service using Node.js instead of Bun
exec node --loader ts-node/esm services/adaptor_apis/adaptor_apis.ts "$@"

Modify: The service loader to detect this and use Node.js

Option B: Use a different JSDoc approach

Replace @openfn/adaptor-apis with a custom implementation that:

• Uses TypeDoc instead of JSDoc (TypeDoc is TypeScript-native)
• Or pre-generates documentation in a separate build step using Node.js
• Or uses a JSDoc fork that works with Bun

Option C: Run entire server with Node.js instead of Bun

Change the start command in package.json:

{
  "scripts": {
    "start": "NODE_ENV=production node --loader tsx platform/src/index.ts",
    "dev": "NODE_ENV=development node --loader tsx --watch platform/src/index.ts"
  }
}

Trade-offs:

• Option A: Most surgical fix, but adds complexity with mixed runtimes
• Option B: Best long-term solution, but requires significant refactoring
• Option C: Simplest immediate fix, but loses Bun's performance benefits

[josephjclark]

Thank you @hanna-paasivirta

I'm not really convinced of this. It's totally fine locally - I don't see this error at all. After a few searches I can't find anything about this incompatibility.

And yet so far as I can tell, production is consistently failing

[josephjclark]

I've got a stack trace on staging and it does seem to lean towards what Claude is saying

TypeError: targetModule.load is not a function. (In 'targetModule.load(targetModule.id)', 'targetModule.load' is undefined)
      at load (/app/node_modules/requizzle/lib/loader.js:102:16)
      at requizzle (/app/node_modules/requizzle/lib/requizzle.js:96:27)
      at <anonymous> (/app/node_modules/requizzle/index.js:22:21)
      at <anonymous> (/app/node_modules/jsdoc/jsdoc.js:31:5)
      at <anonymous> (/app/node_modules/jsdoc/jsdoc.js:34:36)

Really don't understand why I'm not seeing this locally 🤔 If it is because bun.js is missing some features, it should fail consistently