Skip to content

CLI fails with Bedrock models unless #37

@cventus

Description

@cventus

When running RAGChecker 0.1.9 using ragchecker.cli in an EC2 instance, I ran into an issue where I got a rather non-informative error from botocore via litellm, and the process gets stuck in an infinite loop waiting for 10 seconds and then retrying, and my logs contain this over and over:

 [sleep 10 seconds]
litellm.APIConnectionError: 'NoneType' object has no attribute 'split'
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/litellm/main.py", line 2788, in completion
    response = bedrock_converse_chat_completion.completion(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/litellm/llms/bedrock/chat/converse_handler.py", line 385, in completion
    prepped = self.get_request_headers(
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/litellm/llms/bedrock/base_aws_llm.py", line 677, in get_request_headers
    sigv4.add_auth(request)
  File "/usr/local/lib/python3.12/site-packages/botocore/auth.py", line 430, in add_auth
    canonical_request = self.canonical_request(request)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/botocore/auth.py", line 371, in canonical_request
    cr.append(self.canonical_headers(headers_to_sign) + '\n')
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/botocore/auth.py", line 306, in canonical_headers
    value = ','.join(
            ^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/botocore/auth.py", line 307, in <genexpr>
    self._header_value(v) for v in headers_to_sign.get_all(key)
    ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/botocore/auth.py", line 318, in _header_value
    return ' '.join(value.split())
                    ^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'split'
 [sleep 10 seconds]
litellm.APIConnectionError: 'NoneType' object has no attribute 'split'

I tracked the issue down, it's the "Host" header which is None and this is because RAGChecker passes bedrock/meta.llama3-70b-instruct-v1:0 as the api_base to RefChecker by default:

parser.add_argument(
'--extractor_api_base', type=str, default="bedrock/meta.llama3-70b-instruct-v1:0",
help='API base URL for the extractor if using vllm deployed open source LLMs. Default: bedrock/meta.llama3-70b-instruct-v1:0'
)

It seems that this is not the intended behavior, and was likely added by mistake in commit 671b71b where a default value was added to --extractor_api_base when it was meant for --checker_name.

I can work-around it by by passing in --extractor_api_base explicitly as https://bedrock-runtime.<region>.amazonaws.com. I would prefer to not have to do this, however, I would rather rely on the default AWS SDK behavior which uses the default region and api base.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions