FEAT: Remove dependency on fastchat for model conversation templates#1049
FEAT: Remove dependency on fastchat for model conversation templates#1049varshini2305 wants to merge 9 commits intoAzure:mainfrom
Conversation
romanlutz
left a comment
romanlutz
left a comment
There was a problem hiding this comment.
Thank you! I've kicked off a couple jobs and will report back on how they turned out.
| tokenizer = AutoTokenizer.from_pretrained( | ||
| params.tokenizer_paths[i], token=params.token, trust_remote_code=False, **params.tokenizer_kwargs[i] | ||
| ) | ||
| params.tokenizer_paths[i], token=params.token, trust_remote_code=False, use_fast = True) |
There was a problem hiding this comment.
What motivated this change?
There was a problem hiding this comment.
char_to_token() is only supported by fast tokenizers in Hugging Face’s Transformers library, to resolve this switch to a fast tokenizer using use_fast=True
| allow_non_ascii: bool = False, | ||
| num_train_models: int = 1, | ||
| devices: list = ["cuda:0"], | ||
| devices: list = ["mps"] if torch.backends.mps.is_available() else ["cuda:0"], |
There was a problem hiding this comment.
Is this just something you used because cuda wasn't working or do you think this would be an improvement in some cases?
There was a problem hiding this comment.
Its just an additional device mapping to also support torch running on macos devices - https://docs.pytorch.org/docs/stable/notes/mps.html
I originally added it to do a local test run on my macbook device. It can be removed, if we'll always run gcg attacks only on CUDA gpus via cloud.
There was a problem hiding this comment.
I have no problem with support more than cuda, but I worry that it may now us mps even when cuda is available (?)
Maybe I don't know enough about when mps is available vs cuda or whether both can be available at the same time. If you know please reply here 🙂
There was a problem hiding this comment.
I already removed the condition (this is marked as outdated version of train.py) - with just cuda mapping. However its mostly mutually exclusive - to have cuda or mps, it can't be both at the same time, as mps is seen as a cuda equivalent in mac device with MPS backend i.e. the apple silicon devices.
3 participants
Replace FastChat’s custom prompt logic with Hugging Face’s apply_chat_template() and tokenizer_config.json.
pyrit/auxiliary_attacks/gcg/attack/base/attack_manager.pyto call apply_chat_template(), generating model-specific prompts according to the conversation template.