Checking how device_map="auto" lays out model TheBloke/guanaco-65B-HF in memory on 96GB VRAM workstation (2x 48GB) and ~500GB RAM.
Deps:
pip install -q -U bitsandbytes
pip install -q -U git+https://github.com/huggingface/transformers.git
pip install -q -U git+https://github.com/huggingface/peft.git
pip install -q -U git+https://github.com/huggingface/accelerate.git
pip install scipy


