Metadata-Version: 2.4
Name: accelana
Version: 0.1.1
Summary: Lana-driven adaptive SafeTensors weight residency for giant models on cheap hardware.
Requires-Python: <3.14,>=3.13
Description-Content-Type: text/markdown
Requires-Dist: torch==2.14.1; sys_platform == "darwin"
Requires-Dist: accelana-torch-runtime==2.14.1; sys_platform == "win32"
Requires-Dist: safetensors>=0.8
Requires-Dist: huggingface-hub
Requires-Dist: transformers<6,>=5.18

# Accelana

Accelana is an **inference-time memory manager** for AI models.

## Install

Requires Python 3.13. Supported computers are Apple Silicon Macs with macOS 14+
and Windows 11 ARM64 PCs. Windows uses CPU inference; Macs can use Apple GPU acceleration.

Use a fresh virtual environment:

```sh
python3.13 -m venv .venv
source .venv/bin/activate
python -m pip install accelana
```

On Windows, create the environment with `py -3.13 -m venv .venv` and activate it
with `.venv\Scripts\Activate.ps1` in PowerShell. Then run the same pip command.
Do not install another PyTorch distribution in the Windows environment.

```sh
accelana --version
accelana pull Qwen/Qwen2.5-0.5B-Instruct
accelana run Qwen/Qwen2.5-0.5B-Instruct
```

Lana is bundled. Windows also installs `accelana-torch-runtime` automatically;
Mac installs the official PyTorch package. Models are downloaded separately.

## Commands

### `accelana list`

Show cached models and whether each can run chat, plain-text continuation, GGUF, or another supported task:

```sh
accelana list
```

Output includes the Hugging Face model ID and a status such as `runnable (chat, plain, ...)` or `unsupported: reason`. Cached GGUF files appear as separate rows with `--gguf-file NAME.gguf`; without llama.cpp, their status is `requires llama.cpp`.

### `accelana pull MODEL [--gguf-file NAME.gguf]`

Download a model from Hugging Face into the cache. By default, Accelana downloads selected SafeTensors, tokenizer, and configuration files:

```sh
accelana pull microsoft/Phi-4-mini-instruct
```

`MODEL` can be a Hugging Face model ID or the short name `Phi-4-mini-instruct` for `microsoft/Phi-4-mini-instruct`.

To pull one published GGUF file, name it explicitly. GGUF is outside the bundled SafeTensors flow. Python wheel installations need a separate llama.cpp runner (on macOS, `brew install llama.cpp`):

```sh
accelana pull ggml-org/SmolLM2-135M-GGUF --gguf-file SmolLM2-135M-Q4_K_M.gguf
```

This downloads only that GGUF file into the Hugging Face cache. It does not convert a SafeTensors model or select a quantization automatically.

### `accelana run MODEL [options]`

Start interactive chat with a cached SafeTensors model (pull it first if it is not in the cache):

```sh
accelana run microsoft/Phi-4-mini-instruct
accelana run Phi-4-mini-instruct
```

`run` does not download a missing model. Its error shows the matching `accelana pull` command, including `--gguf-file` when needed.

Run a cached GGUF file or one already on disk:

```sh
accelana run ggml-org/SmolLM2-135M-GGUF --gguf-file SmolLM2-135M-Q4_K_M.gguf
accelana run /path/to/model.gguf
```

If a cached repository has one GGUF and no SafeTensors checkpoint, `run MODEL` selects it. Multiple GGUF files require `--gguf-file`. SafeTensors remains the default when both formats are cached. GGUF runs hand the terminal to llama.cpp, so its prompt and exit controls apply; `--plain` and `--vram-gb` are unavailable for GGUF.

For a text generation model without a chat template, continue plain text instead:

```sh
accelana pull Intel/tiny-random-gpt2
accelana run Intel/tiny-random-gpt2 --plain
```

Plain mode keeps one growing text. Each new line follows the previous prompt and generated output; it does not add chat roles. `--plain` also works with chat models when plain continuation is wanted.
Generation stops at an end token or the model's context limit. If the context fills, the next line cannot continue that session.

For SafeTensors runs, enter a message at the `>>>` prompt. Enter `/bye`, `/exit`, `exit`, or `quit` to end the session. Ctrl-D also exits; Ctrl-C interrupts.

Options:

| Option                    | Meaning                                                                                                                                                                        |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `--gguf-file NAME.gguf` | Run this named GGUF file from a cached Hugging Face repository. Omit it when the repository has exactly one GGUF and no SafeTensors checkpoint.                                |
| `--plain`               | Continue plain text with a SafeTensors model without requiring a tokenizer chat template.                                                                                      |
| `--vram-gb GB`          | Set a positive GPU weight budget in decimal GB for a SafeTensors model. Automatically retain as many layers as fit. Without it, the weight budget is the full checkpoint size. |

Example:

```sh
accelana run microsoft/Phi-4-mini-instruct --vram-gb 8
```

Accelana automatically chooses how many layers remain loaded between tokens. Every remaining layer still runs: its weights load just before its calculation and release afterward. All model layers participate in every generated token; memory-saving mode does not shrink the model or skip calculations.

Accelana keeps non-layer weights, such as embeddings and the output head, loaded throughout the session. The budget must fit these weights, the retained layers, and the largest remaining layer. Budgets that cannot fit this working set fail before checkpoint weight loading. Automatic selection retains the largest prefix of layers that fits, using their actual sizes. When the entire checkpoint fits, all layers stay loaded.

`--vram-gb` limits managed weights, not total process or GPU memory. Conversation cache, calculation results, model buffers, and runtime allocations need additional memory. Retaining fewer layers reduces weight memory but increases repeated loading.

### `accelana delete MODEL [--yes]`

Delete all cached revisions for the exact Hugging Face model ID, including any GGUF files in those revisions:

```sh
accelana delete microsoft/Phi-4-mini-instruct
```

Accelana shows the estimated reclaimed space and asks for confirmation. Type `y` or `yes` to continue; any other response cancels. Use `--yes` or `-y` to skip the prompt:

```sh
accelana delete microsoft/Phi-4-mini-instruct --yes
```

Use the full model ID for deletion. If the model is not cached, the command reports that and exits successfully.

## Python API

The Python `generate()` helper uses chat formatting. The package also exposes task helpers for model snapshots or Hugging Face IDs. These functions load and close the model for each call; pass `vram_gib` to use limited-memory residency.

```python
from lana_memory.tasks import classify, embed, generate, rank

model = "microsoft/Phi-4-mini-instruct"
reply = generate(model, "Explain gravity in one sentence.")
```

Task helpers:

```python
from lana_memory.tasks import classify, embed, rank

vectors = embed("owner/embedding-model", "hello", vram_gib=8)
scores = rank("owner/ranking-model", "query", ["candidate one", "candidate two"])
labels = classify("owner/classification-model", "hello")
```

Replace each `owner/...` example with a cached model ID suitable for that task, or a local snapshot path.

For reproducible offline use, pass a local Hugging Face snapshot directory instead of a model ID. If given an ID, these helpers require it to be cached first; download it with `accelana pull MODEL`.

## Supported model files and common limits

The SafeTensors path requires a compatible text-only causal architecture supported by the installed Transformers version and a checkpoint Accelana can map for weight residency. Chat also requires a usable tokenizer chat template; plain mode does not. Custom remote model code is disabled. Quantized SafeTensors checkpoints and multimodal chat models are not supported by this path. GGUF uses an installed llama.cpp runner instead; Accelana does not manage its weight residency. Single-file, text-only GGUF models are supported in this path. Hugging Face filter tags alone do not guarantee compatibility. Unsupported models are reported by `list` or `run` with a reason.

The default Hugging Face download includes selected model, tokenizer, and configuration files used by Accelana; it does not download arbitrary repository files. `pull --gguf-file` downloads only the named GGUF file. Internet access is needed for `pull`; `run` requires a cached model or a local GGUF file.

## Errors and exit behavior

- Invalid command arguments are rejected by the CLI, with help text for the command.
- `run` returns a nonzero status for missing cached models or runtime errors and reports unsupported snapshots before starting a session.
- Ctrl-C during a run or delete returns status `130`.
- A cancelled delete and a delete of a model that is not cached complete without deleting anything.
- If a model cannot be loaded, check `accelana list` for its support reason. For GGUF, confirm llama.cpp is installed and the selected file is supported by that runner.

## Bundled Lana

Accelana wheels include the Lana 4.1.0 Rust executable, compiler bytecode, standard library, and a private copy of its Python bridge. They come from [oponite/lana v4.1.0](https://github.com/oponite/lana/tree/v4.1.0), commit `4e1606a390f44778029ff45bf0b8af7ff9b41272`. Lana's Apache-2.0 license is included beside the bundled runtime and bridge.
